We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Why Software Engineers disagree about everything

00:00

Formal Metadata

Title
Why Software Engineers disagree about everything
Title of Series
Part Number
4
Number of Parts
86
Author
License
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Why are there are so many disagreements in software? Why don’t we all converge on the same beliefs or technologies? It might sound obvious that people shouldn't agree, but I want to convince you it’s weird that we don't. This talk will be a philosophical exploration of how knowledge converges within subcultures, as I explore this question through the worlds of software, online fraud, and poker.
35
Software engineeringElectronic mailing listAngleDifferent (Kate Ryan album)CASE <Informatik>MereologyNumberXMLComputer animation
Drill commandsGame controllerControl flowField (computer science)Dependent and independent variablesRational numberSign (mathematics)MereologyRevision controlFigurate numberElectronic visual displayGroup actionOptical disc driveDiagramNatural numberBitComputer animation
Computer programmingDifferent (Kate Ryan album)BitMereologyKnowledge-based systemsNormal (geometry)CodeBootingView (database)Computer animation
Physical systemFormal languageRelational databaseSet (mathematics)Endliche ModelltheorieComputer programmingPressureScaling (geometry)Personal digital assistantInformation securityBellman equationComputer animation
AnalogyWebsiteLimit (category theory)Game theoryXMLUMLComputer animation
Source codeProgrammer (hardware)Confidence intervalComputer programmingXMLComputer animation
Software frameworkDifferent (Kate Ryan album)Functional programmingObject-oriented programming1 (number)Functional (mathematics)Source codeMereologyDivergenceParticle systemQuicksortComputer animation
Multiplication signMathematical optimizationState of matterQuicksortPhysical systemStrategy gameOrder (biology)Data miningInheritance (object-oriented programming)Plastikkarte3 (number)Computer animation
Ring (mathematics)MIDIGame theoryStrategy gameTwitterState of matterData miningSet (mathematics)Direction (geometry)Point (geometry)BitCurveNatural numberPlastikkarteInternetworkingComplex (psychology)Graph (mathematics)Sound effectTelecommunicationMultiplication signDifferent (Kate Ryan album)Physical systemInformationGroup actionRight angleWordSlide ruleGoodness of fit40 (number)Internet forumVariable (mathematics)Adventure gameVideo gameComputer programmingOrder (biology)WeightMathematical optimizationMoment (mathematics)Automatic differentiationPlanningComputer animation
Right anglePhysical system
Fraction (mathematics)PressureLine (geometry)Population densityNumberMembrane keyboardWater vaporCellular automatonPhysical systemQuicksortState of matterComplex systemComputer animationXMLUML
Prime idealMachine visionComputer animation
Numbering schemeEscape characterStandard deviationQuicksortCross-site scriptingImplementationBit rateWebsiteDatabase transactionGame controllerProduct (business)Type theoryOrder (biology)Data managementLimit (category theory)Scaling (geometry)Patch (Unix)TelecommunicationRule of inferenceSquare numberLevel (video gaming)Traffic reportingMultiplication signBitComputer animation
Computing platformPoint (geometry)Goodness of fitPatch (Unix)Vulnerability (computing)Observational studyHypermediaDispersion (chemistry)Numbering scheme
Right angleMathematicsWave packetComputer animation
Functional (mathematics)Level (video gaming)Degree (graph theory)Multiplication signWebsiteQuicksortNetwork topology1 (number)Right angleMathematical analysisXMLComputer animation
Data conversionQuicksortSoftwareMultiplication signWave packetModulare ProgrammierungDifferent (Kate Ryan album)Open sourceXMLUMLComputer animation
Meta elementLevel (video gaming)Object (grammar)SoftwareUser interfaceSequelMathematical analysisComputer animation
Web applicationMobile appFront and back endsMultiplicationMathematical analysisWave packetReduction of orderProduct (business)Computer animation
Degree (graph theory)SoftwarePhysical systemMereologyOpen sourceXML
MathematicsPhysical systemMultiplication signWeb 2.0Software frameworkConfiguration spaceArithmetic meanDirection (geometry)Different (Kate Ryan album)DistanceFormal grammarWave packetRight angleComputer animation
Order (biology)InformationDifferent (Kate Ryan album)SoftwareTerm (mathematics)MathematicsShared memoryGraph (mathematics)Characteristic polynomialTypprüfungWebsiteExistencePatch (Unix)Internet forumFacebookMultiplication signQuicksortMaxima and minimaMereologyOffenes KommunikationssystemDomain nameComputer programmingCodeTraverse (surveying)Level (video gaming)Software developerPointer (computer programming)Integrated development environmentFormal languageSoftware frameworkField (computer science)GodSpacetimeNetwork topologyInstance (computer science)Dependent and independent variablesGroup actionRepresentation theoryWave packetThermodynamisches SystemVariety (linguistics)Event horizonTuring testPoint (geometry)Right angleInformation securityProcess (computing)ChainDispersion (chemistry)Sampling (statistics)Speech synthesisWeightCompilerGreatest elementType theoryView (database)Form (programming)Endliche ModelltheorieThread (computing)Service (economics)Computer animation
QuicksortGroup actionOpen sourceBlogParameter (computer programming)Library (computing)Different (Kate Ryan album)TheoryComputer programmingIdentity managementCovering spaceUniverse (mathematics)ResultantNormal (geometry)DemosceneSurfaceLevel (video gaming)Drop (liquid)Boundary value problemSoftwareInformationOperator (mathematics)Traverse (surveying)RoutingShared memoryAreaSpacetimeFilm editingRight anglePairwise comparisonFacebookSubstitute goodExecution unitForcing (mathematics)Degree (graph theory)Software developerMathematicsMereologySystem identificationProcess (computing)File formatSource codeStrategy gameProper mapTwitterInformation securityCASE <Informatik>Java appletRing (mathematics)Cartesian coordinate systemVulnerability (computing)Category of beingIdentifiabilityNumberMoment (mathematics)Goodness of fit
PressureFood energyOrder (biology)Office suiteGroup actionDivergenceMereologyIdentity managementCASE <Informatik>QuicksortGoodness of fitAuditory maskingBitAnalogyComputer animation
Food energyBound stateRight angleFlow separationMereologyIdentity managementSoftware developerAuditory maskingGroup actionDifferent (Kate Ryan album)Wave packetResultant
SoftwareMaxima and minimaWordSoftware developerMobile appDegree (graph theory)Identity managementHill differential equationInheritance (object-oriented programming)Cartesian coordinate systemIdentifiabilityComputer programmingProcess (computing)Natural numberFormal grammarMaterialization (paranormal)UMLComputer animation
2 (number)Identity managementArithmetic meanSoftware engineeringLipschitz-StetigkeitService (economics)Real numberProcess (computing)Hill differential equationReflection (mathematics)MereologyRegular graph1 (number)Dressing (medical)Right anglePattern languageGroup actionStaff (military)Computer animationXML
Network topologyJSONXML
Transcript: English(auto-generated)
How's it going? Hope lunch was good. Was lunch good? Okay, awesome.
It's okay. You can say no. That's totally fine. I don't work for these people, so I won't get offended. Cool. So today I'll be giving a talk on why software engineers disagree about everything, okay? My name is Asib Qureshi. I'm a software engineer at Airbnb. I work on the risk team. I'll be talking a lot more about that and a lot more about kind of my own history
and the different things that I've done in my past because that really informs a lot of what this talk is about. I don't know that I'll totally answer the question of why software engineers disagree about everything, but I think this talk is going to be kind of an exploration of this question from a number of different angles. Another thing that's going to happen in this talk is I'm going to disagree pretty
strongly with DHH's keynote, so that might be interesting. Hopefully, at the very least, if I'm not convincing you of what I'm saying, maybe get you thinking about some things. This talk, in large part, is going to be about philosophy.
I know it's a RailsConf. Usually people come up here and they talk about controllers, or I guess someone else is talking about sorting and augmented reality. I'm not going to be offended if you get up right now and want to go to another talk. That's totally cool, but we are going to go way into the weeds. That's what I'm all about.
Specifically, I want to talk about the field of philosophy known as epistemology. The easiest way to define epistemology is epistemology is the nature of knowledge, of justification, and of the rationality of belief. Actually, this diagram here is of phrenology, which is this old science of trying to figure
out what parts of the brain are responsible for what, which is a very old and ancient version of epistemology, which we've now thankfully displaced. Essentially, in epistemology are the two questions. How does anyone know anything, and how do they know that they know it?
These two questions are kind of navel-gazey, but that's fine. Let me tell you a little bit about me. I studied English and philosophy in school. Before I ever came into the tech world, I used to be a professional poker player. I did that for about five years. I kind of took a very different path into programming than most people.
Then I worked as a programming instructor. I taught programming at a coding boot camp. For the last little bit over a year, I've been working as an engineer on the risk team at Airbnb, basically fighting fraud. Because of this, I've kind of snaked my way through many different subcultures.
I've been something of a chameleon in that I've learned the different norms and the different beliefs and the different knowledge systems that many different worlds have and try to assert on you if you become a part of that world. One thing that I've noticed, and you can't help but notice the more worlds you step in and out of, is that knowledge is deeply cultural.
In the world of programming, the kind of things you might hear people tell you when you come into this world, they might tell you that full set JavaScript is the future. They know this, for sure. They'll tell you that everyone should know C. If you don't know C, what are you doing? Go out there, crack open a book and learn some C.
They'll tell you Rust is the best language for systems programming. They'll tell you that relational databases don't scale. They'll tell you that TDD is a fantasy and nobody serious tries to do it anymore. It's not practical. All these are interesting questions. They're all things that people disagree on, obviously,
but they're things that people hold with very, very strong conviction that this is clearly and just irrefutably true. I'm not interested in convincing you of any of these claims. Each of these could probably be great talks in and of themselves. What I'm interested in is why we disagree about them.
Why isn't that we all converge on an answer to these questions? It's funny because when I was in the poker world, I remembered this. This was very familiar to me when I was learning how to be a poker player because in the poker world, people will tell you things like, no limit is dying, you have to learn mixed games. Or they would tell you, everybody needs to use a HUD. Or they would say, only fish play loose passive styles.
They would say, GTO is a fantasy, nobody actually plays like that. There's a wonderful analogy between these two worlds, not just in that people argue a lot and that people are maybe kind of whiny and excessive, but that there are all these things that people fundamentally disagree on, and each side of them holds tremendous conviction
that their side is clearly and obviously correct. When most people hear stuff like this, when they hear someone say, hey, everybody should know C, and if you don't know C, you're not a real programmer, their natural reaction is, oh, God, is that true? If that's true, what should I do about it? For me, having gone through this song and dance so many times,
my reaction is, why do they think they know that? And what is it that implanted so much confidence in them, so much conviction that the thing they're saying is actually true and universally true? And that's very interesting to me. It's almost like I'm asking, I tend to ask a more evolutionary question
than a question about the world as it is right now. The thing that's kind of weird about programming in particular is that nobody agrees. There are so many things that people disagree on. They disagree about functional programming, object-oriented programming, TDD, old, robust, proven frameworks,
shiny new ones that solve new problems in different ways, serverless, event-driven, blah, blah, blah, blah, blah. You guys know all this. Nobody agrees. Why? That's really weird. Now, you might think it's obvious that people don't agree. That's an understandable reaction. You might say, well, disagreement is a normal part of a society,
and the way that we, this divergence is a source of discourse, and that's how we end up having a healthy society. So you might think that's obvious. Of course people shouldn't agree. I think it's weird. I think it's weird that people don't agree. And I want to develop in you an intuition to also believe this is weird,
that people don't agree. Because in a way, it makes more sense for people to agree than for them to disagree. Another way of putting what I'm saying is that systems in general tend to converge. When you see a system, you should assume that over time
it's going to converge on what's optimal, what's sort of the optimal state for that system. Let me give you an example. So I'm going to give you an example related to poker, but you don't need to know anything about poker in order to understand this. I can explain it in a couple sentences. So in poker, there's a strategy called set mining.
And set mining is very simple. So in Texas Hold'em, you get two cards, and if you want to set mine, what you do is you wait for two cards that make a pair. So let's say you get a delta pair of threes. Then what you do is you wait to see if you make three of a kind. If you make three of a kind, you bet it really aggressively, and if you don't, you just fold. That's set mining. Super, super simple. So set mining was a strategy that was pretty simple,
and pretty stupid. Almost anybody could do it. And it worked. It worked really, really unreasonably well, given how simple the strategy was. And so pretty soon, what you would see in the world of online poker is that pretty soon, once people started talking about this, almost everybody started set mining at low to mid stakes for ring games.
The strategy just kind of took over like wildfire. And what you could see is that the game converged on set mining. Everybody saw that set mining was the high ground, and they all moved in that direction. And so this is kind of one of the features that happened to poker
after internet poker took off. Internet poker took off really around 2004, 2005, maybe a little bit before that as well. And if you think about it, before internet poker ever existed, there was live poker, right, playing poker in a brick and mortar casino. And live poker was fundamentally really different than online poker in ways that people didn't really predict before it happened.
So you can imagine live poker, let's say in the 80s and 90s, there were some people playing poker out in Phoenix, maybe. There were people playing poker in Vegas. There were people playing poker in Dallas, people playing poker in London. And these different groups of poker players weren't really communicating with each other. Ideas that were germinating in London weren't going over the pond
and showing up in card rooms in Dallas. People were kind of isolated to their little groups, and they were just playing poker and figuring things out as they went. But with the advent of online poker and the communication that it enabled for a lot of people, sorry, there was like a fly running around me. With the advent of that communication, what happened was that it allowed the system to converge,
where suddenly somebody has an idea for a strategy somewhere in Dallas or New Hampshire or wherever, and suddenly they can share it and everybody can learn about this strategy very, very quickly. And so what this meant was that the state of poker strategy, for a long time it was pretty static.
The way that people played in the 40s and 50s was not that different from the way people played in the 80s and 90s. Information just wasn't able to evolve that effectively. But with the advent of online poker, suddenly if you look at the curve of the complexity of poker strategy, it just takes off right after 2003 when online poker comes to town.
And so right here is a graph. That audio is not playing, but that's fine. He's saying, look at this graph. This is a joke that didn't play audio. That's fine. We're going to roll with it. So this is actually a graph of, this is actually just a graph that goes up and to the right. It has nothing to do with what I'm talking about, but I thought it would give a sense of legitimacy to the slides.
It's like, oh, this guy knows what he's talking about. Look at this graph. Cool. So the point is, the point is, online poker converged in a way that live poker never did. It was never able to. And this to me is, it makes a lot of sense. Of course online poker is going to converge. When you have these people able to communicate,
to see what each other is doing, and move around in this terrain, of course they're going to settle on what's optimal. And nature is full of convergent systems like this. So let's say for example you're walking downtown and you buy a loaf of bread. Maybe you're French. I don't know. And let's say you eat half a loaf of bread
and you toss the rest on the street. Well, what happens is pretty soon a bunch of pigeons, it's not like they're all in one place standing over watching you throw the loaf of bread. But pretty soon one pigeon comes, three pigeons come, and then a chorus of pigeons from presumably all over town just come in and start binging on the piece of bread that you've left.
And this kind of makes sense. The pigeons are able to very quickly come down to whatever is the optimal place for them to be. They eat as much food as they can, and they disperse back to wherever it was they came from, maybe to the next best piece of food that they can find somewhere downtown. So this is an example of a convergent system.
So there's also this great Ramstein album that actually I loved when I was a kid. This is actually just cell membrane in German because that was the only one I could find like ten minutes ago. But the cell membrane, you guys might remember from high school chemistry, you know, there are water molecules on one side,
they permeate through this membrane, and pretty soon you have the exact same pressure, the exact same density of water molecules on each side of the membrane. The system converges. And you see the same sort of thing in stock markets, a lot of natural phenomenon. It seems like this sort of thing is everywhere. So when I started working as a risk engineer at Airbnb,
working in the fraud space, naturally I started looking for convergence too, because it just seems like one of those things that any sufficiently complex system should eventually have some optimal state, and it should find convergence. So working in the fraud industry is its own little subculture,
its own little world. And the subculture is not just the subculture of people working against fraud, which is interesting, but that wasn't really what fascinated me so much. I mean, it is very interesting, but the really fascinating thing about fighting fraud is that you're actively fighting against a community of people.
You're actively fighting against a culture that's optimized to take you down and to basically exploit your defenses as effectively as they possibly can. And so, you know, really there is actually a subculture that I can't see and don't have direct access to that's organizing and trying to attack all these, you know, major online companies, and they're trying to make money.
And that's really interesting. And so I wondered, you know, how do fraudsters figure out what to do? I mean, what they're learning is pretty non-trivial, right? Like, they're learning how to script, how to do, you know, sometimes cross-site request forgery stuff, XSS, whatever. They come up with all these different ways to try to probe our defenses
and attack us and ideally try to make some money. How do they learn this, and how do they communicate this to each other? You know, how does this knowledge spread in the world of fraud? Fundamentally, the question I want to ask is, is there convergence in fraud? And this actually, pretty soon after the new MacBook Pro replaced the escape key,
we saw fraud go way up. So that was, one person laughed at that. That was, all right. There we go, there we go. Just a little bit, that's all I need in order to keep going. Cool. Awesome. Awesome. So I just want to run with you through a quick example of kind of what a standard fraud scheme looks like. This is some site called Shipmunk.
Does anyone know what Shipmunk is? Anyone familiar with this esteemed company? Great, then we're going to shit on them. That's great. So Shipmunk, let's say they cut some corners. They kind of implement a feature they don't really know exactly. You know, they're not really thinking about fraud when they implement this, right? Like some product manager is like, hey, you know, we've got this story. We've got to get it up at the end of the week.
We're going to all look good once we fill this thing out. Great. So essentially engineers who work on this just push it out, don't really think too much about it. So this feature is essentially some micro deposit where essentially they verify that you own this account because you deposit two random amounts. You report them back, right? Pretty standard for some financial institutions to do this sort of thing. So let's say that they don't implement rate limiting.
Easy thing to overlook. Well, you know, they go to sleep at night feeling great that they want this new feature. And pretty soon someone gets alerted, you know, let's say at 2 a.m. that night, that somehow we've lost $100,000, which really should not have happened.
And the only way you could lose that amount of money, obviously, is if fraudsters are at scale repeatedly hitting this API, repeatedly micro depositing money into their account, and then just keeping it. And so people scramble. You know, people are like, oh my god, I can't believe this is going on. They go and they try to patch this whole, implement some rate limiting, do some rules, maybe reverse whatever transactions they can. This is a common type of attack
that, you know, a very easy vector that if your site implements something like this, a kind of attack you would be vulnerable to if you didn't implement rate limiting. So let's say they get that under control. That's fine. Now what happens is that once you patch that vulnerability, of course the fraudsters disperse. They don't keep hitting it. Maybe they try to verify that in fact it is patched.
They try a few ways around it. It doesn't work. Cool. You've patched the vulnerability and the fraudsters disperse. But kind of like the pigeons, they don't just go randomly to different places and just kind of, you know, leave your platform. What ends up happening is that they then go to the next best place to defraud you. So what they'll do is they'll find, okay, that end point wasn't that good. But there's this other thing that we can do,
like this maybe longer, more complex, or more expensive fraud scheme, and we're gonna go do that. And basically what you see is that the fraudsters head down to the next highest peaks in this terrain of fraud, which, you know, this totally makes sense. This is convergent behavior, right? This is exactly what you'd see if the terrain suddenly changed,
this peak became a valley, and they would go towards the next highest peak. So this seems to me like convergent behavior. It seems like fraudsters converge on what's optimal. But the more that I thought about this, the more I realized that this didn't totally make sense. And the reason that this totally didn't make sense is because I asked the question,
why are they defrauding us at all? Why are any fraudsters attacking Shipmunk? This might seem like a weird question, right? Like, of course they're doing it so that they can make money, and that's what fraudsters are motivated by. But if you think about it, the terrain of all the companies that they can defraud is huge.
There are many, many, many companies that are vulnerable to fraud, and pretty much all of these companies experience fraud to one degree or another, right? There are all these different sites that you could attack if you so wanted. And so what you can say is that there's some topology to that fraud, right? There's some peaks and valleys. There's some sites that are really, really lucrative to defraud,
and other ones that are really not so much, and they're not a good place to be spending your time if you're a fraudster. And so you'd think that everybody would converge on the peaks. What you'd think is that almost all the fraud would go towards the very most profitable, most attackable websites. But what you see instead
is that fraud is just kind of dispersed everywhere. There's just this sort of ambient level of fraud that if you start an e-commerce site, chances are you're gonna get some amount of fraud. Why? This is weird. This should strike you as weird. Why, is it like those fraudsters just don't care about optimizing?
Are they not trying to make the best use of their time? Is there something that they're misapprehending about the terrain? Like, what's going on that we don't see this conversion behavior within fraud? Now, you know, it occurred to me that actually you see the same sort of thing in software. You know, even just looking at open source solutions, there's so many different software packages
and so many different open source solutions to many, many different problems that are effectively trying to do the exact same thing. Why? It seems like software doesn't converge either. When you don't get this just one, okay, this is the best way to solve this problem, and everybody converges on that solution,
what you see instead is many, many, many competing solutions, and it's not really clear which one is supposed to win. And when one does win, it's actually really surprising. You know, React, you could say kind of won front-end UIs insofar as you can say anyone won front-end UIs, and we're genuinely amazed by that on some meta level. We're like, holy shit, someone won.
No one ever wins anything anymore, but React just won front-end UIs, you know? And, you know, maybe that kind of relaxes you because you're like, oh, hey, you know, as long as I learn React, I'll just be employable forever and, you know, that's fine. But I think this is somewhat counterintuitive, that more things aren't like this. Why is this such a rare story
that React wins or SQL wins or one particular way of solving a problem just is clearly the best and we all adopt it? So you might have an obvious objection to this analysis, which would be that, well, of course, you know, things aren't gonna converge because software doesn't just solve one problem, right?
The obvious answer is that there are actually multiple terrains. So, you know, you could say that there's a terrain for Prud apps and that looks different, and then there's a terrain for the payments back-ends and then one for social networks, and maybe the tools and the solutions that you use for these different apps are different, and that's why you see this, you know, these multifarious solutions
to effectively the same problem. But I don't think that's sufficient because even within a single terrain, you don't see convergence. You know, even just looking at Prud apps, right? Prud apps are, you know, they're the majority of what people build. The majority of web apps are just, you know, glorified Prud apps, maybe with, like, Elasticsearch on top,
and, like, because they're the vast majority of what people build, they're the vast majority of what people disagree on. So it doesn't seem to me like that's satisfactory to explain why we don't see convergence. And the thing is, you should want convergence. Convergence is actually good because convergence means that we all see
the underlying terrain. We all understand it, and therefore we all go and do the best thing. In the world of software, for the most part, we're not actually competing with each other. We're actually all kind of, at least to some degree or another, motivated by each other succeeding. You know, when someone invents a great open-source solution, actually everybody benefits from that,
and most software kind of works this way. So the question I want to ask is, why do these systems not converge? I think there are four reasons, and I'm going to go through each of them. So the first reason why these systems don't necessarily converge
is because the terrain is actually unstable. The terrain is changing. It's not just one configuration that you can just see something out in the distance, decide, okay, that's a peak. I'm going to go over that, right? The fact that the terrain is changing means that you're not really sure if you do go in this direction
that by the time you get there, the terrain isn't going to be different, right? It also means, of course, that your terrain is moving underneath you. So by the time that you, when you originally came into Ruby on Rails, you might have thought, wow, this is the hottest, newest, shiniest web framework anyone has ever come up with, and now it's 2017, and Rails is not the shiniest, newest, coolest framework anyone has ever come up with.
And so the terrain has changed underneath you, and that makes it hard for people to actually converge when there's so much change going on. There was this article that someone wrote on Medium a little while back, how it feels to learn JavaScript in 2016. Even if you didn't read this article, I'm sure you have an intuition of what this thing said, right? And we see this becoming more and more,
I don't know if I want to say a problem, but more and more of a characteristic of software that things are changing very rapidly, and it might be that that pace of change is even increasing, which wouldn't be that unreasonable to expect, actually, as technology grows more and more rapidly.
And you see the same sort of thing in the world of fraud. So say, for example, that Facebook has some kind of hole, some kind of things that fraudsters can attack. Let's say it becomes really easy to spam Facebook and do some kind of referral, click fraud, whatever stuff. So maybe if you're a fraudster, you're making $20 an hour defrauding Facebook doing this. Well, Facebook goes in, they patch the hole,
and now instead of making $20 an hour, you can only make $3 an hour. But in order for the fraudsters to disperse and find the next highest thing, they might be just incentivized to say, hey, I don't actually know how easy it's going to be to defraud anything else because that might get patched too, and the terrain might just change out underneath me,
so you know what? I'll just keep defrauding Facebook for $3 an hour. That's fine. At least I know this is working during the time that I have it. So maybe that's some part of the explanation why you don't see this optimizing across the terrain. So the second reason why I think you see a lack of convergence in some of these domains
is because of high switching costs. So let's say that this is the domain that, let's say this is software. This is the map of the software world. And let's say that you live all the way on the bottom right, and the bottom right, let's say, is Rails Land. And you go and talk to some wide-eyed, bookshelf developer who tells you, you know what, you just need to learn Haskell.
And if you learn Haskell, all these type errors and nil checks and da-da-da, all that stuff is just going to go away, and you are going to live in a land of just pure programming bliss. Nothing will ever go wrong. Your code will be way more performant. You are just going to, yeah, whatever. Things are going to be great. The problem with that, of course, is that for you to actually engage in that switch, let's say even you see what this person tells you,
and you agree with them, but this peak that they're pointing to is very, very far away. So in order for you to get there, you're going to have to go down into a valley, traverse a really, really long space, until you can finally actually reap the fruits of what they were claiming was so great. And who knows, because the terrain is changing, by the time you get there,
it might not even be a peak anymore. Something else might have become a peak. Or maybe the place where you were at could have become a peak. Who knows? And so this instability and this uncertainty makes people really unwilling to incur the risks of traversing the terrain and exploring. You see, of course, the same thing in fraud, just like I just mentioned with the Facebook example. And this whole thing, of course,
is exacerbated by specialization. The more specialized you are, the harder it's going to be for you to convince yourself to engage in those high switching costs, right? Because really, specialization is just basically you finding your way to some local maximum. That's what specialization is. It's climbing as high as you can onto a local maximum.
Once you're there, it just becomes really uncompelling to climb all the way down from that peak of specialization you've arrived at to go find the true global maximum, which might be very far away. And we've already talked about it, the fact that it's changing and uncertain. And so this makes it harder, the more specialized we are. Same thing with fraudsters.
There are fraudsters who are specialized in attacking one site as opposed to another, and it's hard for them to switch. If we learn a lot of new things, they have to start over in their knowledge. So the third reason why I think you don't see convergence in fraud or in software or in these other fields is information sharing is a very important part
of how you get convergence, right? If you think about it, not all of us can actually clearly and lucidly see that underlying terrain. There are these peaks and valleys, we know they're there, but they're sort of like a fog of war, right? We can't see just kind of beyond our local environment because we just don't know that much about what Haskell Mountain looks like
or what some other language or framework that you're not familiar with. So in order for us to really get that sense of what the terrain is like, we have to share information with each other about what the terrain is. That's how we learn what happens if we go out far enough into the terrain and whether the costs are going to be worth it. So different cultures have different amounts of information sharing,
and that makes it harder or easier for them to converge on different things. So if you imagine a graph of different cultures, you can sort of graph them on how closed versus open they are in terms of information sharing. So if you look at a very, very closed system, a good example of this is the fraud industry. So if you're a fraudster,
then actually it's very hard to learn and get access to the information that you need in order to learn how to become a fraudster, right? So there are all these underground fraud industries. So fraud is an industry in many places in the world where basically you can get access to courses, you can buy them, you can get primers on how to hack this site or that site.
There are various tools that you can buy, you can pirate, whatever. There's all this stuff you need to get up and running as a fraudster, and it's not easy to get this stuff. You actually have to make your way into communities. You have to prove yourself. You have to gain reputation. You can't just decide, hey, you know what? I'm going to go on Amazon and buy a textbook on how to commit fraud, right? It doesn't exist. You can't do that. You have to go in through a very specialized way,
and not all information is actually readily up for grabs. There are some fraud rings that just don't share that information with anyone outside of it, and it's not for sale. And so that makes it very difficult if you're somebody who's wanting to learn more about fraud to actually figure out what is the optimal place for me to be spending my time defrauding people. Now, somewhat more on the open side
is you can look at a world like poker. So poker is kind of a more open system. So there are all these forums. There are different places where people can exchange ideas. There are certainly books written about poker that you can just buy if you want to. But the very best players, the very best ideas, the very best theories and strategies about poker, they're generally not for sale.
The people who hold them and the people who profit the most from them tend to keep them close to the chest. So you get a lot of resources that are okay or that are really crappy that are openly available, but the very best stuff sometimes is hard to find and hard to actually gain value from. And then on the other hand, if you look all the way to the right, you find the world of software. The world of software is in a lot of ways
kind of staggeringly open. You have companies that are just releasing the source code for their entire application. Or security libraries that are, again, completely open source. And companies will just say, yeah, we use OpenSSL, the movie. If you find a weakness in OpenSSL, that's a weakness in us. And this is really about as open as you can get.
And there are blog posts. There are all these things that are shared about software that make it seem like, wow, there's an enormous amount of information sharing. That should really make it so that people see really quickly what is actually the best solution for any different problem. Somehow in the world of software, it doesn't really seem like that happens all that well. And so, I don't know. I think there's somewhat of an open question
of even though there is a lot of sharing on the surface, if in fact there are some things that people aren't that open about sharing. I think when it comes to what a lot of large companies are doing when they're putting together a lot of open source solutions to solve problems, they actually don't immediately go out and tell people, oh, hey, we solved this problem.
Here's how we did it. Very often, the way that companies share this information is pretty selective and pretty strategic. And the moment you solve a cutting edge problem, you generally don't go out and share it unless you think there's some strategic value in doing so. And so, that I think to some degree kind of exacerbates the problem of why is it that we don't get this convergence in the world of software.
So reason number four, and I think this is a really interesting reason that kind of goes to sociology, is basically the problem of group identities. And we heard some this morning from DHH about the value of group identities, and I'm kind of going to go at it from a completely different angle. I'm going to talk about more the dangers of group identities.
So you can imagine that the world of programming is kind of, you know, it's demarcated into these different kind of arbitrary groups. And one of the groups might be Rubeus, or Rails programmers, or whatever, which you categorized myself in there. Then you have Java lovers over here, and Pythonistas over here, and then Scala's over there.
And they're kind of these special norms that dictate what you can do inside these different worlds if you want to fit in into these groups. These groups kind of say, well, if you're a Java lover, you can explore this area, but you're not really supposed to go over there. That's kind of weird stuff that we don't really do in Java land. And so, you get these kind of arbitrary cuts
across the terrain that make it hard for you to just freely traverse and explore this terrain without violating some kind of social norm associated with your group. Turns out, you get the same kind of thing with fraud rings, right? So there's a fraud ring that just defrauds Facebook. That's all they do. And all they do is, you know,
they're talking on some sort of secret channel where only fraudsters that are part of this group can communicate, and they share information about just how to defraud Uber, or just how to defraud Google. And if you're one of the members of the other groups, maybe you can't get into that group. Maybe there's just, fraudsters have just decided, nope, we're this group, and you're that group, and you're not going to get our information, and so if you want to explore,
you only get to explore the terrain over there. And I think in the world of software, you kind of see this when, you know, instead of having like an explanation, a blog post, it's just, here's how Kafka works. There's the blog post, Kafka for Rails Engineers, right? Or the blog post, you know, XYZ for Rubyists. And this again is like kind of reinforcing
that demarcation, that hey, you know, I know you want to go explore that stuff, let me show it to you in the way that's appropriate for our group. And I think this is really fascinating to me, because being somebody who's relatively new to the subculture of software, I can immediately recognize this behavior. And I think it's pretty well explained
by this theory in psychology called Social Identity Theory. So the idea of Social Identity Theory, it's pretty simple, essentially suggests that the way that we construct our identities as human beings is largely as a result of the groups that we adhere to. So this kind of goes in several stages. So the first thing you do is
you start categorizing the world into social groups, okay? So you first have to say, okay, so these people are the Christians, these people are the Goths, these people are the meat lovers, okay, whatever, whatever you want to, however you want to draw up those boundaries in the space of what people can be. So first you have to draw those boundaries. Next, you have to identify
which of those groups you belong to. Do I want to be a meat lover? Do I want to be a pythonista? Do I want to be somebody who loves red or whatever? You have to decide which of those groups you're going to identify with, okay? Then, once you do that, the last step is social comparison.
Now you have to do the pretty hard work of deciding why the other groups are bad and your group is good. You have to make this distinction between your in-group and the out-group and invent some kind of story or narrative that goes along and reinforces why you're good and they're bad. So there are all these classic examples of this sort of thing where basically there's some arbitrary distinction that you've arrived on
as being important to your social identification and there's no intrinsic reason why that should be important, but we're Rubyists and they're Java lovers and because of that, they're bad and we're good and we have to come up with some sort of story why that's the case and they have to do the exact same thing. Now, you might think the social identity theory
would suggest that, okay, well that should mean that all Rubyists and all Rails developers are the same, but I don't really feel like that. I'm not the same as the people around me. You know, if you look around, you all don't look like a completely homogenous group of people and this is true. And so there's this other theory that kind of complements this really well and it's called differential psychology.
And differential psychology essentially examines the way that people within groups try to make themselves different from each other as a way of somehow strengthening their bond as being a part of that group. So for example, if you've ever seen the movie West Side Story, so you look at these characters, they're all part of a gang in West Side Story. They're all together, part of the same group.
And you can tell, they kind of have a look. Right, if you just saw these people in the street, you'd be like, okay, these people are doing something together. There's something that somehow unites them. But notice, they don't all wear exactly the same outfit. And they could. They could all wear the exact same outfit. They could all style their hair the exact same way, but they don't.
Why don't they? Why don't they do that? You'd think that maybe that would strengthen their group identity. If they all literally did the exact same thing, they would be a stronger part of that group. But it turns out there's something intrinsic to us as human beings, that even though we're a part of groups, it's important for us to differentiate ourselves. We actually spend a significant amount of energy just differentiating
ourselves within the groups that we're in as a way of almost masking our identities within that group. As a way to not make ourselves feel like, hey, I don't have any identity outside this group because I seem to be wearing the exact same thing everyone else is wearing, and doing the exact same thing everyone else is doing. We'll expend a lot of energy in order not to feel that. So that's exactly what you see these people in West Side Story doing.
And so I want to draw a little bit of analogy here, is that there's something kind of similar going on when you look at something like this, where we're expending this energy. If you imagine this underlying terrain of software, and let's say this right here is Rails Mountain, there's a lot of energy going into making it so that
it kind of looks like we're exploring this big terrain, but really everything is actually still within the bounds of this group. Even though we're talking about Kafka or Elixir or whatever it is, we're still keeping you as a part of this group, and that identity is actually reinforced by you being here in every single way. Even the fact that you're going to these different talks, talking about
different technologies, you're still seeing it as a Rails developer. And I think this is bad. I think this masks the severity of the problem of social identities making it harder for us to actually converge and actually find what is genuinely optimal. It allows us to kind of distract ourselves with this story, that like, hey, we're exploring these different things, but really
underlying it, we're not. So I think we should really want to find convergence. We should want to find the true global maximum. So as software engineers, what ought we to do about this? I don't know that I have perfect answers to this. I think these are all really intrinsically hard problems. But I do have a couple pieces of advice
that hopefully might be instructive to some degree. So the first piece of advice is an adage from Paul Grammer, initially, where he said keep your identity small. And really kind of what this means is to, as much as you can, as much as is possible, jettison the labels that you've very
easily come to identify yourself with. And so that's to say, don't think of yourself as a Rails developer, or as a Ruby developer. But instead, think of yourself as a software engineer. Such that whatever ends up being the right tool for the job, and that tool might be Rails, my tool might be Ruby, might be something else. That's what you fundamentally use. You solve problems
in the world of software. And right now, it might be very beneficial for you to go climbing up this hill of learning more about Rails or more about Ruby. But eventually, you will not be. You can imagine 10 years from now, you'll be working on something, and Rails probably won't be the tool you want to use 10 years from now. In fact, I would, right now, I actually consider myself to be, I love Ruby, I love Rails.
I think they're really awesome and wonderful tools. But I would be probably pretty disappointed in myself if I was a Rails developer 10 years from now. And that was what I considered myself to be, I'm a Rails developer. You know, when DHH was talking about the article about COBOL programmers, that there are people still making money for banks working on these super antiquated COBOL
applications, you can bet that there will still be Rails apps 10 years from now. And I'm sure that you'll probably be able to fetch a pretty penny basically managing these 10 year old, 12 year old, 15 year old Rails apps. But is that fundamentally what you want to be doing? Or is what you want to be doing to solve problems with software, however those problems end up changing
and however those tools end up changing? The second piece of advice I want to give, besides keeping your identity small, is pretty obvious. It's just to explore the terrain. And exploring the terrain, to me, means more than just kind of paying lip service to different things, like, oh okay, I don't know what Kafka is, I'm going to go to this talk, or I'm going to go to that talk.
It means, fundamentally, to do things you've never done before. Okay? It means to do things that are kind of scary to you. It means to take real risks. And when I say real risks, I'm juxtaposing that against fake risks, which I think are a real thing and something you should caution yourself against. A fake risk is one where you actually retain all of your safety,
all of your comfort, all of your prestige, all of your knowledge, all of your abilities, where it's like, you know what, I'm still a really awesome person, everyone respects me, I know everything that I'm doing, but I'm also taking this risk. No you're not. No you're not. Right? The risk comes when you give up, you actually walk down the hill. And walking down a hill is uncomfortable. It's scary.
It makes you nervous. And if you're not actually doing that, then you're not really taking a genuine risk. Another way of saying this is, go to DjangoCon. Don't actually go to DjangoCon, it probably sucks. But there's probably, like, if RailsConf is the only conference you're
going to this year, reflect on that. Reflect on what that means. Whether it means that you are actually taking risks that are important. So. And finally, of course, I think the most important part of exploration is just to have fun. And when you let go of the idea that you constantly need to be moving up, and that in fact it's okay to move down, and to
take risks in a way that potentially make it harder for you to get your job done. That that's okay. And allow yourself to have fun doing it. And I think that makes the whole process a lot easier. So, that's it for me. I'm Haseeb Qureshi, a software engineer on risk at Airbnb. If you're a senior software engineer or a data scientist, we're always hiring.
And, yeah, thanks for listening.