Pragmatic development at Stack Overflow
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 133 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/48931 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Software developerSorting algorithmQuicksortSoftware developerElectronic visual displayStack (abstract data type)BitBuffer overflowRight angleDataflowProcess (computing)AreaGoogolComputer animationLecture/Conference
02:07
Software developerStack (abstract data type)Programmer (hardware)Data modelBuffer overflowCompilerStack (abstract data type)Programmer (hardware)CodeQuicksortComputer programmingComputer scienceTheorySoftware developerNeuroinformatikRight angleWordWindowCartesian coordinate systemProcess (computing)Scaling (geometry)Computer fileSingle-precision floating-point formatWeb 2.0Java appletMultiplication signScripting languageDataflowArithmetic meanWebsiteMathematicsServer (computing)Goodness of fitComputer animation
06:55
Software developerMetropolitan area networkTouchscreenPointer (computer programming)Slide ruleComputer animation
08:09
Software developerGame controllerEndliche ModelltheorieCartesian coordinate systemRevision controlFinite-state machineVideo gameEngineering drawing
09:12
Software developerComputer programmingService (economics)Computer architectureReading (process)DataflowObject (grammar)Branch (computer science)Web-DesignerWritingOrientation (vector space)Flow separationGraphical user interfaceCASE <Informatik>Multiplication signBlogWebsiteMereologyQuicksortSoftware developerScaling (geometry)Stack (abstract data type)Selectivity (electronic)Revision controlCommitment schemeRight angleNetwork topologyForm (programming)Cartesian coordinate systemSoftwareMultiplicationData structureProper mapKey (cryptography)Physical systemConfiguration spaceElectronic mailing listUnit testingReal numberRegular graphSoftware testingTerm (mathematics)Interactive televisionTask (computing)Functional programmingWeb pageOrder (biology)Axiom of choiceNumberHome pageGame theoryMarginal distributionProcess (computing)CodeServer (computing)GoogolTransport Layer SecurityObject-oriented programmingWeb 2.0Instance (computer science)Repository (publishing)StapeldateiSocket-SchnittstelleComputer hardwareScheduling (computing)1 (number)ProgrammierstilCache (computing)Computer animation
18:32
Software developerEquals signString (computer science)outputFluid staticsRange (statistics)Price indexFunctional (mathematics)Web pageRun time (program lifecycle phase)WebsiteLogarithmBroadcast programmingStapeldateiProcedural programmingGroup actionOperations researchRadiusCodeSpacetimeResource allocationFrequencyStandard deviationRoutingMultiplication signCartesian coordinate systemOperator (mathematics)Query languageNewsletterSoftware frameworkQuicksortString (computer science)Parameter (computer programming)CodeNumberFunctional (mathematics)Computer programmingCompilerEvent horizonFunctional programmingMetaprogrammierungReflection (mathematics)Sound effectHeegaard splittingSystem callOrder (biology)CASE <Informatik>Real-time operating systemObject (grammar)Web browserSocial classServer (computing)Exception handlingStapeldateiResource allocationInformationWeb 2.0Attribute grammarCategory of beingLoop (music)SequenceElectronic mailing listFluidProcedural programmingCommunications protocolTrailWeightObject-oriented programmingMereologyProgrammer (hardware)Decision theoryThomas BayesSocket-SchnittstelleQuantum stateStack (abstract data type)Limit (category theory)Network socketRight angleFerry CorstenDemosceneIntermediate languageWordSpeciesWeb pageDivisorSource codeSpacetimeWebsiteInsertion lossRing (mathematics)Coefficient of determinationSingle-precision floating-point formatValue-added networkNatural numberUniform resource locatorProjective planePerturbation theorySource code
27:48
Software developerFrequencyEquals signResource allocationLimit (category theory)Computer animationSource code
28:51
Software developerRootMathematical optimizationMathematical optimizationStatement (computer science)RootMultiplication signPerimeterBasis <Mathematik>CodecComputer animationLecture/ConferenceMeeting/Interview
30:15
RootMathematical optimizationSoftware developerOpen sourceProfil (magazine)Group actionSubsetMultiplicationBranch (computer science)DataflowCASE <Informatik>Point (geometry)Right angle2 (number)Multiplication signDifferent (Kate Ryan album)Software developerSequelStack (abstract data type)Buffer overflowComputer animationSource code
31:49
Software developerSystem callSequelWeb pageStructural loadClient (computing)Mathematical optimization2 (number)Proof theoryDifferent (Kate Ryan album)Cartesian coordinate systemSelectivity (electronic)Right angleData storage device
33:04
Software developerLocal GroupMaxima and minimaAverageClient (computing)Profil (magazine)HypermediaData storage deviceSource codeHome pageGodPlanningLoginLink (knot theory)WeightRight angleComputer animationSource code
34:46
Software developerLocal GroupMaxima and minimaAverageClient (computing)Multiplication signComputer clusterQuicksortComputer animationSource code
36:26
Software developerAverageLocal GroupMaxima and minimaClient (computing)PixelInterface (computing)SequelRight angleRoutingProfil (magazine)AverageRow (database)Order (biology)MereologyWeb browserServer (computing)PlanningSquare numberInformationResultantFreewareOperator (mathematics)Query languageWeb 2.0Computer animation
38:37
Software developerState observer2 (number)Query language1 (number)Multiplication signAveragePlanningCASE <Informatik>Server (computing)NumberSequelNormal (geometry)QuicksortCore dumpProcess (computing)Computer animationLecture/Conference
40:39
Software developerState observerElectronic mailing listRight anglePlanningServer (computing)SQL ServerQuery languageComputer animation
41:28
Software developerState observerException handlingServer (computing)Multiplication signPhysical systemReal-time operating systemComputer animation
42:19
Software developerDemosceneStaff (military)CodeServer (computing)NumberWebsiteResource allocationStatisticsComputer animation
43:11
Software developerCommunications protocolCharge carrierComputerComputer networkScripting languagePoint (geometry)Link (knot theory)StatisticsError messageWebsiteInternetworkingDialectMathematicsCharge carrierForm (programming)Automatic differentiationUniform resource locatorDisk read-and-write headMultiplication signCommunications protocolServer (computing)Link (knot theory)Point (geometry)Web 2.0TwitterSpeicherbereinigungStandard deviationKey (cryptography)Software developerMetreResource allocationPunched cardCondition numberWeb pageDifferent (Kate Ryan album)Message passingRight angleSemiconductor memoryVideo gameParameter (computer programming)InterprozesskommunikationFeedbackArithmetic meanGame theoryComputer iconAverageCodeLatent heatEmailHome pageCASE <Informatik>Computer animation
49:53
Software developerInterface (computing)Inheritance (object-oriented programming)Sound effectDifferent (Kate Ryan album)CodeBuffer overflowoutputRight anglePerspective (visual)Computer programmingSoftware testingQuery languageSoftware developerMultiplication signInteractive televisionParameter (computer programming)WebsiteMeta elementSign (mathematics)Point (geometry)MereologyMeasurementGraph coloringStack (abstract data type)NumberHypothesisGenderMathematicsHome pageAddressing modeFeedbackLattice (order)DataflowOnline helpHand fanCycle (graph theory)Noise (electronics)Uniqueness quantificationSpeech synthesisSquare numberFunctional (mathematics)Greatest elementNeuroinformatikOffice suiteLecture/ConferenceMeeting/Interview
56:35
Software developerArchitectureModal logicMeasurementFunction (mathematics)Arithmetic meanData managementUser interfaceObject (grammar)Video gameBitView (database)CodeGreatest elementProduct (business)Functional (mathematics)Stack (abstract data type)Software developerProcess (computing)TwitterBuffer overflowPoint (geometry)Boss CorporationBuildingMultiplication signInterface (computing)Cartesian coordinate systemMereologyData structureComputer architectureModal logicMeasurementDifferent (Kate Ryan album)VacuumDecision theoryConnected spaceTouchscreenRight angleArmVaporRepresentation (politics)TheoryGodSystem callNumberBlock (periodic table)DataflowBit rateInferenceSound effectCAN busWebsiteComputer animationLecture/Conference
01:03:18
Software developerComputer animation
Transcript: English(auto-generated)
00:06
So, hello everyone, I'm Marco, I work for StackOverflow. I'm going to talk to you a little bit about pragmatic development today. So it's funny how there are a lot of late comers in this thing
00:23
and then the audience looks like this side is completely packed and there are tons of places there. Cool. Anyway, so they told me that English people are sort of they don't display much emotion, so they're hard to involve but they told me you like humor. Do you like humor?
00:45
No. No? Okay, there you go. So I'll start with a joke and you're required to laugh, right? Okay, so here we go. So this is XKCD, which is a comic of course and it shows some ineffective sorts, right?
01:05
So the half-hearted merge sort is like the sort that they ask you when you're trying to get a job and you're going for you know, they ask you, okay, can you implement a sort? And you try to do that and then it doesn't work so you sort of wing it around.
01:21
And there's no fool of this ridiculous sort. The alt, so XKCD has a tradition of having an alt, an alternate text to each of the jokes which has an extra joke. And the extra jokes in this one was the stack sort
01:41
and the stack sort basically means you run Google and Google Stack Overflow look for any JavaScript in there run it and then that's a sort, right? And someone actually did it.
02:00
Yeah, so stack sort connects Stack Overflow such as for sort lists and so on. So this guy actually did it and if we try, let's see if it works. Okay, sort, there you go, right? So you find the second answer, the second answer is code and the code compiles.
02:21
So this full Stack Overflow programmer. Anyways, this is an example of pragmatic programming though and that's why I brought it up. It's not about finding the right answer according to computer science but it's about finding something that works
02:40
and this sort works. You know, we can try again. It actually works. Let's try. It works. And it's fast so why not? Right, so what is pragmatic development? Pragmatic development is something that I first encountered
03:01
when reading this book, The Pragmatic Programmer and basically means not to be wedded to any particular technology. So have a broad background and then you can pick and choose the right technology for the right tool for the job and you do so not based on a theoretical knowledge
03:23
but based on measuring, right? So you actually try something. If it works, it works and it's good. If it doesn't work, it doesn't work. It doesn't matter what the theory says. If you don't do that, then you are wedded to this guy.
03:45
That's Sean Connery. So let me give you a few examples. This is a stack. And this is also a stack. But this is our stack, Stack Overflow stack.
04:02
So this is what we build the site on. So Windows, IAS, SQL Server and C Sharp. Every time I show this, everyone is like, yeah, you know, Windows. Why don't you use whatever, PHP? Why don't you use Node.js?
04:23
So much better, right? Node.js and Mongo is web scale. So the first answer to this is, because we are pragmatic, we started with this and this works. And I'll show you later what it means it works.
04:40
But also is that we don't really do WISC, right? If I have to put everything in here, it looks more something like this. Yes, we use Windows, IAS, SQL Server, but hey, we use Elasticsearch. Is that Windows? No. Is that .NET? Nope. We use Node.js for a bunch of things,
05:02
not for running the site, but for example for compiling our JavaScript into a single file or less. We use Redis, which is certainly not a Windows application. It's a Linux application sent to us.
05:20
So basically, we don't really care about being a WISC shop. We use it because it works. If it didn't work, we would change it. And we did change it, right? We needed something faster for caching and we decided to use Redis. Which is not Windows.
05:41
But we won't change what works. And by the way, when we say something works, it doesn't mean it's just good enough but a bit crappy. You know like a car that takes you from A to B but is sort of losing a piece of the car while you walk and it doesn't start when it's cold in the morning.
06:00
Now we mean something like this, right? A Porsche. That's what we mean by it's good enough. That's one of our developers, by the way. We should work here, right? Let me give you another example. So, MVC. Who uses MVC here? Hands up.
06:21
Like everyone. Is this MVC? No? This actually comes from Wikipedia on MVC. Just saying, hey, don't go. Okay, I can't see anything anymore. Fantastic. Yeah, so this theoretically is MVC.
06:43
In practice, my computer is going to sleep. It's dead. Be patient.
07:12
I swear I just see a black screen here. It's what?
07:23
Well, that is not frozen but I'm moving the pointer here. I think that's just, I don't know, just went blank.
07:51
I can describe the next slides if you want. Just, yeah, is it?
08:05
I don't, yeah. Yeah, wonderful. Oh, here we go. Yay, we're back. Okay, so this is our version of MVC.
08:26
So, does anybody recognize MVC in here? Well, there is, let me see. Models. Yay.
08:41
Use. Controllers. So, yes, that's an MVC but it's an MVC adapted to an actual application. Don't, it did it again. Bloody hell.
09:03
We need to threaten it again, I think. I don't know what's going on. Now, let's see. Okay.
09:21
This is going to be fun. Okay, so, but this is a real case, a real world MVC, right? When you build an actual application that does actually something useful and is not a blog post that you want to show to your colleagues, you actually need to have, for example, a scheduler, right?
09:41
So, basically any website I can think of needs a scheduler that runs at regular intervals and runs batches. You can't run everything online. You need probably some form of caching, right? And caching is not really part of MVC.
10:01
How do you fit it in? There are different concerns, right? So, there is the DB and the DB at our scale needs to be adapted. In our case, for example, we have a read-write instance and a read-on instance. We also have web sockets.
10:24
Good luck doing web sockets with MVC. So, reality is much more complicated than the fantastic theoretical world that you read on blog posts or you learn at university.
10:41
Let me give you another example. This is how theoretically our deployment process works. By the way, how many of you do continuous deployment? Still quite a number, better than last time I asked this, but still a few people.
11:02
So, this is continuous deployment setup, right? So, at the center of everything you have Git and, you know, you press a key, you build. Actually, you commit something to Git, it's built automatically and it goes on a dev instance of your site or your application. That's done automatically, right?
11:24
It's quite common to have that. On top of that, we have one-click deployments to the main network, Stack Overflow, and to canary or meta, which basically means that we have a few sites
11:41
which receive sort of beta versions of our software. We put it there so we can see it working at scale with a bunch of users. Before we deploy to the 150 sites that we have. This is quite a typical idea, it's not innovative in it,
12:01
except this is not exactly how it works, right? This is a rosy picture. Reality is that you probably want to have some feature switches. Feature switches are basically switches in software configuration, which is that disable some features.
12:22
Why? Because not everything can be just applied and it works. Sometimes you need to commit or to push stuff multiple times to your source control repository and you don't really want people to see it until you're ready. For example, a typical example is stuff that needs maybe some CSS tweaks.
12:44
If you do web development, you know that it's really, really hard to put everything together at the same time. So you're going to commit something and push it to the designers, designers are going to put up the Chrome. So you need to have switches.
13:02
But even switches are not enough. Why? Well, because sometimes you have features that take a really, really long time to be built. And once you have that, you probably want to be working in a branch.
13:21
But then if you have a team working in a branch, that team needs to have a development server so they can collaborate. So there you go. You have a branch on top and another build server, dev2, and then you need to merge and it goes to Git. And then once it goes to Git, it can go to Dev and then to Meta and SakuraFlow.
13:41
Oh, and you also have a switch. So you can turn it on and turn it off. Complicating enough? It's not enough. Because some stuff, actually, you can't really test the way you think.
14:01
For example, one of the tasks I was doing last year was changing the homepage. The homepage of SakuraFlow, if you're familiar, is a list of questions. Nothing complicated, right? Except this list of questions is a selection based on some notions.
14:22
For example, some chosen tags. So it's a selection of 10 million questions. We have 10 million questions in SakuraFlow. People go on their own page and we are going to show, maybe, all the ones in the JavaScript tag. And that is actually not simple to do.
14:41
It's something that, for doing that, we actually have a specialized server that we built that just does this. And it runs separately. ADB is too slow. So when we were developing this,
15:02
we didn't really want to put out new versions just for everyone. That would be really bad. Especially because if you break the homepage on a site like SakuraFlow that leaves off Google, that means you go out of business. Right? So we don't really want to do that. So what we did was an alpha program.
15:23
So some of our users were signing up and saying, OK, I want to see the crazy stuff. And then to them, they have a switch and then we can serve the pages to them. And they see, actually, the site vNext, right?
15:40
To use a Microsoft term. This is good because they can see, actually, the engine working on 10 million questions. If you go on Canary, if you go on Dev, and so on, we never, we don't have the hardware to actually run site on real numbers. So that's why we built that.
16:02
Again, this is a system which is adapted to the circumstances. So other things, other examples of pragmatic development are our coding styles. All the companies I worked at before SakuraFlow,
16:22
they sort of have their own company style. Some companies are like, yeah, we'd like to do functional programming. Or we are really into unit testing. Or we are really into object oriented. We like to do layered architecture, microservices, and so on.
16:41
And SakuraFlow is different, right? Programmatic development, in general, is different. It doesn't mean you do X. If someone comes around as SakuraFlow and says, I want to do functional programming, everybody's going to go. And this is not because there's something wrong
17:01
with functional programming. It's fantastic. But because we adapt our coding style to the circumstances, functional programming is great to do certain things. And we use it when we need to do those things. But in some other cases, you need to have other styles. You need to have more choices. And we use all of them. So a little game.
17:22
Are you guys up to it? Yes? Interaction? Too hard? OK, that's scary. OK, so I'm going to put up some examples here. And then you can tell me what kind of style
17:41
do you think it is. So for example, I'll start with this one because it's easy. This one is the part of our code that does the badges. Do you know what badges are on SakuraFlow? Yeah? They're like achievements. So if you ask a question that a bunch of people
18:02
liked and upvoted, then you get a badge. Yay. There are tons of badges, like 100 badges. And this is the code that runs it. Well, the structure of the code. As you can see, this is proper object-oriented code.
18:22
Right? You have a tree. You have a cache badge. And then you have SQL badge that derives from cache badge. For example, some badges, for example, the one that required the number of upvotes are typically dependent on a SQL query.
18:41
So all of those are 73 badges. They are SQL badges. Some other badges are code badges. And code badges actually talk to other applications via API. So they don't depend on SQL. They just depend on an API call. And then some event badges depend on some particular events
19:03
happening. In particular, there is one informed. It only happens when you go through our tutorial. So you go to the tutorial. At the end, it says, hey, you already won one badge. That needs to happen straight away. It can't happen in a badge.
19:20
And so it's an event badge, which is different. So object-oriented. Next one. You guessed this time, OK? So what about this coding style? Any takers?
19:40
Is that here functional? Let's see. Yeah, functional. Lambdas, fluid syntax, enumerables, and so on. Yeah, this is some code that does localization. And lots of lists there. So it makes total sense.
20:01
What about this one? Sorry? Nope. Yep. Aspect-oriented. So runtime behaviors defined by decorators. We have tons of them. For example, in this case, we have a track newsletter,
20:21
which means every time this route is invoked, we're going to track that someone has come from the newsletter and actually clicked on something, so we know whether our newsletter is useful. Another example of that is our ghetto-style event system,
20:43
which basically ties in some particular method calls with some events being fired elsewhere. For example, I was telling you before that we use WebSocket. Do you know what for? We use it to update the question you're on.
21:03
So if you're on a question and someone edits it or someone answers, you see a pop-up coming up that says, this answer has been edited. That is done through WebSocket, which basically sends the browser an event in real time.
21:21
That event is generated when, for example, a new answer is posted. So the new answer posted method will have a decorator on top, an attribute that says, fire an event to the browser, and then that will do that.
21:41
So we use a lot of this. It really, really simplifies stuff in a bunch of ways. More. What about this one? And here we're getting into the crazy shit, yeah? You don't have to read it all. It's just boring.
22:05
Procedural. Well done. So these are batches, right? So batches are a sequence of operations. You want to run the next one if the previous one succeeds, and that's what you do, right? One after the other.
22:23
You don't need to have an object-oriented framework around this. You don't need to have functional programming or anything else. It's the simplest thing, 80-style programming. It's effective. It works. So we use it.
22:40
What about this one? You have to speak up because I can't hear. Meta-programming. Meta-programming. That's it. Yes. So code that writes code. So why the hell would we do that? So we do that because sometimes it's needed to speed up operation.
23:09
For example, in the case that you are transforming JSON to objects, you don't know in advance which kinds of objects you're targeting.
23:22
Therefore, you can write it in a simple way if you want by looking at the JSON, looking at the objects, using reflection, and pairing up the properties to the piece of string that you get. And that works, except it's really, really slow. Why is it slow?
23:40
Because you're doing a lot of work that you don't actually need to do every time, right? So you don't need to... If you deserialize something on, I don't know, on an object that you know multiple times, you literally need to do the reflection only once, only the first time.
24:06
The other times you don't need to do the reflection, which takes a long time. And the way we do it is we do the reflection the first time, and then we generate a class. So we generate the code and the intermediate language,
24:23
so the next time there will be no reflection. It's just like a class which is written by a programmer. This may sound esoteric, but I can assure you if you use any JSON deserializer which is not completely crap, it does that behind the scenes, right?
24:42
Even Microsoft ones, that's how it's built. And we do a lot of that, right? We have our own JSON deserializer, but we also have protocol buffers, for example, and we use that. We use this kind of thing in other places,
25:04
in really, really hot places. For example, in the engine that I was telling you about, the tag engine, the engine that, given a tag, gives you all the answers. That tag engine needs to scan through 10 million questions every request, literally. So that loop needs to be really fast.
25:23
And some of the code inside that loop is actually written via metaprogramming. But don't use metaprogramming if you don't need it, because this is crazy code, yeah? This is really, really hard to debug, by the way.
25:41
So this is really crazy. Okay, you won't catch that, but... This is weird stuff that saves millions of allocations. So it's not really a coding style, but it's some of the things you need to do. So what happened here? What happened is that Microsoft made a bad decision
26:03
when they created the API for string splits, right? So when you split a string, you have to pass some sort of character that tells you what to split it on, right? So you can split on a space.
26:21
The problem is that if you just pass a character, if you write that character in code, so if you literally write string.split, brackets, space, that's fine, okay? Because the compiler is smart enough to know
26:42
that it doesn't need to create a string object for you in order for you to do the string, if you do the split. It realizes that there is a string there, strings are immutable, therefore it just creates one copy of the string. So if you have a web server that serves millions of pages, all that do the string split, it will know that.
27:07
It will know that it doesn't need to create a million strings with a space in it. So this is called interning, it's part of .NET standard. However, string split doesn't take a string,
27:20
it doesn't take a single char either, it takes an array of char as a parameter. An array of char cannot be interned because it's not immutable. So we created our own array of arrays, of chars, so they can be made static,
27:40
and we do our own ghetto interning to go around the limitation, and this is dead again. The limitations of .NET, I swear that was the last one.
28:32
Still the same problem.
28:40
Could it be the connector for some reason? Maybe this one is better?
29:02
That's also true. OK. OK, so that was the last one. Anyway, so Donald Nutt said,
29:26
the premature optimization is the root of all evil. And I sort of don't agree with that at all. It's a bland statement. Because what is premature optimization?
29:41
When is optimization not premature anymore? Literally, it's impossible to tell. You know, you can do every time, I don't know if you tried this, but I did, every time I tried to say, let's optimize this. Everybody's come up being a smartass and saying, premature optimization is going to be the root of all evil.
30:00
Because you can always say that, right? The truth is, fact-based optimization is important. Right? When you have facts and you can prove that you need to optimize, then it's the time to optimize. So, how do we do it? We measure everything. And let me show you.
30:21
So the first thing is mini-profiler. So all these tools that I'm going to show you right now are open source tools. You can use them, download them, and use them on your own setup. And I actually encourage you to do that. So this is Stack Overflow. I'm sure you've seen it before.
30:42
The only difference from your vanilla Stack Overflow is this stuff over here. Can you see it? Up in the top right corner. There's a tool called mini-profiler. Does anybody use it? One, two, three. So what is it?
31:01
It's a profiler that runs every request or on a subset of requests up to you. In this case, it runs because I'm a developer. So what does it do? What it does is profiles every request. It tells me exactly what happened, right?
31:21
So what happened in this request? You know, I spent 21.1 milliseconds in the on-action executing branch. We did a lot of things. And then in the end, we spent some time doing SQL, sometimes accessing Redis, for example,
31:41
in this case. I can see, let me see. I can go in and see, OK, these are the SQL calls that have been made. It tells me that this one is actually a duplicate of this one, so that's bad.
32:00
You don't really want to have multiple calls which are the same in SQL. It's doing different selects and whatever. As you can see, it's giving me executable SQL, right? So I can actually take this,
32:20
copy it and paste it in my SQL client and run it and replicate. It's telling me this took two milliseconds. You know, if there is a problem, I can actually go here and see what happened. This is very important because the consequence of using a tool like this
32:42
is that as you dogfood your own application, you find the bottlenecks straight away. You know, you go to that shitty page that doesn't load and you know why. You know, you can't fix it. And it's the proof, right? It's not premature optimization anymore.
33:02
We do a bunch of other things with it, of course. For example, we store a few... Well, media profiler does that. It stores a few copies
33:21
so you can actually share it, for example. You know, and see it in a probably more readable way. You can also compare this with the other. So if there are ten snapshots for the homepage, you can actually compare how long it took and see the worst.
33:43
Sometimes it's useful. So this is the first tool. This tool is called... So mini profiler. And well, this is the link. You can find it on GitHub if you need it.
34:03
It's available for .NET, for Ruby, for JavaScript, Go, I think. So useful. Another thing is... OK, so wouldn't it be really cool
34:21
if you could do something like this, right? So if you want to see, for example, OK, what am I going to optimize today, right? What am I going to fix? So it would be great if I could just go to the logs and just do a SQL select that says,
34:41
OK, give me the stuff that takes longest. Argh! We actually do that, maybe. I can show you.
35:00
I'm so sorry about this, but there's nothing I can do about it. It seems to me that every time something goes to sleep, this dies.
35:22
I think when it goes to sleep, it doesn't come back properly. It's sort of disconnected, so...
36:15
Yep, something happened.
36:54
OK, let's see if it works. Cross our fingers.
37:09
OK, so we actually built that. So we have something that you store, we store the results of the profiler or part of them,
37:25
we store them in a SQL database, and then we have an interface where you can go and query everything, right? So you can get something like this out. So you can say, OK, the route comments forward slash comment
37:41
runs and it has 19 SQL queries in it, on average it takes 12 milliseconds, and it does 38 Redis lookups and so on. So you can get this kind of information, this enables you to focus your work, not only on that route that you don't like very much,
38:03
but on the routes that actually matter, the routes that people are actually looking at. So that's the other side of it. There is a tool that allows you to run SQL from the web browser, it gives you all the bits and pieces,
38:22
it gives you the query plans, and everything is called Data Explorer. Again, this is a free tool that you can download. Another tool is Ops Server, and Ops Server is our status monitor. Let me see if it works.
38:42
I can show it here. If it works, it works. No. Sorry, I'll show it here. Basically what you can do with this, I can show it to you later after this talk if you want,
39:01
you can monitor all your servers, see which ones are performing, which ones are not performing, in particular if you go on SQL, which is in many cases the bottleneck that you have, it can drill into SQL, see which SQL queries are actually hosting your server. For example.
39:21
And you can see these are sorted by average microseconds per minute that the query is taking. So it's the number of times something, some query happens times the average time it takes. This thing takes about 20 million microseconds per minute,
39:41
which is a lot. But if you look at it, it's just got 30 seconds. So it's not that relevant maybe. You can see here the query. If you drill down in the query, you can see the query plan again. You can do stuff like removing the plan.
40:02
And this reminds me of actually a joke, but it's not a joke. So there is one speaker, I don't think it's here, but speaking in another section, John Skeet is our number one user. And we have, it creates problems, right? Because, you know, a normal user on average,
40:22
maybe answered one question, two questions, something like that, he answered maybe 10,000 questions, something completely insane. So what happens is that, no, he was not asleep.
40:43
So what happens is that when John Skeet is the first guy that goes to see, for example, his list of questions, SQL server is going to see that and say, hey, this user is trying to see his list of questions.
41:00
How am I going to optimize this query? I'm optimizing it for, because users have 20,000 questions, 10,000 questions, right? I optimize it for a lot of questions, which of course is great for John Skeet, but it's really bad for everyone else. And everyone else gets the same query plan as John Skeet.
41:21
So this is the John Skeet problem, right? And so when we hit something like this, we can simply go, you know, okay, it's going back by itself. That's how it works. You don't need to do anything, it just goes back. So you can go here, click on remove plan,
41:40
just hit problem, it's gone. And that's actually there for that reason. And this is real. So observer, again, is a real-time server that shows you all the status of your thing. Another thing it does that I'm not showing here, for example, it shows all the exceptions happening on your systems, right? So if something goes bad,
42:01
and you do that bad deployment that everybody does from time to time, you see a shitstorm of exceptions like, bam, 10,000 exceptions coming in. It does that for you. Another tool that we have is Bosom, which is a real-time serious explorer.
42:21
Let's see if it works, this one. No, it doesn't. So what this does is it takes all the stats of your servers. For example, one thing that is a problem for us,
42:42
and I'm not saying it's a problem for you, and it likely is not a problem for you, is the number of allocations, right? Why are the number of allocations a problem? Well, normally you couldn't care less. You just write code any way you want, and it doesn't really represent a problem.
43:02
When you're at scale, though, if you write code that creates a lot of allocations, and this happens for every request. It happens for every user that comes to the site. And this means that if there are too many allocations, there are too many times 1 million,
43:22
too many times 10 million. It means that basically with too many allocations, you are allocating a lot of memory, and then the garbage collector has to clear all that memory, and this can become a problem because you have garbage collection stalls, right? At a certain point, garbage collector is going to say,
43:43
OK, screw this, you know? Wait a second, I'm going to clear up all the mess that you made, and then we can start from scratch, which is great, except it takes like a minute. And so that user, or those users that requested a web page in that minute will wait for a minute.
44:01
And that's not good if you're serving a lot of developers, as we do. So we need to monitor the allocations, and we need to monitor the garbage collections. This is an example of what this tool does. It monitors all the garbage collections happening. So you can do things like, I go and see a web01, that's a web server in Colorado,
44:24
that we have. I want to see how much time, on average, is spent on garbage collection. And so you see for, these are two minutes, and, you know, how much time, 10% peak there, but on average, it's much less.
44:41
So you can see it probably here, something happened between here and here. So this is more normal, this is not very good. We monitor that because that affects our holding style. For example, a lot of times it would be beautiful to write something in a link, right?
45:02
Or in a functional style, except, you know, functional style has issues with allocations. It tends to allocate a lot. And if you do it over a big list, it becomes a mess. So we can't do the things the way we want, but we have to do it in a different way
45:20
because we have to adapt to our condition and we can monitor it with the tools. Again, this tool is free. So you can download it, Bosun. I'm going to share this on my Twitter account later on. So I've shown you some of the tools that we use to code.
45:43
Now I'm going to show you some of the tools that we use as a company to enable this because coding and measuring everything is not enough. And probably in a lot of cases, maybe you don't care that much about hardcore performance.
46:01
So a tool that we use is RFCs. And if you don't know what RFCs are, this is an example. It's called IP over Avian Carriers. And it's a joke, of course. It came out on this edition of making joke RFCs
46:21
on the 1st of April. RFCs are what governs the Internet. If some entity wants to make a change to any of the protocols like how mail works, they publish this request for comment document and then people comment it, the comment is formed
46:40
and they debate the pros and cons and in the end they come up with a specification which is also called RFC. Of course, and then if you want to run your Internet with homing pigeons, you can do that thanks to this. You may think this is just a joke.
47:02
Is this just a joke? It isn't because someone actually did it. So this is a screenshot of the ping of using carrier pigeons to the Internet and as you can see, latency is a bit high.
47:21
You can imagine, I mean this is literally done by taking a carrier pigeon and printing out the packet on a piece of paper and then putting it on the little leg and then it flies and then it comes in and then you take it off the little leg, put it in the scanner and that's the ping. So, yeah, two ways.
47:41
So we use RFCs and sometimes they are jokes like this one but in most cases there's actually people proposing new things and changing how we use the site. So internally people suggest things all the time. This allowed, for example, one of the things we did yesterday
48:02
is we made our sidebar slightly larger which is like a, who cares, right? We care because in that way the ads that we put there, they are the standard size.
48:21
The fact that we can use standard size ads means that we have more people that want to put ads on. The fact that we have more people that want to put the ads on gives us more leverage to choose which ads go on and therefore we can ensure that we are not showing crap ads like Punch the Monkey, animated ads
48:41
and we just have really intrusive ads. One of the things we do is by following internal standards and that allowed it. And that change came through because a guy who was the head of sales of ads wrote an RFC. He wrote an RFC, he sent it to the company
49:01
and then everybody saw it, everybody got pissed off at it, everybody commented on it and he got really offended and then we talked and then in the end it came out and it was an idea and it was implemented. I sent out many RFCs. For last year I spent a lot of time thinking about how to redesign the homepage so it's more efficient
49:22
and pretty much everybody in the company has an opinion on how the homepage should be and therefore RFCs, I sent maybe three or four proposals so people can give feedback, tell me how much I suck at life and I can tell them, no, you suck at life and then in the end we get somewhere.
49:43
So RFCs are very important, inter-process communication for your workplace. Another thing that's really close to my heart is a way of terminating arguments.
50:02
This is XKCD again. So citation needed. You can say whatever you want but facts trump theory. So you're going to say maybe, I think that this change will increase the number of registered users
50:23
and now the guy is going to say, no, I think it will decrease it because of these reasons. All this bantering is not good for a company because it leads nowhere, because we're all talking about the hypothetical effect of something that we don't know what it's going to do. So how do we do it?
50:41
We do an A-B test. We do a test and then we measure it. Does it increase it? Does it not increase it? That's the end of the argument and so many arguments just end by saying, okay, let's measure it, let's see it and so on. So measuring stuff is pragmatic. It allows your company to proceed,
51:01
your code to be written in a way that's not going somewhere random because someone decided up top. It allows anyone to actually prove their point. And many of these things actually enable developers. Other stuff, I don't want you to think that
51:22
all the work that we do is simply driven internally. As a company, we actually rely a lot on our community. I know that statistically only a minority of our users go to the meta part of the sites.
51:41
Am I speaking Chinese here? Do you know what it is? A few people do. Okay, so I'm going to explain it for everyone. So there is Stack Overflow. You're all familiar with it. We have a site called Meta Stack Overflow because we like Greek. And it's a Stack Overflow to talk about Stack Overflow. So you go there, you ask questions about Stack Overflow.
52:03
You can ask, why do you close my question? You ask that. Or you can ask things like, and that's what we do, is do you like the new homepage? And then when you spend six months doing it, someone answers that, which is fantastic. I've been using it for ten minutes and it sucks.
52:22
It's like, yes, I win at life. So, again, this is also a way of being pragmatic. It's a way of developing with the community, getting feedback. If you have a community which is sizable and engaged, do that. Another thing is pair programming, right?
52:42
Pair programming, Agile. Do you guys do it at your own office? So what is pair programming? One developer next to another developer? One computer, two developers? Yeah, that's not how I like to do it. How I like to do it is one developer on the right
53:03
and one designer on the left, for example, or one QA on the left, except we don't have QA, but I did it before. So across function, do pairing. It's a completely different thing. It allows people to talk. It allows people to create something which is unique.
53:22
When you create a UI, there is the input that a designer can give, which is very important, of course, and very visual, if you like. And then the input that the developer can do is very factual. It's very, okay, so this query is going to suck, so can we change it?
53:42
And this interaction together is something novel that makes everything much better. So these are the tools that you can use in a team. Okay? Let's see, what else do we do? This is actually the sign on top of our toilets at work.
54:07
I'm sorry if it offends you. So another thing that we do is we talk a lot about diversity, and I know people are going to say that's boring.
54:22
Except I think we are quite unique in that regard, in the sense that what do we consider diversity? Diversity means a lot of people with different ideas, people with different perspectives. It doesn't matter what gender you are, what color your skin is.
54:41
Nobody cares. What we care about is how unique your contribution is. And this is important. Again, if you want to make RFCs work, if you want to have a code base like ours, which is full of different things, right?
55:00
It's pragmatic because we have 20 different techniques to choose from. But, for example, I knew nothing about aspect oriented programming before I joined. Someone else did. And their contribution was essential because, hey, look at what they did. They did something fantastic with the code base. Their unique perspective helped me to become a better developer
55:23
and help our code base to become better. So this is intertwined, right? There's no point of having, you will never have a pragmatic company or a pragmatic development if everybody is agreeing on everything. That doesn't work. You want people arguing on everything all the time.
55:42
You want people to disagree. Disagreement is good. Okay? It doesn't have to be nasty. It's not nasty, but people don't agree. That's fine. We measure it and we decide. And it's normal to disagree. Okay? So this is what we think diversity is.
56:01
And this is very important to us. And it's very functional to pragmatic development. Another thing is this. You can notice the fantastic visual arts and the fantastic skills of our CEO.
56:20
Joe Spolsky drew this and he sucks at it, I think. But this is how he sees the company, right? As you can see, it's like there's a CEO, there are VPs and the team leads and so on. Except, so it's like a normal company, right? Except the CEO is at the bottom, right?
56:41
So the bosses are on top, bosses, and the people on the bottom are the managers. On top, there are people that actually do stuff, developers, designers, that they have deliverables. Apart from marketing, which nobody knows what they do,
57:01
that's why they have a dinosaur there. Their team leads and their VPs and the CEO, ultimately, their job is to enable them. Their job is to make them more effective. This is called, it's not something that we made up, it's something that is a lot of theory,
57:22
and it's called pluralistic companies. We are not the only one, S is another company, for example, that does it. And it basically means that decisions are made on top. And on top means by the people who do stuff. And the people on the bottom are the people who enable them. Maybe they give them a general direction,
57:40
but then how stuff is done, how stuff is decided, and so on, is decided on top. Think about it. Do you really want to work in a team where your team lead is going to tell you which coding style to use? Everybody will hate it. I've been a developer for 20 years. If I see a guy telling me how to code, I'm going to be offended.
58:02
I think I know how to code, and I think I can prove it with facts. Sometimes I'm wrong. Most of the times I'm wrong. But at least my voice is heard. And that's what this structure does, and so it's very important.
58:21
So let me recap here the techniques I talked about. You can pick and choose whatever you want. They all work. If you use all of them, it's much better. They multiply, but singularly, they're also great. So first thing is architecture is driven by objective necessity. Remember the MVC with all the bits and bobs?
58:42
If someone came in and said, that's not how you do MVC. MVC, you have only three blocks. It would never have happened. We wouldn't have a site as functional as today. I've had many discussions in the past about architecture, how it should be done. The right answer is this is what we need,
59:00
and we'll do it. So we measure it. Build it a little bit at a time. This leads to measuring everything. And from day one, measure all the bits and bobs of your application. See how much you're allocating, if that's what you're into. See how long it takes to make a request.
59:21
See how many objects is your interface building. See how long your build takes. We take a lot of pride in making our build faster. Whatever you can think that can be measured, do it. And keep it forever if you can,
59:41
because seeing trends over time is very, very useful at times. Another thing is allow discussion across the company. If you can, I know that most of you are developers, so you can't make this difference. But it's important that people talk, that people discuss. Discussion is fine.
01:00:00
And last company I worked for before Stack Overflow discussion was seen as evil. It's like, ah, you are actually disagreeing with your boss? Oh my God, you must be a terrible employee. This is not how you allow good ideas to emerge. Even if I was wrong 90% of the time, it doesn't matter.
01:00:22
Let the facts win, no politics. Once you measure everything, what's the point if you just throw it away and let the person with the best suit win? If you have a community, involve them. You know, so many companies just go in a vacuum.
01:00:40
Never talk to their communities and just come out with products which are not fit to the market. Talk with your community if you can. Ask them what they want. Pairing across functions. Again, let different views influence how code is written, right? If only developers pair across each other and all the developments
01:01:03
is done in a bunker of developers, that's not going to work. I mean, that's going to work, but it's going to be sub-optimal, right? The designer is not going to be happy. If the designers do all their UI design and then the developers are just code monkeys that glue it together,
01:01:23
that's not how you write the best application, really. And this is the main point of, again, of, you know, pluralistic company with diversity. You know, don't tell people how to work, but tell people how to work together.
01:01:40
Our managers spend a lot of time coaching us, developers, to actually not be assholes to each other. Because, hey, I spent all my life becoming a better developer. I didn't spend all my life becoming a better, you know, easygoing person that is good at arguing with people and shaking their hands.
01:02:00
And I'm not the worst, OK? There are people who are really, really struggling, you know? And it's our manager's job to actually help them to be more effective as developers, you know, to be able to express their ideas without calling other people retarded all the time.
01:02:20
Again, this is maybe controversial, but the best ideas do not usually come from your CEO or from your manager. Why? Because, first of all, they don't have all the facts. They have a part of the facts only. You have another part of the facts. And also it's just a low of large numbers. There are ten developers and one leader.
01:02:42
Sooner or later, one of the ten developers is going to have a better idea. You know, you need to acknowledge that. So that concludes my presentation. If you have any questions, I don't think we have time right now, but I'll be around the rest of the day. So if you want to see the tools in practice,
01:03:01
I'm sorry the connection didn't work and the screen sucked. I can make it up to you. I'll be around available if you have any questions. So thank you very much.