How Disqus is using Django as the basis of our Service Oriented Architecture
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 66 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19935 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201466 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
ArchitectureBasis <Mathematik>Web serviceMultiplication signMaxima and minimaTwitterRevision controlComputer animationLecture/Conference
00:33
Service-oriented architectureDifferent (Kate Ryan album)Pattern languageDifferent (Kate Ryan album)XMLComputer animation
01:02
Service (economics)Similarity (geometry)Data recoveryOffice suiteComputer animationLecture/Conference
01:25
Pattern languageStandard deviationAxiom of choiceComputing platformCartesian coordinate systemXMLComputer animation
02:11
Service-oriented architectureService (economics)Computer architectureXML
02:41
System programmingSoftwareService-oriented architectureService (economics)Discrete groupInterface (computing)Function (mathematics)Internet service providerWritingComputer programDatabaseQueue (abstract data type)Mobile appWeb pageSingle-precision floating-point formatFacebookTwitterWeb serviceIntegrated development environmentIndependence (probability theory)ScalabilityData integrityDiscrete groupPlastikkarteInternet service providerComputer programmingINTEGRALPhysical systemTraffic reportingService (economics)Type theoryStreaming mediaAbstractionBell and HowellFormal languageSoftwareMultiplication signRight angleForcing (mathematics)SequelSoftware developerInterface (computing)Computer fileElectronic mailing listCopula (linguistics)Atomic numberOrder (biology)Web pageProcess (computing)Component-based software engineeringQuicksortSource codeView (database)Virtual machineLatent heatCategory of beingMobile appQueue (abstract data type)Java appletWeb 2.0Software maintenanceHeegaard splittingComputer animation
05:48
Database transactionProcedural programmingQueue (abstract data type)Data modelLogicSystem programmingService (economics)International Date LineIDLCommunications protocolHost Identity ProtocolFile formatEmailSoftware frameworkRemote procedure callComputer networkAuthorizationAuthenticationTask (computing)Data managementParsingBeat (acoustics)FrequencyLoop (music)Point (geometry)Insertion lossQuicksortObject (grammar)Codierung <Programmierung>DatabaseMessage passingPhysical systemInformationSoftware frameworkLatent heatAuthenticationPoint (geometry)SubsetEndliche ModelltheorieMultiplicationData dictionaryCodeFacebookLine (geometry)BuildingLoop (music)Semantics (computer science)Library (computing)MathematicsCartesian coordinate systemElectronic mailing listRepresentation (politics)Formal languageServer (computing)Communications protocolData managementSoftwareDependent and independent variablesGame controllerFile formatMappingAutomatic differentiationBefehlsprozessorWorkloadDatabase transactionLogicPattern languageWeb pageComputer configurationNumberOrder (biology)Connected spaceCycle (graph theory)Process (computing)Web 2.0Remote procedure callCuboidRevision controlInheritance (object-oriented programming)Cross-platformTask (computing)ParsingBeat (acoustics)HookingBlogMereologySoftware testingRankingAuthorizationSolid geometryTexture mappingSpecial unitary groupTransportation theory (mathematics)Uniformer RaumElectronic program guideHypertextHeat transferService (economics)Information managementBus (computing)Client (computing)Exception handlingMultiplication signForm (programming)Source codeInversion (music)AreaIntegrated development environmentUniform boundedness principleComputer programmingEmailAverageComputing platformForcing (mathematics)Goodness of fitXML
13:00
Task (computing)Code1 (number)Interface (computing)Product (business)Message passingDecision theoryEndliche ModelltheorieInformationWeb 2.0Lecture/Conference
13:47
CodeEntire functionWeb servicePhase transitionScale (map)PlanningRead-only memoryComputer networkPoint (geometry)Different (Kate Ryan album)Uniform resource locatorMiddlewareContext awarenessObject (grammar)Template (C++)Point (geometry)Ocean currentService (economics)Regulärer Ausdruck <Textverarbeitung>Revision controlImage resolutionDifferent (Kate Ryan album)Data centerSemiconductor memoryRight angleContent (media)Roundness (object)Graph (mathematics)Greatest elementLibrary (computing)Matching (graph theory)Set (mathematics)Forcing (mathematics)MassState of matterVirtual machineElectronic mailing listMultiplication signBefehlsprozessorContext awarenessComputerPattern languageGene clusterCodeComputer fileEntire functionCycle (graph theory)RoutingProxy serverOrder (biology)Line (geometry)Product (business)Projective planeEndliche ModelltheorieFood energyDatabaseMiddlewareObject (grammar)MultiplicationUniform resource locatorRepresentation (politics)Raw image formatMultilaterationMilitary baseStructural loadComputer animation
17:27
CodeRevision controlData integrityFunction (mathematics)Entire functionLine (geometry)ScalabilityIntegrated development environmentIndependence (probability theory)CodeINTEGRALService (economics)Functional (mathematics)Independence (probability theory)Gene clusterEntire functionRevision controlUniform resource locatorLine (geometry)Traffic reportingZeitdilatationUsabilityMultiplication signSoftware testingPattern languageScalabilityMereologyMetadataIntegrated development environmentComputer animation
18:35
Ad servingProduct (business)Cartesian coordinate systemFlash memoryDecision theoryXMLUMLLecture/Conference
19:05
Encapsulation (object-oriented programming)CodeRemote procedure callServer (computing)Series (mathematics)Software frameworkData managementBeat (acoustics)Service (economics)Line (geometry)Cache (computing)Self-organizationGamma functionInternetworkingEmbedded systemScripting languageData managementMassInformationAutomatic differentiationCache (computing)Web 2.0Service (economics)CodeLine (geometry)Different (Kate Ryan album)StapeldateiPhysical systemPerturbation theoryQuicksortRandom matrixGraphische ProgrammierungFront and back endsServer (computing)Boom (sailing)Maxima and minimaMappingDatabaseSoftware frameworkPoint (geometry)Task (computing)Web serviceInternetworkingComputer architectureSoftware testingClosed setEndliche ModelltheorieProcess (computing)Basis <Mathematik>Term (mathematics)Computer animation
21:45
CodeService (economics)Term (mathematics)Physical systemEntire functionSoftware testingScalabilityDisintegrationIntegrated development environmentIndependence (probability theory)Data integrityCodeRevision controlClient (computing)Service (economics)Traffic reportingSoftware testingINTEGRALAutomatic differentiationMilitary baseInformationMultiplication signPlastikkarteFrequencyPhysical systemTerm (mathematics)Core dumpProcess (computing)2 (number)Line (geometry)Library (computing)Goodness of fit3 (number)Network topologyEntire functionComputer animation
23:58
Service-oriented architectureSystem programmingWordPhysical systemIntegrated development environmentControl flowCodeComputer animationXMLLecture/Conference
24:38
Pattern languageDecision theoryPoint (geometry)Communications protocolSoftware frameworkWikiWeb pageLine (geometry)Decision theoryVirtual machineCommunications protocolPoint (geometry)Pattern languageEstimatorGoodness of fitMultiplicationLink (knot theory)Software developerRight angleSoftware frameworkLoop (music)Computer animation
25:37
DatabasePhase transitionService (economics)CASE <Informatik>Revision controlXMLComputer animationLecture/Conference
26:11
Physical systemSoftware frameworkCodeEndliche ModelltheorieSequelDatabaseService (economics)DemonMetric systemStatisticsPoint (geometry)ExistenceFrequencyTable (information)Different (Kate Ryan album)InformationSoftwareOperator (mathematics)Variable (mathematics)Phase transitionCartesian coordinate systemThresholding (image processing)Physical systemMathematicsBitComputer hardwareMultiplication signNumberMetadata1 (number)WordType theoryPlanningFile formatSequenceXMLLecture/Conference
29:17
Physical systemSoftware frameworkMobile appCartesian coordinate systemMereologyCodeSoftware architecturePhysical systemProjective planeAsynchronous Transfer ModeControl flowLevel (video gaming)Set (mathematics)Message passingInformation overloadService (economics)SoftwareDependent and independent variablesFitness functionComputer hardwareMultilaterationXMLComputer animationLecture/Conference
30:27
Remote procedure callGoogolService (economics)CodeMilitary baseFacebookMultiplication signXMLComputer animation
Transcript: English(auto-generated)
00:15
All right, the fast version, because we're 10 minutes in. Apparently Macs don't work every single time.
00:22
Hi, my name is Adam Hitchcock. I work at Disgust as you just heard. You can follow me on Twitter or something. Shit, okay, cool. Sweet, let's just start off the bat. We're hiring, but we're also American. So, if you want to leave Europe, come talk to me, but that would be crazy.
00:42
So what am I going to talk about? First, I'm going to address the issue of why did I lie to you last year? Then I say, what's a SOA? Why should you SOA? What are different data patterns in SOA? How does Disgust do it with both a legacy and greenfields example? And
01:00
Does that work for us? So last year my talk was entitled, How Disgust Does It When It Isn't Django. And it was a placeholder for services. It was a talk about a very similar topic, but saying specifically, hey, you don't have to use Django for everything. It's not what the cool kids are doing. But
01:24
why do I sit on Throne of Lies? That's the main question. So when I got back from EuroPython last year, my CTO basically said, hey, we're gonna make a technology choice and that technology choice is Django. We're gonna redouble our investment in it, because we are a huge Django shop, so we already are really good at it.
01:46
And there's some really great reasons for it. Using standards across the company that are also known outside the company makes it super easy to hire, and there's a huge community. So we want to leverage all that in both our legacy application and the platform we're moving to.
02:01
So I kind of had to challenge my thinking about Django and find some new ways to use it. So here is take two, is how Disguest does it, Django edition. So if I ask you to raise your hands, who will raise their hands? Alright.
02:20
So about 60%. Cool. Who already knows what SOA stands for? Cool. That's like a hundred and twenty percent, because more of you raised... Were you correct? I guess. So it stands for Service-Oriented Architecture, or if you're British, Service-Oriented Architecture.
02:42
And I'm going to define what I think that means right now. SOA is something that I'd like to pronounce, TLAs or TLAs, you know, three-letter acronyms. So a SOA is a system that you've architected in a specific way. That system should have discrete software components, or
03:02
those are basically running programs, and those are called services. Those running programs should have simple, well-defined APIs or interfaces, and those are the APIs through which they're going to communicate. Then you can loosely coordinate or loosely federate these services to achieve some sort of business goal.
03:21
In a SOA, there's two primary roles. There's a service provider and consumers. These are providers and consumers of data, and you can totally do both. That's cool. We're not going to mark you points for that. And it's not a new idea at all. Doug McIlroy, who I think he worked at Bell Labs a long time ago, said this of the Unix philosophy.
03:44
The new philosophy is write programs that do one thing and do it well. Write programs to work together and write programs to handle text streams, because that is a universal interface. And he was talking about Unix pipes and programs talking over file descriptors on the same system.
04:03
We're talking about network pipes and programs talking over file descriptors over the network. So they're really the exact same philosophy, and I think this still holds true today. It's also not necessarily a new idea on web systems. People have been doing this for many years. You're probably already using services in your system. How many of you are Django app developers day to day?
04:26
Sweet, a lot of you. Excellent. You're probably using Postgres if you're doing it right, or MySQL if you're doing it also right. Or Redis. You're probably using queues in your system. Rabbit or Kafka. Kafka is not technically a queue, but it's basically a queue, so I put it on the queues list.
04:44
Or you're using external API. Who knows what Kafka is? Sweet, I will talk about that later. It's awesome. If someone asks me, it reminds me. So external APIs are also services. Everything is a service. Single-page Java apps are great examples of consumer services, usually. But if they do a post command back to your server, that's now producing data.
05:05
So why should you do a SOA? This is basically my report card for later in the talk. I think that if you've done it right, you can have services written in any language. It doesn't matter what machines those services run on, and if those services have a strong API,
05:23
it doesn't matter how they store or abstract that data internally. It's just the API that matters. Services should scale independently of each other, and they should also provide you easier testability, easier deployment. And because you've split your system up into well-defined chunks,
05:40
it should be easier for individuals to maintain conceptual integrity or understanding of the system. So hopefully you can learn how to do all those things in this talk. I think that in a SOA it's really important to know, like when you're saying, okay, sweet, I want to make a SOA. Well, how? So I think that you have to understand the data patterns in your system.
06:00
And I see two primary data patterns. There's kind of transactional data and asynchronous data. Transactional further is divided by into two areas, I think. There's, are you doing transactional data against basically database models, some sort of straight representation of an object in your system, like a user.
06:22
Then REST works really well. It's CRUD semantics against a model. If you need to do something where you're combining data and math or logic to say, hey, recommend something to me or can I do something? Then RPC is a model that works really well, and that's a more procedural thing. If your response is the same every time, then REST might work well.
06:44
That's kind of the way I think about it. Then asynchronous data maps really well to queuing systems or pubsub systems. It's really good when you have high CPU or long-running workloads that you can't do in a web request cycle.
07:00
So you also have to pick your APIs. There's two parts here. You need to pick what are the bytes that I'm going to send to the other person or other service, and how am I going to send those bytes to the other service? How many people know what protobufs or thrifts or message pack or avro is? All of those are examples of non-Json encodings of how do you put an object
07:23
in a format that other languages can read. So picking something from that list is super important because in order to have a heterogeneous environment, you need to have an encoding that multiple languages can work. Pickle is not on this list.
07:42
Yeah, I just spent the last week removing pickle from something. So you also have to pick your transportation protocol. So HTTP, who knows what that is? Literally every person in this room, that was the lowest number of hands.
08:01
So HTTP stands for Hypertext Transport Protocol, since no one raised their hands. Or thrift, thrift provides a network binding as well. Well, I like HTTP and Json because they're super easy to do. Django does them really well out of the box. And HTTP is great if you actually read the spec,
08:23
or don't even read the spec, read the Wikipedia page on the spec. You'll learn a lot. Things like the accept header. So you can start with HTTP plus Json, and if your server respects the accept header, you can support multiple encodings when you go forward. You can say I accept application Json or text Json.
08:43
In the future you can say I accept application protobuf, or protobuf version two, whatever. And then HTTP works great too. It has things like keep alive as a connection option. Negotiating that initial connection a little expensive, but when you're going forward with HTTP,
09:01
it's like if you're keeping the connection alive, it costs about the same as any other protocol. If you need to save the eight bytes you get from HTTP at thrift, I think it's sweet or Facebook. But if not, anyways. So transactional data stuff. REST, Django does it really well out of the box.
09:22
That's basically what it's been doing for the last, I don't know how, like million years. So but in recent years, we used to use like Django forms, and those suck, and other Django things that are easy to make bad practices in. Django REST framework is a relatively recent framework that makes it super easy to keep your code organized
09:43
and clean while providing RESTful access to your database models. It's really cool, check it out. There's a Kickstarter campaign, it's awesome. Or you can roll your own API. There were definitely, we hit lots of performance nags with REST framework, but we also got around them very easily. Come talk to me about how I do awesome caching things
10:02
and versioning in that framework. I need to probably write a blog post about it. Cool, RPC is the other transactional. Like I said, logic heavy APIs, recommendation services, authorization, authentication. I don't like RPC as a first stop because it's prone to over specialization when you're building APIs.
10:21
You make APIs that are really good at doing one thing, and when you're building a platform, you wanna make APIs that can support lots of things to be built on top of them. Sometimes you need a specialized API, but if you can do it in a generalized way, I think that's better.
10:41
In other systems like Thrift and Zero RPC, which are super easy to use, that's their downfall. You can abstract away the fact that you're actually doing a remote procedure call. So I've seen code that it's lots of Thrift requests in a serially, like 50 of them, and they're like, why is it slow? Because you just did 50 network requests and you waited for each one to finish
11:01
before starting the next. So yeah, it's slow. HTTP, you usually know you're using HTTP because you imported HTTP lib or requests or something. ACN are gonna, good for a high CPU long running tasks. There's a couple of ways to do this. I think Django is a great entry point that our management commands, these are highly underutilized on the server.
11:21
We have a pattern of while true, do work in our management command. And so we can basically do something forever. In our ads system, we have to rank ads, basically forever, it's just constant CPU. And we never wanna not do that. So cron jobs don't really work very well
11:42
because there's like a minute in between the jobs. Celery, how many people know what celery is? Excellent, I don't have to explain this then. So yeah, celery is a tasking system, great pattern for asynchronous work, post save hook and a celery task. Celery can also use JSON, which is supporting our heterogeneous requirement.
12:03
So we wrote something called Go celery, our DevOps lead, Matt Robenolt did that because we need to parse celery tasks in Go. So if you're doing that in pickle, you can't do that because nothing else can read pickle. So use JSON, something platform independent. And celery beat is great for periodic tasks
12:21
that you need to do like data import. Django also has been incredibly easy for us to run. It has an incredibly well understood IO loop, which is a wait for a command or outsource the IO loop itself to something else like uWhiskey. It has multiple entry points. Whiskey, it can be run, Whiskey can be run in like Nginx directly,
12:43
uWhiskey, Apache, ModWhiskey and G unicorn and a million other things. So it's great because those things handle the IO loop and Whiskey just has to respond to, here's a dictionary of specific information. Management commands, also you're in control of the IO loop there. Celery tasks and celery beat, the same.
13:03
They're just listening on cues to respond to information. So it's really great because it's basically, each of these is a message passing interface at the end of the day. They either generate or respond to messages. So how do we do this at Discuss? First we'll look at DiscussWeb, which is a legacy product.
13:24
It's a monolithic Django product. It is this many lines of code, 183,000. And that's just because we deleted a lot of code recently. It was great, like 67,000 lines or something. So it's over seven years old and we made a lot of bad decisions along the way.
13:43
But some good ones too. When we're deploying this in our service-oriented way, we deploy the entire code base. We decide it's impossible at its current state for us to break it up into multiple code bases. So we have to deploy the entire thing and that means we end up treating it like a library.
14:02
We cluster machines by purpose. So at the end of the day, those will be our service. And by doing this, we kind of see the CPU patterns emerge, make scaling easier at the end of the day. You say, oh, we need a different kind of computer, high CPU for one thing and high memory for a different purpose.
14:21
And then we route to these services or clusters based on host name. So we use Varnish and HAProxy in order to route requests and then further based on the path to get them to the right machine. That plays into our data transparency point.
14:40
And then when we're deploying, we do a three-phase deploy. So there are multiple versions of the services out at any given time. It goes old version, old and new version, just new version. So we always have to make sure that whenever we upgrade anything, it will play nice in that route. To change the entry points in these clusters,
15:02
we use different settings.py files. I haven't seen this done a ton in the wild, so I wanted to highlight it. But basically, by using multiple settings.py files, we're able to drastically change the behavior of the discuss library. So we can have different URLs,
15:21
different middlewares, different template request contexts. It's something interesting, template request contexts, I don't think they're lazy loaded. So if you put something that you want in one of your templates in a global request context, if you're setting that up in a middleware or something, that's gonna execute every time.
15:41
So even if you're using it 80% of the time, 20% of the time, that's wasted cycles, wasted IO. So being able to separate that out per service. URL resolution is also O of N. So Django will go down your URLs list, doing does this match this regular expression?
16:02
Does it match this regular expression? And so if your most commonly used URLs are at the bottom of that list, you're gonna have a bad time. So we found actually in our API service, by reordering those, we found a 15% CPU saving because our most used URL was at the bottom.
16:22
It was literally the last one. And simply putting it in about 15 others out of a list of several hundred to the top, we got a huge savings. So do that. So much make a cool project that does that automatically. So two examples of services, we have our public API,
16:41
which if you is discuss.com slash API, it's how we use our own products in the wild. And it's also how anyone that wants to integrate with it uses it. It has a ton of middleware, I think dozens of them. It has over 300 URL routes and it has several things that automatically get loaded for templates.
17:02
Our internal objects API. So this is a internal to our data center, very raw model representation. It has no middleware. It only has one URL route. And it has no, it basically has nothing. It's settings up high, it's like 12 lines long. So there's an enormous difference in speed
17:22
between these two setups. Unfortunately, I do not have a graph to show that, but I can probably tweet one later. So did it work for us? Yeah, you know, it kind of worked fine. It was still a large code base, and we got all of the problems of having a large code base with it still. Version conflicts still very problematic,
17:42
whether it's a function or a method that you're changing, you've got to integrate that over the entire code base still, or an external package upgrade. We've had lots of problems with ZooKeeper Kazoo over the last two years because there were several incompatibilities there and people were using different versions.
18:02
So the conceptual integrity is basically still hard of this entire thing because you still have almost 200,000 lines of code. So on our report card, we did okay. We got our heterogeneous environment, and we utilized that. We have the Go celery tasks, data location transparency, independent scalability, and easier deployments.
18:22
Something I didn't say about those deployments, we can actually deploy those service clusters separately. So if we need to deploy just something for one of them, in an emergency, we can totally do that. We usually deploy the entire thing at the same time though. So now the ad server. So this is a product that's about a year and a half old,
18:44
and it was started as a Flask application, as I talked about in my previous talk last year, and other custom stuff. So in the last year, we've ported it from that to Django, and we've made some decisions when we did that.
19:01
We want to make sure that we use Django apps very well. Django best practices are best practices for a reason. They really help you architect your system in a good way. We want to leverage Django beyond WSGI. We want to use all those entry points, like the management commands and celery tasks. And we wanted to split up our code base very intentionally.
19:20
We want to have one code base that can access the database, and I'm talking about model access, like using the ORM, and the other code base could only access that sort of information via the strong API. So we separate the services in that regard.
19:41
But we do have multiple services that can access the database. So this is just some of the services that go into the ad system. We have our data API. This maps really well to rest framework, and it has some minimal RPC endpoints. Because we need to do mass data export for other services.
20:02
We basically just need an endpoint that says, everything, give me everything. And so then we have our ad serving API. This is basically a recommendations API, and it's an RPC endpoint. Then we have our asynchronous tasks, scoring and cache warming, keeping caches hot with recent information. And these are run using management commands
20:22
in that while true, do something. And then the ads data import service, which is celery and celery beet. These are either responding to new objects getting created and having to collect more information about them, or looking for what's changed in other systems on a batch process, like every five minutes, what was created in a different system,
20:41
and import that information for scoring and other purposes. And this is how we organized it. There's the two on top have the ads stuff. Because we're leveraging lots of frameworks, we get to only have 11,000 lines of code, compared to 180,000.
21:02
And then on the bottom, those are the warming services and such. And that's also, they're very close in lines of code, about 11,000. And this is what it looks like with boxes and arrows. So on the far left, you have JavaScript land, then there's the internet, then you kind of have a web serving. Oh, I'm supposed to use the mouse. Okay. Boom.
21:21
Now you have like a web serving layer here. And then you have all of our backend services. And these are again, colored by code base, just so you can see how that goes. And here are the backing technologies for that. So backbone on the front end, we have our Django, UWSGI with Nginx layer,
21:42
and then our backing stuff. So did this work? I think it was kind of amazing. Ah, didn't it play more than once? Ah, it's so good. Why is it not looping? Okay. So did it work?
22:01
All right. I think it worked really well. The only downside is the inverse downside. Well, we had problems that small code bases have. It's hard to share a code when you have lots of code bases. I hate things like get subtree and stuff. Those are just going to hurt you. So we ended up making a third code base for sharing stuff.
22:23
We use pip internally. So we run our own cheese shop. So we just deploy a package out there that is the ads core package for shared information. And that actually even helps us make even better code. Django best practices helped us a lot in the long term. I wish we actually were more strict on those.
22:43
And we definitely made it easy to understand the entire system. Because there's so few lines of code, and everything's encapsulated so well, it's easy to quickly add new things as long as they're being added as an individual item. Easy to test everything. Integration tests are more important
23:01
because you need to integrate with other processes, not just other libraries. So you need to do a really good job of testing your external APIs. So the Django test client becomes way more important than I like it to be because it's slow. Then service APIs live a long time.
23:21
So you're going to learn how to version them and support them for exam periods of time. I definitely have a couple days every couple of months that's like, OK, I'm going to go find everyone else's code where they're using the version of the API I want to delete and fix it for them. We get fast deployments. Our deployments are about two minutes.
23:41
And then our testing is like a couple seconds right now per service. So it's pretty fast in general. And yeah, they scale independently. So we got all of our check marks on our report card. Yay. Is it a success for discuss?
24:02
It's a knockout success. That was really bad. OK. Yeah, it's easier to run overall. It's easier to understand and build new systems in this environment. We can hire an employee that never has to see legacy code,
24:20
and I'm jealous of them. And it's easier to not break existing systems because you're not touching them. You can't accidentally break something that you're not committing code to. Well, OK, you can, but it's harder. So as a roundup, do one thing and do it well.
24:40
You want to examine your data patterns up front and then based on that data pattern, you need to make some decisions. Those are going to be API decisions around protocol, transport, and what methodology are you using to access that data. And then Django has multiple entry points. You should definitely use them. It's not just a WSGI machine.
25:02
And then do one thing and do it well. That is the one line from the talk you should remember. Here's some links. Go support Django rest framework. It's got a Kickstarter. It's got a ton of money right now. I don't know if he even wants more money, but it's a great thing. Support it.
25:21
Here's our Go celery stuff. Django best practices. I like link and loop. They're out of Portland, I think. There are some awesome people and they are really good thought leaders in just how to do Django development in general. And the Unix philosophy. It's a great Wikipedia page. So if that was interesting to you and you hate Europe, come to San Francisco.
25:54
Questions, which I already have one of. About three phases deployment.
26:01
Oh yeah. In case of the service which uses a database directly, that one you mentioned, how do you manage that because actually database is only at one version, I guess? Yeah, so we basically roll forward. So when we want to do something to upgrade our database, we change the code.
26:25
We change our Django models. We make the SQL that will do that upgrade. We run that upgrade. When you're adding a new column type, that's okay. You're adding a new column, sweet. Django will run with additional columns. It will ignore that.
26:40
If you're changing a column type, well, you can't do that because we have, we exceeded the ID variable for one of, or the size of ID in one of our tables. So now we're all like big ID, or big ID, whatever it is. Was it big sequential?
27:01
Some postgres thing. I don't know how to SQL. So the point is you can't do that. You cannot change your models. It's very expensive and basically impossible, but you can totally add to them. So it goes to data planning, but also because it's behind the API, you can always abstract that. So we have models that are,
27:20
we have, it has its one model, it has its tables, and then it kind of has adjunct models that are meta information for that. And as a consumer of the API, I don't even care because I don't even know that that exists. It's up to that service to manage all of that.
27:40
Just a question. What do you do for logging and monitoring? Because you already have lots of things in your SOA and lots of external dependencies. And I dread when something happens in Celery or RabbitMQ and I have to go to the manual and start to read again and I have no idea what's going on. So what do you do for that? So I'm not on the ops team,
28:00
but I have a little bit of insight into how that works because we use the same infrastructure to manage our applications. And we basically use StatsD for everything. Who knows what StatsD is? We're still in the participation phase. Okay, cool. StatsD is basically a daemon that runs locally or anywhere because it's a service.
28:21
And you can talk UDP to it and it will basically over a period aggregate information and then beacon that out to somewhere else. What's usually called a collection service. And then that collection service will take everything that's collected and tell Graphite or something about that. And so we use Graphite.
28:41
We use... Oh, I forgot the thing that actually does the alerting. Nagios? No, can't remember. But we basically have Graphite stats. We alert on thresholds on those stats and we have them for hardware and software. And so we have thousands and thousands
29:01
and thousands of different metrics being tracked. And by aggregating the correct ones, you can have a pretty good intuition on the system. I want to answer one question I was asked last night because I thought it was particularly interesting, which I was talking to Tom Christie and he was saying, what was the most challenging part about designing your new system?
29:22
And I was surprised when I found it was actually just the software architecture. It was making sure that we don't write that spaghetti code because when you're taking an idea and implementing it, you can spread that idea across your entire application or you can encapsulate it. So using Django apps in a Django project as their own mini services
29:40
with their own APIs at a software level, doing that well was our hardest part. And his response was really, I thought I would say, not design it so it doesn't go horribly, horribly wrong when it breaks. And that was actually relatively easy because everything is message passing. When something breaks, it stops sending messages.
30:02
So that means everything kind of turns off instead of blows up, which is, I think, it's still a failure mode, which is bad, but it's a better failure mode. Oh, we need to bring more hardware online to handle the load. It's just kind of a load dies. So those are the two things. And I also have questions for you,
30:20
which is like, how do you guys like RPC? And we can talk about this outside later. But do you realize, how do you do maintainable RPC? How do you guys do service discovery? I like DNS because I also like HTTP. And what do you guys think on the many code bases versus one code base thing? I know large companies like Google and Facebook love it. And I'm split.
30:41
So I'm just curious, what are your thoughts? Okay, I'm sorry we're out of time. Thank you again, Adam. Thank you.