How Disqus is using Django as the basis of our Service Oriented Architecture

Video in TIB AV-Portal: How Disqus is using Django as the basis of our Service Oriented Architecture

Formal Metadata

How Disqus is using Django as the basis of our Service Oriented Architecture
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place

Content Metadata

Subject Area
adam - How Disqus is using Django as the basis of our Service Oriented Architecture Disqus maintains the largest Django app out there. And we love it! It has, however, grown rather large and unwieldy. In the last year Disqus has had an increasing number of smaller services cropping up based on several different platforms. So this talk will be about how we do continuous deployment with our emerging service-based infrastructure.
Keywords EuroPython Conference EP 2014 EuroPython 2014
Revision control Architecture Basis <Mathematik> Web service Multiplication sign Maxima and minima Twitter
Different (Kate Ryan album) Different (Kate Ryan album) Pattern language Service-oriented architecture
Data recovery Similarity (geometry) Office suite
Axiom of choice Standard deviation Gender Pattern language Cartesian coordinate system Computing platform
Web service Service-oriented architecture Computer architecture
Discrete group Web page Service (economics) Copula (linguistics) Sequel Computer file View (database) Multiplication sign Source code Bell and Howell Database Streaming media Computer programming Writing Latent heat Web service Component-based software engineering Atomic number Software Service-oriented architecture Traffic reporting Physical system Mobile app Service (economics) Interface (computing) Software developer Forcing (mathematics) Discrete group Web page Computer program Electronic mailing list Internet service provider Plastikkarte Twitter Single-precision floating-point format Category of being Type theory Process (computing) Facebook Software Function (mathematics) Web service Internet service provider Order (biology) System programming Interface (computing) Queue (abstract data type) Quicksort
Service (economics) INTEGRAL Virtual machine Insertion loss Scalability Formal language Independence (probability theory) Integrated development environment Right angle Service-oriented architecture Abstraction Physical system Data integrity
Email Service (economics) Multiplication sign File format Mereology Database transaction Semantics (computer science) Host Identity Protocol Web 2.0 Data model Workload Mathematics Logic IDL Representation (politics) Procedural programming Endliche Modelltheorie Communications protocol Physical system Area Service (economics) Dependent and independent variables Database transaction Database Uniform boundedness principle Befehlsprozessor Logic System programming International Date Line Pattern language Queue (abstract data type) Quicksort Object (grammar)
Web page Email Transportation theory (mathematics) File format Electronic program guide Heat transfer Client (computing) Hypertext Host Identity Protocol Computer programming Formal language Subset Number Revision control Average Computer configuration Bus (computing) IDL Cuboid Codierung <Programmierung> Communications protocol Exception handling Email Texture mapping Inheritance (object-oriented programming) File format Electronic mailing list Special unitary group Cartesian coordinate system Connected space Message passing Integrated development environment Software Order (biology) International Date Line Software framework Object (grammar) Communications protocol
Email Building Service (economics) Code Authentication Source code File format Host Identity Protocol Computer programming Facebook Logic Authorization IDL Cuboid Remote procedure call Software framework Endliche Modelltheorie Communications protocol Computing platform Form (programming) Physical system Authentication File format Forcing (mathematics) Computer network Database transaction Database Inversion (music) Software Logic Blog System programming International Date Line Software framework Authorization Remote procedure call Quicksort
Point (geometry) Game controller Server (computing) Information management Decision theory 1 (number) Solid geometry Data dictionary Automatic differentiation Product (business) Web 2.0 Data management Goodness of fit Latent heat Web service Software testing Endliche Modelltheorie Loop (music) Physical system Task (computing) Beat (acoustics) Parsing Multiplication Dependent and independent variables Information Interface (computing) Point (geometry) Special unitary group Data management Message passing Process (computing) Befehlsprozessor Frequency Uniformer Raum Pattern language Ranking Task (computing)
Ocean current Point (geometry) Service (economics) State of matter Code Multiplication sign Virtual machine Gene cluster Entire function Food energy Planning Revision control Read-only memory Different (Kate Ryan album) Semiconductor memory Phase transition Proxy server Scale (map) Forcing (mathematics) Code Computer network Database Computer Entire function Befehlsprozessor Web service Order (biology) Right angle Pattern language Routing Library (computing)
Point (geometry) Greatest element Context awareness Service (economics) Computer file Image resolution Multiplication sign Gene cluster Set (mathematics) Mass Template (C++) Product (business) Uniform resource locator Web service Roundness (object) Different (Kate Ryan album) Object (grammar) Endliche Modelltheorie Middleware Context awareness Matching (graph theory) Graph (mathematics) Point (geometry) Projective plane Content (media) Electronic mailing list Line (geometry) Regulärer Ausdruck <Textverarbeitung> Befehlsprozessor Web service Different (Kate Ryan album) Data center Right angle Cycle (graph theory) Middleware Library (computing)
Revision control Functional (mathematics) Code Function (mathematics) Revision control Code Pattern language Metadata Data integrity
Service (economics) Line (geometry) Multiplication sign Gene cluster Code Independence (probability theory) Zeitdilatation Mereology Entire function Scalability Entire function Scalability Usability Independence (probability theory) Web service Integrated development environment Function (mathematics) Revision control Integrated development environment Software testing Data integrity
Ad serving Decision theory Flash memory Cartesian coordinate system Product (business)
Point (geometry) Service (economics) Code Line (geometry) Graphische Programmierung Mass Automatic differentiation Web 2.0 Data management Web service Cache (computing) Different (Kate Ryan album) Internetworking Remote procedure call Software framework Software testing Endliche Modelltheorie Gamma function Physical system Task (computing) Computer architecture Self-organization Beat (acoustics) Service (economics) Mapping Information Server (computing) Closed set Code Maxima and minima Database Perturbation theory Line (geometry) Encapsulation (object-oriented programming) Data management Embedded system Process (computing) Internetworking Series (mathematics) Software framework Quicksort
Web service Scripting language Embedded system Service (economics) Internetworking Term (mathematics) Basis <Mathematik>
Service (economics) Code Military base Disintegration Code 3 (number) Term (mathematics) Entire function Scalability Automatic differentiation Goodness of fit Term (mathematics) Network topology Software testing Physical system Physical system
Service (economics) Code INTEGRAL Multiplication sign Disintegration Code Line (geometry) Term (mathematics) Entire function Scalability 2 (number) Independence (probability theory) Revision control Goodness of fit Web service Process (computing) Software testing Integrated development environment Software testing Physical system Library (computing) Data integrity
Word Integrated development environment Code System programming Control flow Service-oriented architecture
Point (geometry) Multiplication Decision theory Decision theory Point (geometry) Virtual machine Pattern language Pattern language Line (geometry) Communications protocol Communications protocol
Web page Estimator Goodness of fit Wiki Link (knot theory) Software framework
Revision control Web service Personal digital assistant Database
Point (geometry) Demon Statistics Sequel Code Multiplication sign 1 (number) Thresholding (image processing) Metadata Number Frequency Web service Different (Kate Ryan album) Computer hardware Endliche Modelltheorie Physical system Key (cryptography) File format Planning Database Cartesian coordinate system Variable (mathematics) Sequence Type theory Word Software Phase transition Software framework Table (information) Metric system Physical system
Mobile app Dependent and independent variables Service (economics) Code Information overload Projective plane Fitness function Control flow Set (mathematics) Cartesian coordinate system Mereology Message passing Software Software framework Energy level Software architecture Physical system Physical system Asynchronous Transfer Mode
Web service Googol Remote procedure call
the fast version has retained its been apparently Max don't work every single time hi my name down Hitchcock discusses the you can follow me on Twitter or something partnership of vehicle of
start off the bat were hiring also American so
if you want to leave Europe talk about but that decreases and
so when I talk about the 1st idea to address the issue of why did I lied to you last year and they say was so why should so a different data patterns and so on how does discuss do it with both the legacy and Greenfields act example and does that work for us as
last year it my talk was entitled how discussed does it when it isn't Django and it was a police officer verses was it was a talk about the recovery similar topic but it's a specifically pay you have to think of everything is not but the why do I say
otherwise is the main
question so when I got back from Europe Python of
last year my CTO told that basically said hey we're making technology choice and that technology choices Django were we double or investment because we we are a huge jingo shot so we are a really good at it the current is military reasons for it using standards across a company that are also known outside the company makes a super is higher and there's a huge community so we were lovers all that in both our legacy applications into the platform moving to don't have challenge my thinking about gender and find some new ways to use it so here
is take to is how discussed as it J. U. N. G. so if
I ask you raise your hands who will raise their hands so about 60 per cent kill who already knows what so as stands for cool that's like a hundred 20 cent of more you raise it but were you correct yes so is is ever service-oriented architecture or if you're British service oriented architecture
and I needed to find what what I think that means right now so and so it is something that like to pronounce always work to was new through the atoms of a so is the system that you architected in a specific way that system should have discrete software components or those are busy running programs and those are called services those running programs should have simple well-defined EPI so our interfaces and those of the API as to which they're going to communicate then you can loosely coordinated with the federated these services but to achieve some sort of business but in so there's 2 primary roles there's a service provider when source consumers User providers and consumers of data and you can tell both the school were neck neck
it's not a new idea at all but Doug McIlroy who was on the work at Bell Labs long time ago said this of the Unix philosophy of the forces will write programs that do 1 thing and do it well write programs to work together and write programs to handle text streams because that is a universal interface and and she was talking about Unix pipes and programs talking over values represent into some were talking about network types and programs talking over file descriptors over the network so the really the exact same philosophy and I think this still holds true today is also minus the new idea in Weber systems has been people been doing this for many years but you probably already using services in your system of how many of you are doing out developers today thus we all lot of excellent you probably using PostgreSQL if you're right or my sequel if you it also right or as you probably think using a system rabbit copula covers not technically you but it's basically cues like chronic use list of order using actual API who knows the copulas so we will talk about that later it's also if someone asked me right but so actually the eyes are also services and everything is a service so page job category examples of consumer services usually but if they do a post command that server that's not producing data so why should you do so this is basically my my view of the report card for later in the talk of this you know I
think that if you've got a right you can have services written in any language you can it doesn't matter what the machines those services run on and the services of a strong API and it doesn't matter how they store abstraction that data internally it's just the API that matters what services should scale independently of each other and they should also provide you use your testability easier deployment and because we use for the system up into well defined chunks this should be easier for individuals to maintain a conceptual integrity or understanding of the system so hopefully you can learn
losses but I think that is so it's really important to know obviously so what well house so I think that you have to
understand the data patterns in your system I see 2 primary data patterns of there's have transactional data and asynchronous data transactional further divided by into 2 areas I think there's are you doing transactional data against basically database model some sort of a straight representation of an object in your system like a user but then rest works really well described semantics against our model but if you need to do something we're combining data and math or logic to say hey recommend something to me or can I do something then RPC is a model that works really well and has a more procedural things but if your response is the same every time then rest might work well let's have the way I think about it but then it's his data naturally well to queueing systems or pub sub systems and it's really good when you have high CPU or long running workloads that you can do in a web requests like so he also the pick
your API is 2 parts here you pick what are the bytes that I'm gonna send to the other person or other services and how am I going to send those bytes to the other services so
how many people know what program of the thrift or message that averages so all those are examples of non encodings of how do you put an object across the network in a format that all languages can read so picking something from that list is super important because in order to to have a heterogenous environment you need to have an encoding that multiple languages can work pickle is not on this list it's not enough last week moving people from of so but we also think transportation protocol so issue the that as literally every person is that was the lowest number of hands so ECG sensor Hypertext Transfer Protocol if the don't of known recent and this is yeah or thrift thrifty provides binding as well I like it should be a J because they're super easy to do Django doesn't really well out of the box issues if he is great if you actually read that the fact the spectrogram with the Wikipedia page on the spec you'll learn a lot of things like the Accept header so you so you can start with with HDP + j science and if your clients are adheres to respect the acceptor you can say to support multiple encodings when you go forward user except application J. Sun or textures on and off for future because accept application for the bus or per above version 2 whatever the and then if you were to read to you have things like keep alive as a connection options negotiating the initial connection a little expensive but we're going forward with with it should be a slave to keep the this the connection alive it cost about the same as any other protocol on
you know if you need to save the 8 bytes you get from going from
thrift the formation of reference it's weaker Facebook
but of not and so on transaction this stuff rest doesn't
really well out of the box that's basically what it's been doing for the last 1 and a half a million years so but in recent years that we we use use like jingle forms and those sort and other things that we are used to make bad practices and generous framework wrote relatively recent framework that makes it super easy to keep your code organize clean while providing a RESTful access to your database models it's really cool check it out as a Kickstarter campaign source can roll your own API there were definitely we have lots of performance and suppress remarkable we also got around them very easily come talk to me about how I do Austin caching things inversion in that framework program blog goal RPC is also the other transactional earlier that logic heavy the recommendation services authorization authentication I don't like RPC as 1st stop because prone overspecialization we're building API to make API is that a really good at doing 1 thing and if you're using it we're building a platform you 1 API that can support what you know lots of things to be built on top of them sometimes you need to a specialized API but if you can do it in a generalized way I think that's better but in other systems like thrift and 0 RTC which are super easy use that's the the downfall but you can abstract away the fact that you're actually doing a remote procedure call so seen code that lots of thrift requests in like silly like 50 of them and otherwise it's like because you just did 50 network requests waited for each 1 of them is the force on the next so yeah that's http usually no using HTTP because you important if you have requests for something else is
service good for high CPU 1 Running tasks there's a couple ways to do this and in general is a great and and point there and management commands user highly underutilized on server we have a pattern of Walter through do work in our management command and so we can basically do something forever so far in our ads system we have to use rank ads basically forever it's just constant CPU and so on and we never took 1 and not do that so Cronin jobs don't really work very well because there's like a minute in between the jobs of celery committee people not excellent enough it's 1 of them so we have story is a tasking system great paraphrasing his work was a book and solid task of story can also use J. Sun which is a superior heterogenous requirements so the uh we we rose from the Congo celery develops leave macroblock that and because we need to pass tests go so if you do not typical you can't do that because nothing else can be tackled so what user on something happened that has already is great for periodic tests that you need to do like data import but also has been incredibly easy for us to run but it has incredibly well understood I which is wait for a command or outsource the Iowa itself to something else like you was the 1 that has multiple entry points with you can be run with to run in of like and annexed directly you was the Apache model is the and proper G. uniform Emilio things so it's great because those things handle the Iowa and was used as a response to a user dictionary of specific information management commands also you're in control the island their celery
tasks and celery repeat the same they're just listening in on unhcr used but respond to information so it's really great is basically each of these is the message passing interface at the end of the day be either generated or respond to messages so how this and discussed by the however also looking at discussed Web which is a legacy product it's a model of the jingle product but it is this may arise because 183 thousand and is because we don't need a lot of the recently was great lakes 67 thousand 100 something so it's over 7 years old and we made a lot of bad decisions along the way but some good ones to 1 deploying this in our
service-oriented way we deploy the entire code base we don't we we decide to do it impossible and its current state force to break it up into multiple databases so we have to deploy the entire thing and that is when the 2 hit like a library we
cluster machines by purposes so at the other day this will be our our service and by doing this we can see the CPU patterns emerge but they scaling easier the days l we need a different kind of computer high CPU for 1 thing and high memory for different purposes and then we we're out the services of the clusters are this and hostname so we use of bodies energy proxy in order to run out of route request and then further based on the path to get them to the the right machine that plays into our data and the data transparency point and the more when deploying would do 3 three-phase of so there are multiple versions of the services is out at any given time it goes over the old and new versions just new version so you we always have to make sure that whenever we operate anything people play nice in that Our
out to change the the entry points on his of in these clusters we use different setting is not high files but I haven't seen this done a ton in the wild so I wanted to highlight but it basically by using multiple things that files were able to drastically change the behavior of of of the the discuss library right so we can have different roles different middlewares differences of the template request content template is amazing template request and X are I don't think they're lazy loaded so if you put something that want in 1 of the templates in a global request context like it if you if you send it up in the middle where something that can execute every time and so even if you use a decreasing the time 20 % time that's wasted cycles we sit I so having a current that enable separate that out per service but you were already resolution is also a of that so jingle will go down your Urals list doing does this mass regular expression doesn't match this regular expressions and so if you were at the most commonly Europe use your roles at the bottom that list by that time so we found actually in our own API service
by reordering those we found a 15 per cent CPU saving because our most use your all was at the bottom was the last 1 is simply putting it in about 15 others out of a list of of several hundred to the top but we got a huge savings so do that so much make cool products that the project that does that automatically so 2 examples of services with our public API which if you discuss comes slash PPI is how we use our own products in the wild and it's also how anyone was in agree with that uses it but it has a ton of middleware moving dozens of them it has over 300 EUR rounds and it has several things that are applicable it templates are internal auditing so this is Israel's were data center very wrong model of recommendation it has no middleware no 1 has 1 year all around and it has no repair basically has nothing it's a surprise like 12 lines long so there is an enormous difference in speed between these 2 setups but unfortunately not of a graph to show that but I can probably between 1 but did work
for us I can't find
a it was still logical code and we got all the problems of having a large codebase with it's still a virgin conflicts still very problematic whether it's an function or or metadata changing you guys agree that over the entire could be still or an actual patterns of brain we have lots of problems with of the ZooKeeper because you over the last 2 years because there were several incompatibilities there and people were using different versions so the conceptual Terry basically still
hot this entire thing because you still have almost 200 thousand so far apart part we did
OK so we got a 100 years environment and realize that we have the go sorry test dilatation transparency independent scalability and ease of deployment of isolated say about deployments we can actually deployed those services because the service clusters separately so we don't need deploy just something for 1 of them was so we had an emergency we can totally do that we usually deployed the entire thing in time so now the
ad server so this is a product that's about a year and a half
away and it and this was started as a flash application as I talked about my previous talk last year and all the costs and stuff so in the last year we've according it from that to Jane Doe and we made some decisions to do that we want to use GNU acts very well best practices are best practices for reasons they
really help you architecture system in a good way we want to leverage Jane go beyond whiskey 1 use all those entry points like the the management commands and selling test and we want slippery codebase very intentionally we want have 1 code base that can access the database an entire about model access like using your but in the other of the codebase can only access that sort of information via the strong API so that's uh we we separated the services in that that regard but we do have multiple services that can access the database so this is just some of the services that go into the ad system we have our data API this maps really well to the rest framework and that has some minimal RPC points because we needed you mass data export for other services would be just amendments as everything give me everything and so then we have our ad serving guys visually recommendations API and it's an RPC endpoint that we have the our our a service task scoring and cash morning keeping caches of hot with represent information and these are run using management commands in that while true do something and then the ads data import service which is salaries already the sorry the responding to new objects being created and having to collect more information about them or looking for was changed and other service systems unabashed process like every 5 minutes what was created in a different system and important that information for for a scoring another purposes and this is how we organize others the 2 on top have bags of stuff because we're leveraging lots of framework so we get to only have 11 thousand lines of code compared to 180 thousand and then on the bottom those are the the warming services in such and that's also they're very close lines of code of but and as
well as like boxes and arrows so on the far left you have JavaScript land and then there's the internet that you can have a web
service terms was his most but I have like a web service layer here and then you have all of our back services and user again coded by the color code basis evangelicals and here are the back back and technologies for that so that's 1 on the front and we have our Jane Doe it was the with engine that's where and then are back and stuff that so did this work
but I think it was kind of amazing I didn't play than once of good
was OK so how did no work
but I think it works really well be that the only downside
is is the inverse downside while we have problems that small code bases have it's hard to share code between when you have lots of good reasons I hate things like get some tree and stuff those are just going to hurt you so we end up making a 3rd code base for sharing stuff we use it internally so that we use around the shop so we we just deployed package out there that is the ads for package for sharing information and that actually even helps us make even better jego best practices helped us a lot in the long term I was we actually were more strict on those at end we definitely easier to understand the entire system
because it's everything in there so few lines of code and everything's encapsulated so well it's easy to quickly add new things as long as they're being added as an individual item is easier test everything up integration tests are more important because you need to integrate with other processes not just of the libraries so you need to do a really good job of testing you're actually the eyes so you know that the jaded has becomes way more important than I like it to be because it's slow uh then service if the eyes and live a long time so you know 100 version them and support them for for example is a time so I definitely have played a couple days every couple months that's like a defined everyone else's code where the using the version you get widely and fix it for them but we get fast appointments are deployments are about 2 minutes and then our testing is like a couple seconds alright now per 10 per cent per service so is pretty fast in general and yet they scale developed so we got all of our check
marks on a prepared you but is
it a successful word discussed it's not
knockout success I
was with them again and again I see 0 and overall that the it's easier understand and and build new
systems in in this environment you don't have to we we can hire an employee that never have to see legacy code and I'm jealous of them by and it's easier than average existing systems because you're not touching that way would you can't accidentally break something that you're not encode to but UK can but stock but but so as around do 1 thing and do
it well you want to examine year data patterns of front and then
it was based on a data pattern you need to make some decisions those our immediate decisions around protocol transport and what methodology using to access that data and then it changes that multiple entry points you should definitely use them don't it's not just of was the machine and then do 1 thing and do it well it's that is the 1 line from the top and here's some
links that go supporting from estimate guy
Kickstarter it's got a ton of money right now I don't even I don't I don't know if you want more money but you know it's a great thing supported years ago celery stuff to the best practices are likely can to the other Portland I think there's some are some people and they are really good thought leaders and just how to do change development in general and the useful philosophy integrated with the pedia page uh so if that
was if the CDU and you really Europe come sentences that the questions in
was and I already have 1 of about the phases of
deployment that in case of the service which uses the database directly that 1 you mentioned but how do you amended that's because actually databases on 1 version and yes yes so
we basically roll forward so when we want to do something to operate a database we but you will we change the code which intra-genome models we make the sequel that will do that operate we run that operate we're adding a new
column type that's OK like you're adding columns sweet jingle run with additional columns will ignore that if you're changing column type Will you can't do that because you know we we have of we exceeded the ID variable for 1 for 1 of our that either the size of ID and whatever tables so now we're all like big begin by the whatever because it was it was a big sequential that some post press that I don't know how but so the point is you can't do that you cannot change your models it's very expensive and basically impossible but you can totally add to them so goes to dinner planning but also because it's behind the API you can always abstract that's we're models that are we have it has it's 1 model it has its tables and then it kind has adjunct models that our meta information for that and as a consumer of the API I don't even care because I didn't know that that exists it's up to that service to manage all that just a question of what to do for moving and monitoring because you really have lots of the URIs word and was of extreme differences and the when something happens in summary and human a few to the manual sub-tree through the and the model is going to be up there so I'm not on the off season but I have a little insight into how that works because we use the same infrastructure to manage our applications and we basically use stats the for everything who knows what steps the years we still participation phase of a equal status is basically a demon that runs locally or anywhere because it's a service whom and you can talk to the key to it and it is it will be easily over a period aggregated formation and then beacon that out to somewhere else of what usually call a collection service and that collection service will take everything is collected and tell graphite or something about that and so we use graphite we use a so I would forget the thing actually does the alerting the 90 years now can number but we basically have graphical graphics that's Wheeler on thresholds on those stats and we haven't for hardware and software and we had so if thousands and thousands and thousands of different metrics being tracked and by aggregating the correct ones you can have a pretty good and intuition on the on on the system I wanna answer 1 question was asked last night's particularly interesting which are assigned to time
Christine you say what was the most challenging part about designing your new system and I was surprised when I was actually is the software architecture that was making sure that we don't right that spaghetti code because when you're we're taking an idea and implementing it you can spread that idea across the entire application or you can encapsulated so using Django apps in general
project as their own many services with their own API is a software level that like doing that well it was our hardest part uh and his response was really if I would say not designed so it doesn't go horribly horribly wrong when it breaks and I was actually relatively easy because white because from that everything's message-passing when something breaks that's just it stopped sending messages so that is everything turns off this set of blows up which is I think is still failure mode which is that was a better fit it is not the only we need to bring more harm or where aligned to an overload it's just have a a little dies so those are the 2 things that also questions for you which how do you guys like and then we can talk about this outside later but your how you do maintainable RTC
how do you guys do service discovery I like DNS because I also like HTTP of you and what you guys think on the many codebases verses
1 could be there no large companies like Google and love it and I what some scarcely where your thoughts and so all the time thank
you again thank you


  526 ms - page object


AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)