We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Butter smooth, interactive applications with Django and Websockets

00:00

Formal Metadata

Title
Butter smooth, interactive applications with Django and Websockets
Title of Series
Part Number
40
Number of Parts
48
Author
Contributors
License
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Web applications have changed significantly over the years – from simple static pages, to sprinkling interactiveness with JQuery/AJAX, to full dynamic single page apps. Through each evolution, we’re adding more complexity, more data and more asynchronous behavior to our applications. In this new world, where does the synchronous nature of Django’s request-response cycle fit in? My talk will focus on the topics around asynchronous Django applications. I’ll be sharing some lessons we learnt while building and scaling an interactive web application within the confines of Django and django-channels. This topic is interesting because there’s been a lot of interest with meteor-like frameworks that have synchronized state between the frontend and backend. My intention is to show the audience that you can accomplish the same end-result with Django, without the need to learn and deploy a brand new framework. An outline I have in mind: What does asynchrony mean, and why you need it. Traditional methods of achieving asynchrony (delayed jobs using worker queues like celery, long-polling for messaging, etc.) Why django-channels changes the game. How to architect your state. What are the available options for deployment. Gotchas, and what to do when things go wrong. Just a basic knowledge of Django is required, as the topics are transferable to other frameworks. We did not have to monkey-patch any of the drivers to achieve asynchrony, so what you’ll learn at my talk will apply cleanly to a stock Django.
SmoothingInteractive televisionElectronic data interchangeData analysisMaxima and minimaMultiplication signProduct (business)Black boxInteractive televisionComputer programmingNumberSequelWikiLogicData analysisSystem callSocket-SchnittstelleInternetworkingArithmetic progressionSmoothingLevel (video gaming)Cartesian coordinate systemWritingWeb applicationMaizeGenderComa BerenicesWeb 2.0Line (geometry)CodeStack (abstract data type)Server (computing)Assembly languageFormal grammarComputer animation
Query languageHistogramCorrelation and dependenceLinear mapOutlierGoogolMetreAlgebraEvent horizonVisualization (computer graphics)Data typeDependent and independent variablesRun time (program lifecycle phase)Type theoryCuboidShared memoryProduct (business)Context awarenessSequelComputer fileWeb 2.0Frame problemForcing (mathematics)Dependent and independent variablesWeb pageEmailFamilyBenutzerhandbuchFluidSubsetDatabaseLine (geometry)Group actionSpectrum (functional analysis)Web browserEndliche ModelltheorieAnalytic setLaptopConnected spaceDifferent (Kate Ryan album)Interactive televisionMultiplication signNumberOffice suiteState of matterLinear regressionMusical ensembleElectronic mailing listQuery languagePoint (geometry)Regular graphDebuggerFilm editingGoogolOpen sourceCartesian coordinate systemText editorMobile appSoftware frameworkCycle (graph theory)FehlererkennungLastprofilVideo gameSingle-precision floating-point formatIterationException handlingComputer fontData analysisStructural loadWeb applicationTrailWater vaporMappingCASE <Informatik>Service (economics)Capability Maturity ModelView (database)BootingHybrid computerPhysical systemRight angleHistogramCross-correlationNetwork socketError messageRevision controlServer (computing)MathematicsSocket-SchnittstelleRun time (program lifecycle phase)Term (mathematics)Goodness of fitUniform resource locatorComputer animation
Network socketState of matterServer (computing)Device driverFront and back endsStack (abstract data type)Component-based software engineeringGastropod shellArchitectureDiagramStructural loadQuery languageConnected spaceSocket-SchnittstelleMultiplication signMatching (graph theory)Position operatorSequelProduct (business)ResultantDevice driverCuboidBuildingCompilerGoodness of fitClient (computing)Greatest elementComputer programmingComputer architectureMathematicsLaptopHybrid computerConnectivity (graph theory)Server (computing)Network socketGenderPresentation of a groupProof theorySynchronizationCASE <Informatik>Core dumpDependent and independent variablesElectronic mailing listCycle (graph theory)CodePoint (geometry)Projective planeMessage passingState of matterArithmetic progressionCorrelation and dependenceLevel (video gaming)Event horizonProper mapStructural loadDebuggerThread (computing)Right angle2 (number)Software frameworkComputer configurationFront and back endsLastteilungMusical ensembleExterior algebraRevision controlDecision theoryMetreWeb 2.0DiagramExecution unitDenial-of-service attackDisk read-and-write headStandard deviationComputer animation
Message passingFront and back endsArchitecturePoint cloudHTTP cookieServer (computing)EmailClient (computing)Cursor (computers)TimestampSet (mathematics)ConsistencyNetwork socketEvent horizonStreaming mediaBlogDebuggerCross-site scriptingHypermediaComputer fontStructural loadPoint cloudView (database)Message passingPhysical systemServer (computing)Web 2.0Socket-SchnittstelleClient (computing)AbstractionFront and back endsNumberSoftware testingLastteilungCursor (computers)Web applicationData storage deviceState of matterNetwork socketData structureStreaming mediaLevel (video gaming)Ring (mathematics)Buffer solutionMiddlewareWeb pageOrder (biology)Computer architectureDifferent (Kate Ryan album)Bus (computing)Term (mathematics)Graphical user interfaceFrame problemElectronic mailing listDrop (liquid)HTTP cookieElement (mathematics)Standard deviationMobile appWave packetElasticity (physics)Revision controlCartesian coordinate systemLaptopConnected spaceSign (mathematics)Connectivity (graph theory)Loop (music)TimestampTrailTable (information)LogicImplementationPlanningCodeChemical equationProduct (business)DatabaseProjective planeWhiteboardQuery languageoutputBoss CorporationWebsiteShape (magazine)Multiplication signSequenceDebuggerSet (mathematics)QuicksortMaizeAuthenticationDynamic Host Configuration ProtocolAsynchronous Transfer ModeWeightDiagramPoint (geometry)Computer animation
Touch typingMessage passingClient (computing)Cursor (computers)Point cloudHTTP cookieServer (computing)EmailFront and back endsArchitectureNetwork socketEndliche ModelltheorieInternetworkingHuman migrationSequelRaw image formatDatabaseFlow separationAuthenticationQuery languageSheaf (mathematics)Server (computing)MereologyOffice suiteHTTP cookieEmailClient (computing)Condition numberPole (complex analysis)Different (Kate Ryan album)Event horizonConnected spaceStructural loadMessage passingCycle (graph theory)Bus (computing)BuildingCorrelation and dependenceNetwork socketSystem callRevision controlInstance (computer science)Fluid staticsBit rateCurvatureComputer fileFormal grammarData managementLine (geometry)CASE <Informatik>2 (number)PlotterRight angleDebuggerDiagramDependent and independent variablesData modelComputer architectureCuboidLastteilungIP addressWeb 2.0Type theoryVirtual machineData analysisView (database)Level (video gaming)Affine spaceQuicksortNumbering schemeInsertion lossPoint (geometry)Greatest elementVideo gameCodeMeta elementMobile appMoving averageWebsiteFront and back endsWeb applicationLecture/Conference
XML
Transcript: English(auto-generated)
Can you guys hear me good? Awesome. Excited to be here, DjangoCon, my first DjangoCon.
Besides the people who are always Python community is A plus, the food was exceptional. I think a lot of people, a lot of you guys are still in the food coma after lunch, but let's take it easy. So the title of my talk is butter smooth interactive applications with Django and web
sockets. So I'm going to be giving a little preview of interactiveness on the internet with regards to web applications and then build up towards Django channels and web sockets and the kinds of problems that we faced. I'm not going to go too much detail into how web Django's channels work because it's
a work in progress and I believe there was a full tutorial on Django channels and a lot of the other talks talk about Django channels specifically. So I'm going to be talking about the tooling and some of the deployment issues that we've had.
A more formal introduction about myself. I've been working with Python data for over 10 years. I started my career writing assembly programming and just worked my way up to up the stack. So I'm like a full stack dev but stops at the server level.
So I've written programs for GPUs, a lot of that kind of stuff, but I really love Python. I love Django. I love how clean and none of the magic, none of the black box stuff, you can just open up the code and see what's going on. I've been doing data analysis and sequel for the past couple of years.
My claim to fame is I wrote the first wiki engine for emacs and wiki blog engine. So that was pretty cool. It was accepted into main line emacs and this was a long time ago. I currently run Salota which is a data analysis tool for professional data analysts.
Sequel recipes which our product is something like this. This is just to give you a context on the challenges we faced. You type in sequel there, it's a text box and you hit run and it creates these charts
and then you can schedule these charts, share these charts and do whatever you want. So that's essentially our product. Who uses our product? Professional data analysts. So if you're a, it's not sequel like the ORM style sequel. It's for building histograms, correlations, doing forecasting, regressions, all the kinds
of data analysis stuff you would do but using sequel. So if you Google for Salota sequel recipes, there's a full list of I think two dozen recipes you can copy paste, everything is straight up sequel.
So this is a timeline of asynchronous behavior on the web. How many of you guys are familiar with iframes? That's awesome. So about 20 years ago iframes was how you'd be you get interactive web applications.
If you guys are familiar with MapQuest and some of the I guess the pre-web 1.0 kind of applications, you click a button and then it dynamically generates iframe URLs and then it refreshes. I believe in even Hotmail, the first version of Hotmail worked that way. So you click send and then it refreshes its iframe.
So that was like interactive web apps. And then around 99 Microsoft introduced XML HCP request which is the precursor to Ajax. It's a way of creating asynchronous request responses. Around 2004 Gmail launched and Gmail was the first I would say kind of a single page app.
So there's no full page refresh when you hit send to send an email. Around 2005 Ajax, the term Ajax was coined. 2005 was when Django was released publicly.
And then 2012 Meteor.js. How many of you guys are familiar with Meteor.js or heard of? Awesome. So Meteor.js is like was mind blowing when I first saw how it worked. So it was like a interactive fluid web framework where there was a subset
of your database being held on the front end. So you query your front end and it's pretty fluid. It's pretty amazing how it all works. I still don't know if it's a good idea but it's pretty sweet. 2014 React was public, started getting adoption. I believe that's when the single page app acronym SPA was started getting into the mainstream.
2016 I believe there's a small framework called Rails. They now include web sockets support into their main line. They call it action cable.
So that's roughly the outline of interactiveness on the web. So this is a very common way to visualize how technologies grow and get adopted across a wide spectrum. So I would say pre-2004, a lot of innovative apps,
a lot of risky applications were being built. Probably only 10 or 15% of the browsers out there supported the technologies. But that's the innovation and interactiveness being introduced. 2004 to 2014, early majority, big teams, a lot of resources,
very expensive to build these interactive web apps, but it was being done. Case in point, Gmail, Google Maps, a lot of the first version of the web streaming services and so on. 2014 and up to now I believe you're inching towards the promised land.
It's a lot of the best practices have been established. There's still some challenges in the fringes, but more or less we understand how to build a single page app. What are the concerns? What are the difficulties? There are at least three or four mature front-end frameworks
that work out of the box. Vue.js is a popular one. React.js is another popular one. A lot of folks use Ember.js, so that's pretty awesome. So I think we're getting there. Between 2014, around then, there have been a lot of the commercial applications of these fluid push notifications and that kind of stuff.
So Socket.io, Pusher, Firebase, Meteor.js, these are all a hybrid of open source and a back-end as a service kind of system to have interactive web apps.
So the main question is, why do you need this? Like what's the need, right? Because Django has been around for ten plus years and it works pretty sweet. Why would you want to change something that's working perfectly fine?
So instead of answering this broad question about why, I'm going to be scoping it into our web app and why we saw the need for web sockets and interactiveness and why just regular web apps didn't cut it.
So as I mentioned, our product is a SQL editor, essentially. You have a text box, you hit run, and then it creates these charts. And then you go through this workflow over and over and over again. So that's basically what our product does.
So if you were to draw a schematic on how this works, you have the app, which is the web app, running the browser. There's an AJAX request that reaches the back-end, and then the back-end, which is Django, executes the SQL query and then returns a HTTP response.
So that's basically the life cycle of your one single iteration. If there's a mistake in your SQL, your execute SQL throws an exception and then it's sent back as a HTTP response to the right error code. And your front-end handles that. So this, this worked for, you know, version 1.0 when we had no users.
Everything worked great. So it's, it's always pretty fun to see you hit run and then it, it works. Okay, so awesome. So let's go to production. The minute we hit production and once we got a couple of users, we started seeing these problems that were creeping up. So I'm just going to enumerate some of the problems we've had with this regular
request response cycle. The first problem is unbounded run time for the request. So here, essentially what's happening is we let the user type in arbitrary SQL.
So we do not know beforehand how long this SQL is going to take. It could take, you know, three minutes, it could take 30 minutes, it could take half a day, nobody knows. Because it's up to the user what, what kind of model or what kind of analytics they're running. So the problem was some of our users were typing their SQL, hitting run, and
then closing their laptop because they knew it's going to take half a day to run the query. And from our side, you know, having that connection, that request response, coming back and restoring the state was being was a big challenge. So that's the first difficulty.
The second difficulty was what is this, what we call 8 AM traffic spikes. So in our product, you can organize the SQL queries into a dashboard. So you can have half a dozen to a dozen different charts and create a dashboard. And at 8 AM, our users load up these dashboards, and
you have 15 queries hitting our backend, and they're all trying to run. And so the, the ratio between the peak load to the low point was about 7x. So you had to provision 7x the number of servers and resources to handle that load. So when I casually talked to my non-technical friends, I was like, yeah,
I was at the office at 8 AM, I was like, oh, good for you. You wake up at 6, hit the gym, go for a run, make your breakfast, and then you were at work at 8 AM. No, actually no. I I get paged at 7.55 that the servers are down, and then I rush to the office to fix them. But I don't tell them. I just say, yeah, I hit the gym and, you know, do all that stuff.
The third challenge was about deployment. Because the Django servers were actually executing these queries, you couldn't really swap out new versions of the code. So if you have, let's say, five servers that are executing these SQL queries, and they could take anywhere from 30 minutes to let's say, three hours.
Deployment was a problem because deployment would kill that executing query, and the user would never see the results. So, you know, we believe in active deployments, quick deployments. The minute the unit tests change or they succeed, they just push into production. But we were not able to do that because our users' queries were failing or
were being terminated. So that was a big challenge. So that is, you know, a summary of the challenges that we've had with synchronous request response cycle. So we had to look for alternate solution. So looking around, you know, I think I met Andrew last year or
the year before, and he was just starting out. Maybe this is like version .001 of Django channels. And I was pretty excited about it. It's like, okay, Django channel seems like the holy grail, right? What can go wrong? So let's play with it. And so high level how Django channel works is that you still have your
synchronous request response cycle, which is on the top half. And then you have a persistent bi-directional connection based on web sockets in the lower half. And so you send a message to your backend through the Ajax request.
And then the backend sends in the response through the HP response, just as usual. But at the same time, you can send messages back and forth through the web socket. So your server can initiate a message without having a matching HTTP request or Ajax request. So that's conceptually how web sockets or Django channels work.
They don't replace anything. It's just an add-on to your existing workflow. So the method we used was you continue to, so when our users hit run, it still makes a HTTP request to your backend.
And then your backend, which is Django, offloads it to this worker queue, executes a SQL, and once it's done, using web sockets notifies the frontend that the results are ready. So we're able to push it out of band.
So just as a thought experiment, the alternate old school approach would have been using a thread on the frontend to pull your backend for results. And if there were results, you'd get a response. But then that's undue load, right?
You can't set the polling interval based on the query because you don't know. So you're going to hit the backend, let's say, every 30 seconds, and that's going to break if the user queries only take two seconds to run. So the long polling approach had its own issues.
So pretty amazing. Just went to the documentation, copy-paste the examples into our code, two days, and we had it all working. At least it was working. Everything was working on my laptop. Okay, amazing. Let's go out for drinks because this project is done.
You know, where can I sign up for my bonus check? Well, no, not really. It took us about six weeks to make it production-ready. So going from our laptop to our cluster of servers took a lot of time. So you can copy-paste examples from the Django channels, but the rest of my talk is going to go into the challenges we faced
taking it to production, which I guess is the meat of my presentation. So it works on my laptop, you know, an MVP. It's a proof of concept. The challenges seem to, you know, it looked promising. So this is when you sit down and write a business case
for how to re-architect your app, because this is pretty expensive. You're writing the core or rewriting the core of how you're doing work. So we came up with this, a list of bullet points, and I'll just walk through them. So the big overarching aim of this project was to make the back end the driver of work.
The front end, just because the user says run this, we're not going to obey that command. We're going to schedule it and work on that query when we see fear, when we see it as a good time. So we're going to be, the back end is going to be the driver rather than the client.
Same handling of state, you know, the state could be a query is in progress. The query has failed. The query has succeeded. The query is ready to be run, and then there are disconnections in the middle because people could go to trains. People could shut down their laptop.
So you just have to, you know, map out all the possible states and try to understand what's going on. Just a clear picture. Silver side events, there are lots of approaches for doing this, but the silver should be able to communicate new data to the front end without the front end asking for it.
I think that's a very powerful concept. No proprietary solutions, not that I'm against proprietary solutions, but I feel that when you're working with brand new technology, it's a lot harder to debug proprietary technology solutions. So one of the difficulties I've had with socket IO is
you had to run socket IO on the back end, which is no JS, and it was pretty challenging to debug what was going on because I come from a Python back end and debugging in the external stack was a challenge. Keep investments in Python plus Django stack.
I guess what a lot of other people would have done is they would have just thrown in meter JS or a node JS stack, and then offload this asynchronous component to node JS and let node JS deal with this because there are lots of frameworks on node JS that help you do this. We did not want to do that
because every time you have an architectural decision and you add a new layer to the stack, I think your product becomes kind of a Frankenstein product, and it's just too much to keep in your head because now if you're trying to hire someone, you're asking them for React front-end experience, you're asking them for Django experience, you're asking for node JS experience
and the deployment options. It's just too much. So we wanted to keep everything standard, out of the box, Python plus Django. Off the shelf AWS components, our stack is completely on Amazon Web Services, so we do not want to build something custom,
which means no recompiling NGINX to add web socket support because we wanted to use Amazon's load balancer, for example, because it has a lot of benefits. It gives you health checks. It has a good logging framework. It does DDoS attacks, a lot of good features,
and we wanted to tie into that ecosystem, so it didn't make a lot of sense for us to bring in our own infrastructure components. So roughly this is our architecture diagram. So we have our clients on the bottom, which could be,
I just found this in some kind of stencil program. It's a phone, it's a laptop, and it's a desktop, so three kinds of clients, and they connect to a load balancer, which has web socket support, and the load balancer then starts to load balance to a backend, which could be the web sockets
plus the regular GUnicon, so it's a hybrid with the synchronous and the asynchronous part, and then Redis is the persistent layer where the channels are being shared. So if you go through the Django channels documentation, this is one approach that's recommended,
using Redis as a backing server, and of course you still have your standard Django components. You have a database. You have the ORM and all that stuff, but roughly this is the architecture. So now the challenges. Why did it take us six weeks?
That's a little odd. Did I go on vacation for four weeks during the project? No, not really, because we were iterating through the solutions and we found that something was wrong, and then we had to go back to the drawing board, and then we were trying to swap out this stuff. I think there was some conflict with our initial motivations in our business plan where we wanted to stick to just the Python stack,
so it was a lot more effort to get it to work, but I think now I'm really happy with how it's all working out, and going forward it's going to be a breeze to maintain because there's no custom code anywhere. So the first challenge was cloud deployment. I'm going to go step by step
into the challenges we faced. So if you remember the architecture diagram, we need a load balancer in front of GUnicon and the asynchronous web server. The problem with a lot of these standard off-the-shelf load balancers is that they're HTTP load balancers,
but what WebSockets needs is a TCP load balancer. So all the stuff, if you're familiar with Django's cookie-based authentication, cookies are a HTTP layer concern. So if you have a TCP load balancer, you can't really attach that cookies, like it can't read the cookies.
So if you're trying to have sticky sessions based on cookies, it's not really going to work. And I was surprised to see this, but a lot of the load balancers out there on the cloud were all HTTP load balancers. So Google Cloud, Azure, Amazon, but fortunately Amazon launched a brand new load balancer last summer
called the Application Load Balancer, which is a TCP level load balancer. But the problem with a lot of Amazon's products is that the documentation is just straight out wrong, or they say, yeah, we don't recommend you doing this, but that is exactly what you need to do.
So I was kind of fortunate that I was taking the train from Vancouver, BC, which is home for me down to Portland for PyCon, and one of the elastic beanstalk engineers got on the train. So I just fed him like tons of beer and asked him like, why the hell is this not working? What's wrong with this? Why is the documentation wrong? Why do you have three versions of the same stuff,
which are all like in different layers of compatibility? So the lesson here is, make sure it's working with the Application Load Balancer. The deployments on your laptop are very different from the deployments on the cloud. And try to figure out how Application Load Balancer works with the rest of your stack.
The second challenge we had was to do with missed messages on disconnect. So here's an example. Let's say Django periodically sends a server send message to your front end, right? So it's moving upwards. And so the green bars is when your client is connected to the back end.
And then for some reason it's disconnected. Either the user goes through a tunnel or there's some kind of disconnection notice, and the server sends a message. But this message has nowhere to go because the client is not connected. So that's the stop sign or the no entry sign. And then the client reconnects. And it's missed those messages.
And then it happens again. So if you're going to be missing a lot of the messages that your back end sends you, and the back end has no kind of system to acknowledge that these messages have been received, you're going to be left in a very bad shape. Your client state is just going to be in an invalid state. It's pretty challenging.
And I was surprised to find that on the cloud it's actually quite frequent to disconnect. Persistent connections, there's just not a guarantee. There is an NPM package called Reconnecting Web Sockets. Essentially it does a for loop. And if it's a disconnect, it tries to reconnect. So that was easy.
But what we had to do is actually fix the missed messages on the back end. So our solution to that was to serialize all of the messages being sent from the back end, timestamp it, and implement a cursor on the client side. So if the back end sends message one,
message two, message three, message four, the client keeps track of one, two, and then it receives four. Okay, you know what? There's a problem. So go back to two and give me all the messages since two. So that's roughly how it's essentially a table that we store on the back end. And then the client implements this logic
where if there's a gap in the sequence, it can go back and rewind that cursor. So what we did here as an implementation level detail is not to store it in our database, but Redis has this data structure called sorted sets.
And so you can actually pop and push into this Redis structure. And it's actually really fast. So there's no delay in trying to store this ring buffer. We also wrote a middleware because we wanted to make sure that every message going from the back end was going through,
was being timestamped and sequentialized. So it was important to go through that single stream. Our third problem was frontend architecture. Our frontend is a single page react app.
And these messages from the back end, you don't really know when they're going to come. They could be out of order. It could be different components because it's a single bus that all these messages come in. And so we were actually pretty confused in terms of how do you update your frontend state. And so what we did there, this is an example.
Let's say that's your web app and you have a notification system that gives you a drop down with all the new messages. And then you have a feed which tells you all the new elements that are inserted into your feed system. And our web socket handler gets all these messages and then you use standard jQuery's on messaging.
So you essentially implement your own messaging system on the frontend to dispatch these messages. So if you guys are familiar with how iOS apps are built, there's a single bus and you get these messages and then they're dispatched to different views. And these views listen when they load it
and then they destroy the handlers when they unload. So it's the same kind of concept. That way your notification system and your feed are not tying directly into web sockets. There's an abstraction layer there. And for whatever reason, if web sockets is not supported, you can always fall back. This is just an architectural idea
on how to organize your frontend. Problem number four, debugging. Debugging, testing, all of this was a big challenge because you cannot use cURL to mock requests. A lot of the Django test clients
and all that stuff doesn't work. So what we did is, you know the example with the middleware that serialized the messages? We just stored that as our fixture. So we were able to bring up a frontend client and then feed it messages from the backend based on this fixture. And then it's pretty important to keep that stream
of messages on your backend and the stream of messages received on the frontend and push them both into a centralized data store. So you can reconcile the messages that have been sent from the backend and the messages that have been received from the frontend and see if there are any gaps. So without this unified view across the client
and the server, it's kind of difficult to figure out what's going on. Here's another tip. Chrome has this pretty awesome WS tab. I spent two weeks debugging Chrome web sockets without knowing Chrome had this built in.
So you can just click on the frames and this gives you a sequential list of all the messages received through the web sockets. It's either outgoing or incoming, no problem. It just shows it to you. And we just use JSON as a transport. So you can inspect it, you can predefy it, whatever you want.
So guys, that concludes my talk. And I am open to questions. Thanks for the talk. In one of your earlier diagrams,
it looked like you were basically accepting the initial HTTP request via AJAX, running the SQL query like on a different Django box on a worker somewhere. How are you able, is that true, first of all? Were you doing it like in Celery in the background or?
Yeah, that one. There we go. Right. Are those two separate Django web servers? That's exactly right. So we were offloading the query to a Celery worker. So how are you able to basically keep the web socket alive but when you move it from the initial machine
that got the request to the Celery web server? So the Django channels documentation recommends Redis as a persistent layer. So from any web server, you can talk to a client no matter what actual web server it's connected to. So it gives you a unified view
of all of the clients connected. You mentioned that you use an event bus on the front end with React. Did you use something like Redux or what was the, did you hand roll one for doing that? We just used something built on jQuery.
jQuery event emitters. Okay. Yeah. I have a question. You mentioned sort of the hiccup in the original deployment when you were just using Ajax request response and I wonder how you handle sort of that
in this architecture where you still need to replace the code in that worker that's running the query. How do you handle the deployment of the worker that may be still doing work? Very good question. So our deployment cycles are different
for the Django app and the Celery workers. The workers, it's okay to run with let's say v1 of the code and then you can actually do delayed deployments. All new workers have version two and version one is still running, no problem. It's not, you're not killing the existing workers.
So one worker, one instance of the worker is one query. So it's a lot easier. How do you get the static data in there? Do you just use, do you just do a regular ETL to a Postgres database and then have them as unmanaged? Like the meta unmanaged Django models
so that there's a connection to the front end. Did that question make sense? So you must be running SQL on some static type of data, right? Like if you're doing data analysis could be gigabytes of data from flat files you loaded into Postgres.
Then do you somehow introspect and generate unmanaged models and the reason I say unmanaged is because you don't want to spoil any of the static data and accidentally have you know what I'm saying
but unmanaged is confusing but it's the meta tag for the model to not have it migrate you have to do a migration for it. Very good question. So the question was we execute the SQL against unknown schema and his question was do you import that schema into your Django models
or does it pollute your clean sanitized curated schema or not, right? Because it could be unknown and the answer to that question is we don't manage any of our customer's data. They don't even touch our Django. Our Django knows nothing about it and so we literally send that raw text
in that execute SQL box through a SSH tunnel to the customer's database which they host. So we don't import any models. We don't do, it's nothing to do with Django. Our customers don't even know we're using Django on the back end. So the clear separation between our app and the customer's data model.
Does that make sense? How did you solve the authentication issue? Maybe you answered it and I just didn't hear it but you said that we weren't sending the cookies over and then you kind of like just maybe jumped ahead. There's a full section in the Django channels documentation on authentication
but essentially what it does is it passes off the HTTP session to the Django channels session and that handoff is done during connect and so the subsequent messages don't have any kind of cookie header or anything. So but the Django channels documentation goes into a lot of detail there.
So you just have to add some decorators to your channel views essentially. You mentioned that one of the reasons you wanted to adopt a solution was the 8 a.m server load. Other than the removal of the long poles and are you ready yet type messages
did this solve that problem in any other way? So we definitely partially solved the problem in the sense that we still have that big spike of AJAX requests but the difference now is that that full life cycle there is less than 50 milliseconds.
It's not a long-standing connection. So you can connect 7,000 you can open up 7,000 connections that are 50 milliseconds each rather than having 7,000 connections which are three minutes long. That's the difference. You mentioned you're using TCP load balancing
and there so in the TCP I mean you're so essentially are you able to get the same level of affinity that you would in sort of a higher level scheme for load balancing
such as HTTP and cookie and session-based affinity? That's been a challenge for us. Sticky session is an unsolved problem in this scenario. Another challenge that we have is the IP addresses of the clients keep changing and so you can't really reconnect
to the same web socket connection. So I think that's the reason why there are a lot of disconnects and reconnects. I haven't gotten to the bottom of that yet but that is definitely an unsolved problem in this scenario. How do you manage lifetimes of connections because I can imagine that the reconnection example that you showed could look very similar to a user
saying oh I'm done with your website. I don't actually want any updates. Which portion? Which lifecycle from the web app to the back end or the back end to the customer's database? So the web socket connections if they go away they can be intentionally going away
or it could be a connection issue. So when they go away it unloads the app so we disconnect the web socket. Right now there's no way for you to just go offline essentially right? There's no way to go offline. We're you know pretty aggressive in reconnecting
and trying to reconnect. We have an exponential back off there just in case the back end is actually down then you don't want to like hammer your back in with every trying to reconnect every 50 milliseconds or 500 milliseconds. So you back off 500, 1 second, 2 seconds, 4 seconds just like Gmail. The Gmail you see is reconnecting
in 3 seconds and then if it can reconnect reconnecting in 7 seconds and then reconnecting 12 seconds reconnecting tomorrow and so on right? So it's the same kind of deal. Thank you again for sharing your expertise. Awesome.