We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Asynchronous network requests in a web application

00:00

Formal Metadata

Title
Asynchronous network requests in a web application
Title of Series
Part Number
78
Number of Parts
169
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Lauris Jullien - Asynchronous network requests in a web application Introducing asynchronous calls from within an endpoint in a web app can be desirable but hard to achieve. This talk will explore different solutions for this (running Twisted event loop, Co-Routines, Asyncio, …) and how well they play with the different parallelization models of common WSGI web servers. ----- In the more and more popular SOA paradigm, it’s common for services to have to compose responses with resources from many different services. Everyone’s first idea will probably be to call each service synchronously with your favorite python HTTP library. This unfortunately doesn’t scale well and tens of successive network calls will make your endpoints painfully slow. One solution is to parallelize these network calls. If you are already using an asynchronous web app (such as Tornado or Twisted), more asynchronous in your asynchronous shouldn’t be much of a challenge. But if you chose not to dive into the madness of chained Deferred calls, and used a standard prefork/threaded WSGI web server (such as Gunicorn or uWSGI) to run your Django/Flask/Pyramid application, you might find yourself wondering how to manage these asynchronous calls. This talk will explore different solutions (running Twisted event loop, Co-Routines, Asyncio, …) and how well they play with the different parallelization models of WSGI web servers.
World Wide Web ConsortiumFront and back endsSoftware engineeringBayesian networkCartesian coordinate systemWeb applicationSoftwarePhysical lawWeb 2.0Local ringLecture/ConferenceMeeting/Interview
StatisticsCodeServer (computing)SoftwareWeb applicationTotal S.A.BitOrientation (vector space)Mobile WebSynchronizationReplication (computing)Computer animation
Replication (computing)CodeEndliche ModelltheorieDifferent (Kate Ryan album)View (database)Point (geometry)SynchronizationWeb applicationSoftwareWeb serviceLecture/Conference
View (database)Web serviceWeb serviceLevel (video gaming)View (database)Server (computing)Computer animation
Web serviceSynchronizationWeb serviceMultiplication signSummierbarkeitTotal S.A.Web pageStandard deviationLibrary (computing)Lecture/ConferenceComputer animation
Concurrency (computer science)Thread (computing)Virtual machineBefehlsprozessorNumberRevision controlDimensional analysisProtein foldingProcess (computing)Thread (computing)NumberSoftwareSynchronizationCASE <Informatik>Condition numberReading (process)Virtual machineXML
Concurrency (computer science)Thread (computing)Virtual machineNumberBefehlsprozessorRevision controlMobile appAtomic numberWeb applicationComputer programmingQuicksortWeb 2.0Power (physics)Run time (program lifecycle phase)Limit (category theory)Event horizonPosition operatorStudent's t-testSinc functionSoftware maintenanceLocal ringNumberVirtual machineCore dumpRight angleThread (computing)Lecture/ConferenceXML
Mobile appPressureData modelThread (computing)Event horizonLoop (music)Mobile appProduct (business)Server (computing)SoftwareComputer configurationSymbol tableLecture/ConferenceComputer animation
Data modelPressureServer (computing)Gateway (telecommunications)Mobile appSystem callComputer configurationServer (computing)Web 2.0Endliche ModelltheorieCoprocessorVirtual machineProcess (computing)Flow separationMultiplication signCartesian coordinate systemMobile appDependent and independent variablesSemiconductor memorySoftwareSystem callCategory of beingThread (computing)Right angleProxy serverPhase transitionStandard deviationIntegrated development environmentForm (programming)PiReverse engineeringReflection (mathematics)Lecture/ConferenceComputer animation
Web pageDefault (computer science)Interpreter (computing)Mobile appVirtual machineSource codeNumberDirectory serviceProcess (computing)Limit (category theory)Maxima and minimaComputer fileNetwork socketAddress spaceRevision controlServer (computing)Operations researchFault-tolerant systemAsynchronous Transfer ModeThread (computing)Online helpNoiseGraphics tabletSystem callSingle-precision floating-point formatAlgebraCore dumpEmailComputer networkEuler anglesPhysical lawSummierbarkeitHardy spaceRaw image formatSoftwareSystem callServer (computing)Insertion lossRevision controlDisk read-and-write head2 (number)Multiplication signScripting languageBitWeb 2.0Lecture/ConferenceSource code
CoprocessorThread (computing)Process (computing)Mobile appSemiconductor memoryProcess (computing)Thread (computing)System callStress (mechanics)BitLimit (category theory)Mixed realityCoprocessorParallel portMultiplication signComputer animation
System callComputer networkEmailCore dumpOnline helpThread (computing)4 (number)Error messageLine (geometry)Computer fileTask (computing)Dependent and independent variablesComplete metric spaceSynchronizationException handlingVirtual machineNumberBefehlsprozessorDirectory serviceProcess (computing)Limit (category theory)Read-only memoryWeb pageNetwork socketAddress spaceRevision controlDefault (computer science)Interpreter (computing)Server (computing)Fault-tolerant systemOperations researchIcosahedronMobile appAsynchronous Transfer ModeAlgebraAreaWebsiteLocal ringEuler anglesMetropolitan area networkComa BerenicesProcess (computing)Thread (computing)Maxima and minimaGoodness of fitBitLecture/ConferenceComputer animation
Process (computing)Thread (computing)System callMobile appPower (physics)Point (geometry)Thread (computing)Process (computing)Revision controlCodeRight angleLecture/ConferenceComputer animation
SynchronizationDependent and independent variablesClient (computing)Loop (music)Mobile appProcess (computing)System callLibrary (computing)BitSoftware bugRule of inferenceSoftwareProcess (computing)CodeWeb serviceUniverse (mathematics)WordComputer animation
SummierbarkeitLevel (video gaming)Dot productMobile appMaß <Mathematik>Line (geometry)Error messageSystem callComputer file5 (number)Dependent and independent variablesSynchronizationTask (computing)Complete metric spaceException handlingComa BerenicesVirtual machineSource codeNumberBefehlsprozessorDirectory serviceLimit (category theory)Process (computing)Web pageMaxima and minimaNetwork socketAddress spaceDefault (computer science)Revision controlInterpreter (computing)Server (computing)Fault-tolerant systemOperations researchAsynchronous Transfer ModeAlgebraThread (computing)MaizeComputer networkGraphics tabletRead-only memoryStructural loadSystem callCodeGame theoryLengthSlide ruleDisk read-and-write headStudent's t-testScaling (geometry)Lecture/ConferenceSource code
SummierbarkeitThread (computing)Computer networkSystem callException handlingNetwork socketAddress spaceLine (geometry)Dependent and independent variablesInterpreter (computing)SynchronizationServer (computing)Computer fileAsynchronous Transfer ModeAlgebraError messageMagneto-optical driveLocal ringDefault (computer science)Revision controlFault-tolerant systemOperations researchMobile appCore dumpEmailProcess (computing)Online helpExecution unitComputer iconInterior (topology)Multiplication signParallel portSystem callDifferent (Kate Ryan album)Level (video gaming)Server (computing)Traffic reportingLecture/ConferenceSource codeComputer animation
Error messageLine (geometry)Ext functorComputer networkSystem callEuler anglesMetropolitan area networkComputer fileSynchronizationDependent and independent variablesCore dumpEmailThread (computing)Quantum stateSummierbarkeitMaxima and minimaComa BerenicesNumberThread (computing)Multiplication signDirect numerical simulationImage resolutionLecture/ConferenceSource codeComputer animation
SynchronizationDependent and independent variablesClient (computing)Loop (music)Mobile appConvex hullComputer programNormed vector spaceComputer programmingEvent horizonConditional-access moduleLeakSystem callThread (computing)Computer configurationLoop (music)Broadcast programmingEndliche ModelltheorieComputer programmingThread (computing)MathematicsEvent horizonMereologyResultantBitPhysical systemPoint (geometry)Crash (computing)Wave packetComputer clusterCodeSingle-precision floating-point formatMobile appComputer animation
EmailCore dumpSystem callGateway (telecommunications)Online helpThread (computing)Computer networkMobile appError messageLine (geometry)Computer fileWebsiteDependent and independent variablesSynchronizationComa BerenicesDirectory serviceVirtual machineSource codeBefehlsprozessorNumberProcess (computing)Limit (category theory)Web pageNetwork socketAddress spaceDefault (computer science)Revision controlInterpreter (computing)Server (computing)Fault-tolerant systemOperations researchAsynchronous Transfer ModeExecution unitTask (computing)Exception handlingMultiplication signThread (computing)SequelProcess (computing)Source code
Level (video gaming)Pairwise comparisonThread (computing)Image resolutionMobile appDirect numerical simulationThread (computing)Event horizonImage resolutionRevision controlLoop (music)XMLLecture/Conference
Level (video gaming)Pairwise comparisonThread (computing)Image resolutionMobile appMobile appCore dumpCoroutineLibrary (computing)Server (computing)Set (mathematics)MathematicsSynchronizationLecture/Conference
Event horizonMobile appPatch (Unix)Mobile appCuboidLibrary (computing)Commitment schemeLecture/ConferenceComputer animation
Partial derivativeEvent horizonMobile appWide area networkEvent horizonComputer configurationThread (computing)Green's functionImplementationProcess (computing)Library (computing)Cartesian coordinate systemSoftwarePatch (Unix)System callComputer animationLecture/Conference
Line (geometry)Local ringError messageDependent and independent variablesSynchronizationTask (computing)Exception handlingMobile appComputer iconInterpreter (computing)System callAsynchronous Transfer ModeNumberElectric currentValue-added networkComplete metric spaceRule of inferenceProcess (computing)Web pageComputer fileMaxima and minimaNetwork socketAddress spaceThread (computing)Server (computing)Fault-tolerant systemOperations researchDefault (computer science)Table (information)Connectivity (graph theory)Lipschitz-StetigkeitSource codeLecture/Conference
Pointer (computer programming)MIDIProcess (computing)Level (video gaming)Thread (computing)Event horizonSynchronizationError messageLine (geometry)System callComputer fileDependent and independent variablesException handlingDefault (computer science)Reading (process)Interpreter (computing)Network socketAddress spaceRevision controlServer (computing)Mobile appAsynchronous Transfer ModeComputer networkEmailCore dumpTask (computing)Complete metric spaceOperations researchFault-tolerant systemMagneto-optical driveComa BerenicesGraphics tabletLocal ringInterior (topology)Thread (computing)NumberPoint (geometry)Default (computer science)2 (number)Image resolutionMultiplication signDirect numerical simulationSource codeComputer animationLecture/Conference
EmailCore dumpComputer networkThread (computing)Computer fileLine (geometry)Network socketAddress spaceRevision controlInterpreter (computing)Exception handlingServer (computing)Operations researchFault-tolerant systemDependent and independent variablesSynchronizationMobile appDefault (computer science)Key (cryptography)System callAsynchronous Transfer ModeProcess (computing)Magneto-optical driveLocal ringEvent horizonError messageExt functorSuite (music)Bulletin board systemSpecial unitary groupClient (computing)Maxima and minima3 (number)Loop (music)Task (computing)CoroutineObject (grammar)CAN busSummierbarkeitArmHecke operatorBitNumberSurjective functionThread (computing)Process (computing)Source codeComputer animation
Core dumpBitException handlingSystem callLoop (music)Event horizonTask (computing)Line (geometry)Computer fileComputer networkEmailThread (computing)Cone penetration testComputer networkTorusPhysical lawSummierbarkeitCAN busMaxima and minimaHydraulic jumpPerfect groupProcess (computing)BitConcurrency (computer science)NumberDifferent (Kate Ryan album)TheoryCurvaturePhysicalismCASE <Informatik>Lecture/ConferenceSource codeComputer animation
Maxima and minimaImage resolutionEvent horizonQuicksortCartesian coordinate systemMultiplication signConcurrency (computer science)CASE <Informatik>Event horizonDirect numerical simulationImage resolutionRule of inferencePhase transitionProcess (computing)Proof theoryResultantLocal ringString (computer science)Lecture/Conference
Image resolutionEvent horizonCartesian coordinate systemSoftwareComputer configurationLecture/Conference
Event horizonLoop (music)Regulärer Ausdruck <Textverarbeitung>Thread (computing)Flow separationEvent horizonThread (computing)BitLoop (music)Flow separationStress (mechanics)Machine codeCodeAbstractionComputer animationLecture/Conference
Event horizonConcurrency (computer science)Loop (music)Regulärer Ausdruck <Textverarbeitung>Thread (computing)Flow separationRaw image formatFlow separationPresentation of a groupFunctional (mathematics)Thread (computing)Forcing (mathematics)Logical constantBitExtension (kinesiology)Ferry CorstenDependent and independent variablesGraph (mathematics)Computer configurationSoftwareCodeLoop (music)Set (mathematics)ResultantSystem callComputer animation
Event horizonConcurrency (computer science)Loop (music)Thread (computing)Flow separationResultantSoftwareCartesian coordinate systemFamilyRevision controlThread (computing)DistanceLecture/ConferenceComputer animation
System callDistanceResultantMultiplication signWeb service2 (number)Medical imagingLecture/Conference
Interior (topology)Flow separationThread (computing)Loop (music)Pointer (computer programming)Event horizonConcurrency (computer science)BitComputer configurationOcean currentLoop (music)Cartesian coordinate systemCodeEndliche ModelltheorieComputer animation
Core dumpSoftware engineeringGateway (telecommunications)Online helpEmailThread (computing)System callComputer networkEvent horizonSocket-SchnittstelleLocal ringException handlingLoop (music)Line (geometry)Social classSummierbarkeitComputer fileDot productMobile appRule of inferenceProcess (computing)Interpreter (computing)Default (computer science)Task (computing)Source codeNumberBefehlsprozessorDirectory serviceWeb pageRead-only memoryMaxima and minimaNetwork socketAddress spaceServer (computing)Fault-tolerant systemOperations researchAlgebraMathematicsWebsiteElectronic mailing listRevision controlAsynchronous Transfer ModeNetwork operating systemLength of stayClosed setMobile appCartesian coordinate systemLecture/ConferenceSource code
Error messageLine (geometry)Computer fileTask (computing)Dependent and independent variablesSynchronizationComputer networkThread (computing)Online helpNetwork socketAddress spaceDefault (computer science)Revision controlMaxima and minimaInterpreter (computing)Server (computing)Operations researchFault-tolerant systemMobile appSystem callAsynchronous Transfer ModeProcess (computing)AlgebraCore dumpEmailComplete metric spaceExt functorException handlingElectronic mailing listLevel (video gaming)Pointer (computer programming)Magneto-optical driveLocal ringMultiplication signCategory of being2 (number)Computer fontLecture/ConferenceSource code
Thread (computing)Error messageLocal ringLine (geometry)Computer fileState of matterDependent and independent variablesSynchronizationComplete metric spaceTask (computing)Exception handlingEmailComputer networkSystem callCore dumpReading (process)Euler anglesKey (cryptography)Server (computing)Thread (computing)Scaling (geometry)FrustrationStress (mechanics)Semiconductor memoryLecture/ConferenceSource codeComputer animation
System callFlow separationLoop (music)Thread (computing)Event horizonComputer networkComputer networkComa BerenicesEvent horizonLibrary (computing)Product (business)Loop (music)Computer animation
Coma BerenicesCodePresentation of a groupConcurrency (computer science)Image resolutionCurve fittingComputer networkMereologyLibrary (computing)Image resolutionDirect numerical simulationPoint (geometry)Multiplication signEvent horizonConnected spaceConcurrency (computer science)Mobile appLogical constantLecture/ConferenceComputer animation
Coma BerenicesBlogComputer cluster1 (number)Lecture/ConferenceComputer animationDiagram
Library (computing)BefehlsprozessorThread (computing)Multiplication signDirect numerical simulationLocal ringImage resolutionLecture/ConferenceDiagram
SoftwareThread (computing)Goodness of fitTwitterLecture/Conference
Electronic mailing listGroup actionProcess (computing)Instance (computer science)Right angleSoftwareMereologyMathematical optimizationMultiplication signCartesian coordinate systemStress (mechanics)Lattice (order)ResultantThread (computing)Computer configurationLecture/Conference
Event horizonPatch (Unix)Human migrationLecture/Conference
Standard deviationBitPatch (Unix)Speech synthesisLibrary (computing)Event horizonComputer configurationLecture/Conference
Vector spaceProgrammschleifeDifferent (Kate Ryan album)Presentation of a groupStudent's t-testPlanningLecture/Conference
Lecture/Conference
Transcript: English(auto-generated)
Let's welcome Louris Jullien. He works as a backend software engineer at Yelp in Hamburg and will present us a talk about asynchronous network requests in web applications. Give him a warm welcome, please.
Right, thanks a lot for all of you being here. Asynchronous network requests in web application, but first, I work for Yelp. It's an awesome company. We're connecting people with great local businesses. It allows you to find what's interesting around you, restaurants, hotels, cinema, everything.
We have 90 millions monthly active users just for mobile. We have over 102 millions review in total. 70% of our traffic search come from mobile, so we're a very mobile-oriented company and we're launched in 32 countries, including Spain. What is this talk about?
First, I'm going to introduce a bit why did I do this talk, what's problematic, and what's actually not so easy about wanting to do asynchronous network in web application. We'll do a quick reminder about what's a deployment server, USB, what's a preferred model, and we'll dig into what's the meat of this talk,
some coding examples, some different ways to do this. I might skim over many details and the goal is to have an overview of different ideas on how you can do a synchronous network request in web applications. That's based on our app, that's what I'm working on,
and we have a service which is publicly available, and from a very high level, people do requests on your services, but that's a very, very high-level view, and that's not really what it looks like. You don't serve on only one server. What many people do is they have an SOA, and at first, it's fine.
You call your session service, you call your business service, you call your user service, and it's horribly slow, and it gets slower and slower as much as you add services, and the user are just waiting in front of their phone, which you don't really want. You want them, you want answers. So what you might want to do, if you can,
is just call all of these services at the same time, and that way, they are not, as your total time spending for your public service endpoint is not the sum of all the other services you have to call but just, hey, what's the slowest, and that's going to be your timings.
We are all doing Python. Standard library contains most of the tool you need for everything, and there is a very nice thing, which is the thread pool executor. I've taken copy-pasted things from the documentation. The example is exactly about doing asynchronous network requests. It's very nice.
It's what you should be doing. You could be using processes, but hey, threads are very nice. As I'm reading from the documentation, thread pool executor is often used to overlap IO, which is exactly what we want to do, and there is also another mention, which is it will default to number of processes on the machine multiplied by five, and that's kind of where things go
into what some problems sort of arise, which is we don't run just one Python program. We're running a web application, and our goal is for a web application to run on as much core as possible and as much power to serve as many requests as it possibly can, and right now, the thread pool executor
is going to be concurrent with this, and also it means that there is only a certain limit until which it can serve requests. Right now, it's going to be number of core on your machine multiplied by five, but this means if you want to do one more request, it's not going to be executed completely in parallel, so that ain't great.
I'm going to do small little things about just to define what we call a web app. If you're running Tornado, Twisted, this kind of evented app, or even something that's directly plugged into gEvent, that's not exactly the position I'm going to take. You're going to learn a lot of things. Don't leave the room, but I'm going to focus on the Wysgi app,
because that's what Django, Pyramid, Flask's app are fundamentally. It's fairly unlikely that you're deploying your Wysgi app with Wysgi's simple server or the HTTP simple server, so we also need to choose a deployment option, just for the sake of us talking
about asynchronous network requests for today. I will be using uWysgi, because it's very used in production. There is a lot of options. You can do almost everything, and it's using the pre-forked model, which is what's most efficient to run web servers.
If you don't know what to, oh, that didn't start at the right place. Cool, so you have the master process, you have worker process, which is going to load your app. It's then going to fork to create the workers, and then going to run your application in several pie threads. That's really, really nice, because the fork has a property of being copy on the right,
which means that you will save all the memory for all these new processes you've spawned, because they are all the same. We'll come back to that. Cool, and when you want to serve requests later, you have something, reverse proxy, or an HTTP worker using uWysgi, and your master process is reparting requests
to workers and to their threads. Cool, no one looks too badly injured right now. That's a simple solution to do what we wanted. It's completely synchronous, I'm not even using the thread pool. It's going to be our base of reflection
for everything that's happening. You have the application, which takes the environment and start response, which is the basic of the uWysgi standard. We are, I'm using request for the network calls, because it's way easier that way. I have on my machine a small server, another server running, that is going to be just,
you give it a time, it waits for that time, and respond, and that's a very nice simplification of long network calls. Let's run this. Great, so uWysgi, that's a basic version.
We're just going to run once. Nothing too complicated. Let's hit H, stop to it, and let's call our web server.
This is supposed to last 1.5, 25 seconds. It lasts a bit more, because overhead, and that's long, very, very long.
Right now, I have another little script, which just allows me to hammer the server, and you will see that it only serves one request at a time. All of these requests are arriving in parallel, so if we ask for three requests,
it's going to wait for 4.5 seconds. It's going to be the exact same as the thing, which is what we don't want. What I will just show quickly is running server in another version. Oops, sorry. So I prepared many, but for, as I'm talking way too much already, we're just going to run what's called the mix
with threads and processes parallelization. Basically, every time you actually process, it should not add too much memory, but it still does, and you're using more cores, and threads-wise, you had a bit more memory, because threads overhead, but with the gills, and everything runs on the same processor core,
so there is a limit to how much threads you can run in parallel. Right now, I'm using a VM for controlling this, and actually, two processes to thread is the maximum I can run, because I made a very, very small move. Sorry, a very, very small one, and let's run this other one with a mix, and you will see we can handle, yeah,
let's throw four requests at it, and let's change the port, and yeah, we waited a bit more than expected overhead, but still, it's good, and if we throw a fifth one at it,
then we're going to wait an awful more longer, because one of the requests is waiting until the other are available. Cool, so that's USB in two minutes and something. Great, you can see that, basically, whether you add processes or threads, it parallelizes in a very similar manner.
A note I'm completely forgetting, it's so obvious, all the code I'm going to present, it's Python 3.5, everything is made here with Python 3.5 in mind. I might tell you if things are available for all the Python version, but use the new Python, it's nice, it's good.
Great, so that's very simple. Let's try to do something a bit nicer. Still, let's go into looking into the standard library where it is available. Hey, I think I heard, nice. So that's, again, a very simple solution. I'm going to use, always do all my examples
with two processes. There is a very good reason for that is, as I was telling in the beginning, it's a fork abusing server, so many things can be a bit different when you have, if you just run one, you can not see everything. Some bugs appear, some strange thing happens because the fork process happens
and some things are loaded. So just to be really clear that it all works, two processes, that's going to be the rule here. All the code is taken directly from the IO HTTP library for doing network calls. If you do not know how this works, there was a bunch of talk at this conference
to know how to do so, and there is more again today. Go and see them. Let's run it. Cool, let's get that. You with the game. You with the game, do you know?
No, yes. Let's attach a stop to it so we see how many threads, what's happening there. Sorry for me muttering in French. And let's curl it, just to prove that
I actually did my job and prepared my slide and the code is actually working as I advised. Yes, this only lasted half a second plus overhead, and that's the length of the longest call in the pile of code we wanted to do.
Victory. And we can see something a bit different as well, which is like all the calls started at the same time and they all ended at the same time. They did happen in parallel. We are thrilled. If we try to hammer it again, and this time I will remember to change the port,
and let's throw two requests at it. Yeah, everything is happening fine. If we throw four, we're waiting. Parallelization on the WSGI server level is completely transparent. There is not really a difference with what we were doing before, so this is fairly nice. You might have noticed, looking at HTOP,
that awful new number of threads appear, which we did not really expect or account for in any way. Let's look at what they are doing. Okay, I'm out of time. Believe me, they are doing DNS resolution.
It's a little problem with AOHTP. It spoons the thread pool executor to actually undo the DNS resolution, and you need to be aware of it because it could create a lot more threads for you. Just looking at the code again,
there is one part which is a bit more important than it seems. It's asyncio.getEventLoop. It's not as transparent as it seems. What we did is we kind of changed the way our program run, and there was very nice schematics in twisted network programming essential,
so I just took it. So single-threaded is what we used to do in the single app system, and what we're trying to do now is event-driven colon, so we are not waiting anymore. We are just piling up, piling up, and everything is happening inside the loop. We changed the way our program run, but this schematic is nice, but actually there is still a lot of holes into this,
so that's showed. My performance didn't crash, so I don't need this. Oh, little point. If you try to use the threading model, you will encounter problems because getEventLoop just changed the main Python thread.
You can go around this with a little trick where you just set the event loop if you getEventLoop fails. If you fail to get the event loop, you can just recreate one, and that way you can run with thread, which we can just prove right now.
So this time you see core two, which means we are back in the situation which we presented initially, two processes, two thread per process, and this time if I sort of request that it was the right port, yeah, it works.
Awesome. We see that there are extra threads from the DNS resolution. This is actually going to be a solved problem fairly soon as IIO DNS is coming in, and it's going to have rather than a thread pool executor using the event loop like the rest,
but right now it's still not available on stable version. You need to use a master version of HTTP IIO to be able to do that, IIO HTTP, sorry, to do that. Cool. We did a lot of things, and as I started mentioning, we did change the way our app run, but maybe we could go a bit further than that
and just change also the way the whole server runs and have everything be core routines. Maybe it would be really nice, and I will continue my bingo of using asynchronous library in Python with g-event, which should make actually a bingo with tornado, twisted, and async IIO.
The g-event app. So here are two main things to notice. We monkey patch the standard library, so it does not come out free out of the box. It's a commitment. We are changing further and further and further the thing,
and no, yes, and we pass the option to you with g to spin g-event, micro-threading to the thing. G-event is not exactly an event loop. It's based on the, sorry, green thread implementation, but for all purposes, it really behaves in the same way as async IIO would.
And you might also notice that I didn't change anything at the network call. It's exactly the same as the one in the synchronous application, which is nice. So we did patch the standard library for it. Again, to process, let's run it.
Let's kill this one. Great. It tells you what you're running. Let's call it.
Yeah, it works. I forgot to attach myself to a worker. You will see why I do that in a second. When you see the number of threads, again, yeah, again, g-event points threads by default
to do the DNS resolution. No, it happens. If we try to hammer, again, the server, let's put a lot more. So I had put in 50 g-event core.
So this should be like the maximum number we would expect for things. 50 g-event threads onto processes, 100. Hey, but that's actually a bit more. That's not 1.5, that's a bit more. Let's do less to see how it goes. No, that's not less.
And yeah, okay, let's do something in between. G-event, the scaling happens in a bit of a different manner, which is you don't get perfect performance, then a jump, then a jump.
You have a slow raise in the number of concurrent requests that you can accept for an extra little cost. And if we go above what we specified, then it jumps again. You have a jump, but there is not a flat, hey, I take request, I take request, I take request.
I take request, I take request, I take request. It's I take request, request. There is some sort of ramp up and extra cost to running it. So in that case, we gained a lot of concurrency, but we lost the gain in timings that we wanted. And we completely rewrote our application. We made everything revolve around G-event.
And hey, oh yeah, that was a good idea. So actually, if you estrace one of the process, you can, I can prove you that, hey, it's doing DNS resolution. You can see local rules getting resolved here.
And yeah, it depends on how good you are with reading estrace. And you are changing everything, and you might not want to do that. What you might want to do is keep your good old application. You don't want to deal with AsyncIO because you believe you're going to mess it up. You want the simplest solution possible,
and maybe just the network calls and all the rest be gone. Like you really want a minimal option. And there is a last technique to do that, and a last idea that I will present. And that's offloading the event loop in a separate thread,
and then interacting with the thread. That way you have, rather than having, we're a bit back to the thread pool executor in the beginning, but rather than having many, many threads, you just have one which does everything asynchronously, and then you interface your code with it using the future abstraction which comes in handy
and is exactly made for this. The code is a bit more involving here. I'm using Tornado because it makes a nice code for a presentation. I will mention another option just a bit. Just wait, hold your answers, it's coming. We have a function that we create
and instantiate the event loop. We make it in current. We have a function that makes its current in the thread start set. We spawn this in a separate thread. We start, I do clean up in a terrible manner with ATX it. Don't do that. It just makes the code nice and doesn't make the presentation crash, but there are better ways to do this.
The long network call becomes a bit more complicated because we need to pass the result in a future. You might be noting I'm using concurrent future and not the Tornado future, if you're familiar with Tornado. It's because we are not running things in the same thread, and so we need the thread safe version of it.
Great. The application is very, very nice, and we get an extra things from being able to do this, and that's, we can do something else.
Right now, all the example I presented, you were preparing all of your network calls, then throwing them at the distance server, and then waiting for result. By doing this kind of things, what you can do is like, hey, I want this thing, then image this one, and then you block for the result you want
at the time you want them. If you have things which are, some calls which are dependent on one another, you need to call the business service to get the idea of a business and then call the second one, that might be really interesting for you. That's what we're using at Yelp for obvious reasons.
Right now, I'm just sleeping, one second. Let's run it. Oh, no, a little thing. You might notice an extra option here, which is called lazy apps, and I'm forced to do this because I'm cheating a bit. If you do remember, the application is loaded
and then forked, and the problem with that is the higher loop is created right away when the code is imported, and that wouldn't work with the pre-fault model perfectly, so I'm using lazy apps to avoid this behavior when just a Python stack is loaded,
then the fork happens, and then the application is loaded everywhere. It's running. Cool, let's curl it.
Yes, one second, it's not 0.5 like usual because we wait one second, but we just have one second. Everything happens in parallel behind. We are happy. If we try to hammer it, we have the same kind of performance
and with the right port. Yeah, and if I put three, we should wait for three seconds, which would give me time to make a new stop on it,
and we see only two threads, which is what we expected. Tornado does it right. There is no extra thread spawn. Victory. Again, our server scales in the exact same way as the original WSGI server did,
and that's very nice because the extra cost we have is just extra thread, but it's not really a very working thread. What it does is mostly waiting, so it does not impact performance much, does not impact memory much, and it's very, very practical, especially if you don't want to get too involved
into asynchronous things. You just want to hide the fact that you're doing asynchronous things. I showed you could, you should really not be using in production, but fortunately, there are a few libraries you could be using. If you are really interested in offloading the event loop, crochet is a very, very nice library.
It uses a twisted event loop, and you can basically run anything in the reactor, twisted reactor offloaded. It's very, very nice, and if you are just wanting the networking part, Yelp does a library, which is called Fido, that you might have heard of if you went to Stefan's talk yesterday,
and use it. It's cool. Final note. So it was very, very short. I skimmed over many, many things, but what the takeaway is use what fit you need. If you have like a very, you need to involve many, many asynchronous connections,
and you want something, do evented app. Go for g-events, go for, do something that help your app, but if you do not want to do it, don't say, I can't, it's like, I need to do this. No, there are many, many solution to keep your app in the way it is, and still get the performance. Remember that you're doing trade-offs
between speed of your endpoints and concurrency, and beware of DNS resolution. It can really be a problem and a pain from time to time. We are available everywhere. I would really like to recommend the engineering blog. It's really nice. We published lots of articles, interesting ones,
and we have a booth at which I will be later if you have more questions that I cannot answer right now. Thank you very much, Loris, for this talk.
We are taking a couple questions. I have a few minutes left. Please raise your hands. I might have a question. Why do you need to avoid these threats for DNS?
You don't need to, but it's an extra thing that you don't really need. Like, if you were, you could also avoid DNS resolution altogether. If I had written 1.7 from the local host directly, I would not need that, obviously. It's unnecessary. It's more four threads that are taking CPU times
that could be just integrated more seamlessly into the rest. Tornado does it, Twisted does it, and not all library do it in, it was more surprising in a way than really completely bad and completely terrible. It's more you can avoid it. Okay, thank you.
Any more questions? Yes. Hi. Hi. Very good talk.
You were doing offloading to another thread inside of UWSGI, but one of the things that the UWSGI documentation talk about is that when you activate the threads, it actually has to create the GIL, and it doesn't if it doesn't allow threads,
which is strange, but it's how it works. Have you thought about running all the network calls in another process, and then get the results back so you don't have that? I don't know if the trade-off is going to be good or terrible, but yeah, maybe. We thought about it, but actually you don't,
so actually you have several options. You could also tie yourself closer to UWSGI, and right now I'm spawning the thread manually, but you could also ask UWSGI to spawn the thread for you, and that way it would really be shared across all of UWSGI instances. It's an optimization, but that ties you with UWSGI, which means that you use UWSGI for your deployment,
and you can do it if you want to, but if you want to switch to G Unicorn or if you want to switch to something else, it means that you need to rewrite part of your application code, and for putting it in a process, you don't need to because this offloaded thread actually doesn't do much.
What it does most of the time is waiting, and waiting in Python, you don't hold the guild that much, so the guild is not a problem with a thread that does so few things, because it's spending this time waiting most of the things. If that were to change, so if you were to offload a lot more things in it,
maybe it would be worth it, but as you're IO bound most of the time, threads is more efficient just memory-wise. Yeah, yeah, thank you. Any other questions? Okay.
Hi, thanks for the talk, and my question is about G Event. Basically, it manky patches the standard library, so that means that messes with internals, and how do you feel? Will it survive to the next Python releases?
I'm not, to be perfectly honest, I'm always a bit wary when you have to patch the library especially because that means that when you enter, sorry, I'm speaking way too fast, when you integrate with all third-party libraries, you have problems, and when you go G Event,
it's like, hey, you need to use everything G Evented, and now that there is a standard in Python which is AsyncIO, I think as people are migrating to Python 3, things like G Event will disappear, but for people using Python 2, G Event, Twisted Tornado, they are really, really good options,
so I would think it's going to go away, but yeah, my personal opinion is it's not going to survive AsyncIO unless there is some huge improvement into the scheduler, so as they are not exactly running the same way,
it's a green thread, unless there is a huge improvement in that regard, things like UV loops for the AsyncIO should be more performant anymore. Any more? There's one in the back.
Thank you for that long walk, and thank you for the great talk. Just a comment rather than a question.
G Event, Python 3 support is being worked on, so they have plans. You're right, maybe they will win over AsyncIO, or maybe we have a wonderfully vibrant ecosystem of different approaches. Cool, thanks.
Okay, then no more questions. Thank you very much, Lorius.