An HTTP request's journey through a platform-as-a-service
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 21 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/19979 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201421 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
Computing platformPerturbation theoryProgrammer (hardware)DialectSpreadsheetPhysical systemService (economics)Computer animationLecture/Conference
01:30
Computing platformSoftware developerWebsiteNumberService (economics)Computing platformOperator (mathematics)Descriptive statisticsBlogWeb pageXMLMeeting/Interview
02:34
Scripting languageSoftware testingBitWebsiteMultiplication signSoftware frameworkDependent and independent variablesLevel (video gaming)BlogSoftware testingTest-driven developmentSoftware developerTraffic reportingLecture/ConferenceXML
03:46
Electronic mailing listSelectivity (electronic)Goodness of fitWebsiteMusical ensembleDisk read-and-write headXMLUML
04:13
Stack (abstract data type)Structural loadWeb applicationLatent heatCASE <Informatik>1 (number)BitProduct (business)Installation artCommunications protocolScripting languageVertex (graph theory)Electronic mailing listSet (mathematics)Process (computing)Local ringMathematicsXML
05:13
NumberDescriptive statisticsCartesian coordinate systemConfiguration spaceVideo gameComputer fileGenderLecture/Conference
05:34
CuboidDatabaseCalculationProcess (computing)Connected spacePointer (computer programming)Form (programming)Context awarenessBound stateSet (mathematics)Electric generatorWeb pageServer (computing)Physical systemDirect numerical simulationWebsiteMoment (mathematics)Variable (mathematics)Graphical user interfaceComputer fileConfiguration spaceClient (computing)Dimensional analysisElement (mathematics)MereologyVirtual machineService (economics)Web applicationGoodness of fitFunctional (mathematics)RootSoftware testingLaptopDecision theoryCodeLastteilungFormal languageRight anglePoint (geometry)CodeBlock (periodic table)Plug-in (computing)SummierbarkeitUniform resource locatorNatural languageWeb browserComputing platformScripting languageInternetworkingBlogObservational studyProxy serverStructural loadFront and back endsInstance (computer science)Scaling (geometry)WordWorkstation <Musikinstrument>Flow separationElectronic mailing listLatent heatPattern languageIP addressNumberWeb 2.0Subject indexingPhysicalismNormal (geometry)String (computer science)Network topologyComa BerenicesModulo (jargon)Single-precision floating-point formatFilm editingRoutingBitOrder (biology)Message passingHash functionDebuggerSystem callEmailChemical equationXML
11:50
Proxy serverFront and back endsScaling (geometry)Structural loadLastteilungInstance (computer science)Virtual machineProcess (computing)Web pageWeb applicationDirectory serviceError messageConfiguration spaceStructural loadSocket-SchnittstelleDatabaseSign (mathematics)Domain nameService (economics)Web 2.0Front and back endsComputer fileInformationPhysical systemWebsiteCodeState of matterScaling (geometry)WordOpticsCartesian coordinate systemDependent and independent variablesCASE <Informatik>Proxy serverLimit (category theory)Sampling (statistics)Set (mathematics)View (database)BitWeb serviceServer (computing)RotationCategory of beingFunctional (mathematics)Web browserSoftware testingSystem callNetwork socketAutomatic differentiationDomain nameRegulator geneMedical imagingComputer hardwareNatural languageMass1 (number)Multiplication signGroup actionModal logicLine (geometry)Electronic mailing listMessage passingSound effectHydraulic jumpDifferent (Kate Ryan album)Normal (geometry)Block (periodic table)Channel capacityXMLUML
18:06
WebsiteSoftwareService (economics)SummierbarkeitNamespaceLength of staySoftware testingWeb 2.0Scripting languageCuboidDesign by contractDatabaseVariable (mathematics)Game controllerUniqueness quantificationLastteilungElectronic mailing listIntegrated development environmentCodeLoop (music)Extension (kinesiology)Process (computing)Error messageSoftware developerMultiplication signZirkulation <Strömungsmechanik>Connectivity (graph theory)State of matterMereologyInstance (computer science)Identity managementStress (mechanics)Limit (category theory)Sampling (statistics)FluxDependent and independent variablesData managementVirtualizationCASE <Informatik>Communications protocolMaxima and minimaBitPower (physics)NumberMoment (mathematics)Set (mathematics)QuicksortAutomationRootGoodness of fit2 (number)Right angleMathematicsServer (computing)Condition numberSign (mathematics)RoutingLecture/Conference
Transcript: English(auto-generated)
00:15
The first of our speakers is going to be Giles Thomas.
00:22
Giles started programming Python when founding a business. He wanted to revolutionize the spreadsheet world by making spreadsheets programmable and then he tried to sell them to financial companies. That didn't work. Then his team moved over to producing the Python system
00:44
that they wanted for people like themselves and that sold a lot better. Giles is also playing the guitar. Today however, he is going to take you on a journey.
01:02
The journey of an HTTP request through a platform as a service. Please welcome with a hot applause, Giles Thomas. Well, thanks for an excellent introduction.
01:20
Thanks everyone for coming. So yeah, the thing that we wrote, the thing that we thought that we were gonna build the Python system that we wanted is called Python Anywhere. It's a platform as a service. It does lots of things. One thing that it does is that it hosts a fair number of websites. So just wanted to get a general feeling
01:40
about how much people in the room know about running websites. How many people here are responsible for the continued operation of a website? Maybe a personal blog or company pages? Okay, so a fair number, yeah. That's maybe 50%. Let's say, let's bring that up a little bit. How many are responsible for several websites?
02:03
Okay, more than 10? Still a few. More than 100? Still one. More than 1,000? Okay, I'm not gonna keep going up because it'll be really embarrassing. It'll turn out that you actually run more websites than we do. We have 24,241 websites running on Python anywhere
02:20
as of this morning. Actually, probably a few more by now. And we've got infrastructure to run this. It's a simple platform as a service infrastructure. I'm gonna go through a description of it pretty quickly, touching on a few of the details. But what I'd like to do is leave quite a lot of time for questions because I think which bits are interesting to drill down into will probably come more
02:40
from you guys than from me guessing on what you're going to be interested in. The websites we have, they range from very basic things where somebody has basically started using a particular framework. So somebody here has been trying out Web2Pi. And maybe they're gonna build something. It'll get a visitor a day. Maybe they're just a hobbyist experimenting.
03:02
The next stage is maybe sites which get a couple of hundred visitors a day. This guy is learning Mandarin Chinese in a particular way. He's sharing his lessons with a few other people. So a couple of hundred visitors a day. We want to spend almost no resources on the first kind, enough resources to keep this kind of site responsible.
03:20
It's very responsive even. Other people are running moderately popular technical blogs. This is my colleague Harry's Obey the Testing Goat. It's a companion site for a book he's written which is awesome and you should totally buy it if you're interested in test-driven development with Python and Django. But he gets maybe 2,000 visitors a day. So the site needs to be responsive. It needs to be up all the time.
03:41
But it doesn't need that much more than that. It's not a high-volume site. This is one of our most fun customers. This guy is running a site and it's insanely popular. It gets dozens of hits every second pouring through there. It's actually quite a good selection of music even if you don't like getting out of your head
04:00
and various things. But it's again quite a popular website. It's not Amazon, it's not Google. But it's got to be there, it's got to be responsive and it's got to be maintained at an affordable price. So, how do we do this? Well, here's a very basic list, it's a set of logos. These are the tools that we use.
04:20
We can do Linux obviously. We use Nginx for our load balancing, for all of our HTTP needs. We use uWSGI which is an absolutely awesome product which manages Python processes for you so that they can serve up web applications. It can serve basically any web application that uses the uWSGI protocol.
04:41
So that's gonna be Django, that's Web2Pi, that's Bottle, that's Flask, that's all of the big ones possibly except for most tornado installations which doesn't play so well with uWSGI. We do use Redis. I'm not gonna go into much detail on that today. We also use Lua for a certain amount of scripting. Now, I know I'm in a bit of danger here for talking about the ones of Lua at a Python conference
05:01
but we do use it. It's awesome for what it does for the specific use case we have. Now, you'll notice that I've got Django and I've got Python there. I didn't mention them. Well, all of our infrastructure uses the tools that I've described so far. All of the configuration is managed by Python. It's managed by a number of Django applications
05:22
which basically spit out the configuration files that all the other stuff needs to run and keeps the cluster live and doing what it's meant to do. So I promised a description of a HTTP request journey through this platform as a service and here are the machines that are involved in that.
05:41
So you can see what I have here is up in the top left. Let's use the mouse pointer. You can see each of these blue boxes is a separate physical machine or a separate instance running on Amazon AWS or whatever. Here's the user's laptop. He's running Chrome. Down here we have a load balancer and a bunch of backend servers. So everything apart from this machine
06:00
up here in the top left is part of Python Anywhere's infrastructure. We run on Amazon AWS but that's kind of, that's not particularly relevant to the context of this talk. Let's say that the person who's running the browser up here wants to view my friend Harry's website.
06:20
They want to go to visit www.abaythetestinggoat.com. Well, their browser makes a DNS request. The DNS request comes back with the IP address of abaythetestinggoat.com which is the IP address of this load balancer down here. So it opens up a TCP IP connection down to the load balancer. It sends the request to the load balancer.
06:42
Now, in order to route it through to the web application, let's say that abaythetestinggoat is this web application here, is this Python process that's running over on this particular physical machine, the middle one on the right hand side if you can't see the mouse pointer. So the load balancer needs to have the intelligence to be able to know that abaythetestinggoat.com is running on backend server two.
07:01
So, we'll just say that's magic for the moment. Let's say it magically knows backend server two. It makes a connection. And now we have two TCP IP connections, one from the client to the load balancer, one from the load balancer to the backend. The backend now needs to identify that the process web app four is the Python process
07:21
that's running abaythetestinggoat.com. It does that. Again, we'll say magically. And makes the connection. The web application code does its calculations. It renders templates. It talks to the database. It does whatever magic it does to generate a page. It sends it back to the backend server. Backend server sends it back to the load balancer. Load balancer sends it back to the client.
07:43
Now if you're used to running normal kinds of websites, the kind of system where I personally, for example, I have a VPS where I used to host my personal blog, then you might be thinking, what's the point of the load balancer in there? Because normally you'd simply have a server that looks rather like the backend server here. It's running a front end web server like Nginx or Apache.
08:02
And it's got a number of web applications running, one or more web applications running as Python processes underneath it. Why do we have this extra step for the load balancer? That's kind of where the magic comes in because it's the load balancer that allows us to scale up, to scale down, to add in resilience and failover and all those other good things that people expect when they outsource running their web applications
08:21
to a third party like us, rather than renting a VPS. Right, so I said the load balancer knows by magic which backend it's got to send that request to. Our load balancer is running Nginx. It's running a specific flavor of Nginx called OpenResty.
08:42
Now Nginx is an awesome web server. It has, it's extremely fast. It's very good at proxying connections through it like we do through the load balancer. And it has a lot of great plugins. OpenResty is basically Nginx with batteries included. One of the batteries that's included is Lua scripting. The kind of Lua scripting you can do is actually insanely powerful.
09:02
You can do any amount of Lua processing inside every single request. It works extremely fast. Lua I think is a nice language. It's not as nice as Python. But some of the design decisions they made that make it a less pleasant language to look at and work with are actually very good for speed and efficiency. So that's why I think they chose it
09:21
for the majority of Nginx scripting. What we do inside our load balancer code is actually really very simple. What I've got here is the Nginx configuration file. Hopefully that's reasonably readable. At the top here, we're saying init by Lua file. When Nginx starts up, it's going to load, it's gonna run that script init backends.
09:41
Init backends basically just specifies some global context which is available to any Lua script inside Nginx saying here is a list of all of the backend servers. That's all it does. As we go down here, we come into our server block. So we're listing on ports 80 and 443. And this location slash block is basically something that's gonna be executed.
10:01
Think of it as code that's executed for every request. So what we do is we extract the host that this request is asking for, www.abaythetestinggo.com. We extract it from the HTTP host, the header from the HTTP request and stash in a variable called root host.
10:20
We then set a backend IP variable to empty string. And then this is basically a function call here. We're calling the Lua function that's contained in get backend IP. Now you can guess what get backend IP does. It returns the IP in this backend IP variable and then we go into this little bit of Nginx magic which is proxy pass. So that says just hand off the processing of this request
10:42
to that server over there identified by this IP and Nginx does the rest for us. Let's take a look at that Lua file. This is an interesting bit of code because it was something we put in for the first cut of our load balance so which we thought we were gonna get rid of in a week or so. It seemed too simple.
11:00
It seemed too, what's the word? It didn't seem complicated enough to work. All it does is hash the hostname that comes in. So that's literally the code that Python uses when you hash a string converted into Lua. It hashes that so you've got a number from the hostname. We then take that modulo the number of backends and use that to index into the list of backends.
11:22
So that means that every single web server we're running, every single website we're running is assigned essentially randomly to one of the different backends. It's stably assigned to the same backend. If we add new backend to the cluster, the modulus number we're using increases and so everything automatically spreads itself out over the cluster again.
11:40
That's the load balancer. I said that the backend server also needs to identify which process is running a particular web application. So this is some really basic Nginx configuration that any of you who've done uWSGI stuff on Nginx will recognize. All we're saying here is, again, extract the domain name from the request
12:01
that's being made, from the request for processing. Different way of doing it but the same effect. And what we tell Nginx to do is delegate all requests for www.abaythetestinggo.com to a particular socket. This is all dynamic stuff. This is what the config actually looks like. It's not a sample there.
12:22
So any request that comes into this Nginx, it will immediately look for that socket in that particular location and expect there to be a uWSGI process sitting on the other end of it running the website that should be on that particular domain. How does uWSGI know that it needs to have a web application running on that socket?
12:42
Well, uWSGI has a directory called, it contains what they call vassal files. A vassal is uWSGI's terminology for a running Python process or set of processes that's responsible for a particular web application. It's configured by a vassal file. And a vassal file basically has various things
13:00
saying where the code is, what kind of sandbox you want to apply, how many worker processes you want. But importantly, it also has this line at the top here, excuse me, which is the socket that it needs to listen on. uWSGI's very clever. If a vassal file is created configuring a uWSGI vassal like this, it will immediately detect the creation of that file
13:20
if it's in the right directory, and it will fire up all the processes immediately. And that means that obviously the web application has started. So what we need to do is start a web application when requests come in. This is where things get a little bit more complicated. What happens if a request comes into one of our backends and there is no process running
13:40
for that particular web application? Well, I told you the NGINX I showed you earlier was simplified. Here's something a bit closer to the truth. When NGINX tries to connect to a uWSGI backend that's not there, maybe there's no socket, maybe uWSGI itself hasn't started the processes, maybe it's killed them because they timed out after a certain amount of inactivity,
14:02
NGINX will internally generate a 502 error. Normally that just goes back to the browser and obviously things look bad. What we have here is a error page handler. If there is a 502 error, we essentially do a go to to this other block here at fallback. Error page 502, if there's an error,
14:21
we wind up inside this code here. And all we do here is we check whether there is a vassal file for that particular domain. So let's say we're looking for www.abaythetestinggo.com. The process isn't running. The first thing we do is jump to this fallback. We see whether there's a vassal file for that domain. If there's a vassal file for that domain, we can safely assume that there are processes running.
14:41
So actually, this was a real 502. Maybe something went wrong inside the web application. So we generate a real 502 error. But if there isn't a vassal file for that particular web app, we know we need to start it. Now you remember that proxy pass from the load balancer where essentially we're saying delegate all work for this request to this IP over there. This is another proxy pass here,
15:01
which is delegating to a little microservice running locally. The microservice running locally is actually a very small Django application. It has access to the database that configures all of the websites we run. When it receives a call on its initialized web app view, then it says, okay, I need to start up that particular web application.
15:20
It goes to the database. It gathers all the information about the user. It works out whether we have a virtual, a container for this particular user running on this particular machine. It starts that up if necessary. It then creates the whiskey configuration. It generates a whiskey.ne file, the vassal file, passes that off to uWhiskey. UWhiskey starts the process up and running,
15:43
and suddenly we can start delegating all the work to that. So why is this interesting? Well, what it means is that we can actually scale pretty much transparently. Let's say we've had a busy day. Let's imagine we've only had, until now, say three web servers in our cluster, and then suddenly things hot up.
16:01
Maybe the web applications we've got are getting more busy or a bunch of new people have signed up and we've got to serve more websites. All we do is we create a new backend server, which is very easy with Amazon. We just fire up a new instance, and then we tell the load balancer about it. Immediately, on telling Nginx to reload its configuration,
16:21
it will start distributing requests differently across the load balancer, across the backends, and any backends that need to start web apps will automatically start them. The ones that are running web applications that they no longer need to run will all start timing out and killing themselves. So dynamically reconfigure the cluster very, very simply.
16:42
Now let's say that something goes wrong. Last night, one of our web servers started showing problems, and so we got pinged by Pingdom saying LiveWebOne was going down. LiveWebOne is a particular server that we have on Amazon and every year about this time,
17:00
hardware starts failing on AWS. I think what happens is that all the regular engineers go on holiday, and the interns haven't been given enough information on how to manage their systems. LiveWebOne started failing, and so all we did was log into the load balancer, remove it from our list of backends, everything then immediately reconfigured itself automatically
17:22
just through the use of this hashing function to run on the remaining servers. That meant that, of course, all of the web apps were running a little bit more slowly because our machines were closer to their load limit, but that's fine. We have a fair amount of capacity for that. We could bounce the web server, sorry, could bounce, fix the broken server, bring it back into rotation with the new IP,
17:41
and suddenly everything worked again. That was a very, very rapid tour through how the whole system works, and now I'd really like to hand over to you guys for any questions that we can drill down on, anything that was interesting. Thank you.
18:06
So we got a lot of time for questions. There's a gentleman here at the front. This is about the initializer of our script.
18:24
Could you just use your WSGI to automatically start the final process instead? Why would you not use that option? Um, the real problem is actually in the amount of work that needs to be done to start the process, because all of our users run inside sandbox environments,
18:44
which we have to have control over the code to actually start them up, and you get the sign when we started using it, and I think still doesn't have the capability to do all of the setup work required to do it. It can start processes quite happily. It can run certain kinds of pre-init scripts. Um, I don't recall precisely what, but there was something that it could not do
19:01
to do with the virtualization, essentially. Well, first question, how do you do the sandboxing? The sandboxing? Okay, it's kind of a roll-your-own thing. These days, if we're starting Python Anywhere today,
19:21
we might think of, we'd probably use Linux containers, or we'd potentially use Docker. We're using Docker for some stuff we're working on now, and think of rolling some changes back in, but the good thing about Linux containers is that it was built out of reusable components. It uses true roots. It uses process namespaces, network namespaces, and that meant that all of these
19:41
have been becoming available for a number of years, and we've essentially rolled our own kind of Linux containers light by plugging those things together. Okay, and what happens if a worker is down, and let's say free requests come in at the same time?
20:00
Is there, there's some kind of a race condition here, right? Sorry, I don't think I understood that. So if you need to start a process, a whiskey process, for a website that was down, but free requests come in at the same time, is there some kind of locking to make sure you only start it once, and you don't lose any requests while it's starting up?
20:21
Oh, I see, yes, yes. Yes, there's locking inside our initialized web apps, which handles that, and uWhiskey does queue things to a certain degree as well. So we've kind of got a belt and braces there. We use the code that starts up our sandboxes in various places, including these, we do in-browser consoles and things like that. So we've kind of got locking on our side
20:41
to protect us from that, and uWhiskey does a certain amount of queuing. Okay, thank you. No, we don't. And that's something we really do want to support. All of our infrastructure does support it to one extent or another.
21:01
I think, I'm not actually, I'm not sure what uWhiskey's support for WebSockets is like at the moment. But the problem is that the whiskey protocol doesn't really support WebSockets. If uWhiskey does support it, it'll be in some kind of extension on top of that. I think that's, if and when we do so, well, we definitely will support it.
21:20
Will we support it? We'll either wind up rolling something of our own to be able to manage long-running, say, tornado processes and use the same NGINX infrastructure to route through to appropriate places. Or maybe uWhiskey will, by that time, do something that we can use.
21:41
You never said how you deal with persistent states or with databases. Sorry, could you say that again? Well, what happens to the databases of the web apps? Oh, I see. Okay, that's managed separately. We have MySQL instances, and we're working on supporting Postgres instances.
22:01
Just separate, they're kind of behind those backend servers. Manages that, right. Do we, sorry, do we have code that's? Could we repeat the question, please? Do you have some services that manage databases for users? On the, yeah, on the MySQL side,
22:23
it's all a little bit messy and was built ad hoc for, we're adding Postgres support right now, and we're basically building that as a, we have a Flask microservice, which runs on one of a set of Postgres servers, which fires up Docker containers, each of which runs one Postgres instance.
22:41
So the Flask microservice does the provisioning. We're hard at work on that at the moment, it's working well enough to pass a functional test, which probably means it's a month or so away from deployment. Last question, please. There's one. Hi, you said that when one of your instances,
23:02
you got an alert on it, then you manually removed them from the load balancer. Is there a reason why you don't set up auto, like automatically removal, and also if you have auto upscale, because you said that you have some sort of limit on your instances?
23:23
Yeah, that's an excellent question. It's really been a matter of development time. One of the features we do need to add is automatic instance killing. When we first created it, it made sense to do it manually, because each instant failure was a rare enough occurrence,
23:41
and was kind of unique enough in the way that it failed, that it was better to have a human in the loop. Whereas now I think we've managed to get a list of the different ways in which instances can fail, and we can probably start building in more automated responses. But yeah, that's just a case of, we haven't had time to do it yet. So, thank you very much, Giles, for speaking here.