How to Write Deployment-friendly Applications
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44945 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
WritingSoftwareVideo gameLoginWeb applicationCartesian coordinate systemUniverse (mathematics)Complex (psychology)Task (computing)Scaling (geometry)Interface (computing)Computer-assisted translationTerm (mathematics)Web 2.0Block (periodic table)Configuration spaceBuildingSound effectWordComputer animation
01:26
Web applicationBuildingOperator (mathematics)Endliche ModelltheorieGoodness of fitBusiness modelMultiplication signDifferent (Kate Ryan album)View (database)Server (computing)BitSurfaceComputing platformLink (knot theory)Mobile appType theoryWeb 2.0Software testingWeb pageSoftware developerCartesian coordinate systemMaterialization (paranormal)Slide ruleCoefficient of determinationBlock (periodic table)Computer animation
03:13
Dependent and independent variablesConfiguration spacePort scannerInformationSynchronizationCartesian coordinate systemConfiguration spaceFunction (mathematics)Service (economics)Software frameworkComputer fileAxiomIntegrated development environmentSoftwareSoftware testingLoginFunctional (mathematics)Different (Kate Ryan album)BitStreaming mediaCodeMobile appSlide ruleGastropod shellWeb 2.0Process (computing)View (database)Goodness of fitIdeal (ethics)CASE <Informatik>Default (computer science)Block (periodic table)Scripting languagePoint (geometry)Data typeSocial classNamespaceInformationBuildingSoftware developerBranch (computer science)MereologyData managementMathematicsWeightFactory (trading post)Sampling (statistics)Module (mathematics)Radical (chemistry)Web applicationStandard errorBlack boxAxiom of choicePhysical systemSpeech synthesisObject (grammar)Interface (computing)Product (business)Hidden Markov modelServer (computing)Stress (mechanics)Execution unitOperating systemCoprocessorWordNormal (geometry)Computer animation
10:35
Key (cryptography)Level (video gaming)Graph coloringFile formatSoftware repositoryMoment (mathematics)Product (business)Configuration spaceRadical (chemistry)Cartesian coordinate systemSoftware developerWeb 2.0BitLoginMiddlewareDatabaseRoutingComputer configurationGroup actionComputer animation
12:12
Server (computing)Variable (mathematics)Integrated development environmentLogarithmTemplate (C++)InformationConfiguration spaceLocal GroupVarianceFile formatComputer fileScripting languageFraction (mathematics)MetreProcess (computing)Network topologyVariable (mathematics)Template (C++)Physical lawSoftware testingConfiguration spaceCartesian coordinate systemIntegrated development environmentState of matterDefault (computer science)FreewarePhysical systemSinc functionDot productSoftwareService (economics)Module (mathematics)Group actionInterface (computing)Point (geometry)Social classParameter (computer programming)Mobile appInjektivität1 (number)Projective planeMultiplication signComputer configurationAttribute grammarGame controllerStructural loadRevision controlLevel (video gaming)Functional (mathematics)Video game consoleProfil (magazine)Directory serviceSystem callInstance (computer science)Key (cryptography)Similarity (geometry)Scheduling (computing)Validity (statistics)Data managementDeclarative programmingSet (mathematics)Special unitary groupServer (computing)Enumerated typeMetropolitan area networkMultiplicationError messageStapeldateiData miningData conversionProof theoryOnline helpMereologyPlug-in (computing)Electronic mailing listComputer animation
19:18
Key (cryptography)Network topologyInternetworkingCartesian coordinate systemIntegrated development environmentVariable (mathematics)Process (computing)LeakPlastikkarteComputer animationMeeting/Interview
19:52
Variable (mathematics)Key (cryptography)Point cloudGoogolStructural loadComputing platformExtension (kinesiology)Moment (mathematics)Goodness of fitLastteilungComplete metric spaceCartesian coordinate systemCASE <Informatik>Multiplication signConfiguration spaceData managementComputer filePoint (geometry)Block (periodic table)Instance (computer science)Integrated development environmentSoftware frameworkMereologyMobile appBitBuildingCoordinate systemComputer programmingImplementationMetadataCross-platformVariable (mathematics)System callControl flowHoaxProxy serverSoftware developerString (computer science)Computer clusterCodierung <Programmierung>Client (computing)Social classEmailFile systemInformationDivisorDatabase transactionCommunications protocolStandard deviationUniform resource locatorFront and back endsReplication (computing)Default (computer science)Right angleComputer animation
26:10
Server (computing)Process (computing)Multiplication signData managementConfiguration spaceDefault (computer science)TheoremBlock (periodic table)Software frameworkTerm (mathematics)Physical systemMobile appCartesian coordinate systemCASE <Informatik>Disk read-and-write headException handlingKeyboard shortcutRight angleMoment (mathematics)Interrupt <Informatik>Computer animation
27:25
Structural loadDependent and independent variablesConfiguration spaceRevision controlIP addressError messageDifferent (Kate Ryan album)MultiplicationProcess (computing)DatabaseLevel (video gaming)Server (computing)Library (computing)Metric systemPoint (geometry)Instance (computer science)Mobile appConfiguration spaceInformationFile systemStandard deviationView (database)RoutingCASE <Informatik>Rule of inferenceMechanism designBitDynamical systemResultantNamespaceGodFlow separationInformation securityGateway (telecommunications)Client (computing)Data managementLastteilungProxy serverMultiplication signChemical equationLine (geometry)Product (business)DeadlockTask (computing)Cartesian coordinate systemRollback (data management)Web 2.0Serial portRevision controlScheduling (computing)Interface (computing)Inheritance (object-oriented programming)Web applicationVideo game consoleDefault (computer science)Type theoryWordStructural loadMereologyRotationGradientService (economics)Doubling the cubeComputer animation
35:24
Integrated development environmentMobile appInstance (computer science)SequelComputer fileService (economics)State of matterFile systemClassical physicsFront and back endsException handlingComputer animation
36:08
Structural loadService (economics)LogicState of matterOvalBoundary value problemMobile appDrum memoryInstallation artPrice indexSimulationSocial classVariable (mathematics)Interactive televisionComputer fileCartesian coordinate systemLaptopMobile appMultiplication signView (database)Configuration spaceState of matterKey (cryptography)Interface (computing)Connectivity (graph theory)Service (economics)MereologyComputing platformIntegrated development environmentGame theoryRight angleOnline chatDatabaseOperator (mathematics)Radical (chemistry)Data managementBoundary value problemComputer configurationSoftwareAnalytic continuationWeb 2.0Perfect groupNumberLogicScripting languageGastropod shellMoment (mathematics)Process (computing)Thresholding (image processing)MiniDiscPoint (geometry)Adaptive behaviorFile systemSemiconductor memoryBlack boxBefehlsprozessorData centerMultiplicationGroup actionPolygon meshAxiom of choiceLinker (computing)WordDecision theoryOnline helpRow (database)Scaling (geometry)Revision controlServer (computing)Video game consoleStandard deviationInstance (computer science)Flow separationConstraint (mathematics)HexagonComputer architectureLastteilungGreatest elementSoftware engineeringWeb applicationOpen sourceoutputSlide ruleStaff (military)Link (knot theory)Arrow of timeType theoryComputer animation
44:41
Web pageDomain nameQR codeTwitterHypermedia
Transcript: English(auto-generated)
00:04
All right, thanks for coming everyone. I'm Hinek and my goal for today is to make your life easier Easier by doing less. I want you to stop worrying about a lot of things in your applications I want you to stop worrying where your logs go and how they are processed
00:21
I want you to stop worrying about where your configuration is coming from and how it's structured and I want you to stop worrying where your application is running and how it got there in the first place and instead of that I want you to make Just think about in a different terms about how to make your application play nicely with others
00:42
Because the task I just enumerated they do not go away. Someone still has to do them but it doesn't have to be you because if you stop thinking about the application as a Django web app that serves cat pictures and Start thinking about as a universal building block with clear interfaces that others can rely on
01:02
Those others can start doing the work that you were just dropped So we are gonna outsource complexity that is not inherent to the value of your application So we have in other words. We are creating a lot of SEPs tonight and as a side effect By the end of this talk your application will also be web scale by accident, which is also not not too bad
01:24
now As with actual building blocks, it shouldn't matter whether you're building a doghouse or an airport And I'm from Berlin. I know everything about airports. Believe me. It's fine Your app should look the same on a laptop or on a cluster
01:41
It shouldn't know the difference and this dramatically simplifies of course development testing operations Scaling and also moving to new platforms and while it may sound a bit academic at first Platform agility may be a bigger issue than you may think because every infrastructure evolves over time Until it goes extinct
02:01
so you will have to touch that CentOS 5 server eventually and Even if you are happy with your state-of-the-art Kubernetes cluster right now I'm gonna bet you money that in five years, it's gonna be a good business model model around moving people off this legacy platform So, how do we get there and what exactly do we gain?
02:22
I'm gonna show you what a very simple web application but before you start leaving the room again This talk is not specific to web or something like that But it's easiest to talk about so I'm gonna use it in this example and most of the things I'm gonna say Apply just as well to other types of applications that run on
02:41
servers And why I'm talking meta as with all of my talks I will be using all of my time because I have a lot of materials So there will be no Q&A, but I will be available throughout the conference. Feel free to come up and talk to me I'm here to talk not to sit around and play with my phone I've also compiled a page with everything
03:02
I'm gonna talk about with all the links all the concepts so you can dive into the topics you find interesting because as always I can only scrape on us on the surface now Behold our beautiful app. This is a very exciting pyramid view and the only reason this slide exists is so I do not have to tell you to
03:23
Imagine a simple view. This is it. We are gonna play with this one. We are gonna make this great So to run this in a Python web ecosystems, you need to create a whiskey Application what it is exactly is not that important, but it's something you have to build and how you do that Depends on the framework
03:42
So this is how it looks in pyramid and it's usually the same for all frameworks So there's a function that does something and then in the end it returns something a whiskey container can work with Or a test framework. That's the nice thing about this I like to put it in a file called app maker because it's a function called make app
04:02
so why not and this initially initialization code once it gets a bit more complicated is notoriously hard to test Because you have to simulate an environment It's running in so the goal is to keep this code as simple as possible with as few if branches as possible so if branches between
04:21
Testing development and production in this case And of course this principle is not specific to web or risky at all Like you always want to isolate the creation of your app where you get information from the outside Instantiate your classes and so on and so on and the rest of your application are just classes and functions that take normal data types and
04:41
Which then means that it's much easier to test So in pyramid you basically Create a configuration object you run a bunch of methods on it and by the end you call Make whiskey app and my clicker is not working. Um This whiskey app is actually just a function
05:01
So if you ever seen like a tutorial from whiskey container like G unicorn, it's it's actually really simple words yeah, someone has to build it and Speaking of G unicorn. This is how you could use it In this case, I'm going to assume our application has the namespace sample There's the app maker is the module and make app is the factory that we are calling and it returns something that unicorn can work with
05:27
it comes up and We can curl it and we even get Apache style Log output which is cool Now if we wanted to deploy this application as it is you could of course take this command line and put it into
05:47
Your system D unit file, for example, but that would bleed the choice of your whiskey container Into the configuration which probably if you are following best practices is not part of the application repo, so it might be something in your ansible or
06:05
Or salt or puppet or whatever repo so maybe you want to Switch out your whiskey container at some point now We have to coordinate your changes between having the new container in your application and changing the configuration
06:21
This is not great. This is error prone. So it's better to avoid. Yeah, my clicker works. So What are we gonna do? So remember we want a building block. We want a standardized way to start applications And it shouldn't be shouldn't matter at all. That is a Python application or is a web application It could be C++. It could be assembly. It could be whatever
06:42
So, how do we do that? Well, we do it the same like our ancestors in the 1970s we write a shell script and This shell script we just check in along with our application You can call it however you want in Docker. It's common to call it Docker dash entry point dot SH for this talk
07:02
I'm just gonna call it run app SH because that's what it does and it's shorter. It fits my slides better. So One thing I would like to point out is this exec thingy here This means that your shell processor that is started here is replaced by your application Which is very important if you want to receive signals and you do want to receive signals
07:23
Otherwise you run into all kind of trouble This little tidbit Also straight from the 70s means that the standard error output goes to start out which means that you have one stream of output one stream of logs Which is much nicer to handle and this way a shell script becomes the adapter between your application and its environment
07:44
It's a very simple adapter, but still someone who wants to run your application at this point just has to run a shell script and this works just fine in system D and Docker or In proc files, which you may know from four man or forgo or honcho does a bunch of those
08:03
It's quite popular through because Heroku uses them internally or for deployments, but it's also very useful in local development So what do we have we have our black box, which is our application It's an it's a building block because it's very easy to run
08:22
It Exposes its service on localhost, which is the default for unicorn in this case And it locks to start out which is great. We are done at this point because in development This is your terminal which is exactly what you want If you run it in system D it gets forwarded to syslog which means that you get 40 years worth of Unix experience for handling locks and
08:46
Cluster managers like Kubernetes or nomad have of course first-class support for this kind of locks You can have streaming over the network. So you can just live watch what's happening on your servers and More importantly they will help you to aggregate your locks in something like lock stash or gray lock
09:06
So I would like to stress here that you should not try to walk through files and more importantly Do not try to rotate them. This is something that will eventually make someone very mad Anyhow You have clear interfaces, I think now in the next step
09:24
My goal is to make this example more realistic by adding more features, but to stay as close as possible to this ideal and Let's tackle the most glaring problem here first, which is exposition This this kind of exposition is useful like in two scenarios
09:43
the first one is you really just want to access the service on localhost or You have a local nginx that will expose it to the network or wherever you want That's good, but we need to do better so let's shed those shackles and talk about configuration and
10:02
Here I find it important to stress that There's a big difference between application configuration like your application and Configuration of general-purpose software because general-purpose software like XM Apache nginx They need to accommodate everyone they make you have to make everyone happy your application only has to make you happy
10:22
and your co-workers So what you have to ask you you have to change the question from what can be configurable to? What varies what varies between deployments between environments and it turns out it's very very little So there are things that some people put into the configuration don't really belong there like the routes configuration or middleware configuration
10:44
It's still quite common in a pyramidal world or logging. Let's talk about logging for a moment When you configure logging, there's basically two things that matter first is the log level So in development, you will probably want to have more logs and in
11:01
Production you want less logs. Okay, that's simple enough And that's the log format which is a bit more complicated But again, you just want two options Really? You want one human readable format with colors for your terminal and you want one easily parsable for production Which can be some key value pair thing or JSON? so what you do here is just you define those two configurations in your application and
11:26
Then switch between them using an option that you pass into your into your application This way you can test your logging configuration very easily because it's there. It's not in a different repo. It's not living along the ansible
11:42
Yeah, so What you need to make on the group though is of course exposition you want to tell be able to tell your application where to listen on and Also external resources like web api's or databases. These are the things that may change or they need credentials So this are genuine configuration
12:03
Things that should be configured now, once you've identified those few options. You need to pass them into your application So how do you do that? So you could put them into an any file which is simple enough because Python had support for any files since the 90s, I think so. There's one
12:22
Not one it has multiple downsides So first of all in some environments maybe not in your current one But maybe in one of your future ones it's hard to impossible to inject files It's just a matter of fact and some of those options Belong to the whiskey container some of those option belong to your app
12:43
So, how do you separate them? You could of course parse the any file in your bash script and pass them along But I can I think we can agree on that. This is not great So what do you really want is just passing key value pairs between processes? So if only there were a simple reliable and portable way to do that
13:02
There is it's called environment variables and They are not relevant just for our simplistic problem here. They are universally supported So again system D and any other process manager supports them worker files have first class support on every cluster scheduler under the Sun do that too and
13:20
Just so many tools by now that will help you with that So for example their end will set variables variables and you enter directory and just like 5,000 others to do the same thing service discovery tools like console or etcd They do have tools that will fetch the data you want set the environment variables and run your application
13:44
of course Python does have access to the environment as OS don't environment and If you really really need a file for whatever reason that can be alleged reasons to use files There's also solutions to that. So for example get text which is
14:00
Actually for translating software has a tool that it's called ENP zapst and it does what it kind of sounds like It allows you to do very simple templating using files and environment variables So you can get there just more complicated or more powerful tools for that too Like conf D which supports back ends even redis so you can pull values out of redis and put them into environment variables
14:26
Which is kind of cool console template. It's the official one from the from HashiCorp So there is a very Broad support for this kind of things now back to our concrete problem Um, it turns out that the host port problem is kind of common. So
14:44
It happened to people before they had to that they wanted to configure the host and the port so a standard well a best practice emerged so There are two variable names that are called very appropriately host and port and are supported by most
15:04
Most servers nowadays and this is great. These are conventions conventions are amazing So for example aforementioned for man If you define multiple applications in one profile will enumerate ports for you So you don't have to do it by hand so you can start multiple Applications at once and you do not have to fill around with the port so you don't get conflicts
15:27
The lock level is easy enough to you just grab into the environment do a little bit of get a magic But this is a little bit tedious and ugly and like it doesn't get more global state environment, so
15:40
It wouldn't be me if I give a talk with our plug-in. What another project of mine So let me introduce you to environment conflict So I do realize it does similar projects on pi pi But at least when I started the project there was none that did the same thing So what it allows you to do is to declaratively define your configuration
16:02
including nested groups as you can see there's like a subclass in a class and When loading those names, I just get concatenated along with the optional prefix and Load it from the environment and once it's there You can just access it like normal nested classes the law of the meter be damned. You just use a lot of dots
16:25
since Environment config is based on others and I have new stickers by the way, so if you want talk to me You get a lot of stuff for free like default values validators or converters, for example I love using enums to make sure that I get valid values into my applications and will just explode
16:43
Not start up if you pass something illegal Now I like to put this thing into a file called conflict of pi But the declaration itself does not load it automatically. So where are we gonna load it? We could load it in the make app function I showed you before that creates the application
17:01
but it is not great because make app is sometimes used by tests and You don't really want to mock out an environment like a genuine environment just to create an application. That's that's not great So instead let's create a new file a file that will do the dirty work of Grabbing the environment and then just pass an app config
17:25
Instance into your make app And this one I like to call whiskey pie, which I've seen other people's do too. So just I guess a best practice now and This is how you load it it's it's really simple just a one-liner and now this module is the ultimate interface between your application and your environment and
17:45
Make app at this point only deals with a well-known class with a Python class So if you just grab something that doesn't exist you get an attribute error. This is so great so and it also gives you a full control over your app instantiation and Yeah, you can create whiskey apps for your tests without a lot too much pain
18:04
And what's important here to note is that this allows you to use the lowest common denominator on the outside key value pairs environment variables and Use structured data well-known data validated data in an insight once it's passed this file
18:22
So You may have noticed that now I put the whiskey app into a global variable This has multiple reasons. So it's more flexible. Not every whiskey container does come support this function call thingy it's Also, we need to pass an argument to make up now. So that makes things even more complicated and
18:45
I call it application because there's another convention And because it's assumed that it's called application. It allows us for G unicorn to only pass the module name So our run script got even simpler We just pass the name of the module and it had grabs into it and pulls out application thing
19:04
Now in all the talk about environment variables There's one thing I have left Conspicuously out and it's that time has shown again and again. There's certain things just do not belong into environment variables Because some things you just want to whisper gently into the ear of your application and not make it global to your whole process tree
19:27
Because environment variables can leak There's many ways they can leak they have leaked before they leaked to very smart people in Very sophisticated environments so it can happen to you. It may not be your fault it may be some weird package that you didn't even install yourself because it was a
19:44
Dependency of something else you used and it just dumps environment and now your AWS keys are on the internet So let me be very clear here I want you to I want to ask you to ignore the full factor app manifesto in this point for now Because they got this one wrong and it's not my personal opinion. It's quite quite widely now, so
20:06
Thank you Christian But now things get a little bit hairy because Solutions to this problem are platform specific every platform has its best way It's kind of part of the lock-in to and everybody wants to give you the best possible way
20:22
So what I can really just tell you is to use your platform's best thing And leave it at that because giving you an intro alone to all these things to be a rough overview is a talk by itself But luckily for you this intro exists a talk by my friend Noah country with which he gave two years ago
20:41
Actually also at euro Python and I will link it in my talk. So it's interesting to get an overview For the sake of completeness we do run vault It's vendor independent, it's quite nice. So in any case whatever you do you can use that too It can be a lot cheaper than things like AWS secrets manager where you pay for access and stuff like that
21:05
Anyhow, I'm gonna make Christian sad So since we run nomad And and walls which are both by HashiCorp We get built and templating for free and they're just gonna like mount a special special purpose file system called slash secrets into docker containers
21:23
And I can template my secrets into this file Now this is not perfect like There are ways for For files to leak to but those ways are a lot harder than environment variables So I personally considered a decent trade-off
21:41
Okay, Chris is nodding so it's fine. Okay. Thanks But the safest way to do this is always platform dependent because you want to use the best features That are available. And of course if you want to have dynamic secrets, you have to do definitely the native way and
22:02
What do we do in programming if you want to hide away an implementation detail Well, you write a facade, right? And I click too much write a facade so What I'm asking you to do is just to wrap your secrets client. So for vault it would be HVAC and
22:22
Build on ice API around of it and now you can decide when you load your secrets Do you do it on instantiation or do you do it on access? It's your call your application will not know and if you switch out your secrets back and because it can can happen um Then you just rewrite this one class the rest of the application will not know
22:43
And of course, this is also very easy to replace with a fake in your local development You just return static strings and you are done So yeah, your application should not care about the secrets back end So you have you will have to write more code and with the other things I've said But it's still attainable and you should really do it. Now. I'd like to point out really quickly that I like to encode credentials as URLs
23:06
Which is widely supported and again you prevent a problem Of having to coordinate changes between configuration and secrets if you have everything in just one place it's one transaction once change and You don't run into problems that you change the host in configuration, but the credentials are still the old or something like that
23:26
All right We still have a nice building block Is easy to run we inject essential info that varies across deployment and environments using Using environment variables secrets aren't as elegant but with some effort at some magic. It's good enough
23:44
We exposed based on the configuration is coming from the variables. We'll look lock based on this configuration This is still pretty good Now keen listeners may have noticed that it's impossible To reload this kind of configuration if you go down the road of environment variables, this is right out. I
24:04
Would like to reframe this extensive downside as something good because it forces you to rethink and reconsider So for example for things that change Not too frequently. So let's say once a day. You can just redeploy your application and now this makes you think about zero downtime deployments very early on and
24:25
as someone who Went through this a few times. I can tell you that Caring about zero downtime deployments early on will pay off really really big time by the end And also the longer you wait to think about these kind of things the harder it gets
24:41
so it's kind of nice to have it from the first moment and that you can deploy any time and nothing breaks and the good thing is Thanks to the fact that we now have a building block. It's actually very easy to attain So instead of one instance of your application you have two and
25:02
You can run them on the same host and you put in load balancer in front of it for example nginx Now your only applications duty that the only thing that changes for it is that it has to find know the way how to extract metadata that is Communicated by the load balancer about the client because the app is not talking to the client directly anymore
25:22
So it has to know about things like the x-forwarded for header or about Proxy protocol by HA proxy if you are writing a TCP based application, but this is it That's all your app has to care about and if you use standard headers chances are that your framework or your
25:42
Your framework and your whiskey container will take care of this vitamin themselves and at this point you are ready for rolling updates So what you do now is basically just tell your load balancer to ignore one instance and now do whatever you want and Take how much time you just want to take it doesn't matter. Nobody will know
26:03
So first you will probably want to stop your app without disrupting anyone and to do that smoothly You need clean shutdown and I would like to use this moment to reinforce that you should make sure to handle sick term because Python is not great about this a lot of Applications a lot of frameworks just wrap themselves in a try except keyboard interrupt
26:23
Which is ctrl C and do not care about sick term, but sick term is the standard signal for terminating Terminating processes. It's used by default by system D. It's used by default by all cluster managers by Docker Make sure you handle it or make sure that you
26:40
configure your process manager or your Or Docker to use the correct sing signals, which is possible in the worst case if you do this wrong Your app receives a signal ignores it Your process manager waits for a timeout and then kills it using a sick kill which you cannot handle You cannot block it. You just get shot in your head
27:01
this means That you get a very slow slow shut down because there's this timeout and you get no clean up. That's bad Anyhow, let's assume you did everything right and Your application is down. So now you can just deploy a code and take your time. You can change configuration
27:21
Also take your time. You can edit it on a server if you want. Please don't edit it on a server Now once your app is ready, you can put it back into rotation and You're done You've deployed something and nobody knew and this way you also leave the dynamic reloading of configuration
27:43
to load balancer load balancers are really good at this of Being reconfigured while running and this is kind of complex to do yourself in your app This is great to have moving this move to happen to move this task from the app to someone can do it better and
28:01
Another upside is that if the deploy does not quite go as planned So service on fire. What do you do? Well first you breathe and then you don't do anything because that's the beauty of it as long as you don't return This this instance back to the load balancer. Nobody will know that you screwed up or that you're that something is broken
28:27
So you can take your sweet time to do a rollback or reconfiguration or whatever So even if you deploy using git pull on production service like an animal you will benefit from that
28:41
Yeah, you laugh, but I know some of you do it So So everyone benefits from this approach It doesn't matter whether you have a super sophisticated cluster scheduler from the future or whether you have an intern with an SSH client No one will know you screw up Now there's one thing I hand waved over and I need to talk about and when I said that
29:04
It's added back to load balancing once ready. So how does the load balancer know that your app is ready? For that you have to add another interface We need introspection and introspection is some incredible amazing and powerful concept And the default way that's done nowadays that just expose a web endpoint that reaches into application
29:25
And again, it doesn't matter where your app is the web app in the first place. You can always expose a web endpoint There's it's even in the standard library server HTTP server. So you don't even need an external dependency for that So what the load balancer cares about is called readiness
29:43
So what is readiness readiness means that your application is ready to serve ready to be added back to the load balancer So you expose an endpoint that Checks all resources it needs to do its job And if it's fine, you return a 200 if it's not fine, you reach a 500 and the load balancer will not add you back
30:04
Unfortunately, there is no clear standard how to structure or name this kind of endpoint So most of you or some of you may have heard about health Z Which comes from Google and I've heard multiple legends about the Z So my favorite one is that it's a Google grade
30:23
security by obscurity But it's probably just some clash avoidance. I heard they have this whole namespace of disease at the end Mozilla that goes a bit pythonic they use Dunders so down there are heartbeat Other common things is just to take literally the word and put it into a dash namespace
30:44
So dash ready is from promethos dash readiness is from get gitlab And I personally really really like the dash namespace because it makes it super easy to block it in an edge load balancer So at HAProxy, it's one line one rule and
31:01
You can just have it as part as you happen. Nobody will ever see it Now the downside of this readiness check is that is expensive Because if you ping your external dependencies, you have to ping your database with a select one So that's not something I want to do like every second or something So you shouldn't do it too often and sometimes you just want to know is this application healthy?
31:21
Is it still alive and that's where liveness comes into play liveness is very cheap Liveness is just to prove that your app is not in a deadlock or is running at all that it reacts to Requests and this one should be relevant to process managers and cluster managers So they can detect that your application is deadlocked and needs to be restarted or that the startup failed and it needs to be rolled back
31:43
Now again, there are multiple common names one is dash healthy from promethos Which I find very unfortunate because it's as close to health Z as it can be and means the opposite Git lab uses dash liveness and
32:00
Mozilla uses lb heartbeat dundra lb heartbeat. So this is kind of interesting Because it seems like I didn't double check with Mozilla people, but at least according to the to the documentation they are using liveness for the load balancer and I personally do that too because it's cheaper and all my resources I'm using in my
32:23
applications are lazy like database database pools, so What I use for Readiness, actually, it's this this is like straight from production like no serialization. Just plain text No permissions. We just block out it in any way
32:40
So nobody will ever see it and you return a status 200 and your load balancer will know that your app is alive Then you add it to a route and once your application is capable to route this this view, you know, it is alive and So at this point my pool is initialized so it knows how to connect to the database and
33:03
It can either start serving or serving errors But I feel like my errors are better than that like a 502 gateway not found by HAProxy But you may like may vary so I do have the traditional expensive one, too
33:21
but I personally use it for monitoring to just check that the app is healthy and Alert when it's not anymore It's a trade-off so for me one of the reasons also does I use nomad with console console has only one type of check and I have to choose one of the two and I prefer the liveness one. Also. I don't want nomad to roll back my
33:41
Deployments just because the database is down or something Okay, but why should we stop here? We have an interface into our app. This is a powerful thing So what else is possible? Okay, Mozilla has dunder version which contains version information deployment information If you use pool based metrics like I do and love like promethos dash metrics
34:04
Logging there's it again. There's always logging Good, you can get your lock level, but sometimes you want more information right now Or what is happening right now? And if you redeploy to change the configuration Maybe the problem you're trying to Investigate goes away. Well, there you go. Just allow to post your lock level again to endpoint and configure logging
34:26
Basically, you can use this endpoint for anything that you used a Unix signals before so it's a powerful concept now We've taught our app to talk to a load balancer and it's incredible how much freedom we've gained by this way
34:40
Because can now scale up and down as needed. It's like magic and you can of course take this further because If you have so many instances, how about we distribute them over service sure why not now? the only difference is that we do not dispatch over ports, but over internal IPs and The load balancer runs on a separate host, but the principle is the very same one
35:05
But once you distribute your app over multiple servers you run into a problem because they all have separate file systems And while you could use something like NFS or god forbid samba plus some weird locking mechanism It's probably better to step back and embrace the rethink and in this case the result of rethinking is the file system is lava
35:27
So the one of the classic problems is logging but we've solved that we are locked to standard out and it's the Environments problem to solve that for itself if you wants to locks it has to catch them and send them somewhere You don't care. It's not your problem
35:41
other classics are states like user sessions and those are shared or and permanent So they need to be accessible by all instances You don't want to have your users to be logged in in only one of your back ends So what you need to do is embrace other services in your app
36:01
So no SQLite for you know no files that are needed by others except for like temporary files And learn to love the elephant postgres is great for data of any kind Redis and memcache are great for sessions and caching Console and etcd are great for service discovery and for key value storing like a dynamic configuration these kind of things
36:28
So Let's have another look there's one more problem with this thing Now we have four instances on four servers and Managing this can become a pain For simple setups you can use a bunch of shell scripts
36:43
You can use something like ansible and people do and it works great, but at some point There's a threshold and gets very complicated. So let's get some help Well, you didn't think that it would take so long for Docker being on a slide, huh? So I think I'm on record of saying that Docker is kind of mad
37:04
And I stand with it. Although now it's more positive vibe for me because it's just boring technology that works But Docker by itself is kind of a low-level concept But there's something great they did. So first of all, they cost an industry packaging standard. Well, not one by now
37:22
It's like five or something That will abstract away even more for your application But more importantly it created an ecosystem and this ecosystem is what I'm gonna talk about now because part of that Of course cluster managers and cluster managers are a game changer So if you don't know them all top left kubernetes top right misses bottom left DC OS bottom, right? Nomad
37:46
Once you get them running which depends depending on which one you choose can take something between one hour and one year You're in a good place If you know how to keep keep them running, which is also another question
38:03
Because now you can say this is my container that I built. It's none of your business. What's inside and These are my hosts run this container one of those hosts it needs so much memory and so much CPU and it just happens So this means that your applications become ephemeral
38:21
They become it's possible that their lifetimes will be very short because maybe they get shuffled around maybe some Cluster nodes go up and down. Maybe there's some rebalancing Um, once you have this kind of freedom you can deploy like every 10 minutes and nobody will know so why not And this is another reason why the file system should be always lava for you from now on
38:44
but Our application is ready for our multi data center cluster Without knowing about it. All it knows is how to start and communicate readiness How to serve and Communicate health and how to stop and clean up behind itself. So basically we are web scale by doing less
39:05
And this is the point where I could and should start talking about things like service discovery or meshes like linker D Or our envoy SEO, but I do not have the time and they also do not change Anything fundamentally about your application. It's just an add-on on something you've already attained
39:24
So let's have a final look at our application So our app is a black box that is easy to start It's self-sufficient It has a very few varying options that are configured using the using environment variables
39:41
It does a clean shutdown When signals are using standard signals it magically retrieves its secrets from wherever It keeps its data that needs to be shared and That needs to be permanent in external services like databases
40:01
It exposes its services as configured Where it is picked up by a load balancer that will expose it to the world or to your company it also exposes its state using a well-known endpoint and It does look to standard out where it's picked up by the environment and all your terminal undone and
40:23
It does whatever is best in that moment. Your app does not know it does not need to know And for all of this our web view hasn't changed at all. Our application creator just takes one or two classes It doesn't know where those are coming from. It just knows that it's configuration and the secrets
40:41
It's no, it's it needs and it's all The interaction with the environment is limited to one file and one file only And the same application works exact the exact same way on your notebook on a platform as a service or in a cluster Just a matter how you start it and the heavy lifting is done by decades old Unix tools or the business container orchestrator
41:07
du jour so success one final thought If you squirm and what you've tried to achieve here what I think we've achieved is that we want to see our application as a black box With clear interfaces that enable loose company loose coupling with our components
41:24
We do separate IO from logic. We lock the standard out We push configuration from the outside and transform it into a class and we do isolate your process Global state to one spot These are all practices from software engineering it especially
41:45
Reminds me of the hexagonal architecture by Alyssa Cockburn who talks about ports and adapters So I guess the lesson here is actually that your applications boundary is just another boundary So you should treat this as such it doesn't matter that there's a process ending
42:03
It's still just a boundary between like any other There are so because your application is or could be or will be part of something much bigger But as we software architectures what I've shown here is an ideal So not every application fits the constraints not every application can run in a cluster
42:23
I have plenty of applications run on a server because they need to do something on that server and It turns out someone has to write to a disk at some point if you wanted to keep that data So and finally some of what I say today directly conflicts with my advice from last year and
42:42
Neither is wrong. It's just two solutions to two very different problems So what we are doing here is engineering we are making trade-offs But to make trade-offs you have to know the consequences of the actions and choices you make So in other words what I'm saying here is that you should come to all of my talks and make informed decisions
43:04
So one more thing Don't we all crave realistic examples? however companies tend to not expose how they like Deploy things concretely. They just give you a soup of buzzwords talking about continuous delivery chat ops
43:21
But in the end, it's just a bunch of shell scripts from 2002 It's written by someone who's not working at a company anymore and everybody's afraid to touch it But it turns out we do have a great open source example of a Python application And it's modern and I come back whenever I want to learn something and of course, it's
43:44
PyPI It's kind of perfect timing after a keynote, huh? It serves and these are old numbers I got from Ernest like when I was doing this talk in March
44:01
So back then it was like six billion requests per month 1.5 petabytes of data and most importantly fast I mean when Ernest asked me to test my path for the first time oh, I thought he's just some kind of joke because just appeared and It turns out this is how a modern Python web app can look like if you do everything
44:24
Right and everything you need to know about this one is on github all the operational stuff the docker stuff It's there And the tooling around our Kubernetes that Ernest wrote is also on github just somewhere else So if you want to learn something have a look
44:42
And that's all I have for you today, I would like to ask you to have some Okay So, please this is the page I talked about the QR code will get you there too
45:01
Follow me on Twitter get your domains from various media if you speak German the reason why I'm here Have an excellent Europe item. I mean it. Thank you very much