Cloud Native Python in Kubernetes
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33711 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017113 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Software developerOpen sourceProjective planeWebsiteProcess (computing)ForestRight angleMedical imagingGroup actionMultiplication signLecture/Conference
00:58
Point cloudServer (computing)Integrated development environmentLoginVideo gameContent (media)MereologyPoint cloudContent (media)NeuroinformatikService (economics)Computer animation
01:39
QuicksortMessage passingSoftwareSlide ruleService (economics)BitLecture/Conference
02:13
Software maintenanceEvent horizonRouter (computing)Server (computing)Network socketPeer-to-peerFile formatThermodynamisches SystemArchitectureBoilerplate (text)Template (C++)MetadataComputer-generated imageryError messageException handlingLastteilungIP addressQuicksortMessage passingBitCodeSoftwareCore dumpFunctional (mathematics)Event horizonMoment (mathematics)Set (mathematics)Instance (computer science)Cartesian coordinate systemNetwork socketProcess (computing)Selectivity (electronic)Complex (psychology)Normal (geometry)Loop (music)System administratorWrapper (data mining)MereologyExecution unitCASE <Informatik>Virtual machineOverhead (computing)Thermodynamisches SystemAuthorizationService (economics)Keyboard shortcutKernel (computing)Buffer solutionComputer architectureQueue (abstract data type)Server (computing)Group actionComplete metric spaceSlide ruleDataflowOperating systemPosition operatorLogicVideoconferencingMathematical analysisFunction (mathematics)Graph (mathematics)MultiplicationMetrePrice indexCycle (graph theory)Connected spaceClient (computing)Crash (computing)Workstation <Musikinstrument>Gene clusterBeta functionValidity (statistics)Software bugNumbering schemeCategory of beingSocial classBoilerplate (text)Computer programmingMultiplication signPlotterIntegrated development environmentWindows RegistryWritingStudent's t-testComputer animation
11:51
MetadataTemplate (C++)Error messageService (economics)Structural loadScale (map)Concurrency (computer science)Communications protocolProcess (computing)Data modelIntrusion detection systemDefault (computer science)Computer configurationLoginException handlingException handlingPoint (geometry)Pulse (signal processing)Thermodynamisches SystemIP addressCASE <Informatik>Service-oriented architectureMessage passingNeuroinformatikLoginRow (database)Service (economics)Library (computing)Bounded variationBlock (periodic table)Process modelingLastteilungMultiplication signType theoryConnected spaceNetwork socketClient (computing)2 (number)State of matterSampling (statistics)Server (computing)Game controllerPropagatorInstance (computer science)Socket-SchnittstelleQuicksortCodeScripting languagePolygon meshLine (geometry)Closed setCartesian coordinate systemView (database)Communications protocolSelectivity (electronic)InformationObject (grammar)Dependent and independent variablesProcess (computing)Level (video gaming)Statement (computer science)Letterpress printingFinitismusDefault (computer science)Medical imagingMereologyStructural loadScaling (geometry)Concurrency (computer science)Operator (mathematics)Function (mathematics)Standard deviationControl flowChemical equationSubsetWrapper (data mining)Buffer solutionLoop (music)Query languageHookingBitDifferent (Kate Ryan album)File formatConfiguration spaceMobile appLecture/ConferenceComputer animation
21:13
Radical (chemistry)Frame problemInterior (topology)Network socketTerm (mathematics)Event horizonRouter (computing)Software maintenanceProgrammable read-only memoryNumberTotal S.A.Peer-to-peerArchitectureIntrusion detection systemCASE <Informatik>Dependent and independent variablesCloud computing1 (number)Software testingMultiplication signInterrupt <Informatik>Message passingInstance (computer science)Keyboard shortcutVariable (mathematics)Projective planeQuicksortStudent's t-testComputer architectureProduct (business)Cartesian coordinate systemPosition operatorEvent horizonLine (geometry)Tournament (medieval)BitProcess (computing)Metric systemNormal (geometry)Service (economics)ResultantConnected spaceNetwork socketNumberCommunications protocolRadical (chemistry)Scripting languageLoop (music)Scaling (geometry)Exception handlingStandard deviationComputer programmingSocket-SchnittstelleTerm (mathematics)Core dumpCompact spaceThermodynamisches SystemMedical imagingCuboidComputer configurationExecution unitRootLattice (order)DistanceClient (computing)2 (number)QuantumSpeech synthesisInformationHecke operatorReverse engineeringRight angleType theoryVirtual machineSingle-precision floating-point formatBackupNeuroinformatikComputer animation
Transcript: English(auto-generated)
00:05
Good afternoon. I'm Floris, I've been a Python developer for quite a while now. In my spare time, I'm quite often... sorry, can everyone hear me properly? Should I stand closer?
00:24
Is that better? Okay. Sorry. Okay. So, yeah, I've been involved in Python for quite a while. I work on open source as well, kind of usually around pytest, et cetera. I have very recently changed jobs and I'm now a site reliability engineer at Google.
00:47
While Kubernetes originally came after Google, this is not actually a Google project... Google Talk, sorry. This is largely based on my experience from my previous job, where I was working with microservices in Python that we ran on Kubernetes.
01:06
A little note about the title, so Kubernetes is part of the Cloud Native Computing Foundation, CNCF, and that's kind of where the title kind of came from, so Cloud Native Python.
01:21
Okay. So, this is kind of the contents that we'll be covering. So I'll start with a very really, really brief introduction about Kubernetes, like probably the shortest introduction you can ever have, but hopefully that should be, if you're not familiar with it yet,
01:43
that should be enough to kind of follow the rest of the slides, really. Then I'll kind of introduce a little example, just a kind of traditional echo service, so it's a network server, you send the message, you get the same message back, and then we'll kind of take that example throughout the rest of the talk
02:02
to sort of start modifying it a little bit. So, introduction to Kubernetes, like this is kind of Kubernetes in one slide, which is quite a tricky thing to do, I guess, but the idea of Kubernetes is that it's a cluster or orchestrator, really,
02:22
so the idea is that you give it a bunch of machines and it creates a cluster out of it, and then when you want to run your application, you just say, run my application somewhere in this cluster, and Kubernetes itself will decide where the right place in that cluster is for your application to run,
02:42
and kind of the aim that you're aiming for is to, the reason we want to build a system like this, especially with multiple microservices and you have multiple instances of every service, et cetera, is to make a really resilient application, so if one of the machines in the cluster is unhealthy or something,
03:02
you can just take it down, you can fix it, replace it, and your application just keeps running, so if something crashes, a request will just be routed somewhere else, and you create a really resilient and always up kind of application, that's the end goal of what you're trying to do, or trying to run in Kubernetes.
03:22
So the core concepts of Kubernetes is like, you want to run your application, and your application in our case is just going to be a Python application, and Kubernetes runs kind of containers, essentially, so you need to containerize your application, so right now that basically means Docker,
03:42
in the future hopefully that will be also like a rocket and the like, so you create a container out of your application, I'll skip over that part today, and Kubernetes runs this inside of a pod, so a pod is kind of the smallest unit that Kubernetes will run for you, essentially it's just another wrapper for your container,
04:02
you don't really have to worry why they decided to create a pod, it's kind of the idea is to treat multiple, you can potentially put multiple containers inside one pod and treat it as one unit, why you'd want to do that doesn't really matter for today. So a pod, you then tell Kubernetes,
04:20
please run this pod for me, which is your application, Kubernetes will kind of just find a server in its cluster and start to run it for you, but the problem with pods is that they're kind of ephemeral, so if the pod gets killed for some reason, an administrator goes rogue or a machine gets taken down or something like that, then your pod is gone,
04:41
so that's a far cry from our resilient application. So the next concept that Kubernetes introduces is kind of this idea of a replica set, and a replica set is essentially, will kind of continuously look at what's running inside of Kubernetes, and the idea is that in your replica set, you say, I want to run this many instances of my application,
05:05
and whenever the replica set sees that your application isn't running that many times, it will try and make sure that that happens, so if there's not enough instances of your application, then it will create more ones. If there's too many, it will kill a few.
05:21
So that means that, you know, machine gets taken down, the replica set will notice this and will create a new pod instead. So this starts to, like, create your application to be always there, especially if you request multiple instances. The problem then is, of course, again, these pods, they just run on a random machine somewhere in your cluster.
05:41
You don't know anything about them. You don't know how to contact them. So that's where the concept of the service comes in. So service is essentially some sort of, a fixed IP address is the easiest way to think of inside your cluster, and if you need to contact your application, you can contact it via the service, and the service will essentially load balance
06:01
between all the instances of your pod, and it will make sure if you send traffic to the service, it will go to one of the instances of your application that's running somewhere in the cluster. And these are really, like, the basic things about Kubernetes that's kind of enough to follow. Anyway, I'm now going to say, like,
06:20
everything else is bells and whistles. There is lots of other layers. They don't actually even recommend you use replica sets directly at the moment. So there's obviously lots more going on, but that's the core concept, and that's a core concept that allows you to create resilient applications. So that should be enough for today, hopefully.
06:41
So this is my little example application that I'll work with to address. It's basically an echo server, so on the network. I've implemented it using ZeroMQ because anytime you think of creating a TCP socket and you want to send and receive data, you should really use ZeroMQ instead because it takes care of a lot of the nitty-gritty networking details,
07:03
and you get, like, whole messages automatically delivered to your application, and you don't have to worry about everything else, really. Other than that, this is fairly standard. I should point out that all the code I show is kind of slideware, so I use globals. I don't show all the imports. You shouldn't write an application like this, obviously.
07:22
So, yeah, this application, basically, the main loop essentially is, like, I create my socket that I want to listen on. I then use this poller thing, which is basically, if you've done normal TCP programming, it's, like, select. So essentially it's asking the operating system, like,
07:41
can I sleep until the next message is available, and then when the next message is available, that will kind of come as, like, an event, and then I basically pass the server socket, the echo server socket to the event handler for that. The event handler will then basically receive the message
08:01
and send it back to me. So that's all this is. This is basically an infinite loop, receiving and sending the messages. So here are the few little helper functions to make it fit on the slide. So create and bind, really nothing, just some plumbing. So we create a socket, bind it so that we actually can receive connections,
08:22
and we register it also into the poller, again, using globals, et cetera. This is not nice code. And the handler is where the actual work happens. So we receive the message. Again, there's a little bit of ZeroMQ bookkeeping to split off the way ZeroMQ passes you the peer address,
08:43
and then log the message and then send it back to whoever sent it to us. So that's kind of the very simple application. And the first thing to kind of notice is that that's actually kind of sufficient. So we can just take that application,
09:02
put, like, you know, the if name equals main thing in there, et cetera, and we can just containerize that and run that in our Kubernetes cluster, and that will just work. So the first thing to sort of rely on when you're in Kubernetes is sort of just rely on the fact
09:21
that you are running in that environment, so you don't have to put complexity in your application. And it allows you to really have a very simple logical flow, which is kind of what we just saw was, like, a super simple, straightforward internal architecture. And that's kind of true for larger applications. Well, you can sort of skip through all the boilerplate, et cetera.
09:44
And the first thing in that is, like, I didn't write any exception handlers in that code. And sure, it fits on a slide, but that's actually generally quite true. You don't really have to worry about exceptions because your application runs in multiple instances.
10:00
And if you do get something unexpected, like, I don't know, maybe the bind could fail sort of the socket because whatever goes wrong on the machine. Like, I don't really mind. Like, my process will die, and Kubernetes will just make sure that a new one gets created somewhere else instead, and it will probably work.
10:21
So... Yeah, so you don't have to worry about that, especially. You can even go as far as kind of doing that for when you're receiving a request from someone on the service. If this other service is completely internal to yourself, so you're the actual... or your team or whatever is the author of the other service,
10:41
then essentially you can treat that failed request as a bug. Sorry, an invalid kind of schema of the request kind of as a bug, and you can basically crash again. If you're doing this for external applications, so if you're actually receiving user requests, then that is probably a bit too eager, like... or too brittle.
11:00
So in that case, you probably do want to catch the exception for request validation, because there is, depending on application, I mean, in this case, not very much, but there is some overhead in starting up your application again. And the other thing that kind of happens when you just crash and that you should take into account is that if you have network connection,
11:21
so you're receiving messages from some clients, those network connections, they will have buffers in them. So basically, there will be requests queued up already in your local process. Zero enqueue makes that really obvious, because there is explicitly a queue with your socket. But even if you use raw TCP, there will always be internal kernel buffers.
11:45
The kernel may even have accepted new requests already and just queued them up, even though your application has no ID yet. There might be data still on the wire just coming over. And if you just crash, then you kind of lose that data. So those requests and whoever created those requests
12:02
will have to then kind of wait and time out and retry. So that's not very good. Yeah, so you want to take that kind of into account. And that kind of brings you to the moment, like, how do I organize my messaging, et cetera, and you can kind of start playing.
12:21
If you really don't want to suffer from that, you can start playing with message brokers like RabbitMQ or different systems like Kafka, et cetera. But these all come with, you know, trade-offs, expenses. The only thing I would say, like, in here is basically be aware of when you crash. You may lose requests and make sure
12:41
that that's okay in the system you're designing. So in this slide, this actually shows how you actually would create your... This is the recommended way they want you to create your pods. Deployment is essentially just a wrapper around this replica set thing. The reason they have it is because it creates... Updating your application slightly easier,
13:02
but as far as from our point of view, the two important things here is this line that says replicas three. That means we're requesting three replicas, so Kubernetes will always ensure that there's three of us running. And the other really important thing is the very last line that says restart policy always. That line will basically...
13:21
That tells Kubernetes that, you know, if we're crashing, just start this again, please. So that's kind of the first step. The second thing is, like, the script had no concurrency, and while it was a very simple example, this is generally true. You can rely on...
13:40
You can keep your internal code really simple because the idea is to scale via the process model. That's some sort of, if you've ever heard of 12-factor apps kind of methodology. That's kind of the idea of, like, you just create more... You scale horizontally by creating more instances of our application. As we just saw, that's kind of what you already did. We just create more replicas,
14:02
and they will handle the traffic. And that means that internally, we can have really easy debugging, et cetera, because our control flow just gets really simple, and we don't worry about any of the other stuff. Yeah, and basically, your server is your load balancer in this case.
14:21
So this is kind of the service definition that we have. Once again, one gotcha to look out for with this sort of load... The service that you create, which load balances your traffic between your pods, is if your protocol that you're using uses long-standing connections, which, again, like Xero MQ, in our example, points this out very well
14:41
because Xero MQ creates long-standing connections. It connects, and then it tries to reuse that connection for lots of requests and responses. So our Xero service will accept lots of Xero requests from the client on one connection. So this means that, essentially, we don't get the load balancing. So one client will permanently be connected
15:01
to one of the pods or one of our applications, essentially. And so that's not very load balancing-like. Well, on the other hand, if you're using HTTP, then, you know, a new HTTP connection gets created for each request, and that would distribute automatically.
15:23
The trick to use there is that Kubernetes actually allows you to see the layer below services, and it has endpoints. I think objects, it calls endpoints there. And that actually allows you to see which endpoints, basically, IP addresses that are part of your service.
15:43
And using that information so we could update our application to sort of query the Kubernetes API to ask what endpoints do I actually want to connect to. You can tell XeroMQ connect to all these endpoints. The downside is that you have to kind of work with the Kubernetes API. You need to be constantly aware that this can change
16:01
so when an endpoint disappears, you need to tell XeroMQ, hey, disconnect from that endpoint, et cetera. But that's just generally, you know, the sort of thing you need to be aware of when you have long protocols that use long-standing connections, basically. Next on, talk a little bit about logging.
16:21
So you may have seen in my very first example, I cringed at this print statement. So by default, Docker kind of takes standard output of your container as log data, and Kubernetes will again take that log data from the Docker containers and will make that available to use.
16:40
And generally, the idea is that at operations time, you sort of hook that up to a log aggregation of some sort, something like Elasticsearch or Fluentd or something like that. But using simple print statements is not very nice. In general, you want to be able to control log levels at command line level.
17:01
Again, that's sort of a 12-factor app kind of thing. So you really want to use logging libraries. Yeah. So logging libraries are sort of common.
17:21
There's quite a few variations, and they all kind of try to wrap a horrible amount of global state into a nice API, and global state is always quite horrible, so they all kind of are ugly in one way or another. I quite like logbook. I kind of like the way it tries to handle the global state,
17:41
but there's nothing inherently better about logbook than standard library logging, if you like standard library logging. One nice thing about logbook is that you can use the normal curly braces formatting, the new style formatting instead of percent formatting from standard library logging. But the main thing here is to notice that once you start using a logging library,
18:01
you can actually start... You can hook up your logging library instead of printing out to standard out, which is kind of the first thing you would probably do because if you don't have all the infrastructure yet... But you can hook it up to send the log records directly to the aggregator, and this allows you to, like, if you have tracebacks or something, you can send them as a single big block.
18:22
So the next thing that you really want to do as soon as you start using logging libraries is wrap your main application into basically this exception logging. All logging libraries will support some sort of variation of this, but the idea here is that by doing this,
18:41
I make sure that any unhandled exception, as I was advertising earlier, will be captured by the logging library and will be sent as one single log record back to your logging aggregator. So next one is kind of this concept
19:01
that Kubernetes calls health endpoints, and the idea here is that, the central idea here is that when your application starts up, Kubernetes has said, you know, start this container, and as soon as that container is kind of running from a process point of view, it's available, so the service object
19:21
that you created for your application will start sending traffic to you. But there is a finite amount of time that your application is running, and you haven't opened your socket yet, and you're not listening for connections yet. In this case, obviously, it will be very small, but if you have a lot of setup and things you have to do before you start accepting connections from clients,
19:41
that delay might be bigger as well. And the problem is, if at that point Kubernetes is sending traffic to your application, then you're basically saying refused connection, and all those clients are like, well, the service is down. So you don't want to do that, and what Kubernetes does is it introduces these readiness probes,
20:00
and the idea there is basically that Kubernetes is like, after starting your application, it will wait until the probe succeeds, and once the probe succeeds, only then it will start sending traffic to your application. This is kind of how you configure that in a Kubernetes YAML configuration.
20:20
In our case, so there's several different probes you can use. In our case, it's just a simple echo server, so all we care about, really, from the probe's point of view is that the socket is open and listening. So this TCP socket probe essentially says, like, as soon as this socket accepts connections,
20:42
send me traffic. Kubernetes will literally just try and connect to the socket, and as soon as there's a connection close, and say, yeah, okay, I'll send the traffic. So that means there's still kind of a tiny delay in which, like, we might actually have bound our socket, but not yet in the main loop of processing some messages. But because we know the buffering
21:00
and the queuing in our sockets take care of this, that's basically perfectly fine. That will be a short amount of time that some requests get queued up, but we'll start serving them very shortly. If your service is actually using HTTPS and transport, it kind of has built-in support for that, and there is this convention of using this health Z route,
21:24
and basically this is really nice in a way because you can basically tell the readiness probe is kind of completely in line with or can be if you create it as such to be completely in line with your normal request processing. So if that actually returns,
21:41
so basically Kubernetes only looks for the 200 OK on there. So as soon as that returns 200 OK, it will start sending you traffic. And because you can do this completely in line, you actually have quite high assurance that, yes, my application is fully running and can start serving traffic. So the same concept of kind of readiness
22:02
is also... The same concept also happens for during the pod's lifetime, really. So when your application is running, especially when you're running your application at a large scale and it handles lots of requests, sometimes things will just go wrong,
22:22
and one of the many instances will start misbehaving. This might be because of external things, like there's another container on that box that even though things should be isolated, they're not always as isolated as you would imagine, and, like, things go terribly wrong or someone decided to run a backup on that machine
22:43
and things go really slow. All sorts of things happen, right, really. So the idea of liveness probes is that you don't want to be sending requests to an application that's slow to respond or is just completely stuck or something like that.
23:01
It does this in essentially a very similar way, so it just has this liveness probes thing, and this shows kind of the third type of probe that you can use, so we've already seen, like, TCP socket and HTTP GET, and here I decided to use the exec one. Exec is kind of the most work,
23:22
but it's also the most flexible. So the problem is, like, the TCP one that we used for readiness, and it was very suitable there. It's, like, not very useful for the liveness probe, and what we really want is kind of the same as you can use with the HTTP GET. It's like we want to know in line,
23:40
like, are things working correctly? So the trick you do there is this exec command basically says it gives you the command line that you want to execute inside of your container, so you have to provide an extra binary inside of your container, which will ideally, like, tell how healthy or not healthy that pod is,
24:03
and ideally, you want to aim for this in-line checking, so that's exactly what I've done here. I've added an extra script in my application, and I just invoke it with Python. So this is kind of how simple the script can be. Again, obviously, I have a very simple application,
24:21
but literally, I just create a socket connected to the public endpoint. In this case, because I'm running on the same container, it's actually on local host, but we still make the TCP connection, so this still gives us a very high kind of idea of what the public behavior is of our application,
24:42
and we just send a message, and we wait for, like, 500 milliseconds, and if you haven't got a message back yet at that time, then we fail, and then Kubernetes will start taking your pod out of... basically stop sending traffic to our pod and send it to the healthy ones instead.
25:00
You may notice I'm not actually even receiving my response. I just check, hey, there is a response. In this case, I think that is sufficient. So it depends on the protocol you have or the sort of messaging you use. Sometimes you might want to introduce slightly more,
25:20
and often it is nice to build in something like that, the health set kind of rooted in the HTTP convention, just something that, you know, it is working. Yes, just something that, you know, it is working, and everything is fine.
25:40
So that's the idea of liveness. So next one is, like, termination, and again, this is, like, very similar. If you thought about how we, you know, at startup time, we don't really want to... We didn't want to refuse any connections. At termination time, we don't want to drop any connections that exist. So any requests that are currently queued up on our process,
26:03
if we just say, oh, we've been asked... If we don't do anything, basically, our application will just receive termination signal and it will just die, and all the requests that are queued up will basically be lost, and all the clients will be there waiting again. So that's not very good. So instead, we should handle the termination signal, which is done in Kubernetes,
26:20
just like it's always been done in Unix systems. You'd basically get, yeah, via Docker, but you get SIG terms sent to you. And the signal handler here that we created is just a very simple internal socket. It's essentially, I think of it as a pipe, and all we want to do is, like, send this signal. Like, I'm sending a single byte to my main loop,
26:42
and then my main loop knows it has to shut down. Yeah, and it can shut down while trying to handle the connections. I'm using the same signal handler here for both SIG term and SIGINT. SIGINT is the signal that you get when you press Ctrl-C normally when you're running on a terminal.
27:01
So in Python, that normally gets automatically translated to keyboard interrupt exception because you want your application to kind of behave in the same way when you're running on the test, et cetera. It's usually best practice to just bind both the signal handlers. The modifications for my main loop
27:22
are maybe a little bit more messy now, but essentially, it's not that much. I just, again, instead of creating just one socket in my main loop, I create two sockets. So I add this termination socket. I add it to the potter, and when I now receive... The important thing here is when I receive this new...
27:43
this single byte, basically. I don't even care what that byte is, by the way. Again, I just know, oh, I received the message from my termination socket, so I need to shut down. And here, I first unbind, so this means that I will stop receiving new connections.
28:02
And then I basically keep processing. While there are still events in the queues, I keep processing. And also, I set the timeout, actually, to, like, five seconds or something, which may be fairly high. But the idea here is that some requests might actually already be on the wire, and I want to give those a chance to be processed well. And once there are no longer any messages in my queue, the wire loop will finish, and I return.
28:25
So the last thing that I would like to add is kind of monitoring. So Prometheus is another Cloud Native computing foundation project, actually,
28:40
which is kind of why I mention it. And the idea here is, like, to start, you always want to know kind of what's going on. And Prometheus is kind of... offers you this option of doing wide box monitoring in a way, so you can add counters, and you can add metrics to inside your application. You can start enriching your metrics collection, basically.
29:01
So Prometheus works in a very... in a pool-based fashion, so you have the central infrastructure, and it will go around to all your services, and it will do an HTTP request and get back your metrics, basically. It's a little bit like SNMP over HTTP if you remember the SNMP pool medal.
29:21
But at least we learned that, you know, monitoring data is probably even more important than production traffic, so we actually use a reliable TCP transport instead of something like UDP. Anyway... Prometheus is obviously a very difficult and big project,
29:41
but the idea here that I really want to show is just how easy it is to get started with it, and you can just add very little things. There is a last year that was a talk about from Hynek, I think, at Europe, at Python, which goes in a lot more detail about Prometheus. So this is kind of, yeah, the recap. Like, the idea is basically none of this is actually
30:00
strictly required to be able to run on Kubernetes. You can adopt this gradually as you need. And, yeah, just keep your architecture simple. Think about when you lose requests, and you don't want to go blind, basically, so you always want to have some instrumentation and monitoring. Thank you very much.
30:20
I think I'm about out of time, so, unfortunately, there probably won't be any questions, but you can find me outside if you like.