Infrastructure as Code: "pip install" your environment
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 54 | |
| Anzahl der Teile | 169 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/21221 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
EuroPython 201654 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
ProgrammierumgebungInformatikInstallation <Informatik>ExpertensystemProgrammierumgebungPhysikerFontChatten <Kommunikation>ComputerspielDatensichtgerätVorlesung/Konferenz
01:15
FontCodeDienst <Informatik>Nichtlinearer OperatorWort <Informatik>MereologiePhysikalische TheorieAdressraumComputeranimationVorlesung/Konferenz
01:50
FontCodeDienst <Informatik>BenutzerfreundlichkeitOperations ResearchSchnittmengeApp <Programm>MereologieUnrundheitDienst <Informatik>App <Programm>Data DictionaryNichtlinearer Operatorp-BlockSchnittmengeStatistische HypotheseInstantiierungPlotterWeb logComputeranimationVorlesung/KonferenzXML
02:57
SchnittmengeOperations ResearchApp <Programm>Virtuelle MaschineDatenstrukturDatenbankMathematikServerNichtlinearer OperatorSchnittmengeREST <Informatik>Statistische HypotheseFreewareTermServerGraphfärbungQuick-SortAggregatzustandSchaltnetzDatenverarbeitungssystemBijektionMultifunktionNP-hartes ProblemBildverstehenTransformation <Mathematik>ZahlenbereichRobotikMatchingQuaderDigitalisierungKomplex <Algebra>SchlussregelVorlesung/KonferenzTechnische ZeichnungDiagrammFlussdiagramm
07:11
DatenstrukturTaskGraphfärbungSiedepunktBitDatenbankObjekt <Kategorie>DämpfungVirtuelle MaschineEreignisdatenanalyseVorlesung/KonferenzXML
07:55
MomentenproblemDämpfungProgrammierumgebungVirtuelle MaschinePhysikalisches SystemVollständigkeitAggregatzustandSiedepunktParadoxonVorlesung/KonferenzXMLJSON
08:34
SystemzusammenbruchInstantiierungMailing-ListeEinsPhysikalisches SystemSystemzusammenbruchInstantiierungDruckverlaufVorlesung/KonferenzComputeranimation
09:16
SystemzusammenbruchInstantiierungMailing-ListeComputerspielCodeObjekt <Kategorie>CachingSchlüsselverwaltungServerRechnernetzIdentitätsverwaltungDatenbankMailing-ListeInstantiierungAdditionInformationsspeicherungIdentitätsverwaltungComputerspielNeuroinformatikCodeObjekt <Kategorie>SchlüsselverwaltungVorlesung/KonferenzComputeranimation
10:26
Virtuelle MaschineGraphische BenutzeroberflächeCodeVirtuelle MaschineArithmetisches MittelAggregatzustandRechenzentrump-WertTypentheorieBitRadikal <Mathematik>SystemverwaltungHalbleiterspeicherPasswortSchlüsselverwaltungGraphische BenutzeroberflächeCodeREST <Informatik>InformationsspeicherungComputeranimationVorlesung/Konferenz
11:20
Dienst <Informatik>Virtuelle MaschineIdentitätsverwaltungDatenverwaltungAdditionCodeKartesische KoordinatenDienst <Informatik>MereologieVorlesung/KonferenzComputeranimation
11:50
Dienst <Informatik>Virtuelle MaschineIdentitätsverwaltungDatenverwaltungMagnetbandlaufwerkInstantiierungVirtuelle MaschineObjekt <Kategorie>IdentitätsverwaltungPufferüberlaufTreiber <Programm>Keller <Informatik>Vorlesung/KonferenzXMLComputeranimation
12:30
Dienst <Informatik>CodeMultiplikationsoperatorVirtuelle MaschineSoftwareentwicklerCASE <Informatik>TeilbarkeitAggregatzustandOrdnungsreduktionWhiteboardSpielkonsoleFehlermeldungVorlesung/KonferenzComputeranimation
14:10
Kartesische KoordinatenDienst <Informatik>Vorlesung/Konferenz
14:51
Web-SeiteVersionsverwaltungKartesische KoordinatenDienst <Informatik>MereologiePhysikalische TheorieGarbentheorieFrequenzFehlertoleranzComputeranimationVorlesung/Konferenz
15:22
Dienst <Informatik>Dienst <Informatik>MereologieMailing-ListeVorlesung/Konferenz
16:01
Dienst <Informatik>Virtuelle MaschineMeta-TagService providerInstantiierungDatenbankSoftwaretestDoS-AttackeXMLComputeranimation
16:32
Dienst <Informatik>Meta-TagVirtuelle MaschineVirtuelle MaschineMeta-TagDatenbankInstantiierungSchnittmengeInformationsspeicherungAggregatzustandForcingCASE <Informatik>Vorlesung/KonferenzComputeranimation
17:03
ComputervirusImplementierungMomentenproblemDienst <Informatik>Service providerKomplex <Algebra>Virtuelle MaschineREST <Informatik>InstantiierungXMLUMLComputeranimation
17:49
ImplementierungClientProgrammbibliothekDämon <Informatik>StellenringClientProgrammbibliothekProzess <Informatik>DatenverwaltungVorlesung/KonferenzComputeranimationDiagramm
18:20
ProgrammbibliothekClientDämon <Informatik>GeradeEigentliche AbbildungSichtenkonzeptDämon <Informatik>ClientRPCVorlesung/KonferenzComputeranimation
18:54
InstantiierungGammafunktionFächer <Mathematik>ClientParametersystemCodeInstantiierungProgrammfehlerQuaderMereologieBildgebendes VerfahrenDatenbankMailing-ListeFaserbündelExpertensystemRechter WinkelBitProgrammierumgebungTypentheorieNormalvektorTextbausteinClientVariableOrtsoperatorREST <Informatik>Cluster <Rechnernetz>VerkehrsinformationVorlesung/KonferenzXML
21:22
InstantiierungParametersystemInformationsspeicherungInstantiierungKartesische KoordinatenParametrische ErregungKlasse <Mathematik>DatenbankSoftwaretestLogischer SchlussMailing-ListeArithmetisches MittelVorlesung/KonferenzComputeranimation
21:58
InstantiierungMailing-ListeTropfenBitPlotterCodeFaserbündelInstantiierungComputeranimationVorlesung/Konferenz
22:29
MaßerweiterungKlasse <Mathematik>ImplementierungKlasse <Mathematik>REST <Informatik>VererbungshierarchieMailing-ListeDatenbankSoftwaretestInstantiierungComputeranimationVorlesung/Konferenz
23:27
Klasse <Mathematik>AbfrageInstantiierungSpezielle unitäre GruppeDialektInstantiierungKlasse <Mathematik>DatenbankMailing-ListeFunktionalInformationCASE <Informatik>ResultanteOrdnung <Mathematik>RandverteilungParametrische ErregungProgramm/QuellcodeXML
24:27
InstantiierungKlasse <Mathematik>AbfrageInstantiierungTextbausteinFunktionalProgrammierungDatenbankVorlesung/KonferenzXMLProgramm/QuellcodeJSON
25:13
MereologieE-MailSpezialrechnerServerMeta-TagInstantiierungResultanteProzess <Informatik>CodeBildgebendes VerfahrenComputerspielSoftwaretestPunktTextbausteinServerDämon <Informatik>Vorlesung/KonferenzComputeranimation
26:14
KurvenanpassungElektronische PublikationInstantiierungClientProgrammbibliothekInhalt <Mathematik>Physikalische TheorieTypentheorieComputeranimationProgramm/Quellcode
27:10
InstantiierungPasswortZahlenbereichRadikal <Mathematik>OrtsoperatorVorlesung/KonferenzProgramm/Quellcode
27:47
SoftwaretestPasswortProgramm/QuellcodeVorlesung/Konferenz
28:19
SoftwaretestDienst <Informatik>System-on-ChipSoftwaretestDatenbankIntegralProjektive EbeneXMLComputeranimation
28:52
SoftwaretestDienst <Informatik>Dienst <Informatik>Kartesische KoordinatenZweiVirtuelle MaschineInstantiierungVorlesung/KonferenzComputeranimation
29:42
Dienst <Informatik>CodeVorlesung/Konferenz
30:15
SchätzungKette <Mathematik>ImplementierungPERM <Computer>BenutzerfreundlichkeitKreisbogenComputerspielKartesische KoordinatenKette <Mathematik>Elektronische PublikationVerzeichnisdienstRPCGruppenoperationIsolation <Informatik>SicherungskopieDatei-ServerREST <Informatik>XMLComputeranimation
31:12
TypentheorieCodeWasserdampftafelProjektive EbeneGeradeVorlesung/Konferenz
31:43
PlastikkarteCAN-BusBenutzerfreundlichkeitKartesische KoordinatenDienst <Informatik>REST <Informatik>Virtuelle MaschineXMLComputeranimation
32:20
GamecontrollerInstantiierungSpezifisches VolumenVirtuelle MaschineHalbleiterspeicherMultiplikationsoperatorDateiverwaltungQuaderVersionsverwaltungPunktBitProgrammfehlerBenutzerfreundlichkeitDatenbankLastPhysikalisches SystemBildgebendes VerfahrenInformationsspeicherungKlon <Mathematik>TabelleRichtungsinc-FunktionDienst <Informatik>Elektronische PublikationInhalt <Mathematik>FestplatteSichtenkonzeptATMMereologieCASE <Informatik>CodeDimensionsanalyseProgrammierungClique <Graphentheorie>SpeicherbereinigungBasis <Mathematik>MehrrechnersystemWeb ServicesZählenSoundverarbeitungSchnittmengeProfil <Aerodynamik>AggregatzustandProzess <Informatik>Doppler-EffektInterleavingMomentenproblemEreignishorizontEinsVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:00
The next talk of this session, it's Infrastructure as Code, PIP Install Your Environment. Ladies and gentlemen, please welcome Sebastian Neubauer. Yeah, thank you. So last people come in, have a seat.
00:25
So yeah, that's the title, Infrastructure as Code, PIP Install Your Environment. I have two disclaimers. So the first one is this talk is not about PIP. So if you're here because you want to learn something about PIP, I'm sorry, it has nothing to do with PIP.
00:40
That is just a metaphor. The other disclaimer, I'm not a computer scientist. I'm not an expert. I'm a physicist and data scientist, so everything around infrastructure is just auto-detectically learned by me. So if you find something wrong, then just keep it for you and forget it.
01:02
So okay, yeah, I work at Blue Yonder. Maybe you saw that stand over there. So just come by and we can have a small chat. Okay, so a small outline. Maybe a question. So are there, for example, some ops people here?
01:23
Operations? One, two, three. Yeah, okay. So the rest is developers? Nobody, okay. Yeah, so it's more or less two parts. So the first part is more, is a theoretical thing
01:41
or let's say an introduction to all the words. So a glossary, what is infrastructure? Or what is CRUD, for example, here, C-R-E-D. Immutability, what is infrastructure as code? And then the second part, that is really hands-on. So we see code, we do all the stuff. Maybe we do a short question round between those two parts
02:05
if there are any questions so that the first part is really, everyone knows what is infrastructure as code, what is infrastructure as a service, and so on. Okay, so ready, we start. What is CRUD or C-R-U-D? It's trivial.
02:21
So it's just create, read, update, delete. And it's just a simple set of operations but as we will see in the talk, these are really powerful operations. So now this is a hypothesis of me. Nearly everything can be implemented using these three operations. So for example, think of a block.
02:42
You can create a block, read a block, update a block post, delete it. To-do app or EC2 instances, you can create, you can delete dictionaries in Python, you can create an item, delete an item, or LW, so everything what you can think of can be expressed somehow in these terms, create, read, update, delete.
03:03
And now, the next thing also, a hypothesis if it's wrong, you can correct me afterwards, personally. If we implement such things with this simple set of operations,
03:20
you more or less get a REST API for free. And we see that in this, so who has heard of REST API before? Okay, quite a few. So yeah, there's this really one-to-one correspondence between those REST verbs if you use it with HTTP. So post is a create, get is a read,
03:42
put is an update, and delete is delete. So we really see, okay, there's some kind of correspondence between these things. Now, the next thing, what is immutability? So this is really a hard thing, and yeah, don't nail me down. It's not a theoretically
04:04
nailed down definition what you get here. So I have two numbers here, an eight and a nine. So this small question now to the audience. What do we do to come from the initial state, the eight to a nine?
04:26
Would you really do it like that? So I want a trivial answer. So what would you do? Yeah, we remove the left lower one. Of course.
04:40
I mean, imagine it would be Stonehenge and these are stones. We would remove the one, and then we have a nine. But now is the question, if we would have implemented a kind of a robot which has to do it, would we have done it the same way? I mean, just imagine what you would have to do.
05:01
A camera analyzing with some kind of computer vision the initial state, and then find transformations from how to get from an eight to a nine. And then the next thing is, if we want to come from a seven to an eight, or from seven to nine. So it's an explosion of complexity,
05:21
because we have every combinations of initial state to final state. So how would the robot, if we would build the robot, how would we have implemented it? And there's the delete. We would just have a wiper, kill all the matches, sort them somehow, put them in a little box,
05:43
and then put the robot, the nine. So we just need to have, for 10 digits, we just need 10 instructions. How to come from initial state zero, everything in the box, to a nine, to a eight, to a seven, six, five, four, three, two, one.
06:01
10 instructions and we are done. So that's a thesis, I don't know if it's correct, but an update can always be implemented by deleting and recreating.
06:20
But deleting and recreating in the real world is often quite resource expensive. So let's think of a house. Do we really want to tear down the house and build it up again just because the color of one room need to change from blue to red? Writing a book, hmm, provision a server.
06:41
So can take quite a while to have a server running exactly in the state you want to have it, if you do it by hand. But on the other side, making an update, so come from a nine to a 10, can be quite difficult and complex. Think of the combination seven to nine, eight to one. For example, if we want to add a room to a house,
07:04
can be really complex. So maybe then it's easier to tear down the house and rebuild it with another room. Yeah, or change the structure of a database. Everyone knows who deals with Alembic or some kind of things.
07:22
Can be quite hard or sometimes it's impossible. Or if the task would be change the color of Mona Lisa, it's also really hard. So let's boil it down a bit. Humans are lazy, but intelligent, so what we are doing is always the update. So we see there is an eight, there is a nine.
07:42
We just have to remove one. We optimize for least effort. Machines are eager but dumb for their, for them. Deleting and recreating is the preferred way to go. So we wipe everything up and rebuild it from scratch. Question is, what is the current infrastructure
08:01
you are working in at the moment? Just think of what's your daily business in this infrastructure where you are working. This is just a question to think of. Are we, so who is working in an environment which is machine driven? So deleting, recreating, or not.
08:21
So and now boil it down completely. What is immutability? Immutability is a concept of how to change the state of a system. It sounds paradox, but it isn't. So immutability is not about not changing anything, so it's about how to change it, and how to change it is really easy. We have our CRUD, we just delete the U,
08:43
and we are done. That's immutability. So we do not do an update, we just delete and recreate our systems. Now a really small crash course for REST, just that we know what we are talking about if we later go on and do the example,
09:03
but everyone knows already, we know it. So normally we have something like a collection resource, let's say some kind of instances, let's say houses or things like this, or maybe better to-dos. To-dos, so a collection of to-dos.
09:20
With a get, we get a list of all to-dos. With a post, we create a new to-do. Then there is the instance resource, it's the one with the ID behind it. With a get, we get just the to-do, so what to do in this particular thing, we can delete it and we can update this particular instance.
09:43
Now to something completely different. So we are computer people. What is infrastructure? I mean, it's a quote from me, so don't take it too seriously. Infrastructure is everything, what brings your dead code to life? This means everything around deploy or object stores,
10:05
key value stores, databases, routing, networking, maybe identity access, so everything around your code, which is in Git, is what I call infrastructure. In the end, it brings the value,
10:22
in addition with your code. Now, that's the title of the talk, What is Infrastructure as Code? It just means if we can access this infrastructure, so the key value store or the memory or things like this, in an automated fashion
10:43
via machine-consumable APIs. So now we heard quite a bit of APIs, REST API and CRUD and how to change states. So here, it would mean infrastructure as code means to change something, there should be no tickets to a sysadmin doing things
11:02
or you need a screwdriver and drive to your data center, maybe also not an admin SSH terminal because he needs to type in some secret password. Yeah, and of course, no GUI. So we need machine-consumable APIs.
11:23
Now, in addition to infrastructure as code, what is infrastructure as a service? And this just means that the application itself can consume the infrastructure it runs in. So infrastructure as code could also be done, let's say, from the outside. So someone provisions the infrastructure
11:41
but it's everything is written in code and not with screwdrivers and tickets. But as a service really means the application itself can create the infrastructure it runs in. This means the self-service, it's really like in a supermarket, you walk in and say, okay, I need an object store, I need a, I don't know, maybe a EC2 instance and so on.
12:04
For this, we need simple machine-consumable APIs, of course, somehow identity access management, so not that the machine can buy 12 million EC2 instances and you don't know, so you need something to control it.
12:21
Yeah, and of course, no dependency in this whole thing to any screwdrivers or yeah, some Stack Overflow deploy instruction. So it needs to be consumable via machine-readable APIs. So the next thing after infrastructure as a service
12:43
is immutable infrastructure. But now that's good, we already know what immutable infrastructure is. From all the create, read, update and delete things, we just delete the update thing, so we say there is no update of any state.
13:00
What we do is we delete the thing we are working with and we recreate it in the new state. That is immutable infrastructure. Now we learned a lot, maybe you are bored, but why is it so important to have this automation?
13:21
So everything around automation and infrastructure as code. Automation means the machines do it, so we reduce human error, we enhance reproducibility, it's of course cheaper, maybe not always, but sometimes it's cheaper, it's faster most of the time
13:43
and you also get happier developers because sometimes they really have to do messy stuff in a console, and cheap and faster is particularly not the most important thing. Most people think we need to automate it if it's really a cost factor or if it's too slow,
14:03
but that's not the case. So the reduced human error, for example, is really, it's crucial. Also if you build just once your application, if the one who provisions it forgets to correctly configure the backup,
14:21
it's just done once but it's done wrong. So it's very important, I think. And why is this, we can call it X as a service, so this infrastructure as a service, why is it so important? So why is it important that the application itself can consume the infrastructure it runs in?
14:43
And also a quote from me, so don't take it too seriously, it means we can put everything, so the entire application, including the infrastructure under version control, and that's something really completely new. Never before we were able to put the whole application, including infrastructure in version control,
15:02
and that's really, that's a crucial part. So if you go home after this talk and you know this thing, then I think you've learned enough, that would be fine. So that was the gray theory. And I think for the next part, where we really build such an infrastructure as a service
15:24
for our own here, I think it's really crucial that you understood everything, what we talked about so far. Are there any questions to the first part? No? So everything is clear?
15:41
Or not at all? Okay, so then let's go on. We built a Postgres as a service here together. I hope you are already excited. A small requirement list, so what should it do in the end?
16:01
It should provide PostgreSQL instances as a self-service. These instances should be somehow isolated, that means so if one instance get nuked, the other one doesn't get nuked. Or also if I choose a name like test instance one, it doesn't come back with,
16:21
there is already a database called test instance one, but it's not from me, but it's from someone else. So this means isolation, so we really need to consume and we want to set everything. Of course, simple machine consumable APIs.
16:41
And it should somehow persist the state, so if I create an instance, I can then see how many instances I've created and so on. And this we store in some kind of meta database. But, and that's the most important thing, it should be as simple as possible. That should be always the case.
17:03
So first we need a name and I called it the Postgres as a service, so that's good enough I think for the moment. Let's design this whole thing, so a brainstorming. So how could we provide instant PostgreSQL instances?
17:26
Any ideas? Darker. How should the API look like? Of course, so we know already REST API is good,
17:41
just that we already know it should be machine consumable, so the update thing for us might be not so important, it increases complexity. From the beginning we just say, create and delete is everything we need. How do we implement the API? Of course, there's Flask, we could use everything else,
18:03
but let's stick to this one, it's simple and it does the job. So first thing, Docker. We can manage Docker from Python, there is a Docker client library, who knows this Docker client library?
18:20
Okay, some few, but not so much, interesting. So it has nearly all features, what you can access in native Docker from Python, from this client. All it needs is a running Docker daemon, you can connect via local running daemon or remote daemon, you could also use, for example, Swarm, so then there's a whole cluster
18:43
of Docker behind it. And that's a personal view, I think the quality is quite good. I don't know, has someone worked with the Kafka Python client? So I had some bugs and I thought, okay, maybe just, strange.
19:03
One small example and you find one or two bugs. So this is really good, you can work with it. So first thing, we need to create a Postgres instance and that's the whole code. There's of course some boilerplate stuff around it, but in principle, that's the whole stuff, so if you now would type it into a thing,
19:21
it would really run and it would do the job, namely creating a Postgres Docker container running a Postgres in it. So we connect to a client, we take the image, it's just the official Postgres image from Docker,
19:41
then there's this thing, create container, and it does what it tells the name. So we just say, okay, take the Postgres image, name it like we want to name it, give some, which internal ports should be exported, that's a normal Postgres port, environment variables,
20:03
there we can say which user we want to have, which password, and how the database should be named inside the Postgres instance. Yeah, that's it, then we can start this container, give it a port, and everything is fine. Okay, so the create already done,
20:24
now the get, it's also really easy, so once again, we connect to the client, then we can get a list of all running containers on the client, so maybe there are other containers running so we just search for the right thing here,
20:41
yeah, we search for the name, there's a bit of a quirk inside that you need to remove some slash there, but we can ignore that completely, so it's really easy, list of containers, and return the container, delete, much more easy. It's, yeah, just remove that container with that ID.
21:02
Any questions here? Trivial. Okay, then the RESTful API, and we are now really, really experts in it already, so we know we need some kind of a collection resource, so we name it Postgres instances,
21:21
so the URI would be something like localhost, if we run it on localhost or somewhere else, slash, postgres instances, then we give it some parameters, these are the needed parameters, password, user, database name, or the, and the name of the instance, this should be then unique, so this is really my test database application
21:42
Sunday morning. And then just return the created instance name. That's it. Furthermore, get a list of all instances, is then a get on this collection resource.
22:00
Get the details of an instance, is the URI with the ID behind it, and delete, I think that's trivial, so we grasp it already. Then to the Flask part, okay, now, it's a bit of code, so if you want to have a small sleep, you can do it. If you are not interested in how to implement it in Flask,
22:22
but I go really fast through it. So, what I used here in this example is this Flask RESTful package, who knows it? Quite a few. So it's really easy to define a REST API using this package.
22:43
All you do is you implement, so you make a class, which is inherited from the resource class, and then you implement the get, put, post, and delete methods. Set the URI, so our collection resource,
23:03
which names we want to use for these resources, and basically we are done. Furthermore, for this persistence, so that we can get a list of all those instances we already created, we just use SQLAlchemy with some database. For example, a post, maybe in a container.
23:25
Okay, so the first thing, the collection resource, as I said, we build a class, and yeah, it's derived from this resource thing. We can marshal something, but let's ignore this one,
23:41
so this is a get on the collection resource, so everyone knows what's coming back. Yeah, it's the list of all our created instances, so return this thing. Then how to create a post class instance in that case. It's of course a post to the collection resource,
24:03
so we parse some parameters, but let's ignore this. Very important thing here is exactly what we already saw in the beginning, that's our function which creates a Postgres instance using Docker, and then we just save
24:23
all those information things in a database. That's it, really easy, nothing fancy here. So the instance resource, so with the get, we get the details of a particular Postgres instance, for example the user name or things like this,
24:43
and the delete, there's some more boilerplate, mainly what happens if you want to delete something which doesn't exist yet, but we can ignore that, so we just say okay, from the database, get me the details of this instance the user wants to delete, and then let's go
25:02
to our delete function from the Docker part, and then afterwards if it's deleted, we delete the entry in the database, really trivial. Last thing we have to do, we set those URIs,
25:20
so the collection resource to Postgres, and the instance resource to Postgres instances slash ID, exactly like we designed it. So we are done, that's everything, mainly in principle that's the whole code, some boilerplate code here and there, but you saw everything, really easy.
25:43
Now it's a question, does it really work, so what do we have to do to bring it to life, so we need a Docker daemon running, and we pull the Postgres image so that it's locally available, then we start the Flask server, so you can use of course every WSGI server
26:04
you want to use, for example to Unicorn or just for the test debug server, then initialize the metaDB, and keep calm and get you a Postgres. So what to do, we go into our terminal,
26:26
first we, I mean you can do it however you want, you also can write of course a small client library, but here we can just use curl, it's quite convenient to have the body we want to send
26:42
in a small file called, in this case, my postgres.json, it's a quite good idea because then you can check in this file into your git, so you know how you created your Postgres instance. So with the curl command we just say okay,
27:01
it's content type JSON, we want to do a post on the collection resource, we know what happens, it should create a Postgres instance. And really we get back this small JSON snippet and it says okay, there is a Postgres instance running, the ID is number one, it's the first one,
27:22
we also have the Docker container ID here and the Postgres password on the terminal, so maybe keep it, make it so, okay, that's it. And really we can, one important thing is the port here,
27:41
so we get dynamically assigned a port of this Postgres instance, so everyone who creates now on this host Postgres gets another port, but we can connect via this small command line toolkit for Postgres, here it's local host, but the port we got back here with our user
28:02
and our password and ta-da, we have a running Postgres, we can delete it or we can use it in a small application or in some tests or whatever. Okay, I think that's nice, so a short summary, I put it on GitHub, you can have a look there,
28:22
the contributions are very welcome of course. It's just an example, but really it is running now in our company since a year, it's just my hobby project, but yeah, it's really running, so many, many integration tests use it
28:40
because it's really convenient, you can have a new and completely virgin database for every test if you want to, it's quite fast, so you can have one in let's say at most a minute until the Postgres is really up and running, but mostly it will be some seconds.
29:02
For small experiments like I can use here, all you need now is a Postgres, yeah okay, now it's easy. Maybe we also use it for productive services anytime soon, of course it's not bulletproof, but there are ways to overcome also if this thing is not really the best quality on earth.
29:27
Yeah, so far we created I think something like 200 Postgres instances and when I looked it up, it was really 50 Postgres instances running on a two-core, two gigabyte machine, so it really doesn't need much resources.
29:43
It's really nice, you can try it, you can have a look at the code, but, and that's the reason for this talk, you should not just say okay there's this one thing, but it's really about you should take home after this talk, it's really easy, we don't need RBS services for everything
30:01
and wait until they are delivered and exactly doing what they should do for you, but you can do it for your own, you have all the tools now at hand, so as an inspiration, for example, you can take a manual step in your delivery chain,
30:20
so you want to deploy an application and there's always this nasty step where you need to write a ticket to your sysadmin, for example because you need a directory on a remote file server you don't have access to, so you need to write a ticket, can you please make a directory there because I want to put my backup there. Okay, small blueprint, now we have already an idea,
30:44
maybe we use Flask with the REST API would be a good idea, for this we don't need Docker, we can just use the native file thingy, make deal or, and for delete, shutil, rm3,
31:00
for create and delete, update, we don't need it, then we design the API, what do we need, maybe a name of the directory, maybe somehow the Linux username, I don't know, permissions, quota, it's up to you, it's your project, but it's really trivial to implement
31:21
and all you need mainly is something like a rainy Sunday afternoon, so you type those 100 lines of code and you are done and you can and will impress your colleagues with that. Okay, so that's it, a small summary,
31:40
a wrap up, so what you should take home today, CRD is for machines, CRUD is for humans, CRD is trivial to be implemented as a REST API, my hypothesis, most infrastructure can be expressed as CRD, that means we can most infrastructure expose via REST API,
32:04
if it's only CRD, it's immutable, and that's good for machines, and it can be consumed from applications as a self-service and why is it important to be as a self-service?
32:21
Yeah, we can put it in version control. Yeah, and the last thing, Postgres is a nice example, that it seems to be somehow true, what I said here, and yeah, that's it, thank you, as I said, I'm working there, that's our stand, come over.
32:50
Are there any questions or comments? Thank you for the talk.
33:03
It's interesting that you chose Postgres as an example, because I can rather imagine that all other services are easily teared down and built up again, but a database is containing data, so somehow the not updating principle doesn't really work for databases, does it?
33:24
Thank you for the question, I give you money after. No, so it depends, for example, I have a small internal data service, it's the weather service, and I really implemented it the way that it's a firmware database, so the real data is stored as CSV files,
33:42
and I optimize the import of the CSV files, and normally I can do it with an update, if it's a trivial update, like maybe adding a column, but sometimes if it would be something else, I would really just, for example, if now the machines can do it,
34:00
so it's cheap to build new ones, so I can build a complete clone of my application, wait for two days until the whole data is reinserted in the new state, let's say in the beginning it was one table, afterwards it's two tables, it would be really hard to do it in one Postgres to extract the data such that now there are two tables
34:22
from the content from one table before, so I really can have a clone, put in the data there, if everything is up and running, I can delete the old one, and the new one is the next productive thing, so it works, sometimes the only issue is performance or resources, if it's not possible to have a clone of your system,
34:42
then you need to have it like this. Please, people try to be quiet while leaving. Okay, there was another question, but you already answered it. We have two more questions.
35:04
I'm interested when you have this control over your instances and something like that, have you come to a problem, for example, when you have too many instances for the automation or something like that to set up that you need a solution to actually easier control all the instances
35:23
or have you not come to that problem yet? I mean, and to share it with all the colleagues, so you don't have to remember everything and you can, any execution tool from where we can trigger, I mean, if you have, for my example, I'm working similar solution and we have virtual machines and I have around 200 virtual machines
35:42
and now I have the problem how to control them from the point. Did you think of anything from this point of view? No, so for this small example, not yet, but of course it would be possible. So if I understood correctly, it would, for example, if the host would be really, if it would go out of memory or something like this,
36:03
so what? It would be different for each example that you just want to test or something like that. Yeah, so as I said, Docker is a quite nice thing. There is this Docker swarm where you can put a whole cluster behind it
36:23
and it would be, it would, I guess, and I never tried it, but I guess it would work out of the box when I just connect to a Docker swarm thing and then I could horizontally scale the whole thing. So there's, I would just then need some monitoring how, what's the load on the whole cluster,
36:42
but I could just add then machines to it. But it's not done so far. It's just, it's a small example and yeah. We have one more question. So when you delete the Docker container,
37:03
it may leave a lot of garbage in the system. For example, like unused volumes and stuff like that. How do you deal with garbage collecting? Not yet. So it's really, first of all, there are no mounted volumes.
37:21
We could do it, for example, mount a data volume of the Postgres on the host so that even we could reuse the data in the database even if we deleted and created it, but it's not done so far. So it's really ephemeral and I don't know. So I never had problems so far. It is running on this system since November or something
37:41
and memory is still fine. No problems so far. No, I mean, for example, the Postgres image that is hosted in the Docker Hub, the official Postgres image, it has the volume directive in the Docker file. So it means that when you create the container
38:02
from that image, the volume is created for you and the volume, this file system, it lives somewhere deeply in the file system of your host system. And when you delete the container,
38:21
the volume that the file system is still there. It is still on your host system. So sometime in the future, you will encounter that the storage of your hard drive is out. Yeah, so in principle, that's really, it's easy to answer.
38:42
I delete the EC2 instance and create a new one. Yeah, that's it. So you're talking about we are updating the thing and the garbage grows and grows. It's exactly if we work in this human approach where we just incrementally change things.
39:01
If you really delete and wipe everything and recreate, if we grasp that thinking that deleting and recreating is better, then most of these problems vanish from just itself. So maybe it was a joke a bit. I think there are ways, but mostly I would write a bug issue for Docker.
39:23
That's the thing I would implement. Other question? We have probably time for one more question. If this is not the case, we thank Ian and Peter. Thank you.