Continuous Delivery starts with Continuous Infrastructure
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47239 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020366 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Open sourceOpen setGame controllerOpen sourceWordEvent horizonSoftwareData managementSlide ruleSelf-organizationConfiguration spaceComputer animation
01:35
MeasurementShared memoryFocus (optics)Self-organizationMereologyMeasurementComputer animation
02:06
Continuous functionConfiguration spaceData managementProcess (computing)Software testingOperations researchCollaborationismSoftware developerKolmogorov complexityCodeOrder (biology)CollaborationismComputer animation
02:51
Continuous integrationContinuous integrationAnalytic continuationMultiplication signMultiplicationComputer animation
03:27
Continuous functionBitComputer animation
04:10
CodeBlogFraction (mathematics)Operator (mathematics)MereologyCodeComputer animation
04:51
Software developerForcing (mathematics)AutomationSoftwareMathematicsFirewall (computing)XMLComputer animation
06:00
Operations researchStability theoryComputing platformSoftware developerOpen sourceOperator (mathematics)ScalabilityComputer animation
06:39
Strategy gameMusical ensembleDifferent (Kate Ryan album)Combinational logicOperator (mathematics)Computer animation
07:17
Different (Kate Ryan album)Personal digital assistantMereologyState of matterSelf-organizationSystem callMetric systemDifferent (Kate Ryan album)Archaeological field surveyOperator (mathematics)Computer animation
08:01
Moment (mathematics)Chaos (cosmogony)Asynchronous Transfer ModeCASE <Informatik>CodeSoftware testingBuildingMachine codeLevel (video gaming)Software testingComputing platformModule (mathematics)Software developerChaos (cosmogony)Bridging (networking)Product (business)Multiplication signSimilarity (geometry)Integrated development environmentCodeOperator (mathematics)Data managementMoment (mathematics)Game theoryOnline helpSoftwarePoint (geometry)Self-organizationStandard deviationUniform resource locatorComputer animation
10:27
Software testingJava appletSoftware developerBuildingCASE <Informatik>Projective planeDifferent (Kate Ryan album)Medical imagingCASE <Informatik>Revision controlJava appletPhysicalismSoftware testingIntegrated development environmentSoftware developerReverse engineeringCodeComputer animation
11:12
Software developerCollaborationismAutomationCASE <Informatik>Software testingChaos (cosmogony)Operator (mathematics)Arithmetic progressionCollaborationismSoftware developerGoodness of fitCASE <Informatik>Software testingStability theorySelf-organizationComputer animation
12:12
Focus (optics)CASE <Informatik>Operator (mathematics)CodeProjective planeProcess (computing)NP-hardFigurate numberSound effectProduct (business)Group actionPasswordDifferent (Kate Ryan album)RootComputer animation
14:21
Machine codeSoftware testingSoftware developerCASE <Informatik>Self-organizationSoftware testingKanban <Informatik>Operator (mathematics)CodeMultiplication signAnalytic continuationComputer animation
15:02
CollaborationismCASE <Informatik>Process (computing)Transformation (genetics)UsabilitySoftware developerSelf-organizationCASE <Informatik>Operator (mathematics)Computer animation
16:03
AverageData managementDesign by contractSoftware developerStrategy gamePoint cloudCASE <Informatik>Software developerWordOperator (mathematics)Web pageAverageOrder (biology)Data managementCubeCodeDatabaseProcess (computing)Computer animation
17:11
Continuous functionCASE <Informatik>Machine codeMetric systemBuildingOperator (mathematics)Bit rateOrder (biology)Computer animation
18:13
Software developerOperator (mathematics)Set (mathematics)Computer animation
18:55
Revision controlControl flowRepository (publishing)Software testingMachine codeRevision controlSoftware testingCodeHidden Markov modelSelf-organizationOperator (mathematics)Computer fileAverageComputer animation
20:16
Operations researchSoftware testingSoftware developer1 (number)Multiplication signComputer animation
21:05
CodeConfiguration spaceMachine codeData modelInformation securityRevision controlSoftware testingOrder (biology)Computing platformCross-platformSoftware testingMultiplication signCodebuchInformation securityConfiguration spaceCodeEndliche ModelltheorieService (economics)State of matterGastropod shellIntegrated development environmentCartesian coordinate systemConnectivity (graph theory)Scripting languageComputer animation
22:21
Software testingSource codeBuildingScripting languageRevision controlBranch (computer science)Variety (linguistics)Continuous integrationSoftware developerRevision controlOperator (mathematics)Data managementComputer animation
23:06
Formal languagePattern languageData managementElectronic mailing listModule (mathematics)Operator (mathematics)Different (Kate Ryan album)Multiplication signRight angleConnectivity (graph theory)Repository (publishing)Computer file
24:54
Data managementSoftwareContinuous integrationSoftware engineeringExtreme programmingSoftware developerDisintegrationCodeGoodness of fitSoftware developerCodeOperator (mathematics)Axiom of choiceData managementBranch (computer science)Analytic continuationIntegrated development environmentSubsetComputer animation
26:28
Software testingCartesian coordinate systemParameter (computer programming)Configuration spaceCodeData structureIP addressComputer animation
27:35
CodeSoftware repositoryMachine codeSoftware testingSoftware testingFormal languageCartesian coordinate systemCodePattern languageComputer animation
28:25
Repository (publishing)Data managementSoftware testingCodeSoftware repositoryMachine codeBuildingRight angleCartesian coordinate systemSoftware testingComputer animation
29:02
AutomationSoftware frameworkSoftware testingPhysical systemIntegrated development environmentReplication (computing)CodeCartesian coordinate systemComputer animation
29:36
Software testingSoftware repositoryCodeMachine codeIntegrated development environmentComputer animation
30:10
Stack (abstract data type)Projective planeCodeProcess (computing)Computer fileConfiguration spaceOperator (mathematics)Library (computing)Java appletSoftware developerInformationOnline helpMetadataPoint (geometry)Formal languagePattern languageMixed realityComputer animation
31:40
Software testingLevel (video gaming)Revision controlElectric currentVirtuelles privates NetzwerkFirewall (computing)Information securityStack (abstract data type)Open setPoint cloudRevision controlPoint cloudCodeSoftware testingMultiplicationWritingInformation securityConfiguration spacePattern languageDifferent (Kate Ryan album)Computer animation
32:48
Computer-generated imageryMachine codeDatabaseInformation securityComputer clusterBackupDifferent (Kate Ryan album)Computer animation
33:22
Open sourceScripting languageCASE <Informatik>CodePoint cloudSoftware testingCubeStack (abstract data type)Open sourceGame controllerSoftware developerCartesian coordinate systemMultiplication signLine (geometry)Pattern languageComputer animation
35:09
CASE <Informatik>Software developerBuildingRight angleSoftwarePoint (geometry)Repository (publishing)Revision controlArithmetic meanUser interfaceProcess (computing)Pattern languageBitModule (mathematics)Multiplication signFormal languageInternetworkingOpen sourceCodeSelf-organizationClosed setGravitationData compressionComputer animation
39:05
Open sourceScripting languageConfiguration spaceMedical imagingData managementCartesian coordinate systemStack (abstract data type)MereologyDifferent (Kate Ryan album)1 (number)Level (video gaming)Computer fileLine (geometry)InternetworkingClosed setCubeComputer animation
41:03
Point cloudOpen source
Transcript: English(auto-generated)
00:05
So it's 9.02, and the room that nobody can find is full. Welcome to FOSDEM. My name is Chris Bertaert, I'm one of those weirdos who's been coming to this event for 20 years,
00:23
which means I actually have never been in this side of the building, because they only started using it this year. What I do on a daily basis, I help organizations to leverage open source to deploy software faster, to actually achieve their business goals faster. Somewhere about 10 years ago, we figured out a name for that.
00:42
We called it DevOps. Since then, people started abusing that word for a lot of things. And in my spare time, I organize a bunch of other conferences. So if you want more of this fun stuff, this is actually the only pitch I'm gonna do. Monday, Tuesday, Wednesday, we have this other small event in Ghent, which is config management camp.
01:01
It's basically the same as FOSDEM, only focusing infrastructure. And you're all welcome. We're sold out, but we don't check tickets. We basically do crowd control. So just show up if you care about this stuff. For those who have seen this slide before, they might have noticed this new thingy here. DeliveryConf had its first edition last week in Seattle.
01:23
It was awesome. And the idea currently is that we might run one in Europe next year. So given that you folks are interested in getting continuous delivery and stuff, you might wanna show up. I don't have much time, but I wanna focus on what is this DevOps thing.
01:41
To me, it's not the automation part. It is mostly the culture part. Damon and John have been talking about cultural automation measurement and sharing. There's a couple of things. There's no such thing as a DevOps engineer. If you're a DevOps engineer, your organization basically does not understand DevOps. If you think you can get certified in DevOps,
02:01
you've been scammed. That's not what DevOps is about. It's also not the same as continuous delivery. I think you need to do something like DevOps in order to achieve continuous delivery, but you could definitely do something completely different
02:21
and have a healthy culture and still achieve your business value. But if we talk about continuous delivery and what basically is the topic of this talk, it's this. It's the idea that we can constantly deploy software, stable, reproducible, fastly, and we typically achieve that through collaboration
02:42
between all of the people involved. And that smells a lot like DevOps. So how do we get there? Well, I could pull a just humble trick here and ask, who's doing continuous delivery? Raise your hand if you're doing continuous delivery.
03:05
So who's frequently committing to master every day, multiple times a day? The hands that drop, they are not doing continuous delivery because they're not even doing continuous integration. They're doing continuous disintegration.
03:23
And you need to do continuous integration before you can achieve continuous delivery or continuous deployment. And also there's nothing wrong with maybe having that manual step in there if everything else is automated. So what is this thing about?
03:41
What are we talking about? Today I wanna talk to you about how to get there and how not to get there. And I'm gonna talk both about a couple of use cases, stories I've had, experiences I had with larger and smaller organizations, but also a bit of deeper dive
04:03
into how to do this technically. So what is the actual goal we wanna achieve? This is part of the release notes of Deployinator,
04:24
which is a tool Etsy wrote 10 years ago. And we're still not there yet. But this is basically the goal we wanna achieve. We wanna have the ability to deploy code quickly and safely.
04:41
And a lot of people have tried to achieve that. And a lot of people have succeeded, but many more are failing. And I think partly that is because they haven't involved the operations people first. They've basically been trying to do this as something which is a developer only story.
05:02
And I don't get this because if you look at the operational part, for years we've been telling developers like you're not gonna push this to production. You're not gonna deploy this live. You're not gonna do all these things manually. You're gonna have testing in place. We're gonna have automation in place. And we're gonna force you to go through a pipeline
05:22
and automate everything. But on the same time, we've tolerated that. Ops folks were like, I fixed it. Or let's do this manually. Or let's change stuff there. Or reconfigure a network switch or add a firewall rule. And we've been tolerating people to just do that manually.
05:42
Who's still doing that? Why? The people who have worked with me know that I don't tolerate that. And I haven't tolerated that for the past decade. And it's hard. And if you look at the history
06:00
of where this DevOps movement comes from, if you look at the European part, we were mostly senior Ops folks. People who had been, I mean, I've been coming to Fosdum for 10 years. I did open source before that. And we started DevOps Days 10 years ago. So we were doing more than 10 years of open source stuff. Some of us 15 and more. And we were starting out from an operational point
06:20
where we were seeing bad quality artifacts from developer teams. We were having a lot of pain with stability, with scalability. And we were like, nah, this is not how it works. And on the same side, we had the American people. We were like, yeah, but we need faster deployments. And the developers were pushing this. And then somewhere in the middle, we met. There's also the difference in culture
06:42
between the American startups where there's vulture capital, which basically pushes companies to go fast and burn. Whereas the Europeans are more looking at, we have a sustainable business. We're gonna be around in 10 years and be supporting customers. So the combination of breaking things fast,
07:01
but still being resilient and knowing how to run things, that is to me why a lot of people have been tolerating ops to do things manually, as opposed to really doing infrastructure as code, really doing automation. And I have some stories, some cases,
07:22
which basically back that feeling that you need to start with the operations part first. They're not as scientifically measured as Nicole Forgson's state of the DevOps survey where she actually has good metrics. But I've seen this throughout the journey with different organizations. And I wrote down like three of them.
07:43
I hope that's not somebody on call. It's 9 a.m. on a Sunday morning. That must be painful. I'm actually looking in a room to see if there's people from the customers in the audience. I saw somebody outside, but he's not getting in, so that's good. So yeah, let me start with a couple of those cases.
08:00
And the first one is a situation where I walked into an organization and they were in complete chaos. They had about 300 nodes. They had no clue on how they built them. They had no clue how they deployed them. They had a Jenkins hiding somewhere under a developer's desk. Who still has that?
08:20
Not that many people. That's good. The operations people were mostly fighting fires. There was zero standardization and every artifact that was being deployed had different naming conventions was deployed in a different location. It was, well, you know, hell as we know it. And their tipping point was the moment when they had a really, really huge failure.
08:43
Their senior management realized if we have this failure again, it's game over. So they sought help. And what we started doing there was the first thing we needed to figure out is how do you build your software? How do you make sure that your software is reproducible?
09:01
And how do you actually achieve being able to build your build environments again? So we started teaching those people. They were already using Puppet as an infrastructure as code tool. And we started teaching those people, like, how do you build your own CI stack reproducible so that if it breaks, you can still rebuild it
09:21
then you can guarantee that the next time you spin something up, it's gonna be similar or extremely similar to what you built before. And we taught those folks, like, testing. How do you test your things? What they were doing before was they had 300 nodes but had no development platforms. They were just hacking Puppet's code in production.
09:43
And Puppet applied and it broke. Oh, so the idea there was we start with building the infrastructure there in a development environment. And then teaching them that you can promote things to a new environment when things work.
10:00
So we split out the configuration out of the code. We started to teach them how to use the pipelines they were actually building for the developers. And slowly, you saw the developers starting to actually contribute code to the Puppet code base. They were writing tests. They were writing their own modules. And we saw that because they were now using the same tools,
10:21
the bridge between those people was being closed. The gap was closed and the bridge was made, sorry. So we went from a zillion different GDKs to one GDK and one GBOS version. We called this project initially Project Dolly because we wanted to be able to clone
10:41
the different environments, not as in a physical image clone, but we wanted to have the same environment for everybody. And we tried to shoot for like 90% reversibility because that was the initial goal. We knew there were gonna be a bunch of edge cases. And like I mentioned, eventually we had Java developers
11:00
who were contributing back to the RSpec code base to actually having tests on infrastructure code so they were much more sure that what was being delivered was actually what they wanted. So we started with telling Ops folks automate. And they learned the same tools as developers.
11:20
And eventually collaboration improved, progress improved, and they were happy. They had much less downtime, much less issues, and much better collaboration. The second case I went in was completely different. They were already doing CI, they claimed.
11:43
The developers had a bunch of tests. They had some things under their hood, hiding under desks. But the people that they called operations, they had like, yeah, but we cannot deploy this. We don't know where it's coming from. But those developers said that we have good test coverage and we have everything in place.
12:02
But they still had no way into stable deploys. They could not automate their deployment. It was painful. It took me about 18 months in an organization to finally find somebody who I would consider an Ops person, I would actually say, hey,
12:21
if you're looking for a new job, I hire you. And all the other people in that room were basically people who were, they were called Ops, but they couldn't figure out the difference between a root password or a username. It was that bad.
12:43
It was mostly when we eventually found people senior enough to do operations work. It were the Brents, if you've read the FreeNIS project. It were the people who were firefighting, harding knowledge, and they were not being able to move forward. So what we did there is we found a couple of them
13:03
and we moved them out of their office, out of their day-to-day job. We actually moved them to a different city. And there we started teaching them agile. We started, they were longing to do infrastructure as code for years, but they weren't allowed to. So we started them in their own group
13:22
where they couldn't be disturbed by other people. The side effect was that the other people that were trying to get their resources were finally learning how to fix things themselves. And they started to build a pipeline. They started to do CI-CD on their own code base,
13:42
on their own tooling. And they started to think about, hey, what is our way to build something that ends up to production? And even discussions like, what is production? If the ops teams deliver production, is that production for them or is that production for the developers? Those kinds of discussions started happening.
14:02
And after those first successes, we slowly started moving those people back to their teams so they could take the practices they had learned and they could actually help their own teams back in adopting them. But that was like 18 months after we started working with them.
14:22
So over a period of about six months, we ended up having the really old school gray beard people the people were just doing AIX and said, I'm gonna go retire in six years and I'm not gonna learn anything new. And we converted those people to actually being agile evangelists. Those were people who now were running in the organization
14:41
and saying, hey, you should do this Kanban thing because for operations, it might be better than Scrum. And those were the people who were actually writing code tests, tests for the code. So it took them a much longer time to have teams arrive at continuous delivery because they didn't get the ops folks involved first.
15:05
It took them more than a year longer. But once they were there, things started moving. So you can guess what the third case is, right? An organization that still refuses to involve the operations people.
15:22
They claim to do DevOps. They claim to have been doing it for two years. And they have a DevOps team in the middle, which is dictating tooling. Tooling they never use. Tooling they don't know. So the developers, they basically complain about,
15:41
well, yeah, we have these tools set here, but we cannot use it because it doesn't fit our needs at all. They're broken. They don't deliver what they should deliver. And basically what they achieved is that everybody's working around the ecosystem that has been built and still doing things manually because the tools that they're given
16:00
is completely broken. You're still hiring DevOps engineers and the average DevOps engineer who has a clue and accepted the fact that the job title was wrong but he might be able to help stays for about two months and then leaves. Your senior IT management has left twice over the past three years.
16:21
And they have a team where basically there's only junior in-house analysts that stay and all the rest of the crowd is external people. And by the end of the year, by October, November, they're out of the budget so they don't write any code in the last quarter because they don't have any developers anymore. So they're gonna fix this by adopting the cloud,
16:42
which means they have some cube deployed somewhere and they're moving their Oracle databases in there and they're wondering why things don't work. Oh yes, you still need to fill in a four page Word document in order to be able to deploy something. And they still haven't involved the ops folks.
17:02
Actually it's worse. When I was working with them, I wasn't allowed to talk to the ops folks. So if I look back at these journeys and those are just snapshots, the only thing I can conclude is the earlier you have those ops people involved,
17:22
the better your CI CD story is gonna be. It's gonna have a much higher rate of success and it's gonna be much smoother. And if you think about it, there is a reason because people who run ops need to be able to support what the other people do.
17:40
And in order to be able to support these things, they need to understand what they do. And if you've never done this before, how can you do this? If you've never driven a car, are you really gonna tell people how to drive a car and help them fix the car? If you don't know what a steering wheel is, if you don't know what a gas pedal is, you won't.
18:04
And you need those people involved so you can unblock delivery, you can unblock provisioning and can make sure that everything you want is built in. And putting up a bunch of tools is not gonna solve that. Shouting that you're doing CI CD
18:20
is not gonna solve that. So the culture hack I have for this is, well, if you teach your ops folks to set up CI CD for your CI CD infrastructure first, they're gonna understand how this work. And they're gonna be able to coach and teach your developers into doing exactly the same thing.
18:44
And while you do that, you give them a common set of tools, a common set of terminology, a common set of it's broken, let's fix it together. So how do we do this? Well, if you look at older organizations,
19:04
you see it in a CI ecosystem. There is tool for version control, deployment tooling, build tooling, you need artifact repository, you need code coverage tool, you need test tools, you need a lot more than is listed here. And if you talk to the average ops person in a legacy organization,
19:21
how many of these tools do you think he's been using? All? Two? Two, zero, all? All of them, then it's not a legacy organization. Then it's probably an organization that already moved further. But you'd be happy if they use version control. Testing, hmm.
19:42
And I had a friend who tweeted last week even that he was working with an organization and their version control was file names on Dropbox. And it is 2020. So a bunch of these tools, those people have never seen,
20:01
but they are expected to manage them, expected to run them, and they don't know how they function. And that's already the first challenge to overcome. We need to have people who understand how these tools work because they have been using them. So a lot of people still argue against doing CI because of these reasons.
20:21
They don't know what the tools are about. They don't know how they work. And they're like, yeah, but this is costing time. And well, we don't have tests anyhow, so why would we spend time on this? And the biggest ones is we don't have budget for this. We don't have the money for this. No, that's because the money is hiding somewhere
20:41
and you're spending it anyhow, but you're hiding and spending it in firefighting rather than building something resilient. Or there's a different budget. The budget that the development teams need to do quality insurance and to do all of these things are not theirs. So it's a really painful thing. And then there's a fireman for Git projects, obviously.
21:02
So a lot of teams say, yeah, we do continuous delivery because they don't wanna spend time in building out multiple platforms. Yeah, for those in the back who cannot read it, it says everybody has a testing environment,
21:21
so people are just lucky enough to have a mother platform that's not called production. So in order to get there, we need to do automation. We need to do infrastructure as code, and we need to look at what we built, and we need to write code.
21:40
And with me, code does not mean I'm gonna write shell script and convert them to YAML. I wanna basically define the state of an infrastructure, and I need multiple components to this. I need tools that do orchestration. I need tools to do actual desired state configuration. So you're typically looking at modeling your infrastructure, writing code books or manifests
22:01
or whatever you like to call them these days, and have different environments for your infrastructure. So you can also promote the artifacts you deliver and ship them to different stages. And then you eventually end up with a working service which has the application code, the infrastructure code, and security and monitoring, and everything is built in.
22:21
This is a trivial one, it starts with this, but lots of ops folks still need to be taught on how to do version control. Actually, lots of developers still need to be taught on how to do version control. Who has multiple branches here that live for longer than a day?
22:42
Those are the people who are not doing continuous integration. Who has different tools to do dependency management? Who understands how to do dependency management? One person understands how to do dependency management. Two people. The thing is there is a huge variety of tools out there.
23:05
It's so simple if you understand how it works. So who knows what's on the screen? What's on the right side? On the right side we have something that looks like librarian file where there's basically a module defined with a git repository and a reference.
23:23
And the Ruby ecosystem has some things like that, the Puppet ecosystem has some things like that. Every single language invents their own dependency manager. What's the thing on the right side, on the left side? It's exactly the same, and I hear somebody who actually saw what it is,
23:41
but it's exactly the same. It's a list of a module with a reference to something in git. These are git submodules. They do exactly the same as the thing on the right. With one big difference. This is gonna work for every single language out there. This, you're gonna have 35 different languages
24:03
doing 35 different things, and each time an ops person needs to help a tool, it's gonna be like, oh, wait, there's a different variant of how I'm doing things. Because all of those tools, all those people who have been building dependency management tools are basically wrapping on top of one basic simple component and making it harder for people to understand.
24:24
And having those discussions, operations people between developers, and realizing, well, this is the same pattern, but we're just using different tools. If you have that discussion, you can help each other understand it. You could use the same pattern
24:41
and use this for different languages, or you could understand, that's what you're doing, that's what you're trying to achieve. But in a lot of cases, people don't understand the goal of the tools they're using because they've never had to, unless you understand good swap modules, obviously.
25:01
So people never done this before. People have seen a zillion branches in their ecosystem and they've never understood. And when you think about it, the branching makes it more difficult. So anti-patterns started happening.
25:22
I had a really good discussion at delivery conf with the folks from Puppet Lab about eight, nine years ago, they figured out their release management tool to do deployments on a Puppet Master was, well, we have this tool card, r10k, and you're gonna have a branch for your development environment, a branch for your acceptance environment,
25:42
and a branch for your production environment. And I remember being in the room with Jess Humble when they introduced that, and we were like, we're gonna teach the operations people how to do it the wrong way. But their whole ecosystem was built on,
26:01
you use r10k and deployments of your code base are gonna be fine, it's gonna be smooth. Only last week, I was talking to the people who were now rethinking their new continuous delivery tool for the Puppet code, and I was like, don't use the tools that already exist, don't build your own tool, but they started to realize that maybe r10k was not the best choice.
26:22
Maybe there's other tools around that could do exactly the same. So if you look at a typical infrastructure pipeline, what is in there? Configuration, why would there be configuration in a pipeline?
26:51
And how are you gonna test completely different parameters for a stack?
27:01
Configuration is, to me, actually the only thing which is not in a pipeline, because it's close to non-testable. You can test the syntax, you can test whether the structure of the configuration is correct, but you're testing the code that needs to apply the configuration, but you can never test the sizing exactly, and the IP addresses exactly,
27:21
because if you can, for example, have the exact same IP stack, and you're testing everything, then it's actually breaking your production, because now you have duplicate IPs. So configuration is one of the most difficult ones. No, the thing that's in your pipeline is what I was trying to achieve. It is literally the same as what should be an application pipeline. You should check out your code, you should do some syntax checking,
27:41
some style checking, do code coverage. Do tests if, depending on the language you use, you need to build it, you need to build it, you need to do more tests on the artifact. You need to package it, you need to upload it to a repository, and then you need to deploy it.
28:00
And that's literally the same as you need to do in every other language. Hopefully there's more and more testing, but yeah. And every artifact is like that. Your application code, your info code, your metadata, and your tests, they all are artifacts that you want to ship through a pipeline.
28:25
And you wanna have patterns where you basically have development, sorry, development, acceptance, production, or whatever values. You wanna pull things from the internet, you wanna have local mirrors for every single artifact. All right.
28:41
So, upload to a repository, test. And if you're doing infrastructure, you do things like you check your puppet runs if they're still clean, you check if your Ansible stuffs all still works, it's not breaking, you check your monitoring. It's like your smoke test with the applications.
29:02
And your monitoring is crucial. You deploy a host, you deploy an application to a host, you leverage infrastructure as code so you automatically reconfigure your monitoring, you check your monitoring and it should stay healthy. And if you see that your monitoring breaks,
29:21
then all your replication broke or your complete system broke and you should not push to the next environment. And all of this is automated because that's what infrastructure as code brings you. Those are the side effects. So, you promote to the next environment when your monitoring has decided that things have not been broken
29:42
and you increase testing. This is what we do for applications, right? This is what we've been doing for applications for years. And then you go to the next target and the next target. And the more of these environments you need to build,
30:01
the more you end up with, how do I do this? The way we do this still, and we're still really happy with this, is we have a large Jenkins job DSL code base where we do pipeline as code for our infrastructure as code. We basically have one large C job
30:21
that pulls in a bunch of config files for all the projects we have and it generates our pipelines on demand based on a config file. But we have a bunch of libraries that we reuse for every single pattern. A checkout job is a checkout job because if it's Java code or if it's PHP code or if it's Puppet code or Terraform code, I don't care.
30:44
It is a checkout job. Everything we deploy is an RPM, whether it's Puppet code, whether it's Java code or Python code, they're all the same. And the patterns I build in there, like all the metadata I wanna know, where it's coming from, which build it was,
31:01
all of that information is the same. It's the same library. So what we created is a library that's being used by both the developers and the ops folks to build their whole CICD infrastructure. And they have a shared language, a common language that works for everything.
31:21
There's a mix of pipeline DSL and job DSL and job works. But it's the same for everybody. And that's the strength. It's not one team using one thing and then the other team going like, oh, I cannot help you, I don't know how this works. And we even come to the point where we can,
31:40
in parallel, test multiple versions of our code base because we spin them up in containers and we have different versions of our PHP stack, but also of our Puppet code base and their test. And we don't look at the next version because we don't care yet if it's gonna break. But if it doesn't break, we know that we can upgrade.
32:03
And then people come, yeah, but we're in the cloud. I don't see the difference. I mean, you're still gonna write a bunch of things. You're still gonna write Terraform or Chef, whether you're on-prem or in the clouds. You might be talking to a different API.
32:22
Might not be talking to the OpenStack API, but to EC2. And if you need to configure VPNs, you're gonna need to configure VPCs on the other side. You're gonna have monitoring security in there. So all of those patterns which you've been doing, whether you have a Terraform pipeline or a Puppet pipeline, doesn't matter.
32:41
It's the idea. You have to have the same tool base. You have to have the same ecosystem. But we're using containers. There's nothing different. You need to test this even stronger. We need to figure out better ways how to deploy this these days. Because Docker pull YOLO,
33:03
and we still have to ask all of these questions again. Because people who have been just doing Docker on their laptop, they're container naive. They haven't figured out that this is the gap they need to cross. So there is a huge challenge for us out there.
33:22
Because we're an open source conference and I'm still struggling, maybe you don't, on how to do CI-CD on my whole CUBE ecosystem. I wanna be able to spin up CUBE stacks automatically, out of the box, hands off, from source code. Not the CUBE source code, but some infrastructure code.
33:42
It's easy if you do it in a public cloud. Anybody got a solution for me? It's way too early on a Sunday morning, or nobody actually has a solution. So how do we do this?
34:01
How do we do this in a container ecosystem? That is the challenge that I have for you folks. What I still see is people having 30,000 lines of Bash, calling kubectl or calling Helm. That's not how we want to work. That's basically going back 10 years
34:23
and not having tests, because how do we test this? So infrastructure as code, to me, is the first thing you need to do if you want to achieve continuous delivery. And if you want to have full control over the whole stack, it just got harder,
34:42
if you add containers to that. But it still is something that you need to spend time on, because you want to be able to understand the same patterns and the same tooling that the developers that are building applications on top of it need to understand.
35:03
And I hope the three cases kind of prove that. So now it's time for the rest of FOSDEM. Questions? Yes.
35:29
How do you prevent organizational gravity from taking over? Okay, so the question was, in case two, it seems to be an organizational problem. How do you solve the problem
35:41
that if you move the people back in the team, that they'll just get absorbed by the organization again? That's right. Well, it always is an organizational problem. It always is a culture problem. And the way you solve that is by having C-level buy-in. We pull those people out, and the only way we could pull them out
36:01
was because C-level was actually pushing this and agreeing on this. So when they came back to their teams, they were enabled and they were blocked from doing their old work again, but they were supposed to help the people and help their teams.
36:21
And because they had been gone for close to three months, a bit more even, there were people who actually filled in the gaps of the problems they were solving before. So they weren't being bothered that much anymore with their old jobs. They were still, for the highly critical things, they were still being consulted and were still being asked to help. But it wasn't their daily job anymore
36:42
because they were gone for three months, so some other people had taken over a couple of things. Other questions? In the back?
37:06
So the question is that if he understands correctly that I suggest to use Git sub-modules for dependency management? The answer is yes. The main reason to that is because all the other tools around
37:21
basically try to reinvent exactly the same thing. The second reason to that is because if you do that, you get exactly the same pattern that the developers will be using to do the same. I mean, I've seen a bunch of user interfaces build on top of these things, which are all horrible.
37:41
And what I've seen is once that people start doing this, doesn't matter which language they use, they start to care about where their source code is. They start to care about, oh, I'm not pulling in random things from the internet that might be disappearing when somebody actually decides to unpublish a node module. No, they have the code locally, they can rebuild.
38:01
So people start to understand what they're building, and also they know that when they go back in time and check out another commit, they will have close to reproducible builds. Combine that with being able to build your infrastructure and your CI infrastructure, the fact that they can do exactly the same means that at some point in time, I can actually, theoretically,
38:22
because I've never been able to spend the time on that, go back and say, I want to have the build ecosystem we had three months ago because I have the versions checked out and I have the repositories checked out, and then I wanna be able to build the software we built then. You have everything in place. It's not gonna be trivial because typically you don't have enough resources
38:40
and enough time, but in theory, that's what you should be able to do it. Yeah, it works very nice with open source because you have the source and you keep a copy. This is an open source conference.
39:12
Infrastructure deployment or application deployment? Both. So the question is, on this slide, the container ecosystem is nowhere close to the level of automation reproducibility
39:22
where I used VMs. And if I want to talk about application deployment or the actual ecosystem, and to me, the answer is both. I'm still looking, and I've been looking for close to four years for a fully reproducible automated way
39:40
to deploy a full cube stack with everything in the picture. Most of the things that I see just pull random things of the internet. Totally not reproducible. That's one part of the problem. The ones that do work are really cloud-focused. They only support AWS or whatever, and not on-premise deployment. And then on the other side, on the application level,
40:03
we have a bunch of package management tools that are okay-ish, but configuration of the applications inside, there's nothing. Helm is a package manager. It's not a config management tool, and that's the closest thing to reality. We exactly say in master, okay,
40:39
this image is there, this image is there,
40:41
with this configuration, and you can tweak, not only in Helm, but in Helm file as well, for different environments, for example. So how long is the comment line you used? Sorry. Sorry. Sorry, sorry a lot. We are finished. Thank you. Thank you.