Progressive Delivery
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47493 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020310 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
Right angleInclusion mapBlogEvent horizonService (economics)Term (mathematics)Revision controlContinuous functionLimit (category theory)RadiusAddress spaceServer (computing)Green's functionCodeFlagRight angleMultiplication signFacebookMetric systemGame theoryStatisticsDecision theoryTwitterNumberStructural loadComputer hardwareProduct (business)Computer configurationRevision controlInstance (computer science)Gene clusterPerformance appraisalVirtual machineTerm (mathematics)MereologyFrequencyParallel portStrategy gameMathematicsPoint (geometry)MultiplicationArithmetic progressionPlane (geometry)Computer virusTunisIP addressBlogElectronic meeting systemMultilaterationCuboidChemical equationLie groupSpecial unitary groupDefault (computer science)View (database)Open sourceData miningVideo gameThread (computing)Forcing (mathematics)Rule of inferenceComputer animation
07:50
FlagLatent class modelCodeSoftware testingMetric systemError messageSanitary sewerServer (computing)Online chatComputer-generated imageryBuildingGoogolPoint cloudCartesian coordinate systemControl flowSoftware testingMereologyPoint (geometry)NumberPropagation of uncertaintyGoodness of fitMedical imagingBuildingDemonJava appletGene clusterBranch (computer science)Error messageProduct (business)Traffic reportingUniform resource locatorEmailHookingWeb 2.0Scaling (geometry)Group actionFront and back endsStandard deviationData managementBefehlsprozessorDemo (music)Web pageProjective planeEndliche ModelltheorieComputer clusterPlug-in (computing)Metric systemArithmetic progressionMetropolitan area networkUltraviolet photoelectron spectroscopyExistential quantificationWordCASE <Informatik>Graph coloringRaw image formatWeight1 (number)Scripting languageProcess (computing)Semiconductor memoryLie groupGoogolPhysical systemStress (mechanics)Service (economics)FlagBit rateMathematical optimizationMultiplication signInternet forumElectronic mailing listMatching (graph theory)Computer iconMaterialization (paranormal)Reading (process)Computer animation
15:23
Computer-generated imageryBuildingPoint cloudGoogolSystem programmingMetric systemSheaf (mathematics)Thresholding (image processing)Java appletRevision controlMaxima and minimaMathematical analysisWeightCartesian coordinate systemBitService (economics)State observerMatching (graph theory)Touch typingMassWeightUniform resource locatorSheaf (mathematics)Revision controlFreewareOnline helpJava appletMedical imagingHeat transferMultiplication signPoint (geometry)Demo (music)Web pageMetric systemMessage passingVirtualizationProduct (business)CASE <Informatik>FacebookArithmetic progressionNumberMereologyEmailProjective planeSoftware protection dongleLevel (video gaming)Integrated development environmentFlagMoving averageRational number2 (number)Plane (geometry)NamespaceMetreExecution unitNormal (geometry)Right angleBit rateMathematical analysisCircleQuarkSelectivity (electronic)Connected spaceComputer fileView (database)Set (mathematics)Software developerRow (database)DampingPlanningPersonal digital assistantInternetworkingComputer programmingComputer animation
22:57
Java appletOpen setEmpennageWindowBookmark (World Wide Web)View (database)QuarkStandard deviationGraphical user interfaceComputer fileMoving averageQuicksortProduct (business)Cartesian coordinate systemRevision controlLaserShooting methodDemo (music)MathematicsNumberOpen sourceComputer animation
23:47
View (database)Punched cardGraphical user interfaceBookmark (World Wide Web)Window12 (number)19 (number)DreizehnGame controllerBuildingUniform resource locatorJava appletProgrammable read-only memoryRevision controlMathematicsService (economics)Product (business)Level (video gaming)Integrated development environmentComputer animation
24:42
Java appletProgrammable read-only memoryIntegrated development environmentVariable (mathematics)Software repositoryError messageReduction of orderComputer iconRevision controlFunction (mathematics)Computer configurationDirectory serviceExecution unitPunched cardGraphical user interfaceView (database)WindowBookmark (World Wide Web)Boom (sailing)Asynchronous Transfer ModeMIDICodeIntegrated development environmentSoftware repositoryProduct (business)Traffic reportingCore dumpComputer animation
25:13
Bookmark (World Wide Web)Graphical user interfacePredictabilityOpen setComputer fileGraphic designPlastikkarteMoment of inertiaPatch (Unix)Normed vector spaceWindowView (database)Binary fileNumberMetropolitan area networkGoodness of fitBuildingMereologyCASE <Informatik>Reading (process)Lie groupIntegrated development environmentSoftware testingRootOnline chatComputer-assisted translationComputer animation
26:06
Computer fileBookmark (World Wide Web)Java appletVariable (mathematics)Integrated development environmentSoftware repositoryError messageFunction (mathematics)Computer configurationDirectory serviceGastropod shellGame controllerPhysical systemScheduling (computing)Stack (abstract data type)BuildingView (database)Online helpWindowBuildingGreatest elementCartesian coordinate systemRevision controlComputer animation
26:37
Image warpingGraphical user interfaceExecution unitMaizeMaxima and minimaVoltmeterJava appletWindowBookmark (World Wide Web)View (database)VolumeInclusion mapMetric systemRight angleSemiconductor memoryBefehlsprozessorComputer animation
27:12
MaizeMoment of inertiaBoom (sailing)Java appletGame controllerBroadcast programmingScheduling (computing)Operator (mathematics)Control flowCircleBuildingPhysical systemWindowLevel (video gaming)LogarithmMessage passingDisk read-and-write headForceIntegrated development environmentGastropod shellDefault (computer science)Hessian matrixConvex hullView (database)Bookmark (World Wide Web)Computer filePoint cloudComputing platformRankingGraphical user interfaceCluster samplingElectronic data interchangeGoogolPhysical lawSkewnessNumberScaling (geometry)Multiplication signVirtual machineBuildingRight angleComputer animationSource code
28:15
Integrated development environmentMessage passingDefault (computer science)Software repositoryDisk read-and-write headJava appletVariable (mathematics)Reduction of orderRevision controlError messageFunction (mathematics)Computer configurationDirectory serviceProgrammable read-only memoryTotal S.A.TrigonometryGastropod shellSpacetimeView (database)Multiplication signType theoryProduct (business)Projective planeAliasingVotingSource codeComputer animation
29:02
Sanitary sewerGastropod shellWindowScalable Coherent InterfaceDisk read-and-write headMessage passingDefault (computer science)Software bugIntegrated development environmentData storage deviceFile formatHessian matrixSoftware repositoryComputing platformPoint cloudCluster samplingBookmark (World Wide Web)View (database)Graphical user interfaceComputer fileRevision controlMeta elementCloningLevel (video gaming)Open sourceConvex hullRootInclusion mapMaizeRepository (publishing)GoogolVideo game consolePlane (geometry)1 (number)Cartesian coordinate systemWorkloadSource codeComputer animation
29:50
Default (computer science)Computer fileIntegrated development environmentTemplate (C++)Data storage deviceCantor setLibrary (computing)Gastropod shellJava appletOctahedronVariable (mathematics)Revision controlError messageSoftware repositoryFunction (mathematics)Computer configurationDirectory serviceWindowCheat <Computerspiel>Core dumpCartesian coordinate systemMetric systemSource codeComputer animation
30:28
Java appletDirectory serviceError messageIntegrated development environmentFunction (mathematics)Bit rateFlagGastropod shellOnline helpWindowScalable Coherent InterfacePhysical systemRevision controlGame controllerCASE <Informatik>Revision controlLoginDivision (mathematics)Dataflow2 (number)Greatest elementWhiteboardOpen sourceComputer animation
31:18
Java appletMaxima and minimaMassScalable Coherent InterfaceLipschitz-StetigkeitInclusion mapConvex hullWindowPhysical systemRevision controlGame controllerGastropod shell2 (number)Greatest elementRevision controlLibrary (computing)Computer animation
31:50
Computing platformPoint cloudGoogolMach's principleJava appletQuarkStandard deviationBookmark (World Wide Web)Graphical user interfaceView (database)Medical imagingLibrary (computing)Java appletProjective planeArithmetic progressionInternetworkingProcess (computing)Product (business)Computer animation
32:23
Physical systemMathematical analysisGame controllerWeightJava appletWindowGastropod shellLipschitz-StetigkeitView (database)Bookmark (World Wide Web)Telephone number mappingRule of inferenceMathematicsWeightRevision controlProduct (business)Multiplication signCrash (computing)Computer animation
32:51
Bookmark (World Wide Web)View (database)Java appletTelephone number mappingRule of inferenceWindowCAN busGraphical user interfacePunched cardWeb browserComputer animation
33:19
Java appletBookmark (World Wide Web)View (database)Graphical user interfaceTelephone number mappingPunched cardOnline helpRule of inferenceWindowSummierbarkeitComputer configurationPlane (geometry)Internet service providerSoftwareCache (computing)Computer animation
33:48
Punched cardGraphical user interfaceView (database)Hardware-in-the-loop simulationWindowMenu (computing)PiFacebookRevision controlProduct (business)Cartesian coordinate systemMultiplication signOpen sourceDatabaseQuicksortArithmetic progressionEmailRollback (data management)IP addressGoodness of fitMultiplicationControl flowPlane (geometry)Moving averageThomas BayesWhiteboardView (database)Uniform resource locatorFlagMathematicsComputer animation
37:01
Point cloudFacebookOpen source
Transcript: English(auto-generated)
00:08
Yes. Okay. Welcome. Perfect timing. Well, probably you're here because you
00:20
think continuous delivery is a good thing to do, right? If not, you're probably in the wrong room. And I'm going to start with a story, right? Some years ago, we were doing all this automation. We were deploying with Puppet across multiple machines everywhere. And then you're doing this continuous delivery every time we committed some changes to Puppet,
00:40
then this would get deployed until we get to a point where I made a change and this broke the machines. We do some testing, but this actually broke the machines in a way that not only they got broken, but it prevented new updates to get into the machines, right? So this is an example of how hard it is
01:01
to do continuous delivery right. So my name is Carlos Sanchez. I work at Adobe. As Victor said, the important part is the Twitter, C. Sanchez, if you want to say good things, if you want to say bad things, that's not the right one, it's a different one. And I'm here to talk to you about progressive delivery,
01:21
which is a term that was coined a few years ago, just less than two years ago. It started on Lunch Starly. This is a company that do feature flags, so I'll talk about that. So they started with the term progressive delivery. Then Red Monk, the analysts, James Governor, started also talking about progressive delivery,
01:41
and I was like, I'm totally still in this term. I love it. It's exactly what it means. It gives you the right mindset about what we are trying to achieve. So it's a term that tries to encompass all these technologies that are not new.
02:01
This is very important. This is not new technologies. It's just they're being around for a while, but it tries to avoid the problems with this all or nothing deployment capabilities or strategies. And basically, new versions that you deploy, they're not replacing existing versions,
02:23
but you run them in parallel for a period of time in production. This is the important part. You get production traffic, and you do evaluation of metrics to ensure that your deployment is good or not good. So until you do that,
02:42
so you keep getting more traffic, get the metrics, look at the metrics, decide whether the deployment is good or not. And continuous delivery is hard, and progressive delivery is trying to make continuous delivery easier to adopt by reducing the risk.
03:00
This is the key part, right? We are trying to reduce the risk. Nobody wants to get to waking up in the middle of the night because a deployment is going wrong. Nobody wants to decide like, do not deploy on Fridays, do not deploy before a holiday, because then when you start out, do not deploy on Fridays, then well, maybe you shouldn't deploy on Thursdays either.
03:21
So it's a bad game that you would want to get in. So reducing risk, but avoiding downtime, and limiting the blast radius so if something is wrong, then you gotta make sure that it affects the least amount of users as possible.
03:41
But it also gives you another benefit, which is the shorter time since you decide to do something until your production users can see some of your production users, right? So some of the techniques, if you've been to Victor's talk, this is more interesting because I have a Spanish accent, as he said.
04:04
So some things that he also cover for those of you, I knew some people left the room and new people came in after waiting for there probably for an hour or two. So running updates, this is something that Kubernetes does natively. You can do a kubectl command.
04:21
I mean, he's using Kubernetes. It's pretty much a lot of people, okay? So every time, this is not different from doing rollabout days on VMs. It's just doing it with containers, with pods in Kubernetes. But this is natively included in Kubernetes. So it's very easy to do.
04:40
You deploy a new version and then you get new pods in with a new version and it starts taking old pods from the, with the old version from the pool that serve your requests, that get your traffic. So every time you keep going, one pod, two pods, three pods, and so on,
05:01
same thing we used to do with VMs back in the day, not much different, other than it's more automated. Blue-green deployment. What did you learn about blue-green deployment from Victors? Don't do it, okay.
05:22
So if you're doing blue-green deployment, blue-green basically means having two instances of your production clusters or production hardware or infrastructure or whatever, and having something in front of them and then you can quickly switch from one to another. So when you are ready to roll the new release, you point to the new one. If there's any problem, you just go to your DNS,
05:42
load balancer, whatever it is that you have in front and you point to the back. So you have immediate rollback, but all your users are getting the new version. So if there's something wrong, all your users are gonna see it. Canary deployments is a specialized way,
06:01
it's a better way of doing things, I guess, where you get a percentage of your users that get the new version. So you start sending 1%, 2%, 10%, 20%, whatever is the number of your users to the new version. This is something that Facebook is doing,
06:22
Netflix, I mean, all the big companies are doing, but you don't have to be Facebook to do Canary deployments and I'll show you that later. And you can select the users based on multiple options, right? You can just say randomly send some percentage of users to the new version, or you could say people from,
06:44
what a lot of companies do is employees get the new version first before any public user gets it. So you get employees first, then you can use HTTP headers, IP location, anything you want. And another common trend that I've seen is like people,
07:03
I think Facebook does this, they roll it out first to New Zealand. So for whatever reason, New Zealand has the statistic population very similar to the US. So they roll it out in New Zealand, they get the new versions first before it gets rolled to the US or the rest of the world.
07:21
So I guess New Zealanders are the guinea pigs here, but that's something that you can do. And another option in progressive delivery is feature flex. This is something that LaunchDarkly and other companies are doing, where this is very interesting
07:42
because it allows you to do several things. One is exposing new features to a specific people. So you could say, you could ship something to production, disabled by default, and then you could look and use either a HTTP header also or location base or a flag and your,
08:02
I don't know, URL or something. And then you could get the new feature. So you could see how that behaves in production. You could enable that feature for a specific users, for a percentage of users. And it gives you also the speed because if you read them, if you're familiar with the DevOps report from Dora,
08:25
the companies, the teams that deploy faster to production or to master the merge things into master faster or commit directly to master or have short-lived branches are the most productive teams.
08:40
So feature flex allows you to continuously get things into master without exposing users to those things. So let's say you have a feature that it takes you two months to do. You don't wanna have a feature branch for two months because that's gonna be a horrible idea. It's a horrible idea. Don't do that. And feature flex allows you to disable that,
09:03
commit to master, and at some point, if you're starting trying it with users, iterate, fix things, start showing to more users and so on. This is the other way you can do deployments. I don't recommend it. Containers don't really fit in.
09:21
This is actually a picture. How did the deployment, there are a lot of containers in some island in the Caribbean before, and that's how a lot of companies look some days, but just don't do it. And a very important part of progressive delivery
09:40
is monitoring, and I like to say monitoring is the new testing because with testing, you're basically just checking things that you think may fail, right? You have to think about, okay, what can fail? I'm gonna write some tests to ensure that it doesn't fail. Monitoring will tell you how things fail that you didn't expect to fail, right? That's the only way you can know about those things.
10:04
And it's the only way you're gonna find out how your users, because your users are always gonna break your application in very new and innovative ways. So this is the way you catch that. And it also allows you to have a reaction to these issues automatically.
10:20
So you have some data coming in, you have real data. So you know this saying, if all we have is opinions, let's go with mine. Unless we have data, let's go with the data, right? So this will give you the actual data where you can do things based on that data.
10:41
And the bad part is like progressive delivery requires a good number of metrics. You need to instrument your application. You need to know what means success, what means failure. These metrics, it's not only HTTP metrics or uptime or things like that, but also could be business metrics, right?
11:00
If at some point you have a metric that says how many items are people adding to their shopping cart, right? You're saying this metric going well, okay. And at some point, people are not getting things to their shopping cart, then you may think, okay, there's something wrong with the application. Everything gets served, but maybe they get an empty page or some page
11:21
that I'm not getting error from that. So you have to catch that. And I love this quote from DevOpspara, to make error is human, to propagate error to serve in an automated way, that's DevOps. So actually in my previous company, I'm not gonna name names,
11:41
and we have this, we have to do this demo. The CEO was gonna do this demo at the big conference, and we were running these huge clusters with, it was like 2000 Yankees masters running, and I don't know, terabytes of memory, CPUs and everything.
12:02
And at some point I'm getting on a flight to go to San Francisco to the conference, and I'm checking my Slack before the flight, and it's like, oh, I deleted the wrong cluster by mistake. So this was one day before, two days before. So I think if you haven't automatically destroyed
12:23
something by mistake, it means that you are not automating enough, okay? You gotta get to this point where you're saying, okay, I did something really bad automatically. That's the point where you're saying, okay, I mean, automation is good. No blame, no blaming postmortems, right?
12:42
So Jenkins X, what does Jenkins X come here? Jenkins X, who's using Jenkins X? Okay, who's using Jenkins? Who's using Jenkins on Kubernetes? With a Kubernetes plugin probably, yeah? Okay, that's something I wrote a few years ago.
13:01
I'm glad you used it. Don't blame me, okay? So Jenkins X is the new project I like to say that Jenkins X is to Jenkins like Java is to JavaScript. They have nothing in common, okay? Jenkins X runs natively on Kubernetes.
13:22
Kubernetes, did I mention Kubernetes in this talk? I don't remember. And it uses a bunch of technologies that you probably know and some new ones. So it uses Docker, of course, so everything runs on Kubernetes, Scaffold-Rough, I'll talk about those,
13:40
and Tecton Pro. So Tecton is the pipeline engine in Kubernetes. For Kubernetes, it uses Pods and containers to run pipeline steps. This is a project from the Continuous Delivery Foundation now. And it uses this basically, it's a model that you've probably seen
14:00
with GitHub Actions and other tools where you have a pipeline, every step runs in a different container, so you have isolation and you have a scale. You have more pipelines, then you get more containers, and everything is scaled, there's no central point. It scales as much as your Kubernetes cluster does. Prowl is a project that comes from the Kubernetes itself
14:23
that implements chat ops. So it handles the GitHub web hooks. And then you can talk, you've probably seen this now, it's very popular where you got a pull request and the people start slash something, slash something, and then slash test, and the test shows you the test results, slash retest, and you can rerun things.
14:42
So chat ops. Helm is the package manager for Kubernetes. Defacto, standard. Scaffold to build Docker images. This is a project from Google that allows you to do typical Docker builds, but also has support for other backends to the Docker image builds.
15:03
Genki says it allows you to use Kaniko. I like Kaniko a lot. It allows you to do builds without the Docker daemon, so it's safe to use in Kubernetes, or at least safer. And you don't have to open your cluster to everybody to have access to the Docker daemon. And also Jib, if you are a Maven user,
15:23
you can use, I'm a long-time member, well, I committed to Maven for a long time. Jib optimizes, you can have a Maven plugin, and you can run MVM something, and then it will build a Docker image for you, and it will optimize the layers with your dependencies.
15:41
So when Maven downloads the internet, it gets optimized with the different layers for the Docker images, so when you change dependencies, it's not having to rebuild everything from scratch, it caches things. And Draft is a project from Microsoft that generates the Dockerfile and Helm transfer for your project. This is something that Jenkins X uses
16:00
when you import a project into Jenkins X, or when you generate a new project, it will basically create, decide, oh, you have a Java project, so I'm gonna create you a Dockerfile and help charts using the Docker image for the OpenJDK or other OpenJDK,
16:21
and it will generate all this scaffolding for you. What I'm talking about Jenkins X, because I'm gonna show you how to do progressive delivery with Jenkins X, there's a webpage that goes through all the details if you wanna try it yourself. I'm gonna show you a demo.
16:43
It uses Istio. I mean, who's using Istio here? Some people. Are you happy? Yes, okay. Probably because you don't have to maintain it, right? You don't update it, right. That's the point. You install it, that's it, don't touch it.
17:00
Okay, so what I'm gonna use from Istio is it's a service match. It provides you all these cool things, allows you connection between services, allows you to do things like say, oh, I want these services to go encrypted, so securing connection between services, allows you to control the traffic. This is what we're gonna use for Canary deployments.
17:23
We want to control what traffic goes to what service, the old version, the new version. And it also allows you to do observation of your application or your service without having to touch your application. So this is what we're gonna do to get the metrics. Istio uses Prometheus to store all these metrics.
17:42
So this is important bit. You don't have to touch your application. When you run on Istio, your application is automatically gonna get metrics from it, all the HTTP metrics, all the traffic is going through Istio, and you're gonna get these metrics for free. Right, you don't have to be Facebook to use this.
18:01
You can have Istio and all the metrics automatically, without touching your application, are going to Prometheus. And we're gonna use Flagger. Can you say, do you know what Flagger is? No. So. I'm doing Flagger for one year now. He's the creator of Flagger.
18:21
And automates the promotion of Canary deployments using Istio to shift the traffic, to control the traffic, and looking at the Prometheus metrics to decide whether your application and the Canary version is behaving correctly or not.
18:40
So it's gonna go through the different stages. Updating, saying, add some traffic to Canary service, look at the metrics, is this okay? Add more traffic, it's okay. Add more traffic, it's okay. Add more traffic until you finish the rollout. Of course, this is all configured with YAML, that you know and you love.
19:02
You need some extra tools, but okay. I mean, with all the dongles you need for MacBooks, one more is okay. This is cheap, this is cheaper than anything that Apple will sell you. So you, in Genki.sex, you just need to enable the Canary section. You have to say Canary enable true.
19:21
And just recently this was merged, and new projects will have a lot of this already included. But you have to say what host is your traffic coming in, and you use, because we are using Istio, Istio's gonna look at your headers and say, oh, this URL is going to this service.
19:41
And we use Canary analysis, so we can say, how often do we wanna check for the metrics, and how many metrics do we wanna fail, how many times do we want the metrics to fail before we roll back? In this case it's five. Run out of battery. And the step weight and mass weight is
20:01
what are the steps in my Canary deployment? So this means 10%, 20%, 30, 40, and 50, and then do a full rollout. So you can configure all these things in YAML, so no problem. And an interesting part is, what metrics do I want to check, right?
20:22
So you can use any metrics that are impervious. In this case there are two metrics that Istio gives you for free. Don't have to touch your application, you don't have to view Facebook to do Canary deployments. So you do request success rate, and say that 99% of your requests are successful,
20:40
non 500 errors, check every 60 seconds, and request duration means that your P99, you want them under 500 milliseconds. So you can use any of these metrics, you can use your business metrics. If you get business metrics into Prometheus, let's say number of items in the shopping cart, you can use that too.
21:03
And then you can just profit, right? Genki.sex has this concept of environment, some promotion between environments. So in Genki.sex you have pull request environments, every time you create a pull request you get a new Kubernetes namespace, it gets deployed as the pull request, so you get one full pull request.
21:21
You get another one that is a staging where master is always deployed, so staging always is updated with master. And then you have a production where you manually promote things. So in this case I'm saying, take my application, crack hunter java, version 130 into environment production.
21:41
And this is gonna deploy to production. In Genki.sex, by default, this will just role update my deployment. When I have the canary enabled true, this is gonna do a canary deployment. So how this all looks like, what time is it?
22:01
Okay, it was 45, right? Okay, I'm gonna make sure I have time for the demo. So my traffic is coming through the Istio virtual service. I have two services running, primary and canary. Primary is the old version, canary is the new version. Flagger is sitting there, looking at the Prometheus metrics
22:22
and updating the virtual service, sending more traffic to one or another. I'll show you that. And if you don't have enough Slack channels, you can have another one where you can say, show me how my rollout is happening,
22:41
because everybody wants more Slack channels and more Slack messages. This is how you can do it. Also Microsoft Teams. Also Microsoft Teams. Well, the demo is using Quarkus, so if you're interested. Okay, so let me go to the demo.
23:02
So I have this application, this is a stagey, no, okay, let's go to production first. This is an Istio, my URL is going through Istio. This is version 76. And it's this amazing application. I'm gonna just do the same demos, sort of the same demos that Victor did,
23:21
but more fun with animals, okay? Because Victor's demos were really boring. And I'm gonna say now, if he's not here, they didn't work. So I can shoot crocodiles with lasers. What could be more fun than that? But of course there's some people that complain, this is not very environmentally friendly,
23:43
shooting crocodiles with lasers. So we made a change in master. I did have a change before coming here. This is version 77. And instead of shooting crocodiles with lasers, I'm throwing fish to the crocodiles, okay? This will be more politically correct, everything.
24:03
So we're not getting into trouble, okay? So what I want to do, I want to roll out this into production. So I'm gonna do JX, I can do get applications. I'm gonna show you at the top, okay? So it tells me, Genki.x tells me,
24:21
staging is 77, production is 76. So I'm gonna do JX, promote Crock Hunter Java, version 0077, environment, production. And what this is gonna do is open
24:42
a Genki.x, oh, okay. And I will do cherry pink. Ah, because I was messing with it before. Let me do it just manually. This is the, Genki.x uses Gitops. So everything is in Git. If I go to my production environment, it's a GitHub repo,
25:02
staging environment's another GitHub repo. So I can go directly to make it faster here and say, no, 77, I want the 77. And it's the same thing. Either I do it from the command line or I can do it here.
25:21
Let's create a pull request, create a pull request. So this is basically gonna go through my promotion build. It's gonna make sure that my home charts compile and do some testing there. And the most important thing, I have chat ops.
25:40
If this is telling me this PR is approved or not, in this case, in the environment, we have auto approval. So if this passes the build, this will work. I can do the most interesting things that you can do with Gitops, with chat ops, is you can run commands. So you can do meow and then you get a picture of a cat. So that's probably the number one feature.
26:03
But you can do slash test or slash root test. And you can add your own things. So this is running the promotion build. While this is running, I have at the top, I have something getting the version from my application. So you see 76.
26:22
I have Flagger here at the bottom. There's some stack trace. Don't worry, I'll tell you later what happened. And so this is gonna be doing this. And more interesting stuff.
26:41
I have Flagger inclusive graphana dashboard where I can see on the left hand side, this is my primary. On the right hand side is the canary. So I can see all these metrics compare one to another. So request volume, request duration, CPU usage,
27:00
memory usage, whatever. So we could also come here and look at things. Well, I thought it was late, but not that much. Okay, so this is still not merged. Let me see, I think I have Tecton here.
27:22
We can look at the pipeline that is building. The Helm chart. Okay, there it is. This is still doing the build. While this runs, this is all running on GKE, Kubernetes engine. I have a cluster.
27:42
Right now it has three machines. That may be where it will take more time. I have auto scaling enabled. So it was gonna go up and down depending on the number of things that I'm running.
28:00
Last time I checked, it was four nodes, so I'm guessing it may need to go up to four before to the build, but we'll see. What else can I show you? Okay, this is still building. We can take a look at the pods.
28:24
This is JX production. Okay, we have still the primary one running. It hasn't created the other one yet.
28:41
One advice, if you wanna be a 7X engineer, you can alias kubectl to K, and then you say type in seven times letters. And there's a project that will create aliases for use. Like when I do KGPO, that means kubectl get pods. So I'm saving a lot of time.
29:02
You just need to remember all of them. Okay, so this is still working. Still working. Is this merged? Okay, this is merged. So it's gotta, okay, this is doing now the deployment.
29:24
And what else can I show you? I can see here also what workloads I have running on my cluster. I have Istio, I have Jenkins X, and here are my applications. Production, staging, production,
29:42
these are the ones that I'm deployed with Jenkins X. Okay, and this is this. Let's take a look at the pods again. Still not running.
30:06
Oh, that's what is querying my application. The curl request that I'm showing is just hitting, because that's one of the problems. If you do canary deployments and you don't have traffic, then it's not gonna work very well, because there's not gonna be any metrics. I have something that is basically
30:21
just doing curls against this thing. Okay, now it's doing something. Oh, isn't it? Ready, zero of one. Yes. Okay, it has started.
30:43
Why is this what I'm doing here? Let's take a look at the Flagger logs. Here it is. So here at the bottom, you'll see it says, new revision detected scaling up my canary.
31:04
So it has detected that I have deployed a new version, and it has created this copy for my canary. So I have both canary and the old version running. And this is now starting the pods.
31:21
Okay, now I have two pods. Forget about the trigger at the bottom. That's the one I'm doing the close from. I have the canary is the first one. Primary is the old one. So 70 seconds ago, it has started the canary. And eventually, Flagger will start saying,
31:42
I'm sending 10% of the traffic to the new version. While this is happening, yes. So this is running on Quarkus. Quarkus will also allow you to compile to native. So you don't even need the JVM
32:00
or any libraries in your Docker image if you want to do that. So that's a very interesting project if you want to do small Java images on Kubernetes or Docker. I will tell you to buy my book, but it's not my book. It's Victor's book. But I wrote the chapter on progressive delivery.
32:23
Let's see. Okay, here it is. Advancing canary weight 10%. And if you can see here, some requests are getting version 77, while some requests are getting 76. So one in 10 should get the new version. If I go to my production one,
32:45
and I refresh enough times without caching, that's the other thing. There's caching involved, browser caching, things like that. Okay. One in 10.
33:02
76. 76. 26. Jeez. Maybe we should wait until it's 20%.
33:26
Yeah, that's the other option. It should work. Should. Just timing. Yeah, I'm not the chosen one, yeah.
33:40
Okay, let's do this. Network. Disable cache. Anyway, 77, okay. So I have my version in production. Some people are getting this version. Some people are getting the old version. And remember, you don't have to be Facebook to do canary deployments. Anybody can do it.
34:01
So I'll take two questions before finally doing things. Yes? Somebody gets the new version, and they refresh. Can they get back the old version? Can you make this persistent somehow? Yes, you can do with Istio,
34:20
you can do all sort of things that you want. You can say HTTP headers, you can say IP, I think the IP location should work too. I think bookies, everything, yeah.
34:40
Yes? Then you need to reset them so that everybody gets the old version. Well, what happens when the rollback happens? Everybody gets the old version anyway. We stop looking at the headers or anything. Actually, it kills the pod with the new version. So there's no new version to go to. So everything goes back to the old version, yeah.
35:03
Yep, yes. Somebody else, yes? That depends, that's the big concern I have about progressive delivery and rollouts in general.
35:23
You now, you fix one problem, now you have a new and different problem, more interesting problem, right? So what happens when you have multiple commits to master, multiple things going on, multiple feature flags, whatever, right? Now you need some sort of dashboard that allows you to see how am I progressing things,
35:42
where are the different things happening, stuff like that. So that becomes the next challenge. I'm gonna note that nobody asked about databases because Victor already answered about that because that's the typical question. How did you do this? Can I redeploy with databases? And so you have to make sure
36:01
that your application is able to live with two versions. So what typically people do is, you have the breaking changes, you make them with enough time. So if your application can only talk to this version of the database, you make your application now talk to two versions of the database,
36:21
and you deploy your application, and then later on you change your database and do things like this, okay? So one thing I want you to get out here and remember, is like you don't have to be Facebook to do granary deployments, I don't know if I mentioned that before. And you can use, all these tools are available there,
36:41
it's all open source. You can use Genki.sex, you can use Istio, you can use Flagger, you can use Prometheus, you can use something else. But I hope you give it a try, and if it works for you, good. Thank you.