Build CI/CD pipelines as code, run them anywhere
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61497 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202317 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
MereologyPresentation of a groupGoodness of fitComputer animation
00:27
Online helpFocus (optics)Source codeTask (computing)Formal languageCache (computing)Line (geometry)Formal languageQuicksortCache (computing)Multiplication signInternet service providerService (economics)CASE <Informatik>File formatOperator (mathematics)CodeComplex (psychology)Latent heatConfiguration spaceRight angleKey (cryptography)Core dumpStandard deviationSpacetimeMetropolitan area networkParameter (computer programming)Focus (optics)Process (computing)Projective planeFigurate numberPresentation of a groupLogic gateHoaxRepository (publishing)Cartesian coordinate systemOcean currentINTEGRALSoftware developerMereologyDifferent (Kate Ryan album)Revision controlMessage passingoutputSoftware testingRun time (program lifecycle phase)Function (mathematics)Local ringVirtual machineMathematicsWordOrder (biology)Computer fileResultantDefault (computer science)Level (video gaming)Graph (mathematics)Line (geometry)FeedbackInteractive televisionIntegrated development environmentObject-oriented programmingQueue (abstract data type)MetreType theoryForm (programming)CountingObservational studyBlogProduct (business)Uniform resource locator2 (number)GodHypermediaRoundness (object)Portable communications deviceSingle-precision floating-point formatComputer animation
09:56
Demo (music)Regulärer Ausdruck <Textverarbeitung>Client (computing)Cache (computing)Polygon meshMagneto-optical driveLibrary (computing)Order (biology)TouchscreenMultiplication signLatent heatPresentation of a groupSimilarity (geometry)ResultantSoftware testingFerry CorstenCodeVolume (thermodynamics)Medical imagingDemo (music)Computer fileFormal languageAuthorizationDistribution (mathematics)QuicksortSource codeDemosceneWhiteboardKey (cryptography)Machine codeCASE <Informatik>Range (statistics)Ocean currentPartial derivativeComputer animation
12:32
Formal languageSingle-precision floating-point formatQuicksortCASE <Informatik>Electronic program guideService (economics)SoftwareThermal expansionExtension (kinesiology)Personal computerSystem callState of matterCodeImplementationPoint (geometry)Speech synthesisVirtual machineNetwork socketGroup actionPlanningProjective planeSoftware testingIntegrated development environmentRevision controlGraph (mathematics)Library (computing)Data managementOcean currentGodComputer animation
16:29
Program flowchart
Transcript: English(auto-generated)
00:06
first thing first the dagger team was kind enough to send me some stickers this is really the reason why you go to conferences so if you want to pick some up I will leave it somewhere here okay cool so good evening everyone
00:24
thank you for joining my presentation today my name is Mark and for the last couple years or maybe for the better part of the decade I've been helping engineering teams focus on their business building their business applications instead of worrying about things like deployments or CI CD or stuff
00:43
like that and I'm currently my current title at Cisco is tech lead but by the way is there anyone here who saw my presentation in the morning in the Gode room okay so I'm gonna make it up to you guys so and please laugh
01:02
my jokes as well again even though you are recording them so I decided that I would come clean here today that this is a completely fake title and that in fact my real job is a yaml engineer so anyone else want to come clean unburden themselves okay cool oh yeah yeah that's engineer probably that's
01:29
an overstatement so let's talk about CI CD and we do have a bunch of CI CD services available today and it's still evolving continuously but we do
01:40
have a couple of challenges that causes pain to developers and and and that was people every day and that already kind of hints that one of them yellow like man you put a space in the wrong place and it just breaks over and you don't even know why because if the CI tool may not even tell you where
02:03
that extra space is so yaml is really one of the core pains of all the CI solutions today and yeah I know there is Jenkins and groovy which is even worse but yaml is really the the standard of CI CD languages these days and there are some places like there are a couple solutions where it works
02:22
kind of okay for simple pipelines but for more complex cases it's just it's just a nightmare then CI has this tendency to break for no obvious reason like one day the pipeline works and the other way it just doesn't and well
02:44
for for for operations work for deploying your application to a production environment you can always say okay oops problem let let them solve but for CI that's not really a case like developers have to interface with interact with CI and and if the CI is breaking then it's probably
03:01
the developers who have to fix it and the problem with current CI solutions today is that we don't really have like an easy way to debug CI issues like if there is something wrong you probably have to guess where the problem is maybe add a few echo lines to the yaml file push the whole thing
03:21
to the repository wait for the CI to get triggered by the repository and then go through this whole long feedback loop over and over again you see a lot of people nodding so when something goes wrong it's it's your job to fix it and it takes a lot of time and it's just painful and sometimes sometimes it's
03:47
actually not the CI's fault but but your fault the code doesn't or the test doesn't pass the linter doesn't pass and that's that's often caused by
04:00
things like having different versions in the CI and different versions in your in your development environment and there are tools and there are like ways to make those as close to each other as possible but still this is this is happening very often like I had this problem like a week ago so sometimes
04:22
it's just just your code that's that's not working with the CI and you have to go through the same feedback loop trying to push a change hoping that it will fix your problem and of course it doesn't work for the first time so you do it over and over again and until after an hour maybe maybe if you
04:42
sacrifice something to the CI gods it works so I'm pretty sure there are other challenges with CI but let's see how dagger may be able to solve some of these challenges here so first of all who has heard about dagger who knows what dagger is oh cool so dagger is a programmable portable CI CD solution at
05:07
portable is a pretty great feature here because instead of going through the whole feedback loop I was talking about you can run your CI CD pipeline on your own local machine and figure out what's wrong much sooner than by pushing to
05:23
the gate repository and waiting for the CI over and over again so it's it's much quicker that way to debug issues either related to the CI or your code and it's also much easier to build the pipeline in the first place like when you build a new CI pipeline for a new project you have to go through the same feedback loop because you have to add like new steps and figure out if it
05:44
works or not and if it doesn't then you have to figure out how to add the right parameters so even building new pipelines is way easier because the whole thing is portable the other thing that makes dagger great is that you can basically write your pipelines in in any language dagger officially
06:04
supports a couple languages like Go, Python, TypeScript and Q but basically any language that can talk to a graphical API because that's what's under the hood any language that can talk to a graphical API basically can be used to build your own pipelines with dagger and if you combine these
06:24
two traits like being portable and being able to write pipelines in your own language it also points to the fact that you can completely avoid vendor locking or vendor locking like you can obviously you would still need some sort of CI service and you would need like a thin layer of integration
06:41
that would run dagger itself but once you have a portable pipeline written in your own format in your own language not in a proprietary or a CI specific YAML format you are not logged into the CI vendor you are using right now and you may think that you don't really switch like CI
07:01
providers often but but that happens like when they buy your company and then you have to move from one provider to the another and then you have to move again because reasons and the fourth reason or the fourth thing that makes dagger great is caching now most CI services already have some sort of
07:23
caching solution that you can use to cache like intermediary artifacts or dependencies or whatever you want to store in the cache that you don't want to download or compile every single time when your CI pipeline runs but you still have to configure it properly and you have to make sure that you
07:41
have right the right caching keys you have to add the right paths to the caching configuration and you may either end up with a huge cache at the end of the day or you may not use cache at all so if you don't configure it properly it may not work with dagger you get caching by default and by
08:01
default I mean every single step in your pipeline the result of that run will be cached similarly how a docker file works like every single instruction the result of the instruction will be cached in case if there are no changes before that step actually so similar similarly to that
08:21
dagger caches every step in your CI pipeline now how does it do that how does it work how is this portable any any guesses one word yeah Docker yeah that's right so containers so of course containers so in order to be portable and to do all this magic that dagger does it needs to have like a
08:44
reasonable level of isolation so that you can be confident that it will run on your local machine and on your CI the same way so it runs your builds in containers and I already mentioned the dagger has a few official SDKs that
09:01
you can use to build the pipeline in your own code and using that dagger SDK you can talk to the so-called dagger engine which is the API that implements the graph QL specification and the dagger SDK will call this API with the steps in your pipeline and the dagger engine will build a DAG from
09:25
these steps and then we'll pass that to basically to run through a container runtime and that's how your your pipeline will run and the good thing about this is that you can actually change this pipeline so the output of one pipeline can be the input of another and this whole thing goes through
09:44
a single thing called session so in a single session you can have multiple like these container executions and you can change the results into each other if you want to now let's actually take a look at how these things run and the reason why I asked if there is anyone here who was in my
10:04
presentation in the morning because I completely botched the demo and it didn't work at all so let's hope it works this time so the example is in Go but again it could be like TypeScript and Python or even Q and I'm not
10:21
gonna go into that much detail about the Go specifics here but basically you need to import this dagger SDK in order to... by the way can you see the screen or the code? Make it bigger? Okay. Better? Cool. So you have to import this dagger SDK if you want to use Go and then the first thing you need to do is
10:45
connect to the Dagger engine. Now if the Dagger engine doesn't run locally then the SDK will actually run it using a simple Docker container so the first thing you need to do is connect to this Dagger engine and then
11:01
you can start launching these containers and start building your pipelines and if it looks very similar it's because it's basically the same, uses the same language and it looks very similar to Docker files and it works basically the same way so you have like a base image you have a
11:21
bunch of mounted volumes for caching and then you mount the source code and you run some sort of command and that's it that's your pipeline now let's see if it actually runs so I use this make file authority for Go called made so this this is how I have this whole code implemented in a test
11:45
function let's see if it runs so it didn't cool so if I just just for
12:02
let's turn debug lock to see what's happening in the background so it pulls an image with the goal in the image it mounts the code it mounts the volumes and then runs my Go test on the mounted code and then basically exits
12:24
and outputs the result of the test so well that's it if you want to get started with Dagger check out the documentation it's getting better by the day they actually released well either today or yesterday a new quick start
12:43
guide which is pretty awesome it has all some it has all the three or four supported languages and a single document so you can switch between languages if you want to there is even a playground for the low-level GraphQL API so if you don't want to start a new project you can play directly with the GraphQL API with the hosted version of the dagger engine so thank
13:03
you for your attention if you have any questions I thank you so much awesome I have a question with regards to like implementation so let's say
13:22
that you roll your pipeline and you commit that you want to run that somewhere like in a CI environment okay get up actions or God forbid Jenkins or whatever how do you how do you go about that I can imagine that you need to expose the Docker socket to the pipeline or how does it work yeah so basically if you have Docker running in your environment you can run
13:42
this pipeline and you can run Docker anywhere basically today like you have you can run it in Jenkins or you you you have it in GitHub actions actually and you probably have it on your machine as well so wherever you have Docker running today this pipeline will run so you just say invoke the dagger command that that command that you just showed us this is entirely my
14:06
code this is my go binary basically right again and it will be I should and it will find the the Docker API socket and if you just start 20 hours that's thanks before I very cool stuff so before I switch all my CI
14:24
to dagger let's frame it like this what would be the two things that you would really love to see an improved implementation of in the next version can you repeat the question what what are the two things that really need to
14:41
improve about the current state of dagger so what improves to improve dagger yeah in your opinion what are the two things that need most improvement okay so one thing is secret management right now dagger it's not that easy to work with secret so that needs to be improved and they are actually working
15:03
on it so that's great the other thing is that right now if you build something in one language for example if I build a reusable library in go to run my pipelines I can't reuse it in in TypeScript for example today and for that there is actually a feature called extensions extensions so they are
15:23
working on a feature so you can build extensions to the dagger engine so you can build these reliable or reusable pipeline pieces like running liters and stuff like that so you don't have to build that in your own code you just have to build the extensions and you can call it from whatever language you want to call them basically graph API extensions thank
15:44
you last question hi does dagger support spinning up service a concurrently with like service be because the tests need something else to run while the
16:02
test is running and then afterwards you can continue to other stuff right now I don't think it does again this is something that they are thinking about but it's not a trivial thing to do so no currently okay someone is working on it
16:21
thank you of course thank you thank you very much