Building a CI pipeline with Dagger in Go
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 542 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/62049 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 202359 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
InformationNotebook-ComputerDiagrammComputeranimation
01:05
HilfesystemFokalpunktQuellcodeGreen-FunktionKartesische KoordinatenBitMultiplikationsoperatorKontrollstrukturKonfigurationsraumPhysikalisches SystemHoaxNichtlinearer OperatorDokumentenserverMereologieCASE <Informatik>Minkowski-MetrikEinsSoftwareentwicklerRückkopplungComputeranimation
03:50
SoftwaretestFormale SpracheCachingRückkopplungCodePhysikalisches SystemDatenverwaltungKonfigurationsraumStellenringFormale SpracheElektronische PublikationMobiles EndgerätSoftwaretestNeuroinformatikZahlenbereichDefaultDemoszene <Programmierung>Mixed RealityMehrrechnersystemIntegralCachingVirtuelle MaschineGamecontrollerEinsPunktGruppenoperationComputerspielResultanteDifferenteSelbstrepräsentationDokumentenserverVersionsverwaltungComputeranimation
09:19
Schreiben <Datenverarbeitung>ClientGraphMultiplikationsoperatorResultanteSoftware Development KitLaufzeitfehlerVirtuelle MaschineFormale SpracheCodeAbfrageMobiles EndgerätGebäude <Mathematik>RichtungComputeranimation
11:36
BenutzerfreundlichkeitTaskBildschirmfensterMagnetooptischer SpeicherSpieltheorieKontextbezogenes SystemComputeranimation
11:56
Video GenieBildschirmfensterNabel <Mathematik>SpieltheorieTermManufacturing Execution SystemSpezialrechnerCodeSchreiben <Datenverarbeitung>Elektronische PublikationProjektive EbeneFunktionalCASE <Informatik>CachingVerzeichnisdienstLaufzeitfehlerCase-Moddingp-BlockDistributionenraumGrenzschichtablösungBimodulSoftwareentwicklerGebäude <Mathematik>Kartesische KoordinatenQuellcodeProgrammbibliothekBildgebendes VerfahrenWort <Informatik>SoftwaretestMereologieProgrammierungComputerspielComputeranimation
16:56
BildschirmfensterNabel <Mathematik>ProgrammierumgebungSpezialrechnerTermElektronischer DatenaustauschDualitätstheorieEinfach zusammenhängender RaumGüte der AnpassungDemo <Programm>InternetworkingLoginComputeranimation
19:19
GmailGoogolBildschirmfensterSichtenkonzeptLesezeichen <Internet>Elektronische PublikationGebäude <Mathematik>ProgrammschemaSpezialrechnerProgrammfehlerWendepunktNabel <Mathematik>TermInklusion <Mathematik>DualitätstheorieLemma <Logik>Magnetooptischer SpeicherLokales MinimumWort <Informatik>ClientSystemaufrufInternetworkingEinfach zusammenhängender RaumBenutzerfreundlichkeitOpen SourceFunktion <Mathematik>CodeSoftwareentwicklerSoftwaretestInterface <Schaltung>VerschlingungCASE <Informatik>ResultanteMAPBildgebendes VerfahrenSelbstrepräsentationProgrammbibliothekÜberlagerung <Mathematik>Computeranimation
23:01
Lesezeichen <Internet>SichtenkonzeptBildschirmfensterDifferenteElektronischer ProgrammführerCodeFormale SpracheComputeranimation
23:49
Elektronische PublikationSichtenkonzeptLesezeichen <Internet>BildschirmfensterMobiles InternetMAPNeuroinformatikComputeranimation
24:10
SichtenkonzeptLesezeichen <Internet>BildschirmfensterMultiplikationsoperatorKonfiguration <Informatik>Mobiles EndgerätMereologieComputeranimationVorlesung/Konferenz
25:11
FitnessfunktionBildgebendes VerfahrenProgrammierumgebungQuick-SortKonfigurationsdatenbankDifferenteCodeCASE <Informatik>NeuroinformatikPerspektiveStellenringVorlesung/Konferenz
26:50
Flussdiagramm
Transkript: Englisch(automatisch erzeugt)
00:05
Okay, thank you everyone. Our next speaker has some interesting news in for your CI There are better solutions than the YAML you used to and it's gonna mark is gonna talk to us about Building a CI pipeline with dagger in go
00:27
Thank you. Thank you. Can you hear me? Okay, so very important Information before we get started. I have some dagger stickers here if you want to pick them up I don't know. Maybe I can just leave them after the talk or you can come to me and pick them up
00:45
I'll leave the stickers over here People can grab them Important thing stickers are for your laptop not for the room every sticker you put inside a room in false them We have to pay for so keep them for yourself. Yeah Well, that's why we are going to conferences for it
01:02
So thank you again for joining me today my name is mark and For the better part of the decade, I've been focusing on or helping the jury teams focus build Helping them focus on their business applications building their best business applications instead of worrying about things like CI
01:23
And how they are being deployed and I have this fake title at Cisco called technical lead But I decided that I would come clean here today that I'm really nothing more than just a yamal actually Who that feels good anyone else when I'm burden themselves any other yamal engineers here
01:46
All right So let's let's talk about CI CD a bit and Cis or CI systems improved a lot in the last few years. We have new and
02:01
More evolved CI solutions today But we still have some challenges that we face every day like the one I've been already hinting at yamal obviously yamal is one of the biggest problems with CI systems today and admittedly sometimes
02:20
Like using yamal to build the decorative pipeline can be fine, but Man, you miss a space the whole thing just broke. We would think this is very excellent and You might not even know where to start debugging it so yamal makes it often really hard for people to even just touch CI and
02:42
the other thing is CI like tends to break for no obvious reasons like the pipeline that work yesterday may not work today, and you don't really know why and Obviously as developers when something breaks in production, we can just tell the ops people to worry about it
03:01
But with CI that's that's not really the case like we have to interact with CI and if something goes wrong, we might have to be the ones who fix it and With the currently available CI solutions today You can't really like Everything was running in the cloud or in a remote system You can't really have or you don't have tools that you can use to debug effectively
03:25
you have to start guessing and start changing some yamal config and You have to push that to a repository and then wait for the CI to Get triggered and you have to go through this whole and long feedback loop to be able to debug What's going wrong and to be able to fix that?
03:42
And that's a pain like it takes a lot of time It's really a huge waste of time and and it's really painful to do that Now sometimes it's not The CI that's wrong Sometimes it's you like pushing something that you shouldn't be pushing
04:01
to the repository like tests are not passing or the linters are not passing or something else goes wrong and again you May have the tools locally in your machine, but you may not have the same versions. You may not have The same setup as in the CI and it may just break in the CI Even though you run the test locally and all everything was green
04:23
It may still be fail in the CI and you still have to go through the same long feedback loop again and again trying to fix that Instead of being able to run the whole thing locally and being confident that it will just work in the CI as well And probably there are other challenges with CI, but these are the ones that wasted hours for my life in the last couple years
04:48
So how can dagger provide an answer to these problems So first of all dagger is a programmable and the portable CI solution, which means you can run your CI pipelines
05:02
Basically anywhere we will get to how it does that but the important thing is that you can run your CI pipelines Anywhere using the same environment, which means if it runs on your machine Then you can be confident that it will run the same way in your own CI system now
05:20
That's a great thing for a number of reasons because when you start building the pipeline For example using any of the CI systems today You still have to go through that feedback loop like adding some config and then pushing it to the GitHub Oh and trying to figure out if it works or not and then changing until it works Now if you have the ability to run this whole thing locally, it's it's much shorter feedback loop
05:47
So you can build your own CI pipelines much more quickly than using some remote system The other thing is that if something goes wrong, you have the whole thing running locally. So again shorter feedback loop you have more
06:03
Tools to debug so it's much easier to figure out what it was wrong, even if it's either the CI pipeline or your code The other thing about Dagger is that you can actually write your pipelines in your own preferred language now Not any language, obviously some of the languages that the Dagger supports
06:24
But that's already much better than YAML. Like you can write your pipelines in Go or Python or TypeScript ThinkQ, even Q But that's already much better than YAML like you can write your own pipelines in code and you don't have to
06:40
Invent or use some weird syntax, for example to represent dependencies between steps or between different pipelines You can you can just do that in plain code So that's great and all those So the portability and writing pipelines in your own language Points to the fact that you can avoid
07:00
Pandora locking entirely like you would still have like a CI solution like Jenkins or GitHub actions or whatever and you would still run Dagger On those systems, but you would have to write like a very thin integration layer just to run the dagger pipelines and You you can you would be much more confident that the pipelines would run the same way on the CI system
07:26
as on your computer and Yeah, you can avoid vendor locking entirely like you can move to another CI system if you want to and you may say That doesn't happen often but when it does man, it's really painful like
07:41
Converting from one YAML to another or one YAML to I don't know groovy or Jenkins file or something. That's that hurts and lastly Lastly caching so every CI system or most CI systems have their own caching solutions where you can like cache the dependencies of your
08:03
Of your language or dependency manager But that requires like configuration you have to make sure that you configure it, right? otherwise Well, it could either like grow the cache and Leslie and then you will be paying a lot of money for that Or it would just be non-functional at all and it wouldn't cache anything properly now with dagger
08:25
Everything is cached by default. Like every step is cached. You can think about it like a Docker file Every instruction or the result of it is basically cached in a separate layer in the Docker file and if nothing changed between
08:41
The steps then when you run it again, it will basically run the same way and it will come from the cache That's really how dagger works. Obviously you have some control over what you want to cache and how you want to do it but by default dagger Got that covered for you. Now. How does all this work? Behind the scenes if I had to describe it in one word, it's obviously containers now
09:05
We can we can do it ourselves today, right? Like we could just run everything in a container and it would be reasonably Reasonable to say that it will run on the CI the same way What dagger adds to the mix here? Is that you can actually build pipelines with code and that would be translated into?
09:28
A build pipeline so you would use the dagger SDK The language SDK that dagger provides again today. I believe it's for go TypeScript Python maybe queue as well But the underlying API is actually the graph QL. So if you have
09:45
a language client for Graph QL you can actually build your way SDK if you want to or you can just Write graph QL queries and send those directly to the dagger engine But basically you write your own pipelines using this as SDK
10:01
In your own language and then the SDK will basically send graph queries to the dagger engine Now when you run the whole thing locally first and the dagger SDK will actually launch the dagger engine for you All all it needs is really a that docker compatible Container runtime. So if you have docker on your computer or in your CI
10:23
Then you can run your Dagger dagger pipeline basics it so that's that's once more the Portability of this whole thing if you have docker on your machine and docker basically runs anywhere these days Then you can run the dagger pipeline there
10:42
so Locally when you when you launch this stuff for the first time the dagger SDK we launched the dagger engine for you And you send these graph QL queries You'll see a couple examples how that looks like in the SDK and the dagger engine basically builds a DAG Directed acyclic graph of all those steps and then sends it through
11:05
Well, it says an OCI runtime. I believe currently docker is the only supported pipe Runtime but sends through an OCI runtime and runs the whole thing in containers for you And then when obviously when a pipeline is finished you get back the results and
11:21
You can use the results in another pipeline if you want to for example the result of your build pipeline would be used in your deploy pipeline and You could deploy or project or whatever you have so that's how dagger works under the hood and Let's take a look at an actual example. Let's see
11:45
So the example you've been go Because this is that over there room you see it through Can you see it from the back?
12:02
better now Okay, cool so The first thing you need to use the dagger SDK and go is importing this module From dagger. It's the dagger SDK for go That you can use to interface with the dagger engine and
12:27
Once you have that you can basically start writing your own program now in this case. I'm using mage I'm not sure if you're familiar with that, but it's basically like a make file like solution for go you can write these
12:40
functions and mage will basically compile a binary from that and execute it like like It would work with make now You can absolutely import this dagger package in your own application if you want to in case of applications It's probably not a huge deal if you have like an additional dependency in your go modules
13:02
If you are writing to a library though, you might want to create a separate module for example called CI and and Import dagger the dagger SDK in that separate module So you don't import dagger as a development dependency in your in your libraries go that mod file
13:21
I know it still won't be built or still won't be in the final binary If you import that library, but some people get annoyed if they see dependencies that is actually not necessary for the library, so make Make make the life easier for your peers And and if you develop a library and use dagger just create a separate module and and put all the dagger related code there
13:47
the first thing you need to do when you Want to write pipeline with dagger is called is dagger connect Function which will basically
14:01
connect to your docker runtime and it will launch the The dagger engine for you and start the so called session And now within that session you can start building your actual pipelines using These containers now, it's it's pretty similar to how a docker file would look like
14:22
for good reason But what you can do here is basically use Some of the same instructions as you would do in a docker file you can Obviously go from a base image. It will be going in a go project. For example, you can mount your source code. That's how
14:42
You would have access to the source code within the container and then you can run a bunch of commands like Test or you can do the same with The linter for example and the other two here these are the mounted caches you can do that with
15:03
with the Build keep actually so that's that's I believe that's a built-in functionality So you can mount a cache directory to the container that will not actually be Part of the container, but it will be a mounted cache directory From your host
15:22
Now Let's see if I can write this So I'm using the mage minor here I'm telling you to change to the CI directory because it's a separate module and then I'm telling you to use the current yet
15:48
Can you hear me? Okay, don't know what happened there and then I'm basically just telling it to run the test function here again. Similarly how make would
16:03
Make five would look like Now, let's see what happens Kind of hope that I don't have to download all those container images. Let's get some blocks here. I
16:38
Swear this word like a couple hours ago. Oh
16:50
You know what? I think I don't have doctor. I mean, yeah That's a problem
17:09
Yeah, maybe yeah, so I don't have the dagger engine running it. I went let's see. This should start a new container. I
17:48
All right This is not great Can we all pray to the demo gods, please? Thank you. Okay, you'll just have to believe me that this actually works
18:34
Yeah, okay. So here are the logs from the previous run. So this is actually very good work
18:47
Yeah, it says let's see. Okay, that's Again, okay. Let's try that. Do we have internet connection here?
19:22
We do have internet connection Okay. Well, we'll have to work with the logs from here. So well, basically what happens here is
19:40
When it works is it just runs the whole thing within this goal and image Mounts The source code and then runs the go that test comment and just inspect the results Obviously, this is this is the build log like this is the debug log, but normally it would just output the output of the go test
20:03
command Or the all NCI command. Let's see. Let's try from Hotspot that works better
20:31
anyway, if Well, if someone wants to get back their money, sorry folks. This is a free conference
20:44
Anyway Yeah, you will just have to believe me that this works but I will try to make this work after Representation now if you take a look at the code here, this is still not very Like user friendly if you don't know how dagger works or if you don't know
21:02
What happens here then it's not really useful to you We'll have to go to the documentation and understand how this whole thing works when it works so but the good thing is that this is like This is cool. This is not an arbitrary YAML interface you have to use so we can actually make this a bit better if we want to and
21:25
What I did in the last couple of weeks is that I built Higher level library over the dagger SDK So instead of writing all that container mount nonsense stuff. You can just use the so this
21:42
Go link package it's actually called the OCI you can find it here if you want to give it a try and instead of You still have to connect to dagger obviously, but instead of running that whole or writing that whole container code you can just use this much more friendly interface to run your tests or
22:04
Around the goal and see I lean for example And it's much easier for developers to interact with this like if they want to change the cover mode, for example It's pretty obvious how you would want to do that in this case compared to how you would want to do that with the
22:21
With the lower level dagger SDK stuff Let's let's give this a another try now from the oh Still one. Let's go to the well. It doesn't work
22:58
Anyway If you want to give this a try at home, you're absolutely welcome to do that
23:04
The documentation is getting better by the day It has a bunch of different examples You will find these examples on my github as well. I promise it works They've actually just released a brand new kickstart guide
23:21
So so far the documentation they had documentation for the different SDKs In different places now, they have a quick start guide that that is basically the same for all of the languages So regardless which one you want to choose or if you want to try All three supported SDKs you can do that with the kickstart guide and you can actually go ahead and run the code from there
23:44
And finally, they have a playground that works with their Low level draft QL API So if you want to give that a try it's fairly similar to the SDK actually if you want to give that a try
24:00
Then you can absolutely do that and See how dagger works without actually installing on your computer So that's all I have for today if you have questions feel free there is a sticker and Thank you very much for your attention. If there are any questions raise your hand. I'll try to give you a microphone
24:33
I have one over here that's closer Hey, can you? Other than Docker. I'm sorry. Can you can use it with something other than Docker underneath?
24:46
So, I think you can use it with podman at the moment, I think Technically see a time But I don't know if that's currently available as an option But we have someone from the dagger team here who can actually answer that question
25:06
Hi So, how does the portability work when parts of your deployment depend on? publishing a docker image to repo that is external or AWS or terraform or things that require secrets. How does that fit and running it locally?
25:26
so that that's a great question actually, so The code itself should be completely portable so the pipeline itself can run anywhere What you would need to do in that case is you need that some sort of either a central
25:43
secret store that you can connect to from your own computer or You need to be able to load some sort of secrets or credentials from From your environment variables, for example, you can absolutely do that with a dagger pipeline So from that perspective
26:06
No, hello and From locally you can push to another registry or push to a development environment for example So you can you can parameterize pipelines based on where you run them
26:21
You would still run the same code, but you could deploy to different environment From local IPL. We have one more question there Okay, thank you one last applause, please