An event based approach for CI/CD pipelines
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 490 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/47236 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2020369 / 490
4
7
9
10
14
15
16
25
26
29
31
33
34
35
37
40
41
42
43
45
46
47
50
51
52
53
54
58
60
64
65
66
67
70
71
72
74
75
76
77
78
82
83
84
86
89
90
93
94
95
96
98
100
101
105
106
109
110
116
118
123
124
130
135
137
141
142
144
146
151
154
157
159
164
166
167
169
172
174
178
182
184
185
186
187
189
190
191
192
193
194
195
200
202
203
204
205
206
207
208
211
212
214
218
222
225
228
230
232
233
235
236
240
242
244
249
250
251
253
254
258
261
262
266
267
268
271
273
274
275
278
280
281
282
283
284
285
286
288
289
290
291
293
295
296
297
298
301
302
303
305
306
307
310
311
315
317
318
319
328
333
350
353
354
356
359
360
361
370
372
373
374
375
379
380
381
383
385
386
387
388
391
393
394
395
397
398
399
401
409
410
411
414
420
421
422
423
424
425
427
429
430
434
438
439
444
449
450
454
457
458
459
460
461
464
465
466
468
469
470
471
472
480
484
486
487
489
490
00:00
TelecommunicationEvent horizonQuicksortComputer animation
00:22
TelecommunicationEvent horizonCartesian coordinate systemOpen sourceDifferent (Kate Ryan album)MereologyTelecommunicationPlanningComputer animation
01:16
TelecommunicationTelecommunicationDifferent (Kate Ryan album)SoftwareProduct (business)Visualization (computer graphics)Configuration spaceTrailLevel (video gaming)Integrated development environmentSelf-organizationConnected spaceComputer animation
03:02
Product (business)TelecommunicationIntegrated development environmentTrailLevel (video gaming)Product (business)SoftwareINTEGRALMultiplicationDifferent (Kate Ryan album)Software testingRevision controlHierarchyTelecommunicationQuicksortComputer animation
05:20
Event horizonFormal languageDifferent (Kate Ryan album)Visualization (computer graphics)InformationSet (mathematics)Event horizon1 (number)File formatCommunications protocolCASE <Informatik>Cartesian coordinate systemJava appletSoftware testingRight angleSource codeComputer configurationSuite (music)Message passingRepository (publishing)Link (knot theory)ChainMedical imagingComputer animation
08:07
Software testingLevel (video gaming)Maxima and minimaSource codeEvent horizonEvent horizonSet (mathematics)Computer fileParallel portSoftware testingGraph (mathematics)Process (computing)CASE <Informatik>Product (business)Computer animation
09:11
Source codeEvent horizonCASE <Informatik>ResultantEvent horizonSoftware testingQuicksortLevel (video gaming)Extension (kinesiology)Link (knot theory)Confidence intervalComputer animation
10:57
Formal languageComputing platformLink (knot theory)Event horizonSet (mathematics)Event horizonSelf-organizationDifferent (Kate Ryan album)Formal languageVisualization (computer graphics)Stack (abstract data type)Link (knot theory)Open sourceSoftwareComputer fileComputer animation
12:26
Event horizonCartesian coordinate systemMultiplication signContinuous integrationCodeCartesian coordinate systemTelecommunicationScaling (geometry)SoftwareLine (geometry)Total S.A.CuboidSoftware testingComputer animation
13:14
Event horizonCartesian coordinate systemService (economics)Communications protocolComputer animation
13:39
INTEGRALPhysical systemCuboidRevision controlComputer hardwareSoftwarePoint cloudStandard deviationContext awarenessCartesian coordinate systemStack (abstract data type)Computer animation
14:42
Communications protocolCartesian coordinate systemCartesian coordinate systemDataflowCommunications protocolLattice (order)SoftwareVisualization (computer graphics)Context awarenessProjective planeComputer animation
15:54
Event horizonMessage passingBus (computing)Computer animation
16:44
Point (geometry)Type theoryUniform boundedness principleEvent horizonChainDataflowComputer animation
17:23
PrototypePhysical systemMereologySoftware testingCartesian coordinate systemDampingMereologyEvent horizonPhysical systemSoftware testingSoftwareInformationComputer animation
18:42
PrototypePhysical systemCartesian coordinate systemSoftware testingMereologyOpen setEvent horizonService (economics)Independence (probability theory)Musical ensembleMessage passingComplete graphGraph (mathematics)Library (computing)Event horizonRepository (publishing)Communications protocolData storage deviceSoftware testingOpen sourceOpen setBus (computing)Software developerComputer animation
22:03
GoogolYouTubeMaizeYouTubeLibrary (computing)INTEGRALVideoconferencingInformationGroup actionComputer animation
23:40
GoogolYouTubeSelf-organizationEvent horizonSoftware testingImplementationQuicksortSoftware developerFlow separationTask (computing)Communications protocolAbsolute valuePhysical systemMultiplication signLevel (video gaming)Data storage deviceDifferent (Kate Ryan album)Open sourceService (economics)Set (mathematics)Message passingComputer animation
27:56
Point cloudOpen source
Transcript: English(auto-generated)
00:05
And the room is very full that's we're very glad to be here and My name is Emily. Yes, and my colleague is Frederick. We only have one mic. So we're gonna switch in the middle sort of But let's get started
00:23
I have an agenda here and at the beginning we see I Planned to walk through the definition of a pipeline but considering the audience. I think that's unnecessary so I'm gonna skip that and And Me I'm coming from Ericsson
00:41
which is Not open source. Of course. It's a company, but we do contribute open source and that's why we're here today I'd like to start off with describing what kind of challenges we've seen considering the communication between different CI CD tools and Then I'm gonna present the way we solved it
01:02
Yes, I'm gonna speak a little bit louder and also how we Open sourced our solution and how Frederick and his company axis Started using the same solution. So let's skip this part as promised and
01:21
Instead and I go straight to the communication challenges that we've seen at Ericsson So first of all, why does Ericsson care about CI CD? well any company that produces after should care about CI CD obviously and We have a very complex environment. We have a lot of different products going out to our customers and
01:44
Different configuration applied on top of them considering if it's going to be to customer a or if customer B wants them So and we have many different pipelines and different pipeline tools used to produce our software Very often they're integrated into each other. They're called connected pipelines and that's what we like to call them
02:05
and I'll Go through that what I mean by that later We are a very large organization. We have like hundreds of teams developing our products each one of them use different toolings and
02:21
The demands on traceability is very high We need to keep track of our software. We need to go see which stages does it go through? In the in the pipeline No matter if it's a stop set delivery or if it's goes all the way into deployment through customer networks We need to follow and see where does our software go?
02:43
We're not we also need to be able to monitor it Independent of whichever technology we use in the background. We need to be able to keep track of the important KPIs and I Guess also visualize it in a good way and one visualization tool would be nice
03:00
So I spoke about connected pipelines and what I mean by that is we usually integrate our products Into each other sort of so their hierarchies of products So you've got product A and B to the left here and they are produced in their own separate pipelines Before they're together integrated into the next level of the next connected pipeline producing the product a B and
03:27
this is a very common phenomenon at Ericsson where we have multiple integration spots and it's tough to keep track of all of these dependencies to integrate upstream and See, where does this software come from and how it has been tested before we integrate it and again traceability
03:47
I'm supporting here all the way So I've tried to summarize into a few questions of what are the exact challenges we see What is what is the problems that we need to solve?
04:02
One thing is when we use different pipeline environments How do we communicate about when our product has been delivered or released? Let's say our product a is done here. How does somebody downstream become aware of this? How can we announce this to the world?
04:20
and How can we if we have another perspective, how can we? Become aware when others release their software automatically independent of whichever technology stack would use it would be nice to Become aware of oh, hey this version x-boys said is now available. So we automatically integrate it into our pipeline and go for it
04:43
And how can we visualize these connections? How can we see what kind of tests has been done? Upstream but also downstream on our software later on How can we see how many has downloaded our products
05:00
Be nice to know where they're test successful into their environment or not and why? So The solution for this was to be we Focused mostly on the communication. How can we communicate across different tools? So we came up with a solution is like a
05:23
Common language if you will across these different pipelines across different tools across different Visualization tools that could based off the same information And we called it a full knife listen Created in 2012. So we've been using it for a lot of years internally a
05:44
Couple of years back we open sourced it and it's now available on github But what it is it's a message protocol is event-based It's defining a set of events that you can use to communicate about different concepts used in your pipeline Be it artifacts be it tests source code changes. You can communicate about this
06:05
independent of whichever technology you use Doesn't matter if it's a Java application is uploaded somewhere to a Maven repository or if it's a Docker image you uploaded it to Docker Hub And you can still communicate about it in the same thing in the same way
06:22
and What's what I think is great about a felt at least it's you can share a pic whichever events suit your needs You don't have to take all of the protocol. It's a lot of events. You can share a pic the ones that you Specifically need in your use case. So let's say you're only interested in the test case related events
06:41
then you only matter and care about those and The events are also linked together and That's a very important for us at least to maintain the traceability between them So you can trace back from the latest event you can go backwards in time and see the whole pipeline chain Can represent the pipeline and with a full you can answer some of the questions. I post previously
07:08
For example when we have a new artifact available You can send out an event saying that I've now created this artifact You see it to the left of the in the picture. It's called a full artifact created event
07:24
What comes after that is could be a published event whenever you upload it somewhere You can send additional events linking back to the previously created artifact And the linking here is important The protocol itself defines how you can link your events if they must have or if they if it's optional links
07:48
These are all in JSON format against you're wondering and Next question could be how can we visualize our pipeline? It's a pipeline consists of many different steps, right and depend on how it looks you want to maybe
08:03
Tell the world about what has happened. How do we visualize this? This is a very simple CI pipeline, it's a GitLab syntax So you build your product then they test it. It tests it some more and then you upload you build binary somewhere
08:21
Very standard case and Some of the a file events that could be used for this are for example These you can have an artifact created event to signify the first built. This is where you actually build your binary or whatever it might be Then you start some tests and then you might want to send test case events
08:43
so you've got the triggered started and finished events for this and These can be sent of course these jobs are in parallel in this pipeline so you can send them in two sets And then comes the artifact published event as the last one to signify that hey now my artifact is available somewhere for
09:04
Downloading and I'm gonna show you the exact same set of events only connected in an event graph to make it clear So start to the left got artifact created event. This is the first one in chronological order
09:21
Then comes the test case events which links back to the artifact because this is the what what we're testing and The link name in this case is called item under test It makes makes perfect sense to me at least Then comes the test case started event and linking back to the triggering
09:44
And then we've got the finished event test case finished event This one contains the result of the test if it was successful or not now, of course if you have several test cases, you might want to group them under some sort of Label the saying yes, I've tested it to this extent and I want to label my artifact now, this is
10:04
Ready for the next level of delivery somewhere Then you can call you can send a confidence level modified event and this is a Very free formatted in a way that you can name the conference level whatever you want to as long as you standardize it, it shouldn't be a problem and then you point back to
10:25
The artifact that you've created So this is very useful for us as Listeners or consumers of this product. We know once it has been tested enough. We know when it's reached the confidence level we
10:41
wanted to have before we actually start using it and Once the last event comes in it's the artifact published event. We know where we can find it Where is the artifact located? Where can I download it? And I have one last wrap-up here as I said, we are
11:01
we have open source this and It's available on github and We really hope that the community keeps growing and Getting more contributions because we've seen that a lot of different companies and organizations have the same problems that we faced
11:23
And One thing I liked about a file is that it provides one common language across different technology stacks across different tools you might use so you can base a visualization tools on the events in this data, for example, and
11:41
What's also important is the links between the events it enables traceability so you can follow a complete set of pipeline You know what has happened to your software And you know where it might go even further downstream and I'd like to invite Frederick to this side of the room so we can switch mic
12:00
Because he's gonna tell you about how access started contributing and using a fill Set this up here. Okay, so so I'm from from a company called
12:26
Axis communications and and we do Surveillance cameras, which is a kind of a scary thing, but there are a lot of there's no software running in these cameras and Somebody said roughly around 20 million lines of code in the camera, so there's a lot of software going into it this small thing
12:44
it's a Kind of a it's basically a Linux box but but still and we have nowhere the scale that that Erickson has we have about a thousand developers in total and the team that that does the teams that do the Cameras are about 400 developers
13:02
and about 150 testers and we do we like to think we do continuous integration at least because we commit to master a lot of times every day and I'm going to try to tell you our a full story so Emily gave you a good background on what a full is and I now try to
13:22
To go through some kind of a timeline how we how we found out about a fill and how we adopted it and how we now Very much would like to drive our all of our pipelines using it. That's at least as the ambition for for the future and
13:40
First I would like to give you some context on what exactly it is that we do. So so axis is it's quite an old company and We've done in the history of done network connected stuff and and which it's basically just Making a some kind of hardware putting in a box and selling it That's the standard axis way that we've been doing it for a lot of years
14:03
And and it's basically what other companies do also days. They sell cardboard boxes with tech stuff in it but but more and more we've been seeing The demand from our from our business and from our from our customers that they want complete systems they want Cloud integration they want they want a lot of stuff which
14:23
Which in turn gives a lot more software that has to be tested has to work together So that that's where in there that comes the challenge of knowing how which which versions work together with software stacks works best together and so on so That's that's kind of our the similar challenge that Ericsson has
14:43
But back to our story The first recorded presence of the a full protocol at axis As as I've heard there were a couple of engineers Attending some kind of a conference or a meeting where where somebody from Ericsson was presenting some kind of visualization tool how you could visualize
15:01
Your software. I don't think it was called I mean, I don't even think there was referred to as a pipeline but but still some kind of software flow and And they found that this was something called a full which was underneath here and they were they got intrigued so they they started doing some tinkering on their own and trying to trying to make something with a full in the axis context and
15:25
And which was which was fine But it really didn't lead anywhere and then it resurfaced in some kind of a in a research Academia industry project that axis had where some of the researchers Had found out about a full from Ericsson also and
15:40
And it resurfaced and they did some more tinkering and some more engineers got interested But it was still kind of under the radar stuff just engineers like technology enthusiasts that did stuff but nothing that really drove the business or was that was a thing and In Like the next next big thing in my opinion was was when axis actually released something that had to do with a full
16:04
So these engineers had been tinkering more and they wanted to to make something of it They wanted to a full to produce something that was a value. So so they they we use Jenkins as Almost everybody else. So they made a Jenkins plug-in which was very basic. It basically
16:21
produced artifact create that event when we we did a build that was all we did so it was kind of worthless on Its own but still it's kind of footprint that that now we're actually producing a full events We have a message bus where there are events coming out but Anyone there was really nobody listening which was which is kind of a problem because it's it's when doing
16:48
This type of thing so you did the whole point is to somebody listening in and acting on the events and there is like a chain created and a flow So so our biggest problem here was was nobody was listening in and if nobody's listening in a pub sub culture
17:02
It's kind of hard to know if you're making sense if you if you in the beginning at least because because you have to know that that's In the beginning you have to have some kind of acknowledgement that your event. It was useful. There's a value to it so we had to to create some kind of value in our minimal a full implementation and
17:25
This is basically where I where I came in in the in the a full thing I was a part of a team that does test automation and We were struggling with our CI part of the testing part of our CI pipeline It was kind of old and we didn't it didn't really want to do what we wanted to do with our testing
17:45
so we We started looking to Rebuilding it basically or changing it and so on and I had a coffee discussion with a colleague of mine He he was one of those a full tinkering engineers He said you should you should look at a full if you're gonna do things from scratch
18:01
You might as well look at a full because it's pretty cool. You can do cool stuff with it and We looked into a full and we a colleague me and a colleague and we really liked it We we saw that this was this is something that could that could probably not solve our immediate problems But there was a bigger issue of the system of systems talk and knowing
18:23
information about the software that goes in from From everywhere so so we decided to make our new testing part of the CI CD pipeline talk a full and understand a full and In this we started listening to the Jenkins events the lonely guy over here
18:42
That was playing a full music and nobody was listening So we actually started listening to that and we started triggering our tests based on those those events Which which gave us much more understanding of what it is like to have an event-driven pipeline and what it was like to
19:02
adhere to a protocol like a full and and we actually wound up being a great listener today for protocol and during During the development of our our testing pipeline We had to learn a full basically. So we we
19:23
wrote a bunch of Python libraries and Two of them which were kind of important at least we thought we we said we had to we have to Contribute back to since it was an open source If it was an open source Community and open source protocol. We said we can contribute here. We can learn people how to
19:44
teach people how to how to use a full and These two lovers the a full Python lib which is a kind of small library that lets you publish Aful messages and listen to a film. It's just on the message bus, it's it's
20:01
Very useful if you want to get started with a full and then there is something called a full graph QL API, which is basically an API on top of the the The event repository which is basically a storage for events because if you want to look at events Back in history. You have to store them somewhere. You can't just keep them on the bus
20:23
So that's basically our graph QL Which is kind of useful if you want to start tinkering with with the storing events and looking back in history and so on And This this open-sourcing of this got Access more engaged in the community and we actually there's a there's a conference called
20:45
called a full summit Very small conference will be actually access actually hosted it This fall and In moving forward we're going to try to to keep the backlogs working with an open backlog
21:00
Concept as we learned about a full how we are our journey will try to contribute directly to the to the community Rather than just doing stuff inside and then just dumping it outside which is kind of a bad thing and then moving forward As a full is now a thing at access
21:20
We actually have a small team that work with a full and we're going to really going to try to to drive all all of our pipelines We see them being event-driven in some way moving forward and also also, we're going to try to to Contribute more to the a full community trying to to like Emily said want to spread it want to
21:45
Make more people try it out and then see if it can become a thing So we'll try to to contribute more of our experiences moving forward in 2020 and that's that's that's my access a full journey basically and
22:06
We'd like to thank you for listening to us and here's some some contact information the a full Community on github and we have a Google group and a slack channel and a YouTube channel also with some some videos if you Want to learn more or get a more basic?
22:23
Tutorial of what a full is and now I guess it's quick
22:53
So the question was is there some kind of a gitlab integration for for a full Not that I know of currently on on github, I don't always know not yet
23:06
So and I'm I'm not quite sure how get that works So I But it should be possible. There's no I don't see any like big technical Blocking it. It's just that somebody has to to contribute it and do it
23:23
but it's Kind of There are some if you look at the the Python library there is some there is some basic basic
23:49
tutorial how to set that up how to send events and and receive events and then there's also also something called I think it's called a full easy to use or Which is basically a set of docker containers that you could just spin up you get a full
24:04
Like a small pipeline which produces events, but I don't it's it's that's something that Ericsson contributed I don't know if all of it is open source yet might be but it's basically something you can try out To get some messages going and then try to get a feel for how it how it works
24:46
Yes Absolutely. So the question was since we've been sending events for several years now within Ericsson We have of course a lot of data accumulated So we the question was if we can look at lead times and see where are the bottlenecks and the pipelines and yes
25:02
of course, we do have that possibility and that's one of the Great benefits that we've seen internally at least and that's why I think I don't know how many Millions of events we have stored but there's a lot of them so it's indeed possible if you have like a
25:20
implementation of a storage somewhere and then you can look it up and you can see Because all the events are time-stamped so you can also see lead times between the events between different activities in your pipeline which is very useful and Yes, Frederick mentioned and just like to add because aside from the protocol we have some
25:41
Example implementations of services surrounding the protocol helping to send the events helping to Visualize them. I think we also have open-sourced and there are more coming the more planned For that it's just a matter of prioritization and so on Any more questions?
26:00
Yes in the back so the question was have we tried to use Eiffel outside of development to trigger some manual activities if I understood it correctly and
26:27
Not I'm not aware that we listen and act upon These events and then doing some sort of manual step, of course it it's at all possible, but the
26:40
listeners usually are Performing some sort of automatic task afterwards and as far as I know we mostly focus on up until continues delivery sort of deployment Stages don't really trigger on a fill events later on yet at least
27:02
I can add something to that So so there are there are like more generic a full events Like announcement events that say say something that something has happened and you could definitely use that for for for signaling other people in the organization and we actually use that to
27:21
Sort of a debugging thing that we so we know that we've gone a certain path through our testing CI system we do announcements here and there to see okay, we've gone this far because it's It's a microservices architecture. It's kind of hard to keep track of everything. So so it's it's a handy thing But there are announcements events that are very generic so you could use them for for almost anything
27:45
Any more questions? Okay. Thank you