Delivering a crossplane-based platform
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61509 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 202328 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Source codeSoftware maintenanceInternet service providerProjective planeInformation technology consultingPlanningSource codeOpen sourceWordTheory of relativitySoftware developerExtension (kinesiology)Computing platformInternet service providerSoftware maintenanceFlow separationDiagram
01:07
MiniDiscUniversal product codePoint cloudData typeMaxima and minimaDeclarative programmingNumberObject (grammar)Graph (mathematics)Cartesian coordinate systemState of matterData managementRight angleExtension (kinesiology)Scheduling (computing)MetadataPower (physics)Data structureInformationMultiplication signRevision controlPoint cloudCloud computingSlide rulePlanningSource codeMereologyFerry CorstenCompass (drafting)Graph (mathematics)CurvatureComputer animation
03:51
Computing platformCollaborative softwareBuildingExecution unitMedianImplementationRevision controlMiniDiscPhysical lawPatch (Unix)Complex (psychology)Data structureIntegrated development environmentDifferent (Kate Ryan album)Logical constantBlock (periodic table)CodeNumbering schemeString (computer science)Projective planeComputing platformPlanningSoftware testingSoftwareInformationCodePatch (Unix)Line (geometry)MultiplicationEnterprise architectureLimit (category theory)Similarity (geometry)Multiplication signBuildingMedical imagingFluid staticsProduct (business)Error messageData structureRight angleBlock (periodic table)Logical constantGene clusterGoodness of fitTemplate (C++)Hand fanBitDifferent (Kate Ryan album)Software frameworkDynamical systemForm (programming)Object (grammar)Computer fileFerry CorstenRepository (publishing)Parameter (computer programming)ArmNumbering schemeNumberPoint (geometry)Video gameRow (database)Message passingLevel (video gaming)CASE <Informatik>Position operatorSoftware developerElectric generatorIntegrated development environmentCycle (graph theory)Social classComputer animation
10:44
CodeSoftware repositoryError messageSoftware testingCondition numberVirtual realityObject (grammar)Software testingCategory of beingElectric generatorCASE <Informatik>Flow separationVirtualizationGene clusterBoilerplate (text)Different (Kate Ryan album)Game controllerPlanningCartesian coordinate systemPhysicalismBuildingEnterprise architectureLevel (video gaming)CodeDifferenz <Mathematik>Right angleMathematicsSound effectNumberMultiplication signRepository (publishing)BitGraph (mathematics)Set (mathematics)outputSocial classFunction (mathematics)WhiteboardRoutingFilm editingServer (computing)Execution unitConstraint (mathematics)SurfaceArmInformationData managementCross-platformData storage deviceComputer animation
17:32
Extension (kinesiology)Revision controlString (computer science)Software testingParameter (computer programming)Patch (Unix)Type theoryError messageWritingNumbering schemeLink (knot theory)Slide ruleCommon Language InfrastructureInternet service providerCASE <Informatik>CodePatch (Unix)IntegerError messageComputer fileComputing platformOpen sourceValidity (statistics)Link (knot theory)Rule of inferenceSlide ruleSoftware testingLine (geometry)Type theoryField (computer science)Source codeDefault (computer science)Set (mathematics)Universal product codeQR codeGroup actionOpen setRecursionControl flowPoint cloudString (computer science)Multiplication signDynamical systemPhysicalismReal numberSoftware developerRun time (program lifecycle phase)Object (grammar)Key (cryptography)PlanningClient (computing)Barrelled spaceWeightWave packetFerry CorstenNumbering schemeBitComputer animation
24:19
Program flowchart
Transcript: English(auto-generated)
00:19
Hey everyone, I'm here today to talk about delivering a crossplane based platform.
00:30
A few words about myself, my name is Maximilian Blatt, I'm a Kubernetes and crossplane developer and consultant at Accenture in Germany. I'm using or working with crossplane for almost two years, it's two years now, and I'm the
00:49
maintainer of several crossplane-related open source projects, including the provider for AWS, the provider Styra, provider ArgoCD, and I've contributed to many more including
01:00
crossplane itself. Now, since this is the CICD Dev Room, I don't know if everyone is familiar with crossplane, so I just want to spend a minute or two explaining what it is. So crossplane essentially is an extension to the Kubernetes API and it allows you
01:21
to create cloud resources the way you would create resources in Kubernetes. So the thing on the left is something most of you probably have seen once or twice, which is Kubernetes pod, and it's a very common resource that you have in Kubernetes
01:41
and it basically just schedules a container where you can run an application. And on the right you see a bucket as you would create it with crossplane and it represents an actual bucket on AWS S3. And if you look at both of these objects, then you see that they are very, very similar
02:02
because they are both inside the Kubernetes cluster and you have both very common, or the same kind of structure. You have your API version and your kind. You have the metadata that comes with every crossplane, with every Kubernetes object. You have a declarative spec, so where you describe the state of the resources,
02:24
the resource that you want, and then you have the status information about the resource itself. And that is one of the features that crossplane does for you, so it connects external APIs, any kind of external APIs, with Kubernetes and lets you manage your
02:44
whole cloud infrastructure through one Kubernetes cluster. And the second very powerful feature of crossplane is that it allows you to create your custom Kubernetes APIs by using something that is called compositions. And then it's the thing that you can see in the
03:02
middle. It's a very rough and simplified graph to show the way crossplane works, and it essentially always works that you have a user claim for a resource for your API that you have to find using a so-called XID or a composite resource definition. And that is then passed to a
03:24
composition, and then the composition spawns a number of managed resources. Managed resources are something that you have seen on the slide before, which is in a bucket or any other kind of external resource, any other kind of external API. Today I want to talk mostly about XIDs and
03:44
compositions, because that is what you do most of the time when you are working with crossplane. Now, developing a platform with crossplane. If you look at a simple CI-CD pipeline, then you
04:00
have usually build, test, and then deploy, and that is very easy. And for most software projects that is also very easy to understand. But because crossplane is a bit different, you have different things that you do inside these steps. So what you do
04:23
with crossplane is you are first building and pushing a package, and you're not writing code, but you are just writing YAML objects, which are then applied on the cluster, and then they are handled and treated like data by crossplane. And then when you are testing
04:44
your crossplane platform, then you are applying all your compositions and your XIDs to a test cluster, and then you are claiming them, and then you see if they work. And then if that is okay, then you are deploying them, and you're just doing the same, but
05:01
on a production cluster. I don't want to talk about the deployment today, because that is very simple. That is basically just like doing a Kubernetes deployment. You are building an OCI image and then pushing that, and then you are installing that on a cluster
05:20
using crossplane, and that's it. There's not much to tell about, but I want to talk about the building and the testing. Let's start with the building. If you have worked with crossplane before, then that is probably very familiar for you. On the left you see an XID, as you would
05:41
write it, and on the right you see a composition. So an XID, I said it, basically just defines the API that your user has applied to, and it's very similar to custom resource definitions that you are writing in plain Kubernetes. So you have your API schema in the spec of your XID,
06:05
and then in the composition what you do is you define the resource that should be created when the user claims this API, and that can be an arbitrary number of resources. So you don't have to create just one resource, but you can create dozens of them. So I've written compositions where
06:22
you are creating 30 or more resources at once, but that is essentially how it is done. You are specifying a base resource, and then you can modify this resource by copying information from the user claim into the resource that you want to create. That is what you do the whole
06:46
time you are working with crossplane. You are writing an XID, and then you are writing a composition or multiple compositions, and then the user can claim it and then choose the composition that he wants. That now looks easy at first, but when you are doing this
07:04
on an enterprise level, then very easily you end up with compositions that can be thousands of lines of code where you are creating dozens of objects, and then because you are just dealing
07:20
with pure YAML, then you really are starting to get at a limit because you have a lot of things that are very repetitive inside compositions. You have very similar structures, let's say if you are spawning a lot of similar objects on your cluster, but in different
07:40
compositions, then sometimes you have the same patches that you are reusing. For example, if you just want to patch the name of a resource by what the user has given to you, then you are repeating this patch over and over for every resource, for every file you are writing. Sometimes you then have compositions who only vary in details.
08:01
If you have different environments, for example, you are in different AWS accounts and you only want resources to appear in specific accounts, or you have different values like the region or static resources that you want to connect, like the account ID.
08:22
Then you have to write the same composition over and over, but just with different values, and then you see that you are ending up with something that gets really, really complicated because you are just doing a lot of copy and paste. You need something to generate the YAML dynamically. In these two years, I spent a lot of thoughts on how to simplify this process,
08:47
and I have experimented with a bunch of stuff. We've tried out Q, which is some form of JSON-like framework that allows you to build structures and have them validated,
09:02
but it's very complex and not very easy for newcomers. If you have new developers and teams, then it's a bit hard to onboard them on it because the error messages are not very helpful in many cases. The tool that we ended up establishing was Helm.
09:26
I'm not the biggest fan of Helm because it's a bit quirky to use, and sometimes if you have error messages or if you have errors, then it's sometimes hard to detect where the error actually is because it just tells you, oh, there's something wrong with your YAML, but you don't
09:43
know where it exactly happened. But the good thing with Helm is that it can do everything that we need. You can replace common code blocks such as constants with things that you have written out in your values, YAML. You can use templates to parameterize patches
10:06
to save lines of code, and you can even replace the API schemas of XRDs by something that you can generate. That is a really, really cool thing. I just checked the code in our
10:21
repository, and we have about 100 lines of code for Helm. I'm sorry, 10,000 lines of code for Helm, and we are generating 200,000 lines of code of compositions that are then applied on our API clusters. If you are doing this, if you are generating code for crossplane with
10:48
Helm or any other kind of code generation tool, then I recommend you to check these generated YAML bits into your Git because, as it turned out, it's very hard to detect unintended changes
11:06
that you are doing in Helm with your bare eyes. If you are changing one value or a template somewhere, then it might have some side effects that you're not seeing so easily. I really recommend you to check these generated codes, YAML codes, into your Git
11:26
and do not treat it as artifacts. Then if you are in your CI, then what we are doing, and that is really helpful, is that you should regenerate all your package and your generated YAML and see if any diff appears. If that is the case, then you should just treat this as an
11:45
error and abort. If there is no diff, then it's okay. Then you can continue on and push your package to the OCI repository. Now, so much for the building. Now let's look at the testing.
12:04
The first things that you are doing probably when you are starting working with crossplane is that you are writing your composition, and then you are applying it on a cluster, and then you are claiming it, and then you see if it works, if all the resources get ready, and if you can
12:23
use them, and then it's done. That is all manual. That is very easy to do because it requires no additional setup, and you can just use the cluster that you have. But when you really want to do automatic testing or enterprise level testing, then that is not enough.
12:48
Because you have manual steps, you have an outcome that is not reproducible because you are doing the things all by yourself, then also you don't have to find what is actually the expected
13:02
outcome because sometimes even if a resource gets healthy, it doesn't mean that the resource is configured the way you want it. So we also tried and tested a few things, and we started with Go testing, but it turned out to be much more complicated because you have to write a lot
13:24
of boilerplate code stuff, and so we ended up using Kettle. I don't know if some people know it. It's basically a Kubernetes testing toolkit that allows you to define all your test
13:41
cases in YAML, and then just let Kettle do all the work, all the application of the YAML on the server, and then you can define the resources that you expect afterwards. And if you imagine the graph that I showed you before where you have the composition, and then you claim it,
14:03
and you have a number of managed resources that are then spawned, and so you can have the claim as an input, and then you can just define the resources that you want to have created as an output, and then you can let Kettle handle all the rest for you. And then it can do things
14:22
in parallel and such, and this is a really, really great thing. So I recommend Kettle. Just to show you an example how these tests look like. So you have your small bucket claim. If we are sticking to this simple bucket example, then you have your bucket claim
14:44
on the left, which is your test case, and then on the right you are defining all the objects that you want. You have the bucket claim itself, which has a resource status that should become ready, and then you have composite resource, which is an internal resource that gets created by
15:03
Crossplane where it stores some reconciling information, which should also become ready, and then you have your actual bucket manage resource, which also has properties that you are expecting it to have, and it also has a status. And so that is all you need to do testing with
15:25
Kettle for Crossplane. And one thing I want to highlight is because in Crossplane the names of the composite resource are always generated by the Kube API server, so every time you are claiming an API, the name is different. It's always different, and you cannot influence it.
15:48
And so what you can do with Kettle is you can let Kettle identify the objects that you are expecting via the labels. You don't have to pass the name, but instead just tell YAML that you just
16:01
want an object with certain properties and label set, and then Kettle will look for one object, for any object on the server, and if there is one that satisfies this constraint, then you are good to go. One other thing that we've experienced is very good, is you should run your tests in
16:30
separate clusters for every pipeline that you are running. So we are using virtual clusters, OB clusters for that, that are run inside a physical cluster. Of course you can create your
16:45
own physical cluster all the time, but if you're spinning up physical clusters, at least on EKS it can take up to 30 minutes, and that is not something that you want for every test, and it also costs a lot of money. And so you're just spinning up virtual clusters,
17:02
which are Kubernetes control planes that are running as pods inside a cluster, where you can then install Crossplane, its providers, apply the compositions, and then run all the tests with Kettle, and then once you are done with the tests, then you can just delete the cluster and everything is fine. And also you don't have any intervention between two different pipelines,
17:25
because compositions are cluster scope and they are most likely overwriting each other. Now I've been talking a lot about end-to-end tests, and they are really good, and I recommend you to write end-to-end tests when you are building a Crossplane platform. But end-to-end
17:46
tests also take a lot of time to run, if you're considering that you have an API where you are creating real physical cloud resources, and then you always have to wait for your resource to actually start, and then after some time maybe it says that something is misconfigured and then
18:04
you have to look for an error, and if you're really just doing development that it really slows you down, because you have always this 10, 15, 20 minutes gaps between something happening.
18:20
And there are a lot of mistakes that you can make when you are writing compositions, and so I just want to highlight a few things. So you have these composite type rest that reference a composition with the XRD they have to match, and they are only validated at runtime. Then you have the group names which have to match with the XRD name. You have an unstructured open API
18:44
schema, because Kubernetes does not support recursive API schemas yet. Maybe it will come in the future, but as of now it's not supported. The same goes for the resource base which can
19:01
have any kind of field. And then you have the resource patches. By default the behavior in Crossplane is if you want to patch from a field to another field and the path of your source does not exist, then Crossplane's default behavior is that it will just ignore
19:20
the patch and it will not throw an error or anything, and if that is the case then you might easily swallow any errors and then you're wondering why things don't work, but you just have a typo in your patch and it's really hard to find these if you have 2000 lines of YAML code. And then you have types that must match, so if the user's inputting a string
19:44
then you have to make sure that the string is actually expected and not an integer on the actually bucket API for example. And then you have the indentation, the big thing that when you are writing YAML files, that is my big problem if I'm writing YAML files I always
20:04
mess up the indentation and then things get all messy. So we need something to detect these errors sooner, because the sooner you detect an error the easier it is to fix. So what we have done, because there is nothing out there, at least we couldn't find anything,
20:27
we've developed a linter for Crossplane compositions where we are loading actual XID and CRD schemas and then comparing them with the compositions and then applying
20:41
a set of rules like ensuring that the composition actually supports a valid XID type that you don't have duplicate objects which can sometimes happen especially if you are generating things with Helm. And then the most important thing is that it actually validates the patches that you are running against the CRD and the XID schemas and that is really really helpful that
21:06
the first time when we ran this against our production code it turned out to have I think 800 errors that nobody noticed. But somehow our platform still worked.
21:22
Yeah and the other cool thing about our linter is that it's pure CLI and you don't need a Kubernetes cluster or a Crossplane installation you can just run this locally without setting anything else up and you can it really takes maybe one minute or two and then you have all your compositions lintered and that is really really really great.
21:47
If you're wondering where to get it there will be a link on the last slide where you can find the code. Yeah summing things up, so this is our CI CD pipeline that we have developed after
22:02
a couple of years of testing and failing. So we use Helm to write and build our compositions to generate the YAML code dynamically. We use our self-written linter to lint our compositions and we use Kettle to run all the end-to-end tests and then we are just pushing things with
22:26
Crane or any other kind of OCI tool that comes handy. Yeah so much. Here's a QR code for the linter. We are actually making this open source today
22:44
so you are the first one to actually see the code except us. Yeah thank you. Do we have time for questions?
23:02
Okay any questions?
23:25
So my question is more about crossplane then crossplane lint. This looks really good though. How does crossplane compare to things like cluster API and the CRDs that that introduces? Like where's the distinction between the two of them just you know if you're familiar with
23:40
cluster API? So crossplane makes use of CRDs under the hood. So if you are applying your XADs on the cluster then crossplane will generate CRDs which are then used as the API that the user can claim. If there are no more questions then thank you.
24:13
We're gonna make our five minutes break.