Effective management of Kubernetes resources for cluster admins
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61657 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023257 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
SoftwarePrincipal idealPoint cloudSystem callGoodness of fitShared memoryComputer animation
01:19
Chaos (cosmogony)Multiplication signComputing platformOpen setCloud computingOperator (mathematics)BuildingComputer animation
02:21
Chaos (cosmogony)Software maintenancePoint cloudOperations researchData managementMultiplicationData storage deviceNetzwerkverwaltungSocial classOperator (mathematics)Self-organizationMetadataSpacetimeNamespacePhase transitionSystem administratorLevel (video gaming)Template (C++)Range (statistics)World Wide Web ConsortiumIntegrated development environmentContinuous trackBuildingSystem administratorData storage deviceCartesian coordinate systemProjective planeRollback (data management)Control flowWorkloadData structureFluxDirection (geometry)AreaArithmetic meanCASE <Informatik>Mixed realityType theoryDeclarative programmingComplete metric spaceDifferent (Kate Ryan album)Chaos (cosmogony)Operator (mathematics)Point cloudBitTemplate (C++)Order (biology)Spectrum (functional analysis)Social classFault-tolerant systemFocus (optics)Overlay-NetzValidity (statistics)Configuration spaceMathematicsPatch (Unix)Line (geometry)Integrated development environmentComputer fileContent (media)Configuration managementSpacetimeFluid staticsGame controllerRandomizationKey (cryptography)SubsetLimit (category theory)Single-precision floating-point formatServer (computing)Data managementSoftwareInternet service providerComputer animation
12:05
Configuration spaceRollback (data management)Execution unitStability theoryTexture mappingObject (grammar)Process (computing)Integrated development environmentContent (media)Core dumpData managementPoint cloudComponent-based software engineeringFiber bundleTrailSpacetimePatch (Unix)Rollenbasierte ZugriffskontrolleAuthorizationExtension (kinesiology)System administratorGroup actionPartial derivativeRule of inferenceInformation overloadBoilerplate (text)Kolmogorov complexityLimit (category theory)Fluid staticsNumbering schemeSoftware testingDisintegrationAdditionCASE <Informatik>Limit (category theory)Multiplication signNamespaceCodeService (economics)Data managementObject (grammar)Installation artComputer fileSingle-precision floating-point formatConfiguration spaceOverlay-NetzCore dumpPublic key certificateMultiplicationProjective planeRollenbasierte ZugriffskontrolleGroup actionDifferent (Kate Ryan album)Alpha (investment)Extension (kinesiology)System administratorRepository (publishing)Keyboard shortcutExecution unitProcess (computing)Flow separationForm (programming)Boilerplate (text)Software testingSet (mathematics)Operator (mathematics)Green computingValidity (statistics)Directory servicePatch (Unix)Software maintenanceFiber bundleRule of inferenceRollback (data management)Connectivity (graph theory)Software developerRevision controlParticle systemControl flowMathematicsState of matterPoint cloudPattern languageSelf-organizationPay televisionPartial derivativeNumbering schemeFluid staticsData structureRange (statistics)Computer animation
21:44
Point cloudSource codeNetwork topologyPoint cloudOperator (mathematics)Patch (Unix)MappingNumbering schemeCodeDifferent (Kate Ryan album)Revision controlSystem administratorSoftware repositoryMilitary baseSet (mathematics)ExistenceFile formatData structureRepository (publishing)Network topologyRollenbasierte ZugriffskontrolleComputer fileTrailNamespaceIntegrated development environmentSource codeData managementComplex (psychology)CASE <Informatik>WordWorkloadGroup actionCartesian coordinate systemSoftware developerProduct (business)Single-precision floating-point formatOverlay-NetzValidity (statistics)Complete metric spaceLink (knot theory)Pattern languageField (computer science)Attribute grammarCollaborationismOpen setLatent heatComputing platformMereologyInsertion lossPay televisionInterface (computing)Limit (category theory)Entire functionControl flowComputer animation
31:23
Point cloudSource codeNetwork topologyComputer animationProgram flowchart
Transcript: English(auto-generated)
00:07
So, last session for today and we'll make sure it's going to be a really, really long one so that you have to starve and don't get to the drinks. And I'm happy to welcome Tom from Red Hat.
00:22
Have fun and enlighten us. Thank you. Should I have fun or done? What? Should I have fun or done? Both. Okay. Okay. Okay. Hello, everyone. My name is Tom Sofau.
00:40
As you've heard, I work at Red Hat and I think the last talk was a great segue into my talk. So, if you were here for the previous presentation, who was here? Good. Good. So, we were talking about or the talk was about standardization, call for unified platform,
01:02
call for a sharing exchange of ideas, knowledge of findings and how to get to some kind of an open unified sovereign cloud. Well, we've been working on a thing like that for past two years or so, yeah, in an
01:25
initiative called Operate First, building open hybrid cloud platform ready for everybody to consume, to use, to look into operations, to look into metrics, look into whatever telemetry you have to actually do the operations yourself if you want to.
01:45
So, this talk is going to be focused precisely on that, on sharing a story, sharing a lesson that we learned during that time and maybe hopefully take it as an opportunity to not
02:03
just share that with you, but to also encourage you to learn those lessons for yourself and experience our pains and our challenges yourself. So, let's dig in. So, the talk is called Affected Management of Kubernetes Resources, the GitHub's Way,
02:25
GitHub's for cluster admins. So, first we're going to talk a bit about what is a cluster lifecycle and what's the role of cluster operations in that. Then we're going to experience the chaos that is out there in the world and then we're going to talk YAML.
02:41
If you've been on the YAML lighting talk, this is going to be a very slight variation of that, but more Kubernetes focused. And then we're going to bring some order to that chaos. So, we have these three crises of cloud management, right? We usually provision some resources.
03:02
We manage those clusters once they are deployed, once they are provisioned. We then deploy applications on top of them. If you are talking about Kubernetes-based cloud systems, this is the usual three pillars, three graces of what we are experiences. We have tools available for both ends of the spectrum.
03:25
So, for resource provisioning, we have great tools like Ansible, Terraform, Hive for cluster API in Kubernetes. This is an established pattern, established workflow that is widely used across hyperscalers,
03:42
across people who are deploying Kubernetes by themselves and so on. Good, this is a solved issue. This is a known issue. Then there's the application maintenance, application deployment, application lifecycle. Again, very well thought through aspect, very studied place.
04:10
We have tools like Customize and Helm, we have Argo CD or Flux CD to do continuous deployment of your workloads and to provide you with all the goodies like rollbacks to
04:24
previously non-broken deployment and taking it even further with other projects like, and now we forgot the name. What are we talking last about?
04:41
The SRE-driven deployment? No, you don't. OK, let's move on. So, what about the middle part, the cluster management itself? If we are managing Kubernetes resources, what are we talking about? If we are managing nodes, if we are managing tenancy, if we are managing networks, what are we
05:06
actually talking about and how we can manage that? So, we have these four main problems that we want to solve somehow, and we found out that basically nowadays it wasn't the case two years ago, but nowadays it's the case that we can
05:23
solve all of them through Kubernetes native resources, through YAMLs, through deploying YAMLs, applying YAMLs to our clusters. It's done by a few different means. So, we have main areas within Kubernetes API that we can explore to solve these needs.
05:47
We have multi-nancy, so we can solve that by just simple namespaces, cluster roles and whatnot. Cluster upgrades, again, we can apply install operators, talk to those operators and get those
06:01
clusters upgraded. For storage management, we can use operators, we can use storage classes and storage providers and custom resources if we want to deploy our own storage on, for example, bare metal clusters. And for network management, we can do that also through operators.
06:22
So, things like cert manager, things like nmstate, all of that can now happen through Kubernetes API natively. That's great. What does it tell us about the cluster management? It can be all managed as a Kubernetes application. It's in YAML.
06:42
Well, YAML is a mess. We know that, and we know that thanks to multiple aspects. YAML can be defined and stored in files no matter how you structure it. It can be a single file with many different resources.
07:01
It can be many different files, each of them holding a separate resource. And only asterisks in bash is the limit for your kubectl apply. You can do whatever you like on the client side. On the other hand, on the server side, the manifest that you apply to the cluster is not
07:25
the same that you get from the cluster back. It's modified, it's mutated. So, you have things like status. Some operators, some controllers modify also the specs. Some modify also annotations, labels and whatnot. And you don't have the full control over the definition.
07:44
You need to know what subset of the keys and values you can actually define as a declarative manifest for your resource. It's not the same as the manifest applied on the cluster.
08:02
So, how people store manifests online. If we pull random project on GitHub that is deployed to Kubernetes, you will find many of these patterns. kubectl doesn't have ordering, so people solve it creatively through numbering
08:22
their manifests. Some are aware that their application may run in different environments. So, they create different files with duplicit content with the same deployment with just few lines changed here and there. Some combine those approaches and in some projects, and we find that even in some
08:44
controllers for their dev setup, they have a single file with all those resources in line in there. This is not a standard, this is not a good practice. And if we want to manage environment which is live, which should be out of
09:04
instable, which should be approachable to people, this is not the way we should do it. So, in application space, we have basically two choices, how to organize, how to structure our manifest. One is through Helm, which is great if you're deploying applications and you want
09:23
some templating involved, if you want to quickly change many different places of the same manifest or of different manifests. So, you can basically create this template, applying these values, and you get the full YAML that we saw earlier.
09:43
Great, but is it readable? Is it understandable from the YAML itself without rendering? We don't think so, and we want our cloud manifests to be auditable, to be approachable, to be reviewable. So, if we want to be able to explore what those changes do on a PR review without
10:07
actually spinning up a cluster and applying that PR and maybe do some static validation on it, you can do that with this. You would need to render it, you would need to understand it. And if you change the template and you have different values for different
10:23
environments, how would it affect the template itself? So, you need to explore all the possibilities. And this is one of the biggest challenges in the Helm space that we are currently facing in application development. Then we have the other way, the Kubernetes native configuration management
10:42
customize, which is a bit nicer. All of those, all of those manifests are fairly easily organized and referenced through customization. And all those customizations are organized into bases and overlays.
11:01
So, it's a composition type of configuration that we have a base which defines basics. And then we have the overlay, which can patch and mix different resources. These resources in the base are already complete. This is a complete definition, complete declaration of my resource.
11:25
This is reviewable. So, we kind of thought that this might be a way, but before that, we defined a couple of rules, couple of directives that we wanted to achieve with this.
11:40
So, if we wanted to organize our manifest, we don't want to build our own solution. We don't want to build our own CI CD that would understand our manifest structure. We want to use something that is readily available with great community. We want the configuration to be stable.
12:03
So, if I change one manifest, it doesn't break five different clusters and those things that never happen usually happen. So, if something like that happens, I can roll back the faulty cluster, just that individual cluster. I don't need to roll back all the clusters that are working with that
12:22
particular configuration and it's unit testable. So, that's also an important thing. File mapping, also very interesting topic because YAML allows you to inline multiple resources into a single file, but we don't want that. We want the file and its name to fully represent the resource.
12:45
So, before I even open the YAML, I already know what to expect inside. I don't have to guess from a dev.yaml or from namespace.yaml, which also contains like OpenShift project or whatnot.
13:04
Each file is readable without processing. That's so explanatory. I want to be able to open the code tab on my GitHub repository and understand the manifest. If I'm defining the same resource on multiple clusters, if I'm applying the
13:24
same resource on multiple clusters, let's say I have the same user group on two different clusters, I want to apply similar or the same RBAC. I want to apply the same cluster roles, project namespace permissions and whatnot.
13:42
I don't want this definition to be duplicit, to be duplicated. Define differently, maybe differently, maybe slightly differently, maybe the same in two different places. I want to share the same definition. As a practice that we use in programming for ages, this is not a well-established
14:02
pattern in Kubernetes manifests. We want to reuse stuff. And as I told before, the file name already describes what's inside. So we came up with this pattern and this pattern has been vetted through a couple of
14:22
organizations that I'll show you later on. And this is a pattern that we come up to. We have a base for Customize, which references every single object that we deploy to any other any our cluster that requires elevated permissions.
14:44
If those resources are standard namespace scoped things like deployment, config map, secret, whatnot, this is the developer responsibility. They live in their own self-contained namespace and they can do whatever they
15:00
want in there. But if we are talking about creating namespaces or creating cluster roles, we don't want developers to create namespaces on their own or create limit ranges or create resource quotas on their own. We want to do this, set those things for them because we don't want them to
15:26
basically expand and take over the cluster if we don't want them to. So this pattern of API group, kind and name is actually kind of working because
15:43
already from the path base core namespace sovereign cloud or base FOSDEM org talks and I talk, I already know what the resource is about without actually looking into the file. Then I have overlays, which each overlay represents a single cluster.
16:05
And they have customization, which basically mixes and matches whatever resources I want to pull from base. And if I want to change something from the base, I basically just patch it because customize allows us to patch resources and applies either a strategic
16:23
merge patch or a JSON patch so I can do various things with that. This is very helpful if I have, for example, cluster admins group and I want different cluster admins on different cluster. But the group itself is already defined in base.
16:43
Well, this is nice, but it doesn't work in all cases. It doesn't solve all the issues. So we had to introduce two additional concepts. One is components, which is also an alpha extension to customize, which
17:00
allows you to reuse the same manifest multiple times. This is important in cases like RBAC, if we have role bindings that you want to apply to multiple namespaces, like granting this user group admin access to a certain namespace, because if customized by itself wouldn't allow us
17:24
to use that resource multiple times. So this is a limitation of customized in this particular case that can be overcome through components. And then we came up with bundles, which is an addition that basically
17:40
selects related resources from the base, which are always applied together. So imagine you want to install cert manager. It's always a namespace, it's always a service account with cluster role, it's always subscription or whatever.
18:02
Or cluster issuer for certificates. So all of these things come together and they are referenced as bundles, so we don't clutter the overlays too much. And we also introduced common overlays, which are region specific, which are shared across regions, because for some regions, we have a shared config.
18:25
So how such single cluster overlay customization looks like. We reference the common, we take all from common, which also references some things from the base and whatnot.
18:41
Then we can, for example, this way, deploy our custom resource definition for Prowl. We can create an namespace for Prowl and we can apply some R back to node-laborer. We can install a whole bundle for cert manager as is, and this ensures cert
19:04
manager is deployed and configured properly for this cluster. We also can specify a specific version for the particle OpenShift cluster to upgrade it, to do maintenance on the OCP version.
19:23
And if we want to, we can patch certain resources, as I mentioned, the cluster admin. So fairly simple pattern, but there's been a two year journey to get into a state where it's actually working across regions, where it's actually
19:41
working across multiple clusters and when it's efficient in managing multiple clusters through PRs, through GitHubs, through single file YAML based changes so it doesn't break all the clusters. What I didn't mention on this slide, each of the cluster has their own
20:02
separate Argo CD application, so they act independently in the CD process. They reference the same code base, but they are independent. So the rollback is possible. So in conclusion, to evaluate what we did here, we have no duplicity.
20:27
Manifests are readable, manifests are not confusing. The set of rules is fairly simple. It's nothing very complex or bulky. The CI CD is very easy and we can do static validation, we can do unit tests,
20:44
we can do integration tests. All of that can be done fairly nicely. What are the downsides? We have boilerplate in the form of customizations, in the form of components, in the form of very nested path structures, directory structures and whatnot.
21:04
Customize is not always very straightforward. So you need to learn the tools before you can use it. And what also limits our static scheme validation is that manifests in base can be partials because they are not always complete because we expect to patch them
21:25
in those overlays to, for example, set a specific channel for our operator subscription and whatnot. So that's that. We have four organizations currently adopting this scheme and running this scheme. We have Operators Community Cloud, New England Research Cloud,
21:45
Massachusetts Open Cloud and Open Source Climate Alliance all running on this pattern. So this is a lesson that we learned through collaboration in cloud operations. And I hope we may be able to learn more such lessons in the future by
22:06
exploring cloud together. So if you want to know more, you can join us in Operators Community Cloud. You can see our ADRs and how we got to those outcomes. And on the last link over here, you can actually see the code base that we are
22:24
running against all of those clusters. Thank you very much. Thank you for the talk.
22:47
We use the same, more or less the same pattern, but the one of the manifests in completion. We fixed it, we adopted an approach that we define those attributes that are required with customization overlay.
23:02
So like a dummy value and then you have completion and then you know that that particular value, it's a valid YAML because it matches the spec fully. But then you know visually that that particular field will be patched in overlay. So we use that as a solution for the manifesting completion and the static
23:23
validation. We always use customization over overlay. And then we know that we are going to do that. That's just a solution that we... I don't know if there is a better way or a better word to use for that, but that's our approach. We use the same, but it doesn't work in every case.
23:43
We found like in some cases, the scheme is very detailed. It requires this complex nested structure, like, for example, cert manager requires solvers. And if you define a solver, you can't remove it in a patch because it's a
24:06
mapping. So strategic merges don't work that way in Customize. You would need a JSON patch and you would need a long JSON patch. And, you know, it's becoming less and less clear in this regard.
24:20
I think another thing that we do is we have, for example, a common base like you and then have a non-production base, production base. And then, for example, for the admin groups. So we have a group of admins for the non-productions. And then we don't have a full group of admins in the base. And then we need it from the non-production or the production in case we need one
24:44
group or another. That's another approach that we have. Thank you. And then the last one.
25:01
In this case, when you have like a couple of bundles, maybe it's easy, but when you have a cluster with 12 or 15 bundles, it can be a little bloated. Having a single Argo CD app, managing all the applications of a single cluster. And we use this approach.
25:20
We have one for the cluster deployment with Hive. And then we have for each operator, we have his own tree. So we have independent applications. And when, for example, an operator breaks, it doesn't break the entire Argo CD application of the cluster. It only breaks the Argo CD application. Or when we need to break or we think it's safer because we are like really, really scoped.
25:45
And you can not break the entire cluster, just a single application. Yeah, we do the same for operators which have like specific deployments and whatnot. If we can deploy operators through subscription to the OpenShift operator catalog,
26:01
operator hub, we can do that through a single resource and then it's not bloated that way. So, yes, we same lesson that we face the same same issue. And we were solving it very similar in different.
26:21
Good, good, great, great. We should talk after. Yeah, hi, really nice talk. Thank you. Thank you. We build a lot of internal developer platforms and we face the same issue where we kind of lose track of the code bases. Do you implement any repo scanning or file structure scanning that makes sure
26:44
that this is enforced among your customized charts and kind of a two parter? Do you just block all use of the Helm charts? Because everything has a Helm chart nowadays and it would be kind of limiting to have to rewrite something in this format if there's an existing Helm chart or existing customized or is this only for, you know, org internal YAML?
27:06
Thank you. So we enforce this only for resources that require elevated permissions. If you have an Helm chart that is deploying custom resource definition, then we tell you this is not a good thing.
27:21
You shouldn't do that. The API wouldn't allow you to do that, like our RBAC settings. So we basically tell those people you need to get that CRD into our repository, check it in our base for resources which require cluster admin or elevated
27:43
permissions, because if we would reference it from somebody else, from some other repository, they can change it in the repository. We don't want to do that. And we don't want them to be applying CRDs because those are shared on the cluster. And if two people on the same cluster are deploying the same Helm chart
28:03
in different versions with different CRD schema, it can fight. And we don't want that. So that's why we want a single source of truth for all the resources that are cluster scoped or requiring elevated permissions. So Helm charts are allowed for developer
28:21
and application workloads in their own namespace, self-contained or across all of their namespaces if they have more, but not under our watch on the elevated permissions. Thank you.
28:46
You said that when you have several clusters, you can limit what the developers or the user of the cluster can deploy. But how do you manage that? For example, we use Pro, so we have a top interface and we have ownership.
29:01
So each environment has a set of owners, but we cannot limit. So a developer can create a customisation that adds a new namespace and statically we cannot limit what kind of resources it's going to be created by the developers inside his cluster tree.
29:22
How do you handle this? So if it's deployed from our overlays, we would know that. And if it's deployed from his own customisation repository or whatever, he wouldn't have the permissions.
29:41
To create a specific resources? Yes. How do you manage that limitation? So if he's, maybe I don't understand the question, but if I have a developer who has access to a set of namespaces, they can deploy only to that set of namespaces.
30:01
And if they onboard our ArgoCd to manage their application through our ArgoCd, they have their specific ArgoCd project, which also restricts the RBAC, so they won't be able to deploy to any cluster, just to that cluster that they have access to, and just to those namespaces they have access to.
30:21
OK, so the cluster resources are only managed by the operations team? Yes. And then the developers? Yes. In your case, you have a mix, so the developers can create patches and edit part of the tree of the cluster. So we don't know how to handle, like, they only can create a specific set
30:40
of resources and we do that through validation. So they can review, but we need to approve and manually review that they are not creating, like, namespaces or operators or cluster resources or things like that. Yeah, we limit that through a single code base, basically, through a single repository. But we also do this pro with chatops and whatnot,
31:04
ownerships, and that's a great addition. Any more questions? OK, then we call it a day. Thank you so much.