An Approach to Air-Gapped Deployment
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/34568 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
ChefConf 201740 / 45
1
4
5
9
12
16
18
20
21
22
24
25
27
28
35
37
39
42
00:00
SphereComputer networkInformation securityComputerInternetworkingVisualization (computer graphics)SoftwareVirtual machineComputer fontSelf-organizationSoftware developerEnterprise architectureBuildingStudent's t-testMereologyThermodynamisches SystemNuclear spaceComplex (psychology)Configuration spaceDigital rights managementUsabilityCodePublic key certificateRun time (program lifecycle phase)AreaMobile appProjective planeData storage deviceBitContext awarenessStudent's t-testGame controllerProcedural programmingInternetworkingThermodynamisches SystemSoftware developerVirtual machineSoftwareCodeMereologyNuclear spaceInformation securityRight angleOvalTouchscreenWordComputer clusterLogical constantRing (mathematics)Multiplication signMicrocontrollerMetropolitan area networkPoint (geometry)Configuration managementDifferent (Kate Ryan album)Message passingComputer engineeringComplex (psychology)Validity (statistics)VideoconferencingRepository (publishing)2 (number)Cartesian coordinate systemImage resolutionConnected spaceProcess (computing)Term (mathematics)Self-organizationMoving averageIntegrated development environmentUsabilityInheritance (object-oriented programming)QuicksortReal numberConfiguration spaceLetterpress printingBuildingSheaf (mathematics)Graph (mathematics)Proxy serverSound effectVideo gameRule of inferenceEnterprise architectureEqualiser (mathematics)Bus (computing)Run time (program lifecycle phase)GodGradientGroup actionEndliche ModelltheoriePower (physics)Polar coordinate systemBackupArray data structureOpen sourceFinite-state machineJSONXMLUMLComputer animation
09:39
WebsiteInternetworkingOperations researchThermodynamisches SystemSystem programmingState of matterSoftware testingIntegrated development environmentForm (programming)Component-based software engineeringWorkstation <Musikinstrument>Server (computing)Virtual machineComputer configurationInstallation artClient (computing)Electronic meeting systemProduct (business)Process (computing)Similarity (geometry)Local ringCache (computing)Virtual machinePower (physics)Water vaporProduct (business)Group actionServer (computing)Right angleConfidence intervalComputer configurationValidity (statistics)Multiplication signComputer fileShift operatorCodebuchOpen sourceBitSoftware developerInternetworkingData structureClient (computing)Connected spaceMathematicsComputerInstallation artTouch typingLoop (music)Workstation <Musikinstrument>Term (mathematics)Digital Equipment CorporationCovering spaceInstance (computer science)Pairwise comparisonCore dumpWordPoint (geometry)Self-organizationField (computer science)Local ringProcess (computing)WeightForm (programming)NumberVector potentialObject (grammar)QuicksortSet (mathematics)Projective planeComputer programmingThermodynamisches SystemInformation securityWebsiteCircleRevision controlConfiguration spaceElectronic mailing listState of matterCache (computing)Asynchronous Transfer ModeCartesian productSpacetimeIP addressPOKEGoodness of fitOperator (mathematics)Line (geometry)Single-precision floating-point formatDifferent (Kate Ryan album)Computer animation
18:57
Virtual machineCache (computing)Workstation <Musikinstrument>Local ringServer (computing)Configuration spaceSelf-organizationComputer fontInternetworkingProcess (computing)Similarity (geometry)Integrated development environmentSocial classComponent-based software engineeringCodeRevision controlShift operatorDigital photographySpacetimeProjective planeInstance (computer science)Game controllerProduct (business)Computer fileRepository (publishing)Self-organizationQuicksortVirtual machineCountingReal numberDefault (computer science)Set (mathematics)Internet service providerPulse (signal processing)Web serviceSphereData conversionWave packetMultiplication signWorkstation <Musikinstrument>Software developerServer (computing)Execution unitThermodynamisches SystemInternetworkingValidity (statistics)Process (computing)Social classMoving averageVideo gameDevice driverGreedy algorithmMedical imagingCodeGroup actionDifferent (Kate Ryan album)MereologyProxy serverIntegrated development environment1 (number)Open sourceState observerCartesian productComponent-based software engineeringNormal (geometry)Ocean currentInformation securityLoginPlotterData centerOnline chatBitPlug-in (computing)CloningPasswordElectronic mailing listRight angleSoftware repositorySystem administratorComputer animation
28:16
Virtual machineScale (map)Similarity (geometry)Computer-generated imagerySoftware developerServer (computing)Flow separationDigital rights managementCodeLink (knot theory)Public key certificateSoftwareFunction (mathematics)Military operationOperations researchSelf-organizationNP-hardType theoryProcess (computing)Computer fileValidity (statistics)Source codeSelf-organizationServer (computing)Software developerBootingWorkstation <Musikinstrument>MereologyRepository (publishing)Different (Kate Ryan album)Medical imagingSheaf (mathematics)ReliefProxy serverVirtual machineIntegrated development environmentBitCodeSoftware repositoryJava appletCartesian coordinate systemData centerPublic key certificatePlug-in (computing)Revision controlData miningRoutingType theoryProjective planeNP-hardLimit (category theory)Table (information)SoftwareInformation securityVideo gameCycle (graph theory)MultiplicationDependent and independent variablesConnected spaceProduct (business)System administratorSingle-precision floating-point formatOrder (biology)Functional (mathematics)Multiplication signState of matterProcess (computing)CloningControl flowOnline helpClosed setShift operatorSpherePoint (geometry)Right angleParsingStudent's t-testDenial-of-service attackOpen sourceAnalytic continuationInternetworkingFlow separationGroup actionCircleWeb pageMessage passingRule of inferenceExtension (kinesiology)Computer animation
37:34
Computer animationJSONXML
Transcript: English(auto-generated)
00:05
I'm Joseph Gonzalez, I'm here on behalf of Sandia National Labs and if you're here to see an approach to air gap deployment, then you're in the correct place. And if you're here to see a different talk, then you're not in the right place.
00:24
So, who is this talk for? Oh, I don't know. I probably won't tell you. Pretend that I had this on the whole time. The video is going to be dead silent for the first couple of seconds.
00:45
So, who is this talk for? Try to intend this talk to be for everybody. I'm throwing up some terms on the screen. Most of you probably already know exactly what these mean. But I don't want to leave anybody out in the dark.
01:02
Quick note, I'm going to be using the words stand up and provision probably interchangeably a few times. And those words can be used in a few different ways. I just want to make sure that you guys all know that I'm using it purely as the virtual machine infrastructure, not any software set up or anything like that.
01:21
Let's go ahead. So, what am I talking about? I'm going to talk a little bit about who I am because we all love talking about ourselves, don't we? I'm going to talk about our projects so that way you guys have some context of the work that I did. I'm going to talk about architecting the solution. This is the air gapped organization deployment.
01:42
You can see there it says chef countertop. You guys will know what that is in a little bit. And then also because we're running the organizations in deployment or in air gapped environment, we also have to have some kind of procedure or pipeline to validate our cookbooks.
02:03
And I'm going to try to close down with some advice for new developers and hopefully we'll be done on time. And that way we can have some questions at the end. So, that's underlined right there, those two sections. Those are really the bread and butter of what I'm going to be talking about.
02:22
So, who am I? About me, I'm from California. Anybody else from California? Yeah, sweet. Of course we're from California. Super expensive, right? Born and raised in California. I have a bachelor's in computer engineering from CSU Sacramento.
02:42
Recent graduate 2016. My professional experience includes Hewlett-Packard Enterprise. I spent my last year there, my last undergrad year there as a storage performance engineer. So, that was fun. First experience and the first time I had a little bit of money.
03:03
So, it was kind of nice to go from noodles to actually being able to go out and maybe buy Chipotle. And luckily, right after graduation, about a few days, I think two days after graduation, I got hired on by Sandia as a solutions architect. So, I was very happy about that and have a whole lot of down time.
03:24
So, I got to have a lot of Chipotle right afterward. So, I'm also new to DevOps. During my undergrad, you know, I've heard of the term but for me it was just like a buzzword. I'm very new to Chef. I've been using Chef for about eight months now.
03:41
And I've just recently, the past few months, have been using it in a meaningful way. So, that was about me. But, you know, that's more of the professional side. But who am I really? I do a lot of 3D printing in my off time. That's pretty fun.
04:02
Microcontroller programming, MicroPCs. I have a couple apps on the App Store. And of course, we're all lifelong students. If you're not learning from every experience you're going through, then you're not really taking a whole lot away. That's, I don't know if you guys recognize it, that's Groot from Guardians of the Galaxy.
04:20
That's a bust that I 3D printed. I'm also building a 3D printer with my 3D printer right now. So, that's pretty fun. So, that's a little bit about me. What is, I'm guessing a lot of you guys are probably like, what the hell is Sandia, right?
04:41
It's a part of the NNSA, the National Nuclear Security Administration. Got it right, I'm pretty sure. Too many acronyms in government. It's headquartered in Albuquerque, New Mexico. I'm from Livermore, California. That's where I work. Started in 1948.
05:00
We mainly work with government agencies, also industry and other academic institutions. Our primary mission, let me get this right. The synergy and interdependence between our nuclear deterrence mission and broader national security missions forge a robust capability base and empower us to solve complex national security problems.
05:23
Rolls right off the tongue. So easy to say. What does that really mean? That means that we, our missions basically support national security. And that whole message was brought to you by Control C, Control V. So, that's a little bit about me, a little bit about Sandia.
05:42
Any quick questions real fast? We're good? Cool. So, let's actually start talking about some stuff. Why does this matter? Configuration management is awesome. And if you don't think that, then you're probably at the wrong conference.
06:01
It obviously adds ease of setup, adds visibility, traceability. Those kind of go hand in hand. So, you know, you upload your infrastructure into a repository as code. But it also adds network dependency. So, when you're trying to set up Chef and you're using Knife,
06:21
Knife goes in the background and tries to download packages off the internet. And if your infrastructure doesn't have any internet connections and the recipes you want to run on it are to set up its configuration for internet use, then it's like chicken and egg constantly. And you're like, what the hell do I do first? So, yeah. And you might have other restrictions like proxies, man in the middle issues.
06:44
And we all know that moving infrastructure code is not always easy, especially when you're going from development to production environments and the differences between those environments. So, yeah, Chef orgs require internet for their first setup.
07:02
And I'm sure most people here know that. So, when you first run Knife and you do a bootstrap, it's going to check your system and then try to download the packages off the internet. So, how do we get to where we want to be, where we want to use everything and it not break because of internet issues? We want to install packages and set up our systems.
07:24
But to go to the solution first, we actually have to talk a little bit about the project so that way, the project I'm working on so that way you guys have a little bit of context. So, my role in the project was and still is basically to architect the way we use Chef.
07:42
And also, if we're going to be using Chef, we also need some sort of validation process, right? We need to have some kind of pipeline. And the great thing about Sandia is that you never know, the next day you might have a new project on board or you may have tons of new roles the next day. So, everything is so exciting.
08:04
So, the project deliverable for this project that we're working on is basically a full stack delivery. We're delivering the virtual machines, which is the infrastructure, the run time, we're using Docker containers and craft.
08:22
We're also delivering the applications that are running inside those containers as well. The machines need to run in a private facility and we don't own that facility for security reasons. The environment that all the machines are going to be running in
08:41
are completely air gapped for security reasons. And all needs to be set up as infrastructure as code, which is Chef's part of the machine, obviously. We have a lot of machines deploying into many different environments. So, we have to avoid any manual machine installation possible
09:03
because that's a pain in the butt. We need code traceability, not only for me, I'm the main developer with Chef, but not just traceability for me for seeing what's going on and uploading stuff to repos and stuff, but also our internal security process
09:21
is that we have to get everything approved before we send it out since we're a government agency. So, it has to go through a ring around a little bit. And when you can point somebody towards recipes that define the machine, it makes it a lot easier and a lot faster to get through that process than having an intern or somebody else
09:42
go through the machine and poke around and write a list of what the hell's on there and version numbers. And finally, it needs to be installed on site and we cannot deliver it over the internet, for security reasons, again. We have to sneakernet it over, drive over with a disk and give it to them.
10:01
Quick note, this is actually from Galen's talk. It's easier just to assume that all development is also air-gapped when you're deploying to an air-gapped environment. Instead of just, oh man, I'm gonna change this, change this, change this when we move over. Just assume everything's air-gapped from the get-go.
10:23
It'll save you a lot of time, save me a lot of time. Advantages of Chef for this project obviously makes it easier for operations. And basically, because we can predict the states of the machines with our chef recipes,
10:44
they shouldn't sway too much. So as I said before, we're also delivering it and we can't touch it. We don't own the environment it's running in. So being able to have some kind of confidence that we know what's happening when we leave site kind of just eases our anxiety a little bit.
11:02
The cool thing about this too is that now we have two forms of delivery. We can deliver large updates, we can deliver machine organizations, chef organizations, as virtual machines already ready to go and running,
11:21
or we can deliver small updates by just adding cookbooks and just shipping the cookbooks to the customers. Reproducibility, reproducibility, I think that's pretty obvious with Chef. That's not just for our project, that's just Chef in general.
11:40
All right, so architecting the solution. So this is where things start to get a little bit fun. So the two main things that I developed was Chef Countertop and that's what I used to deploy to air-gapped environments. And then the simple pipeline that I created
12:00
to give ourselves some kind of validation and confidence with our cookbooks. So, mouth is dry, excuse me. So the goal of Chef Countertop is to, like I said before, turn existing machines, I'm not provisioning the machines,
12:21
they're existing machines into Chef organization machines without any internet. So Chef Countertop basically includes all the packages. It needs server, node, ChefDK artifacts. They're all packed into the cookbook itself. So right now it doesn't rely on artifactory or nexus
12:41
or anything like that to pull down packages. Another one I was investigating was Pulp. I don't know if anybody's familiar with Pulp. Open source. But that is right now. It's probably going to change in the future with more investigation, one step at a time, right?
13:00
It's a comparable solution to Chef. So has anybody here ever used Chef Ingredient? No, not a single person? No? Okay, a couple people, thank you. So it works kind of similar to Chef Ingredient where you basically define your project in a recipe, and in your recipe you run your,
13:21
you know, Chef zero, whatever you want to run, and you provision all of your machines that way. But this is a little bit different. This is without any internet. The main resource I'm using inside the Chef Countertop Deployer is the converge only option of the machine resource,
13:40
and it's probably one of the other actions that people don't use as often. But basically, it doesn't install Chef before trying to do a convergence. All right, can we dim the lights a little bit? This is a little hard to see.
14:04
So I'm going to run through a little bit of animation so you guys can kind of see the process a little bit. There we go. And then could you click the bar for me? Thank you very much, sir. Yeah, the space bar's fine.
14:21
Yeah. So I feel bad for you guys, but just try to look over here, I guess. But go ahead and click. Oh, that's good. So this is like your local workstation, right? These are the potential target machines that are going to become the Chef organization,
14:41
and you can see those IPs are awesome, aren't they? One, two, three. So basically, you run a Chef Countertop on your local machine in zero mode. I think it used to be called Chef zero, client zero, I don't know, the exchange. Let me hit it a couple times. Hit it a few more times.
15:01
Two more. So the first step is that your local machine, the cookbook tells it to push over the Chef RPM. We're using Red Hat machines. Push over the Chef client RPM to the target machine. Go it again. Go it again. There's a lot of clicks in here for some reason.
15:24
A couple more times. And then it basically tells it to install Chef. Very simple. Hit it like three times, please. I wish this would be a lot easier if I was allowed to use a clicker, but I can't plug USB devices into that computer. Yay, government.
15:43
The next step after installing Chef is to push the client Ruby file. So that client Ruby file will tell the target machine to point back towards our local machine as its Chef server. So go ahead and click it about three times. So we're running all this in Chef0,
16:01
so we have a Chef0 instance. So it points back towards the Chef0 instance. Go ahead and click it a few times. And the red line is basically a Chef connection. Hit it three more times. So it checks into the server because it thinks that your local instance is the actual Chef server. Go ahead and click it.
16:25
And it tries to do a convergence, so it pulls over and caches Chef countertop. You guys are all good. Can you guys see me a little bit? Like, everything's good over there? If anybody's confused, just raise your hand and I'll try to answer your question. Go ahead and click it a few times.
16:42
So when it starts converging, it's going to try to converge with the recipe inside Chef countertop that is dictating its role. So the first machine is supposed to be a Chef server. So when it starts converging, go ahead, it starts converging on the Chef server recipe.
17:03
It's going to load, and your workstation is the server. So install Chef server, configure Chef server, and then after it's done with that, it's going to cut the cord by removing the client RB file. Go ahead a few times.
17:21
So now it's kind of lonesome all by itself. It doesn't know what to do. But it is a Chef server now. So click it three times. Now it's a Chef server. Yay! It's a red circle. It's a production server. So this is our production set, right? These target machines are going to be our production set.
17:41
Be what you were meant to be. Yay! So that's basically the process. I can go through now that they got it. Thank you. No worries. Making sense, right? Is the animation beautiful? So the production workstation basically does the same thing.
18:05
All the target machines connect as nodes first. Checks in, caches the Chef countertop cookbook. Converges. Converges this time on a workstation recipe to set up workstation. So that installs Chef DK, and it configures the connection
18:22
to the new Chef server. So the production Chef server that you see on your right side is where it's going to start connecting to now. So it cuts the cord to your local instance and then connects to your new Chef server.
18:41
And when it cached the Chef countertop cookbook, this may be a little hard to see, but inside that Chef countertop cookbook existed inside the files folder as artifacts, the production cookbooks that are supposed to run on the nodes inside this organization, and also the policy files as well.
19:03
I know policy files are a little newer. Is everybody familiar with policy files? I see. So I'll just explain it real fast. So policy files, it's a really easy way to dictate the run list for your machines. So it's basically instead of using roles like you normally would when you're setting up your org,
19:23
you can use policy files, and that way you can push those to a repo and you can version your policy files now. So it takes your policy files and your production cookbooks, and it'll push them to your new Chef server.
19:40
So now they exist, and you use them in your org. Yay, it disappeared. So now you can see the Chef server and the Chef workstation can talk to each other. Basically what it was doing in the background is it was configuring your knife setup and everything.
20:01
So it's basically just a normal workstation, so you can use it like one, and it can be used to update cookbooks or policies later on. It can be destroyed, whatever you want. Production nodes. So production nodes is exactly the same process, just a little bit of a tweak.
20:20
First, it checks in your end caches cookbooks from your zero instance, just like before. Sets up on the first convergence, it'll set up itself as node. On the second convergence, it's a little bit different though. The second convergence is an actual production convergence, so whatever its role was supposed to be,
20:43
it'll also do that as well. So during this, it also gives you that assurance that this organization is going to work because it should work. All the production cookbooks should have been validated, should have went through a pipeline. You never know, something might have squeezed in that was bad, and this will let you know
21:02
because it will fail. A lot of clicks on this. So yeah, now you have a Chef node. Now you have a production set, and when you're done running the Chef zero instance against these target machines, your organization that's on the right side
21:22
should be fully running. So does that make sense to everybody? Cool. I think it's pretty cool. But I'm also a nerd. So this may be a little hard to see, but this is actually a photo of,
21:41
oh, thank you. This is a photo of Chef Countertop being used. Chef Countertop is actually a lightweight resource provider, so you use it per project. So if you have project A, project B, project C, you find it in your default recipe for that project, and you use it as a resource. I'm going to give these guys a chance real fast
22:01
because I feel like I'm ignoring you guys. You guys are like the child that's living in the attic. So right here you can see that the first resource is a Countertop server. The next one is Countertop workstation, Countertop node. And you can see it's pointing at certain machines already, and then it's also got production admin password.
22:22
So when it's setting up the Chef server recipes, it knows how to set it up. So you can log in right afterwards. And then if you see here, this is a little bit long, but bless you. You also point it towards the production cookbooks and the production policies that you want to run,
22:41
and those live as artifacts inside the current cookbook. So that way, nothing is reaching out to the internet. The production cookbooks that are running shouldn't be reaching out to the internet, but also the organization cookbook that you're running right now shouldn't be reaching out to the internet. So it's very similar to Chef Ingredient.
23:01
Chef Ingredient runs in a very similar manner. You define your projects. Thank you. You define your projects, and instead of using Chef Ingredient, you use Chef Countertop. So some lessons that I learned, talk about Chef Ingredient.
23:22
It was actually easier to stand up machines beforehand instead of using provisioning drivers, just because we have so many proxy issues and stuff like that, and we have a few different security environments. And it's also, we didn't really need it.
23:41
We didn't need that added flexibility, just because our customer is actually providing us with images that we have to use, so we're not going to be provisioning our own images anyways. So that was Chef Countertop. Any questions about that? Cool.
24:01
So Chef Countertop is now running the production cookbooks. Oh, question. Oh, do you have a question? So, any Chef Countertop? Yeah, so we're actually manually moving them. That's part of the recipe. And then, I don't know if anybody knows this,
24:22
but Chef Zero instance, you can pass it whatever you want. It'll automatically accept it, and then it does something. So that's what's cool about Chef Zero instance. Yep. That was a really good question. Nobody's asked me that before.
24:40
You know what you're talking about. So, before we get to, you know, getting to Chef Countertop and deploying it, we have to have some kind of pipeline validation process for our cookbooks. So I had to come up with something. It's fairly simple. We're using GitLab. GitLab Runner is the GitLab CI.
25:01
Everybody's familiar with GitLab or at least heard of it, right? Cool. It's an open source, not just a repository server. It has a built-in CI, and it's like DevOps-style CI, which is nice. It gives control to the developer. And then, so these are the three classes of cookbooks
25:22
that go through the pipeline. A resource cookbook, these are self-defined. A resource cookbook is very, very simple. It can be used by many projects. It's like, you know, turning a service on, turning a service off, or installing a service. Role cookbooks are for machines, are per machine.
25:42
So before policy files, I know a lot of people were using roles to Chef server to dictate what the nodes should do. Instead of that, I moved towards a role cookbook style. So this whole cookbook is per machine, so every machine should have a cookbook, and it makes it a lot easier to manage
26:01
when you're trying to figure out what's wrong in the end in our development environments. And then the organization cookbook. And the organization cookbook is basically countertop. The image I showed you before, this image right here, is not actually Chef countertop.
26:20
It's a project, project A, project X, using Chef countertop as a resource, right? This is what I would call the default recipe inside a Chef organization cookbook. So they're all dependent on each other. That's why they're scaling down like that.
26:42
Organization cookbooks, the production set that's inside the organization cookbook are just a set of role cookbooks. So the main components of it are obviously the cookbooks that are going inside of it, the pipeline. And then I have a Chef organization set up in our vSphere data center
27:03
to be tested on. That's obviously the technology that I'm using, Chef and vSphere. And they're all being ran by the GitLab CI, which is called GitLab Runner. It's very nice. So these are the steps that the cookbooks go through.
27:22
First, we want to deploy to our target organization. So whenever code is pushed, whenever a cookbook is pushed, the GitLab Runner will pick it up. Excuse me. The GitLab Runner will pick it up, and it uses knife vSphere to clone the Chef org that I have sitting in our data center.
27:42
I'm not sure how many people have used knife vSphere before, plug-in. Basically, it uses the vSphere API in the background to control your data center, which means it's super easily scriptable if you're using Jenkins or something, which is very nice. So I could control a whole data center,
28:01
which is probably not what I should be doing, but I am. So basically, it clones the organization that I have floating there. It turns off the original organization so there's no IP conflicts or anything like that. And then it takes the cookbook, the target cookbook, and it uploads it to that new cloned Chef server,
28:21
places the policy file on the node, and then invokes a run of that node. So you have a cloned organization sitting there, and we're using it like you normally would. We're putting the recipe inside the server. We're telling the node that the node should run on that policy file.
28:42
We run InSpec on it, of course, because if you don't have any kind of validation of any testing, you're not really doing a pipeline correctly. So we run InSpec against the node, and we say, hey, did it pass? We move on to the next state. The next state, we basically just destroy the temp organization that we made.
29:02
We boot back up the original org, and if it was successful, we upload that new cookbook into that original Chef organization so now it could be used as a dependency in our role cookbooks or in organization cookbooks. Make sense? Lesson page, cool.
29:23
Everybody just ate too and had lunch, and it was like, ugh, nap time. So lessons learned. Machine persistence is a huge lesson that I learned. Basically, I opted for the kitchen style. So kitchen, you spin up machines, you run your code, you run your InSpec,
29:41
you destroy the machines. I just wanted to bring that to a bigger scale because we're using custom images that were provided, and I want to actually test them on security-hardened machines. So it's very, very similar. We create machines, we destroy them. So we always know the state of the machine.
30:01
One downside of using a pipeline and constantly pushing cookbooks to it is that I found myself writing inverse cookbooks to try to put the machines back into the original state, and I look back and I'm like, why am I doing this? Why don't I just destroy and make, I have access to a data center,
30:22
and I'm like, okay, I'm going to do this. So that took a little bit of time, but GitLab CI allowed me to do that very easily, and knife vSphere as well. Really cool because it enforces the cookbook order. So we had resource cookbooks that live inside of role cookbooks. Role cookbooks live as the production set inside organization cookbooks for your project.
30:41
And because the Chef server, because the developers are not controlling the Chef server, we're really making sure that the machine that gets uploaded to the Chef server actually works. So my favorite part is that we're separating the developer from the Chef server. We're making the CI the workstation machine
31:04
that uploads the Chef server so nobody's uploading any broken code ever. And since none of the developers have to do any of that, it makes it easier, especially because certificate issues behind the middle when we have different classifications
31:21
for networks internally. It makes it kind of hard to publish Chef servers. Advice for new developers, and probably some current developers. So I came, I used Jenkins a little bit before,
31:44
but honestly I'm in love with GitLab and GitLab runner. Just because Jenkins you have to set up, that's a separate server, that's a separate login, separate everything. So you kind of have an admin managing that. With GitLab though,
32:01
GitLab's runner you use, it's more DevOps style because the developer can upload the YAML directly to the repository and then that's how you dictate what the pipeline should do. So you're giving every single developer access to controlling their own pipeline. So that's very nice.
32:21
Very different style than Jenkins. The life cycle of the machines, keep that in mind when you're doing development. And containerized software. So one of the big things, and I might even be able to talk about it because I'm looking pretty good on time, is that our application development,
32:42
we have multiple repos per application. We have an application repo for the actual Java code or whatever the application does. And then we have a counter repo that basically gets triggered and builds Docker images by pulling all the source code
33:00
from the other repository. And then when you're trying to deploy that with Chef, it just makes it a lot easier because with Chef you don't have to install a bunch of different prerequisites for your software. You can just tell the Chef recipe to install this Docker image and it just makes all of your recipes so much cleaner and less tangled.
33:21
That's what we're doing right now. And it works very well. And it kind of relieves a lot of headaches as well. This is probably super obvious, but it wasn't obvious for me. Attach enough NICs to your virtual machines. I separate NICs per responsibility.
33:41
So whenever I create a virtual machine or we're actually handed virtual machines, I edit them slightly. I add extra NIC for the Chef connections. So that way, if there's any operational functionality that's clogging one of the NICs or something like that, you can always go back and fix it with another recipe or change it with a recipe.
34:01
So you constantly have a persistent connection with the Chef server of that organization. And that limits downtime. It helps you stay up. I do this using IP tables and routing rules. It's not super hard. Red Hat makes it a little bit more difficult to achieve that, though.
34:22
So quick recap. This talk was about the Chef Countertop. Chef Countertop is how I deploy my organizations into air-gapped development environments that mimic our production environments. You should always assume air-gapped for everything,
34:41
not just for your production environment and how you're going to migrate to that. Countertop contains all its own dependencies. For now, we're probably going to use something like Pulp or Nexus later on. There is an organization cookbook per project. So the organization cookbooks define your projects, right? Define your machines that you're setting up.
35:01
So that's per project, per enclave, whatever you want. And it's good to keep that hard classification between the organization and the role cookbooks and the resource cookbooks, because having a steady standard going through, especially because we're all new to Chef in my group,
35:22
it makes it a little bit easier. So hard classification types for cookbooks. And if you are going to be making your own pipeline, I don't know what you're using, but if you can and if you do have access to your vSphere data center or knife vSphere plug-in or whatever,
35:45
or if you want to just do kitchen in your pipeline, I highly recommend it just because trying to write counteractive cookbooks is a pain and you probably shouldn't have to do that. And also, I don't want to harp on this too much,
36:04
but people just uploading versions of cookbooks to Chef servers I do not like. It's a pet peeve of mine. But doing it in this way and letting the CI have access to it, and that is the workstation that's uploading
36:22
is probably the best route. It keeps everything from breaking, and trying to fix that break and go through your Chef server is a pain in the ass. So, in closing, I'm a Chef newbie still.
36:41
I'm still learning a lot. I included that last section to try to help everybody else out. And if you're working for government agencies and you have security issues and proxy issues, I'm hoping some of you guys can relate to this. So we talked a little bit about the pipelines, the restricted internet, and our development process for this is still evolving
37:02
and I'm still doing research on it. And I'm learning more every day. And I was very glad to be a part of this world. And especially because I'm so new to DevOps, so this is like a learning chef and everything, so this is really exciting for me. And we have a few minutes for questions,
37:22
if anybody wants to ask any quick questions. And I'll also be out in the hallway as well right afterwards, so if you guys want to ask anything. Thank you.
Recommendations
Series of 3 media