An Approach to Air-Gapped Deployment

Video in TIB AV-Portal: An Approach to Air-Gapped Deployment

Formal Metadata

An Approach to Air-Gapped Deployment
Title of Series
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Pitfalls, brick walls, and struggles I faced while navigating the seas of Internet dependent software in an air-gapped environment. This discussion will be from the perspective of a new member of the Chef community, Sandia National Laboratories. Our team has been researching and adopting DevOps techniques to automate our workflows and deliverables. Being one of the architects tasked with exploring Chef as a solution to configuration management, I have had the pleasure to design some of our development process and architecture. This process is continually evolving to suit our specific needs. All of our products are deployed to air-gapped production environments. We are working toward not only using Chef to build our environments but also delivering our environments as fully functional Chef organizations to ensure our contracted work is predictable after delivery.
Polar coordinate system Point (geometry) Context awareness Backup Mobile app Enterprise architecture Open source Software developer Multiplication sign Letterpress printing Virtual machine Sheaf (mathematics) Microcontroller Student's t-test Sphere Mereology Computer 2 (number) Array data structure Thermodynamisches System Term (mathematics) Visualization (computer graphics) Software Computer engineering Endliche Modelltheorie Self-organization Area Enterprise architecture Computer font Touchscreen Building Software developer Gradient Data storage device Sound effect Computer network Bit Virtual machine Word Internetworking Video game Self-organization Procedural programming Quicksort Information security
Building Group action Thermodynamisches System Building Software Bus (computing) Finite-state machine Right angle Student's t-test Mereology Mereology God
Complex (psychology) Group action Context awareness Run time (program lifecycle phase) State of matter Code Confidence interval Equaliser (mathematics) Multiplication sign Nuclear space Set (mathematics) Mereology Configuration management Mathematics Different (Kate Ryan album) Oval Website Information security Workstation <Musikinstrument> Public key certificate Nuclear space Software developer Complex (psychology) Electronic mailing list Bit Mereology Virtual machine Connected space Message passing Thermodynamisches System Process (computing) Internetworking Ring (mathematics) Repository (publishing) System programming Website Self-organization Software testing Right angle Quicksort Information security Point (geometry) Game controller Open source Image resolution Virtual machine Codebuch Rule of inference Field (computer science) Power (physics) Number Revision control Thermodynamisches System Internetworking Computer programming Touch typing Integrated development environment Data structure Digital rights management Proxy server Form (programming) Operations research Pairwise comparison Shift operator Graph (mathematics) Validity (statistics) Run time (program lifecycle phase) Server (computing) Weight Projective plane State of matter Code Usability Computer network Configuration management Cartesian coordinate system Component-based software engineering Loop (music) Integrated development environment Form (programming)
Pulse (signal processing) Group action Installation art Multiplication sign Plotter Workstation <Musikinstrument> Set (mathematics) Water vapor Client (computing) Cartesian product Configuration management Computer configuration Electronic meeting system Core dump Circle Process (computing) Data conversion Local ring Covering space Workstation <Musikinstrument> Bit Instance (computer science) Computer Virtual machine Product (business) Connected space Process (computing) Computer configuration Self-organization Normal (geometry) Right angle Quicksort Spacetime Point (geometry) Server (computing) Computer file Virtual machine Number Product (business) Power (physics) Goodness of fit Cache (computing) Term (mathematics) Installation art Validity (statistics) Server (computing) Client (computing) Configuration management Vector potential Similarity (geometry) Cache (computing) Word Digital Equipment Corporation Object (grammar) Local ring
Ocean current State observer Server (computing) Group action Real number Multiplication sign Workstation <Musikinstrument> Execution unit Virtual machine 1 (number) Device driver Login Mereology Product (business) Online chat Medical imaging Thermodynamisches System Internetworking Different (Kate Ryan album) Integrated development environment Process (computing) Information security Proxy server Self-organization Greedy algorithm Computer font Shift operator Projective plane Counting Similarity (geometry) Digital photography Internetworking Internet service provider Self-organization
Process (computing) Internetworking Validity (statistics) Integrated development environment Process (computing) Instance (computer science) Similarity (geometry)
Server (computing) Game controller Open source Computer file Software developer Projective plane Virtual machine Wave packet Medical imaging Web service Integrated development environment Repository (publishing) Self-organization Video game Social class Moving average Quicksort Social class
Medical imaging Default (computer science) Computer font Shift operator Component-based software engineering Projective plane Self-organization Counting Set (mathematics) Social class Product (business) Self-organization
Web page Server (computing) Validity (statistics) Code Server (computing) Code Sphere Revision control Component-based software engineering Message passing Data center Self-organization Cloning Extension (kinesiology)
NP-hard Group action Code State of matter Java applet Multiplication sign System administrator Workstation <Musikinstrument> Source code Sheaf (mathematics) Mereology Public key certificate Medical imaging Type theory Different (Kate Ryan album) Military operation Single-precision floating-point format Circle Process (computing) Information security Link (knot theory) NP-hard Public key certificate Software developer Closed set Flow separation Parsing Virtual machine Connected space Data mining Type theory Process (computing) Repository (publishing) Software repository Self-organization Right angle Cycle (graph theory) Point (geometry) Server (computing) Open source Link (knot theory) Software developer Computer-generated imagery Virtual machine Control flow Online help Student's t-test Rule of inference Product (business) Revision control Flow separation Software Analytic continuation Digital rights management Proxy server Self-organization Scale (map) Operations research Shift operator Dependent and independent variables Inheritance (object-oriented programming) Server (computing) Projective plane Code Denial-of-service attack Limit (category theory) Cartesian coordinate system Sphere Similarity (geometry) Software Integrated development environment Function (mathematics) Video game Table (information)
kind of that in the in the area of the new place in a lot in this form so that the sum of of
at end thank you everybody pretend that I have this the DSM-V dead silence the 1st couple seconds of so was a stock for a tried and I intend talked before everybody of the of some terms on the screen most of you probably already know exactly what these mean but but at lead anybody on the dark of quick note of a monetary using the word stand up a provision probably interchangeably few times of and those words can be used in a few different ways I just wanna make sure that yes or no them using purely as a virtual machine infrastructure not any sulfur celebrate life that effect secure the so what am I talking about on the talk about a little bit that who i am because we all love talking about ourselves to there are many talk water projects that we assess in context of the work that I did someone talk about architect in the solution this is the air gap point so you see there's as chef counter tops and as a lower that is a little bit and also because we're of running the organizations in deployment during there yet environment we also have to have some kind of procedure or pipeline to validate cookbooks and try to close down with some advice for new developers in and hopefully will be done on time and I would get some questions at the end so that's underline right there those 2 sections of those are really the bread and butter 1 and talked about so who am I but only from California anybody else from California In the this week England and of course for people with this source right on arrays California have a bachelors in computer engineering the using of from SEC Sacramento a recent graduate to those 16 who my professional experience includes a Hewlett-Packard enterprise I spent my last year there of my last undergrad you there as a sort performance engineer and so bells by 1st experience the 1st time I had a little bit of money so it's kind I from models to actually being able to go out and they might tuple led this that and likely but right after graduation but few days I think 2 days after graduation I got hired on by Sandia as Solutions Architect so 3 happy about that and have a lot of down time so I can have a logical a grade afterward so I'm also knew that what's the during 2 of my undergraduate part of the term but from you just like a buzzword of a very new to share by using Schaeffer about 8 months now and i've just recently the past few months they have been using it in a meaningful way so those about me I but you know that's more the professional side but who am I really but the it a lot of 3 D printing in my off time as refined and microcontroller programming backup PCs have collapse in the App Store I'm and of course were all lifelong students if you learning from every experience you're going through the anomaly taking away that's a lot
of years recognizer that groups that from God reality that's a bus that a 3 D printed also building is a 30
printer with my 3 printer right now so that's pretty but so that's over what we what is I'm guessing lot against public with the holes India right that's a part of the NSA the the National Nuclear Security Administration got right don't richer but to me acronyms in government that it's a headquartered in Albuquerque New Mexico and from Livermore California that's where I work I started in 1948 I we mainly work with government agencies and also industry and other academic institutions our primary mission
I he it's right the synergy and interdependence between nuclear deterrence mission broader national security missions Ford robust capability based on powers to solve complex national security props rules right off the top so is users what I really mean that so we are our missions basically or national security of now whole message was brought to you by control c control the so
that's a little about me a little bit about India um n equal questions real fast called so let's actually start talking about some stuff so why does this matter of configuration management is also and if you don't think that then you're probably wrong conference the obviously as he is a set up as visibility traceability this try your hand in hand so you can upload here own infrastructure into repository as code the but they also adds no dependency so we're trying to set up a Shefford using knife I knife goes the background and tries download packages of the internet but and if you're infrastructure doesn't have any Internet connections and the rest you wanna run on it or to set up its configuration for Internet use then it's like chicken and egg constantly like what the hell do do 1st and so yeah you I have other restrictions like proxies of Mandel issues and we all know that moving infrastructure code is not always easy specially with from development to production environments and the differences between those environments I so the action of course require in it for the 1st set up in a true most people here know that uh so the first one knife thing you do of the trap it's gonna check your system and try to download the packages of the internet so how do we get to where we wanted the horror 1 use everything in it not break because of Internet issues that what we wanted so packages and set up a systems but to go to the solution 1st we actually after to talk will be about the projects so that way on the part of our counselor we assembled of context so my rule project was and still is basically to architect the way we show and also if for abusing chef we also use some sort of validation process right you have some kind of pipeline and ready Ross idea is that you never know the next day you might have you project on board or you may have tons a new role the next day so that is so exciting and so the project deliverable of for this project they're working on is the full-stack delivery our delivering the virtual machines uh which is the infrastructure the runtime were using a darker containers and graph but there were also delivering the applications that are running inside those containers as well the machines only to run in a private facility and we don't own a facility but for security reasons of the environment that it's going to the all the machines review running in a are completely air gap for security reasons In in for sure any always be a set up as infrastructures code which is just part of the machine obviously but we have a lot of machines depleted many different environments so we have to avoid any manual machine insulation possible can entertain the but when code traceability not only for like me and that I'm the main developer whichever up nitrate just traceability for me free no seeing what's going on in uploading set to repose and so but also are internal security process is that we have to get everything approved before we send it out were like government agency and so it has built a ring around a little bit of and when you can give somebody points every towards recipes that define the machine makes it a lot easier what faster to get through that process then having an interna somebody else go to machine program right a list of what the hell's on their and version numbers the the and finally needs to be installed on site and we cannot delivered over the Internet security reasons again we have the sneaker net it over drive over with the desk and give it to them but quick note the all this is actually from the dealings talk fields here but it's easier just assume that all the development is also here gapped when you're when you're the point in a draft of environment a set of just all them change this changes changes will move our just assume everything there adapted from the get go on middle city allotted time saving a lot of time advantages of chefs for this project obviously makes it easier for operations of and basically because we can predict the states of the machines but with our big fish with our of chef recipes missions way too much so as I said before were also delivering and we can't touch at this we only environments running and so being able to have some kind of confidence that we know what's happening when we leave site of kind of just eases are anxiety a little bit pull thing about this too is that now we have 2 forms of delivery of we can deliver large updates we can deliver machine organizations such of organizations from as virtual machines already you ready to go on running or we can deliver small updates by just adding codebooks just should because works to the customers the reproducibility Reproducibility of I think that's pretty obvious was just that's not just for a project such a shift in general alright so architecting the solution so this is being circular but fun so the 2 main things that I developed was chef carrot top and that's my user deployed to air gap environments and then the simple pipeline and I created to give ourselves upon a validation confidence of the cookbooks so structure excuse me the so what the goal of Shut carrot top is to like as support turn the existing machines are not provision machines their existing machines on into chef organization emissions without any Internet the so check can't uh basically includes all the packages it's me it needs server node shifty K artifacts they're all packed into the cookbook itself so right now doesn't rely on artifact neccessray and I got a pull-down packages of another 1000 best Giddings pull the matches the based loop hole open source but that that is right now it's probably get a change in the future of with more investigation 1 step at a time right it's comparible solution to chef and so as anybody everybody here we use a chef ingredient no messing impression that capable of making so it the worst kind of similar to shift ingredient where you basically Dov that define your project in a recipe and in your recipe you run your you know ships 0 what everyone around and you provision of all of your machines that way but this is all different this is without any
but the main resource and using inside uh the chef carrot top Deployer is the cover of converge only option of machine resources and it's probably 1 of the other the rest of actions that people use as often but but they say basically doesn't install usher before trying to do urgent power can be delights little bits is low will so we're going to run through a little bit of animation so yes you can see the process a little bit it killed thinking that that or from thank you very much yes these 1st time the idea that so FIL bad for you guys but just try to the trailer go here yes but when the number of so this is like your local workstation right these are the potential target machines that are going to become the chef organizations and like he's are also like they want to 3 of so basically run a chef carrot top on a local machine in zero-mode a DEC culture you of announced a point 0 have no the exchange of think it couple times In a few more times 2 more so the 1st step is that your local machine the cookbook tells it to push over the chef Arkansas easier Red Hat genes which over the years chef-client are appealing to the partition during good again for the from someone couple times and it's basically tell the to install show how very simple it like 3 times what is the personal so realize 0 follows a lot a clicker like of USB devices and the computer from Yale governments could the next after its is to push the client Ruby file so that that kind of will tell the target machine to point back towards or a local machine as its Chef server for going too quickly about 3 times so we're running all this ships 0 so we have a shared 0 instances answer points back towards the ship 0 instance could in terms the end the red wines BCI shelf connections on hit it 3 more times so check sent to the server could things that your local instance is the actual shot server good that it was and it tries to do at convergence of pulls over and Katusha character Yisrael the KSC me a little bit like of these you know if it was Confucius region of for a question ongoing quite a few times so when it starts converging it's going to try to converge but with the recipe carrot top the that is dictating its role so the 1st machine is supposed to be a chat server so when it starts converging go it starts converging on the ship server recipes in right now it's acting as a node in at workstation is this server so consortia server configure Chef server and then after its stone that it's going to cut the water by removing the client for of hobby file a few times so now it's kind of lonesome all by itself doesn't want to but it is adjusted so I could 3 times now to show server J you can see because it's red the red circle the production server so this is our production set right which would be sort of machines are going to be our our production the which always meant to be yeah so that's basically the process so I think the good now I'm thinking it making sense right the animation beautiful so the production workstation basically does the same thing that goes and cashes the cookbooks I connectors noticed that 1st of all the target machines connectors nodes 1st objects and caches Bishop countertops the book converges converges this time on a workstation recipe this of workstations so that installs chef decay and a configures the connection to the new ships so that production starships release see on your right side is words this are connected together so it cuts the core to your local are instance and then connects to new ships sir In cash the chef carrot top cookbook is maybe a little harder to see um but inside that ship character cookbook existed inside the files folder so as artifacts the production could books that are supposed to run on these on the nodes inside this organization and also the policy files as well the unipolar suppose a little where a is a really familiar policy files I see us under 6 literal fast of so father to really easy way to a dictate the less for your machines so that space instead of using the roles like normally would always say setting of your work the it is also falls and and that we can push those 2 you know everybody conversion here Paul spells out the so takes your pulse evolves in your production books and it'll push server so now they exist and I can actually use them in your work but it disappeared so that you could see the chef server the chef workstation can talk to each other obviously will soon background is it was configured in your knife set up everything so it's Visitors normal workstation of the user like 1 they could use that they could books policies later on I can be sure that everyone production and so productions with exactly the same process to just a little bit of a tweak there's actually 2 different convergences that occurred on the notes 1st you know checks in your and caches cookbooks from Europe 0 instance just like 4 sets up on the 1st convergence it'll set up itself has on the 2nd convergence is will be different but the 2nd convergence is an actual production convergence so whatever it was its role was supposed to be it also do that as well so during this it also gives you that assurance that this organization is when work because you know it's it should work all those that production cookbooks shouldn't validation winter pipeline but you never know so they might that was bad and this will let you know because they will fail plot clicks on this and I have a chef node they have production set and when you're done running the ships 0 instance I guess Osaka machines your organization that's on the right side should be fully running but if
so that makes sense a running all I think it's critical this but also in the set so this
may be a little hard to see that is actually a photo of that thinking this is a photo of shift on top being used just count subsection sexual lightweight resource provider up so you use it per project so the appointed a Project B Project C you define it in your befall recipe for that project and use it as a resource of harmony give these guys a chance real fast is like for young and knowing so here's like the child at that's willing addicts 50 sorry to see that their 1st resources that can't observer the next ones can talk workstation count top node and it's the it's pointing at certain machines already and it's also got you production at passes so when it set up the once chat server recipes it knows how to set up the article login right afterwards and then that if you see here this is a little bit of but the you also pointed towards the production cookbooks the production policies 1 around and those live as artifacts inside the current cookbook I'm so that way nothing is reaching out the internet that production could put the region of the internet but also the organization cookbook that you're running our notions of which are the so it's it's very similar to shopping Reshef agreement in a very similar manner you define your projects I think you define your
projects and you do instead of doing using chef grating user can talk the of so some lessons I learned Russia greedy but creating machines that was actually easier this uh stand up machines beforehand as of using uh provisioning drivers but just because we have so many policy of proxy issues of like that and we have a few different security environments but it's also we didn't really need it any that act that added up and flexibility just because our customers action providing us with images that we have to use so we're not gonna be provisioning our own images in units so that was should contact up me questions about the so shut carrot top is now running but the production cookbooks of question that think of this over time so in Japan yeah so we're actually manually moving that's part of the recipe and
then if I don't know if anybody knows this bunch of 0 instance you can pass it whatever you want it automatically accepted so you can pass a garbage but is on the path that something that the school Scheffer ships your instance there's really a
question it was asking for do it either and so before we get to you know getting to shift can count up and we have we have to have some kind of pipeline validation process for cookbooks so
I have come up with something it's it's fairly simple reason get lap and Ghil'ad runners the idea let's CI and there is you familiar with the latter early sort of it right who it's open source of Matisse repository of server it has a built-in CI I mean it's like deadlocked styled CI which is nice this control to
the developer and then so these are the 3 classes of cookbooks that go through the pipeline of resource could be the itself define a resource cookbook is very very simple to be used by many projects so it's like you know turning a service on train service offered songs service role cookbooks but or for machines are permission so before I for policy files that I know what you were using rolls and to show server dictate with the notion a instead I I move towards a role cookbook style so this whole cookbook is per machine so every initiation of a cookbook and makes life easier to manage uh when you're trying to figure out what's wrong in the end you know our development environments and then the organization of book and the organization cookbook is basically a carrot top of the image I showed you
before it this image here is not actually chef can top it's a project project a Project using shift count up as a resource right this is what I would call the default recipe inside a chef organization cookbook so they're all depended on each other
that's why the scaling down like that organization cookbooks that production said that inside the organisation cookbook are just the set of role cookbooks so the main components of it are
obviously the cookbooks they're going inside of it of
the pipeline and have a chef organization set up in our the sphere dataset to be tested on an that's obvious technology that using feature and the 0 being ran by the get let's see I just called
give ever run experience so so these are the steps that the cookbooks go through but 1st we want ploy to our target organization so whenever code is pushed whenever a cookbook is pushed the giver run will pick it up excuse me the galette runner will pick it up any user's knife of the sphere to clone the shaft work that have sitting in descendant of much only the 1 used by the CIA before plugging so basically this is uh the the stair API and background to good for your data center on which means it's super easily scriptable if you're use like Jenkins or something which is very nice so I control the a center which is probably now what I should be doing this but I am so I basically it comes organization are floating it turns off the regional organizations so no IT conflicts and the like that and then it takes the cookbook the target cookbook and uploads it to that new cloned Chef Server places the policy polymer node and then invokes a run of that node so you have a clone organization sitting there and were Usenet like normally would were putting the recipe inside server were telling the node that the node should run on a that policy fire or urine inspect on it of course because if you don't have any kind of validation of any testing and you're not really doing a pipeline quickly us we inspector gets the node and we say that a pass we move on to the next day and the next day we basically just destroy the temp organization that we made the blue back up the original work and if it was successful we upload that new cookbook into that original shot organization so now to be used as a dependency in our role cookbooks or an organization cookbooks extensive bills and page who everybody just a to have lecture was that
also lessons learned machine persistence is a huge lesson that i learned how to basically I opted for the kitchen style so kitchen you he spent machines you run your code you run inspect you destroy the machines I just wanted to bring that to a bigger scale because but were using custom images that were provided and I want actually test them on those security hardened machines so it's very very similar we create of machines we destroy him so we always know the state of the machine 1 downside of using a pipeline and constantly pushing cookbooks too it is that I found myself writing inverse cookbooks to try to put the machines back to the rooms they and of the bakelite 1 way doing this why don't I just destroyed and make have access to the user so it's like 1 of my students and so that a little time but uh do let's see I allowed you do that very easily and make the sphere all on really cool because it enforces the Cookbook or so we had resource cookbooks and that live inside of brawl cookbooks role cookbooks live as the production inside organization cookbooks for your project and because the chef server but because the developers are not controlling the chef server but were really making sure that everything gets uploaded to the Chef server actually works source separate my favorite parser were separating the developer from the Chef server our making the CI the workstation machine that uploads the ship server so nobody's uploading any broken code ever and you know since you know the works at now the developers have to do any of that makes easy especially because certificate issues that little or we have different classifications for of networks internally a Mexican hard to publisher chef servers who had advice for new
developers and probably some current developers so I came I use Jenkins a little bit before but but honestly I love give up a giver have rented Justice J. did you have to set up you know that's a separate server that's supper separate Ellogon separate everything so you can have like an admin you know managing that right with give lab go get labs runner use it's more Dev style because the developer can upload the the Yamal directly to the repository and then that's how you dictate what the pipeline do so we give every single developer access and to control their own pipeline that's very has a very different style than Jenkins from the life cycle of machines and the keep that in mind when you do you get you doing development and continuous software so 1 of the big things from and I might even be able to talk about it is on looking pretty at a time and is that were are application development we have multiple repos the upper application we have application repo but for the actual Java code or whatever the application does and we have a counter rebo that basic uh that basically but gets triggered and build darker images by pulling all the source code from the other repository of and we trained the point at which it just makes it a lot easier because which if you have to install a bunch of different prerequisites for your software you can just told the chef recipe to install the stalker image and it just makes all of your recipes so much clearer and less tangled on that ordering a right now it works very well In his Connolly's lot headaches as well this is this is like out of this policy problems there was an obvious for me of detection of next to virtual machines but I seperate links for responsibility so whenever I create a virtual Xin or were actually handed over machines I edit them slightly I add it should make for the chef connections so that way if there's any operational of folks now that's clogging when the Knicks or something like that and you can always go back and fix it with another recipe or change it with the recipe and so you possibly have a persistent connection which that Chef server of that organization I knew that limits downtime and help to stay up yeah but I do this using IP tables in writing rules and that's super hard uh Red Hat makes a little more difficult to achieve that the so quick recap of this talk was about the chef countertop shipped out of his house I deployed my organizations into here get development environments that many are production environments you show always assume gaps for everything not just for your production environment hiding migrant I can talk contains also dependencies for now Republican something like whole or of a nexus later on but there is an organization cookbook per project so the organization cookbooks defined projects right define refuses said not onsets per project per enclave whatever you want it's going to keep that hard classification between on the organization and the role cookbooks and the resource cookbooks on because the the uh having a steady standard going through especially because we're all new to chef at in my group and makes it'll be easier to hard classification types for cookbooks but and if you are going to be making your own pipeline and unaware using but if you can and if you do have access to a year the spirit is an war knife of the circle you or whatever if you want a strong do kitchen in year of a new pipeline I highly recommend it just because trying to write counteractive cookbooks is a pain and you probably shouldn't have to do that and the and also I don't harp on this too much but up look at people just uploading versions of books to shove servers I do not like I the it's pet peeve of mine that out but doing it in this way but in the C I have access to it and that is the workstation of flooding is probably the best rock of fix everything from breaking and trying to fix that break and go through year Chef server is the previous so bad no the closing in I'm a shift and we still and so learning a lot but I include the last section to try to help everybody else out and the if you were conferred government agencies never security issues in proxy issues and I'm hoping some years to relate to this I so we talked them up pipelines restricted Internet and are development process for this is still evolving and still doing research on it and and learning more every day and I was very glad to be a part of this world and especially from so need to about so this is like a a learning chef and everything so this is you know really exciting for me in and we have a few minutes for questions of 31 st questions else'll beyond the hallway as well I'm right after serving as well 1 as beam Fig