Chef Vault: A Deep Technical Dive

Video in TIB AV-Portal: Chef Vault: A Deep Technical Dive

Formal Metadata

Chef Vault: A Deep Technical Dive
Title of Series
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
The challenge of balancing the need for security with the need for usability is nothing new. Managing secrets when using configuration management tools like Chef is no exception to this rule. Add in the fact that there are multiple tools attempting to solve this problem - each with advantages and drawbacks - and the balance becomes even more precarious! This talk will provide a brief overview of secrets management and then take a deep, technical dive into one tool in particular - Chef Vault. You will walk away understanding how it works - what theories and technologies drive it - as well as how to use it and evaluate whether Chef Vault is the right tool for your particular need. You will also walk away knowing the limitations of Chef Vault - it is not the right tool for every secrets management situation - and how to evaluate whether you safely can work around those limits or need to look at another tool.
Web 2.0 Structural load Order (biology) Database Lastteilung Database Water vapor Information Vertex (graph theory) Cartesian coordinate system Information Technology Infrastructure Library
Classical physics Sensitivity analysis Information management Variety (linguistics) Limit (category theory) Student's t-test Usability Internetworking Different (Kate Ryan album) Authorization Integrated development environment Information Information security Physical system Information management Dependent and independent variables Information Chemical equation Usability Database Bit Limit (category theory) Type theory Arithmetic mean Integrated development environment Universe (mathematics) Order (biology) Data center Point cloud Self-organization Information security Data management
Covering space Moment (mathematics) Voltmeter Core dump Limit (category theory) Software maintenance Focus (optics) Software maintenance Twitter Exterior algebra Integrated development environment Software Self-organization
Workstation <Musikinstrument> Computer file Information Key (cryptography) Structural load Workstation <Musikinstrument> Database Database Open set Parameter (computer programming) Cartesian coordinate system Message passing Casting (performing arts) Personal digital assistant Password Order (biology) Lastteilung Encryption
Rotation Workstation <Musikinstrument> Distribution (mathematics) Information Key (cryptography) Distribution (mathematics) Workstation <Musikinstrument> Shared memory Limit (category theory) Process (computing) Encryption Vertex (graph theory) Error message Information security
Server (computing) Open source Link (knot theory) Workstation <Musikinstrument> 1 (number) Online help Coma Berenices Client (computing) Disk read-and-write head Rotation Latent heat Semiconductor memory Different (Kate Ryan album) Authorization Symmetric-key algorithm Energy level Computer-assisted translation Error message RSA (algorithm) Source code Focus (optics) Key (cryptography) Information Server (computing) Moment (mathematics) Directory service Software maintenance Public-key cryptography Arithmetic mean Computer cluster Speech synthesis Object (grammar) Table (information)
Distribution (mathematics) Server (computing) Server (computing) Order (biology) Encryption Usability Online help Encryption Information security Information security Usability
Workstation <Musikinstrument> Asynchronous Transfer Mode Server (computing) Computer file Key (cryptography) Workstation <Musikinstrument> Client (computing) Database Client (computing) Public-key cryptography System call Personal digital assistant Musical ensemble Key (cryptography) Musical ensemble Asynchronous Transfer Mode
Noise (electronics) Workstation <Musikinstrument> Information Workstation <Musikinstrument> Password Encryption
Slide rule Server (computing) Computer file Information Code Gender Multiplication sign Set (mathematics) Database System call Metadata Attribute grammar Word Password Statement (computer science) Library (computing)
Covering space Workstation <Musikinstrument> Server (computing) Key (cryptography) Workstation <Musikinstrument> Voltmeter Database Directory service Theory Message passing Mathematics Personal digital assistant Self-organization Video game Text editor Key (cryptography)
Mobile app Key (cryptography) Integrated development environment Structural load Different (Kate Ryan album) Lastteilung Database Limit (category theory) Cartesian coordinate system Data management Flow separation Product (business)
Mechanism design Group action Scaling (geometry) Order (biology) Limit (category theory) Vertex (graph theory) Group action Limit (category theory)
Blog Scaling (geometry) Moment (mathematics) Summierbarkeit Vertex (graph theory) Key (cryptography) Group action Information security Limit (category theory) Login Information security Physical system
Group action Dynamical system Service (economics) Key (cryptography) Multiplication sign Projective plane 1 (number) Limit (category theory) Software maintenance Number Exterior algebra Core dump Data structure
Rotation Complex (psychology) Group action Dynamical system Sine Scaling (geometry) Multiplication sign Group action Sequence Mechanism design Integrated development environment Core dump Self-organization Quicksort
Information Chemical equation 1 (number) Core dump Online help Rotation Software maintenance Usability Software System programming Self-organization Information Information security Information security
Point (geometry) Server (computing) Thread (computing) Beta function Open source State of matter Multiplication sign Modal logic Client (computing) Login Perspective (visual) Field (computer science) Revision control Mathematics Energy level Software testing Information security Dialect Information Key (cryptography) Software developer Horizon Shared memory Planning Plastikkarte Bit Database Directory service Limit (category theory) Cartesian coordinate system Category of being Data mining Integrated development environment Personal digital assistant Blog Formal grammar Statement (computer science) Self-organization Summierbarkeit Right angle Quicksort
the 2 shaft involved a deep dive know all of us to practice configuration management particularly when were 1st learning it have probably found
ourselves asking this question what do I do what I need to share static data among my nodes was idyllic in
example infrastructure let's say we had 2 application nodes and there's a separate database that they talk to you and then there's a load balancer in front of those application knows that takes in that that traffic from the web and distributed among the application nodes no both of these ap nodes are going to need the credentials to access the database now we could manually SSH and each node and added but that feels like it defeats the purpose of automated configuration management and it becomes even more complicated when I want to add a 3rd node or for throw at 1 or 5th water 1 them dies and i have to replace it in order to keep things sustainable I need some way to share those credentials among my nose you know a
lot of us are probably familiar with the classic chef approach user data bag but what is that information is sensitive database credentials are usually pretty sensitive what if we need to protect that data from unauthorized access by encrypting its 1 then it gets a little bit more complicated this is because any tool which manages secrets must not only keep the data safe it needs to find a balance between security and usability no occasionally when I mention things about securing its secrets in the cloud a response I get is well if you really want to be secure just put in a room without Internet access shut the door don't let anyone in and then you're fine well that technically might be secure but it's definitely not usable has anyone else heard things like that i up the other got a few hands up so this is because preventing on authorized access is only half of any kind of security management the other half is an enabling authorize entities to access what they need when they need it a way I like
to put it is a system that is secure but unusable is inherently insecure this is because people still need to get work done people will go around it if it's not usable for them and often the more secure something needs to be the more critical it is to people completing mission whether that's a new feature in a corporate environment or enabling a university to protect the students information but be able to access it when someone whose authorized means that or even a national security agency needs to keep its Datasafe but also needs to give the people on the ground or in the data centers or whatever wherever access to the information when they need it so when there are a few 1 actually have more than a few secrets management tools out there in order to know which 1 is best for you you need to understand 1 your own environment what mission is your organization attempting to fulfill what are the risks associated with your type of work this is critical for picking the best tool for your particular needs and the 2nd thing you need to know is what you know your environment when you're looking at school you need to be able to evaluate the capabilities and limits of that tool in on assess whether it is the best 1 for you the this is because no tool is 1 size fits all the reason we have a variety of secret management's tools is not because everyone just keeps trying to invent the wheel reinvent the wheel but because different tools meet different needs in different environments so in this
talk we're going to focus mainly on shaft Baltes but we will how it works what environments that works well in and what limitations it has but we will also briefly explore alternatives this is because even if shaft bald is not the right tool for you there are others that might fit your needs better and we'll discuss common questions any organization can ask themselves to figure out which tool is best for their needs before we dive in no was cover who I am I know
Cheryl Harrington I'm a quorum maintainer on both habitat in supermarkets and also the co-host of the food by podcast if anyone was a so that that's a chef related podcast by you can find me on Twitter @ @ notion umbrella or if you have questions or such after this I feel free to e-mail me at en sham relic shef . I O and looks like some people are taking pictures of that all pause for a moment but I will be playing this back up at the end of this talk yeah so before shoving Volt there were
encrypted data bags let's
using this example of an infrastructure again we have a application of a load balancer in front of them and we have that common database that all the need to be able to access mimicry in
encrypted data bag let's say I've created a Jason file my workstation that includes the database password the In order Crick this I firstly to create at the in this case I'm creating an Open SSL key called Mikey on my workstation and once that's on my workstation I can go ahead and create that dated back to the j on file using the knife data back from file commands but when a cast 1 more arguments that commands I need to pass the past to that key on my workstation that I want to encourage that data bag with and once I do not allow have the data with my item inside and it's infected with my if I wanna look it was in a state of about I use the knife databank show commands and again I would pass at that same key I need a pass at that particular key anytime I want to decrease that did about and then the info within that item will be passed back and rendered on a workstation so this looks fairly simple if I'm only trying to access that data back from my own workstation but 1 of multiple people need
access or edit that information let's say we have 3 what people's workstations that all need to access it I We need this distribute that exact same key to all of those workstations is not just workstation that will be that the every node that needs to be able to access information that data that will also need that identical key to decrypt it so we have 1 key that we need to distribute to all workstations and all nodes and I is the main limitation of encrypted data that you have to control the sharing and distributing of that team manually and this is security risk what if someone suddenly leisure or particularly involuntarily we will then need to generate a new key real that data that with the new T and then redistributed that new key to every workstation and every node that needs access to that information so this is a highly manual and very error prone process additionally if
that is compromised somehow just on 1 person's workstation on 1 node the everything that is that data will potentially be compromised so what I think this highlights is that the rotation is
hard there's no error from cats managing access to data and making sure the correct people and knows can access it and unauthorized ones cannot is very hard but it's not impossible someone's writing tables upstairs I think but it's not impossible there are tools out there that help with this problem know I'm not going to tell you they make the problem easy but they do make it easier so chef vault is 1 approach to solving this problem
and shaft fault is a tool that was originally developed by Nordstrom they saw a need with their use of shaft created a tool to meet that need and then open source that it's kind of the open-source dream right there so ownership and maintenance of vault was transferred the shaft in 2015 and it is completely open source still if you wanna check out the coordinates are maybe check out some of the new things that Tom mentioned in the keynote today head on over to get home dot com slash shaft slash bolts yeah now how chef ball works on a high level is let's say we have a sharp server no shuffled does work shot so but for this example we're gonna focus on using a shot server we have 2 nodes that are managed by that Chef server and a node is managed by chef server that Chef server has an associated client object on it for each node that's how chefs server is aware of and how it manages each node no additionally what are known as managed to shove server it uses an RSA key pair to authenticate it's the node itself will retain the private half of that RSA keepyour pair and the client object on the shaft server will maintain the public half of that pair that's how the node noses talking to the crankshaft server and how the chef server noses talking to the correct no know like nodes we have a user accounts on a shelf server each user is also going to have a client object on that chart server and each user is also going to have an RSA key pair the user will keep the public or the party the private have that keep here on their workstation usually when we're doing Scheffler worstation will have a . shef directory and yet the the key in this is the key that you're putting in it and likewise speech our client object for that use of the shop server will maintain the public half of that keep here so when shut all does is it takes advantage of this existing the set up and uses it to encrypt data so let's say we have a evolved evolved is very similar to a data bag it also contains items in it so in this example I've evolves called my ball in an item in a called my items you know when I generate this bolts look will go over how would you do that exactly in a moment but when I generated shaft fault generates a shared key and encrypts the ball with so so far that probably sounds pretty similar to fit the data that is but we're about to see the difference the so let's say that boulders ownership server and we have a user account and know that need be able to access that vaults or memory each user account and each node has their own RSA key pair associated with that so much of what will do is it'll make a copy of that shared key for each user and each node that is authorized to access it then what it does is link credits that shared key with the key that is particular to the user and the key this particular to the node so if we have 1 user 1 no we're going to have 2 copies of that share key and each 1 is going to be infected by that entities specific key pair so that means of user 1 needs to access to that ball from that their workstation The 1st going to use the key to decrypt that copy of the shared key and then there will be all of the décrit the bolt itself and access the information inside of it and it's the same thing with the node when that know these axes information sigh the vault it's going to use its key pair too decrepit copy of that shared key and then use that shared key to décrit the vault itself know what this highlights is that the approach a shaft ball is to
use layers of encryption to balance that need for security in usability and help make the distribution again if not easy at least easier so now we know how shovel works let's take a look at an example of creating a bolts so let's
say we have a shot server and have to nodes registered with it and to user accounts on it yeah no order you shaft vaults I needed installed on my
workstation now the use the chef decay I have good news for you you already have shuffled on your workstation and you're good to go if you had a few developed shot without the ship decay however you need to install it as a jam on your workstation so once I have that installed creating a ball is very similar to creating a data bank I 1st call the command knife ball create that I pass the name of that ball in the name of the item with that bolts and the next thing I passed as specify by which user accounts I want to be able to access that ball using their key pair associated with their shops server accounts along with that I also pass in the specific nodes that I want to be able to décrit that bolts In this case is node 1 and node 2 next I need to specify which mode I'm running chef Bolton again shaft all worse with bow Chef Server setups and ships solo setups if I'm using the chef server I'm going to pass client as the mode if I'm using it was chef so low I would pass so as the mode and finally I will pass it in the
file with that data that J is on file I have on my workstation that has those database credentials in it so must I run this I'm going to create a bolt with my item inside it but shuffle itself is going to generate 1 more thing is going to generate an item called my item keys and with this includes is all the public keys for all nodes and all users who are authorized to access that bolts now
if I wanna look at what's inside the ball and I'm using a workstation that has an authorized he on it I would use the command knife both show and again from authorized to view its I would get back the D cryptid
information that is within that item in the vaults no fire running this from a workstation that did not have an authorized the I would just get back this noise I would get back being clicked data but it would not too cryptic for me and it doesn't tell me that much about what's in that item itself so we've
covered how ship all words we covered high trainable was talk about how you would actually use the ball within your own chef cookbooks and the 1st thing you need you to
usable in a cookbook is you will call the gender shaft ball from your metadata . be file then when you get into your recipe you firstly to specified that the gender needs to be installed at compile time this is because we are going to require Shadbolt even the recipe itself and that will only work if we install the gym at compile time now if I were doing this requires statement in a supporting library somewhere outside the recipe then at the gym would install at converge time as I expect but for the sake of keeping this example simple let's say we're going to call it from the recipe itself I can download that bolster my recipe using the shaft vault item call passing in the name of the and the item within it that I want to use and finally I can use to set node attributes member example contains a database password so I can set that at for battery by pulling that information from the vaults now I do wanna mention if any of you are wondering if I realize that setting a known at tributes my secret means the secret be stored unencrypted amish server yes that is correct you can get around this by using certain resources but again for the sake of keeping the code on my slides simple I'm going to use this as a the ongoing example so now we know how to create the
ball we know had hi Usinor shaft is in chef cookbook now as cover how you maintain a evolves because it's not done when you create it if I find I
need to any of all items say the credentials for that database change all I need to do is call the knife ball edit commands identify if I have an authorized workstation it will open their eyes in a text editor and I can edit it however I please you if I need to add a user nodes the bolts to authorize and access that ball I can do that using the knife vault update command and pass either the chef user name of the user and ad or the name of the node that I want to be able to access that item within the balls now some of these organization or we need a 2nd node out of it all I need to run is knife vault removed and again passed in either the username or the node name associated on the shelf server now occasionally for me use a chef server we need to regenerate keys I in my case this was right before come 2 years ago when I was not in the directory that I thought I was and I instantly blew away the private half my our city there so if you need to it right if you have to change your key on your shop server for some reason you're going to need to really fresher involves to let vault know that that T is now 1 that's authorized to be cryptic so you do not letting the nite volt refresh commands and that will go through and pull all the key is odd that are new destroying evolves requires a few more steps I firstly the delete the items within that ball before I can delete the bolt itself so if I run knife ball to leave my vault my I it's going to delete both those items within the ball but is gonna leave that and ball out there so the get rid of that now empty vault I need to run the knife data bag delete command I do need a user data that command to delete the outer bolt itself and once I run this demands then my bald is now gone so we've seen examples we seen theory let's look at a real life
example however with public supermarket as I mentioned under quarantine upon on what supermarket in general but for public supermarket we use ball to manage our secrets so this is
loosely what the infrastructure of productions production public supermarket looks like we will load balancer we few application knowns and then we have a separate post press database and a separate readers catch no lower with this production environment December AWS account we also have a full staging infrastructure the staging infrastructure uses all load balancer its own application nodes its own posters database and red catch so this means we need to be able to manage different secrets for different environments so how we do this is we use of vaults that we call apps then within that bolt we have 2 items 1 4 stupid for supermarkets staging and 1 for supermarket prod no along with these items when we created them shelf ball created supermarket staging keys and supermarket prod keys for us so this means that staging nodes will not be able to access the supermarket prod secrets and prognosis will not be able to access the stadium secrets that's 1 way we can keep them as separate as possible so shaft fault does work very well for our needs with a public supermarket but it doesn't limit us in certain ways and I think it's important to not
only understand how shuffle works and how to use it but you also need to understand its limits in order to know if it is the right tool for you and the biggest limit is with
autoscaling groups whenever you add a new node to chef balls you currently need to add that node manually you need authorize that node being manually so this means Chevy Volt is not compatible with autoscaling groups or other self healing mechanisms now you could
theoretically have 1 note generate all the keys and distribute those as you add new nodes but that is a major security risk if is that notice compromise your entire system will be compromised a see some pictures sums in a pause for a moment and lastly
shuffled does not keep audit logs of who accesses the bolts this is a significant limitation when your compliance mandate requires lots of who or what access certain secrets so not only on the limits of Chevy Volt let's talk about a few
alternatives and the 1st I wanna
highlight is hashi core bolts yes it is also called Baltes yes it is very confusing and a practice this talk a number of times just to not say the wrong 1 I'm at the other 1 so the cool thing about how she core bald is allows you to create dynamic secrets secrets which are only created when you actually need them and then can be removed immediately after no it's very cool stuff but the price and that's cool cool stuff is adding complexity along with the aim to be dependent on another service there is a higher amount of set up and maintenance than there is with other solutions additionally there is set and the all which is a project from shot community member Noah Kantrowitz so rather than using encryption keys to control access to secrets in uses the AWS I am roles feature now this is very useful and hosted shelf actually use citadel because hosted Japanese use autoscaling groups the major limit but this 1 is only worse with AWS if you have structure outside of AWS is probably not the best tool for you so there are a few other tools out there but these are the 4 major ones encrypted data bags shaft bald hashing corbel and Citadel that I've seen used within the chef community and that
begs the question which tool is best How do you know which 1 to invest that time in setting up and learning and the answer like a lot of things and technology is well it depends it depends on your needs it depends on your organization's environment it depends on the complexity that you're willing to live with or need to live with based on your environment but there few common questions no matter what your organization and I what yeah environment that can help you choose the best tool for you and the
1st is where is your infrastructure citadel only works in AWS if you infrastructure outside of AWS sin and all is not the tool for your needs next question is do you need autoscaling groups or some other sort of self-healing mechanism if you do as I mentioned before chef ball is not the right tool for you and finally do you have a kind of environment that needs those dynamics secrets sequence that all only when you need them and when you're using them if you are willing and able to take on the complexity associated with that she core bald is by far the best that so as we wrap up I want to repeat this yes key rotation is
hard managing access to data coming into the correct people nodes can access it and the incorrect ones cannot access it is very hard but it's not impossible there is help out there there are tools that can help you solve this problem and again I'm not going to stand up here and tell you they make it easy they do make it easier remember whichever tool you use it is crucial to find that balance between security and usability both to keep your information safe and so allow the people in your organization to fulfill their mission and with that that's who
I am again that's my contact info and or are there any questions this or that
it is curious on and using test kitchen along an axis a vault was a little difficult for me I was curious if you have a suggestion in a better way to do that I am not accessable for actually so uncertain of it but I if you figure out a way to do it make a blog post or I do talk an x here but I will look into it and I find another way of beta details what I did do got another hand out there that will go over to the other side but high on a think you've got some blog posts that have a lot of this the information and I've thread through those and found to be very useful so I've kind of implement some of this already on but there was the thing that you have in there about the setting the compile-time thing with the regions is that a newer development because I don't remember that being something that I had to do what all of its cardinality is changes in shaft itself and is the changes in the field of it do you know at what point that became a necessity because I don't think I'm doing that and I'm not sure nullify need to be I don't know offhand if it's working for you probably don't do it but is not so I would say that there is a somewhat of a with this like you can lock in a hand up over the year and month is missing security guard that doesn't know how to a gentleman who stated that it my it's a great talker and think so the scaling issue is a big deal for us but we manage thousands of bare metal nodes and I'm constantly adding new nodes into my environments as you see in the in intuitive solutions that are trying to tackle that problem or the chef everything on the horizon that's going to make shuffle work a little bit better for that I'm of sinking recipes is or something I can write that would mean your around the 2 sisters so your thoughts on how old how is that problem to be solved share higher with shuffled not anything currently on the horizon that doesn't there won't be right now shuffle is undergoing a major rewrite the fact that it was just the least common shares to make it usable with the AWS he hears and some of the things we do realize that is the pain points and it is something we're going to be considering but I don't have a a good answer when that will be at this time if your AWS as they were considered all of but of a metal so yes I think the earlier this year so working don't have I don't have the time but we do know it's a pain point and you know it's all thanks world actually this thing with they might have enough I think it so what was ahead is is there a way to move around or back of vault in the latent cryptid state between which you between servers organizations right in supermarket I'm I'm not here is that this point we do for the corrected data I mean of private property OK so we do that that we could also matter S 3 environments I think actually that will again if you're interview as the whole world is not going to thing but I I even even back up in some sort of status or in its present state x look at it at the yeah any other questions as an adjustment of the name you I think so no logging for that is that going to a change in the future or at suffer from permanent I would say it permanently anything in open source on there are no plans to change this but they did the that got them in the working really clustering on with things so this is the question isn't used to comment for the bearded gentleman back from version has chef try notify it's a recipe and with a little bit about that on Monday met with looking into that problems in this which re try efficient also available so to Paris paraphrase Bisley we shuffle trying to fight this recipe that can work grammar but has some security concerns in the some cultures around it of shuffle tried notify emotion get OK the and the directory here in the sum of that was the does there has she cooked solution do audit logs of belief in unassuming they don't have the solution for the static secret for out scaling I will leave it to you but I'm not certain that they have the right look I have not heard of that as the limitations of the will of the thanks that's to clicker phrases
properly that so the vault itself was secured by who can access it from client perspective right so if the ship is running the Vulcan say that hosts can access this thought Adam that OK an from a user perspective the user's does no desktop can access the vault because of that relationship as well right on because they have the right key here who associated with their accounts have server and that your account may have been authorized to act against so we we have a use case where we're trying to run the client on a particular host and then we'd like to build to in the recipe specify the user that's going to access the database as well is that possible I don't know actually I think it's very interesting use case I can see that being possible yes we haven't found out how to do itself I want to look at it but yes I do what you want posters you talk here any other questions out there some of can you add the nodes of the 2 evils via the bishops the itself minister name tag you can write on him and people do that all our personal level he has noticed copper mines in the name of that node the change and if it does a search across all of them a potentially pickup when no that's not supposed to so yes you can do that right now I just don't recommend recommended as approaches the that here the biggest 1 was consumers wanting a effort takes a wild cards and stuff of fretting like hundreds of nodes of time to evolve and you can use wildcards yes that is a lot of the day but thanks it any other questions all right well it's not statements learner and unaware leaves the early so I think the snacks enjoy them and I'll be hanging out here if you have any more questions what come and talk and I think