Credit Union, AIX, and DevOps Oh My

Video in TIB AV-Portal: Credit Union, AIX, and DevOps Oh My

Formal Metadata

Credit Union, AIX, and DevOps Oh My
Title of Series
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
This talk focuses on introducing a modern, DevOps approach to infrastructure to a legacy organization. We will cover how to convert "curmudgeonly" team members to a new product and workflow, as well as introducing Chef and DevOps to non-infrastructure teams.
Web page Oval Data storage device Design by contract Cartesian coordinate system
Scripting language Server (computing) Mathematics Divisor System administrator Data storage device Bit Moving average Cartesian coordinate system Sinc function
Standard deviation Game controller Group action Service (economics) Open source Multiplication sign System administrator Average Mereology Product (business) Mathematics Process (computing) Physical system Task (computing) Source code Standard deviation Projective plane Process capability index Instance (computer science) Incidence algebra Cartesian coordinate system Type theory Data management Process (computing) Integrated development environment Chain Window
NP-hard Satellite Building System administrator Multiplication sign Direct numerical simulation Medical imaging Mathematics Different (Kate Ryan album) Square number Office suite Error message Information security Scripting language Satellite Decision theory Data recovery Structural load Point (geometry) Data storage device Physicalism Bit Process (computing) Order (biology) Self-organization Right angle Cycle (graph theory) Quicksort Point (geometry) Multitier architecture Server (computing) Service (economics) Virtual machine Product (business) 2 (number) Workload Goodness of fit Operating system Integrated development environment Backup Form (programming) Red Hat Server (computing) Projective plane Computer network Frame problem Uniform resource locator Software Window Address space
Point (geometry) Multitier architecture Server (computing) Vapor barrier Multiplication sign Online help Perspective (visual) Hypermedia Memory management Negative number Integrated development environment Backup Physical system Rotation Shift operator Decision theory Server (computing) Data recovery Chemical equation Point (geometry) Multitier architecture Sound effect System call Integrated development environment Musical ensemble Address space
Scripting language Point (geometry) Area Boss Corporation Divisor Multiplication sign Projective plane Fitness function Product (business) Power (physics) Data management Process (computing) Integrated development environment Lie group Process (computing) Data conversion Cycle (graph theory) Social class
Point (geometry) Building Game controller State of matter Multiplication sign Mereology Product (business) Number Subset Wave packet Machine vision Different (Kate Ryan album) Core dump Videoconferencing Operating system File system Integrated development environment Process (computing) Codierung <Programmierung> Endliche Modelltheorie Eccentricity (mathematics) Social class Physical system Area Projective plane Fitness function Staff (military) Median Bit Lattice (order) Line (geometry) Cartesian coordinate system System call Word Data management Process (computing) Integrated development environment Universe (mathematics) System programming Freeware
Server (computing) Game controller Service (economics) System administrator Multiplication sign Product (business) Number Subset Data management Mathematics Different (Kate Ryan album) Single-precision floating-point format Gastropod shell Energy level Integrated development environment Software testing Physical system Scripting language Default (computer science) Shift operator Demo (music) Server (computing) Projective plane Fitness function Infinity Data management Process (computing) Internetworking Integrated development environment Logic Order (biology) System programming Connectivity (graph theory) Configuration space Website Right angle Quicksort Arithmetic progression Resultant
Laptop Point (geometry) Functional (mathematics) Server (computing) Service (economics) Computer file Source code Branch (computer science) Mereology Mathematics Computer configuration Energy level Software testing Physical system Computer icon Execution unit Information Demo (music) Weight Software developer Directory service Cartesian coordinate system Annulus (mathematics) Inclusion map Personal digital assistant Repository (publishing) Order (biology) Configuration space Convex hull Right angle Game theory Cycle (graph theory)
Computer chess Peer-to-peer Presentation of a group Mathematics Process (computing) Integrated development environment Code Forcing (mathematics) Multiplication sign Source code Order (biology)
Satellite Context awareness Multiplication sign Decision theory Range (statistics) Image registration Client (computing) Mereology Web 2.0 Direct numerical simulation Mathematics Peer-to-peer Different (Kate Ryan album) Oval Square number Bus (computing) Website Extension (kinesiology) Information security Physical system Satellite Data storage device Instance (computer science) Flow separation Process (computing) Phase transition Configuration space Pattern language Right angle Figurate number Turing test Slide rule Server (computing) Service (economics) Patch (Unix) Control flow Event horizon Power (physics) Revision control Frequency Operating system Energy level Default (computer science) Stapeldatei Standard deviation Dependent and independent variables Shift operator Focus (optics) Red Hat Mathematical analysis Computer network Word Software Integrated development environment Network topology Factory (trading post) Data center
so and I am lesson and and I think X is underappreciated after insistence that might be to speed and so on they so I said were from the Federal Credit Union had been with its you for 8 years prior to that I was going to determine an annuity contract in my background is the eggs and little storage but primarily it so I have found a soft spot for her that's not really that came on top of the page in half an so I know them with a focus in for 3 years now before that I was at the hospital didn't and so we have been working with I started with a about 12 years ago when I started there and that's all they had was a hack so that's what I I jumped into and I learned the hard way a lot about an axis not as much as most of that I learned about
it and then we started bring insulin X so my experience was basically trying to be as lazy about administrations possible I think that's that's a good way to be an administrator so I I tried to script everything I couldn't so my experience from the very beginning was trying to automate as much as I could usually linkshell scripts and so that's my experience so little bit
about and see you and we are at the world participating in them occur at least when we're we're at 7 million members since I've been the federal and and growing about half a million members a year and since it's the obviously when there's growth that rolls downstream more customers more members means more applications more servers or storage and so on and so forth and not always more people to help you manage factor and they've been around for it's math 84 years and it says you can imagine it as an older companies there's a lot uh Legacy tentacles the applications and infrastructure and I think of it like a science experiment or anything it's just kind of spun out and and you can turn something off and you know what that's going to impact it's exciting and so that's that's kind of what we're dealing with on the underside of the house
Artstein and you have 10 Unix administrators that 5 of whom have primarily x background and this is not including myself a three humble primarily a Red Hat Linux background we have to add a new to next in general so they're learning both sides and their years of experience is heavily slanted and XI and we add up everybody's time and 136 252 and there is an age joke in there that we don't have to get into and we're very tight change control process and work is heavily and measured and scandal then planned and we have at least 2 applications are attract work and an HP patting system in each HP project management system they don't talk to each other but they should 1 your projects are planned in in advance and you it plan an 2 weeks roughly in advance and internally when we started the chef project we started asking the question Can you are engineers contribute to open source instance where benefiting from it and that that's become a question if not yet inserted and that question is is going to the annex to the Gartner research for you to figure out what what we can do that and and how and how the company as a whole would still leverage open source in the environment now that we run into what would we do that and then obviously as a financial institution we do have to meet the PCI advertising standards and that the keynote that was kind of touched on that makes things difficult and it just does everyone knows that the firms that change
control we have at least 6 types of work request that engineers have been handled and potentially impactful production chain animal change with a standard change that is a non impactful teams to any environment and and that more mean standard and tell you do poorly and in fact production and we we call request fulfillment in a service that Q were some of directly the service not up to now we have changed tasks where you're working on some announces change we have incidents around the things work we have problem management tasks where you're working on a wide scope problem new task and the fun part about that is they're all in different cues in the same to consistent so when you log in you have to check at least excuse to see you in to do that they have as we discuss production changes about 10 days in advance were allowed to work on production and 1 or 2 days a week Tuesdays and Sundays and that obviously compete with other groups are trying to get at same window to take down the same systems to do other things we average the moratorium among because the service strong forces and their paydays are blocked out for our work because it's obviously more important had their money posts that we capture system in November-December closed for changes holidays people on there
and services how we get a request to build a server yeah so speaking of all those change request everything this is so so the thing obviously is going to be was right so this is Kansas is is where we started off with them and that it's a it's a process right and so should the MOS image we have uh forms of people can fill out and those those requests come down to us so in order just a server to be born out in a lot of people give me like blank looks I say this because for most people it's really just click a button and it's good but for us our customers to go in and they have to submit a form and on the form they have to attach the project but that server exists within her will be built for those of you don't have a project again and services a whole load of stuff that goes on even before this we can get to this point where we start then we we don't do Windows but it's the same process and so once they do submit that form becomes us as the admins actually assigned host names so we have to figure out what the product is if they already have those names and we have to provide something like an mind you have a very set standard aware of or so we have to provide that and we as the admins don't have any hoax in during the analysis of so it goes back to impose set to approve the two-year so they decide whether it's a non-secure secure the CID and z that based on the form that was submitted and once that happens goes to the DNS to get their DNS approvals so as far as time frame we're talking about here is not the seconds that most people are used to so the customer submits a form that takes this a little bit of time usually we get this request to get the hostname pretty quick so I mean that's what then that could be within the same half a day and then there are ample cycle take 1 to 3 days to decide what here needs to be an no DNS cannot be provided on a moratorium day so that could take anywhere from 1 to 3 days to get the innocent and then we finally get to the point where we're building our servers and and that's where we actually can take over and that's where we started with trying to make the difference In organization to try and make things move faster than show you the build process here is that there is a so good so
old old old build processing Kansas was or was somewhat automated but mostly manual but this should look familiar to anybody who's been building an X for a while and satellite on the left was a little bit more automated was Linux and always a Linux is easy to do anything Linux because hard on the right hand side so we use an end to actually install the operating system before we do that we have to go actually defined the ELP on on the machine so that's physical clicking in and out of forms and within what we call the agency so that takes some time once we get that help or about defined it gives us a the storage of the max so we need to get the store designs and if you stored standard store assigned and then eventually is ready to get installed this is when we use an and so that process takes at least a day and then once the OS is actually installed which takes you know that the few minutes that you would expect then we had and some sort of automated but we had a cake office automation right so we had to we had scripts they were written over time but some of them were in central locations where everybody you some people had their own their own individual scripts that they used and so build processes at this point become very so there very diverged for also there can vary diverged from and then that whole process could take us with storage in getting everything done depending on the workload from those other you know 5 ticket cues we have to work on they could take another there's wanted on with that and we have to submit tickets to back up and then to a monitoring we get that done so the whole build process that we tell people lead time we need so this is the the advertiser 5 data in fact reading very fear server and if it's a moratorium that obviously can take longer because you're waiting on that the network can have the ability to give phials you can build up into the network and waiting for the assignment new waiting for before any evacuating the information security to review the form where it states what kind of data you holding so they can manually square it and say the I know it's secure what's this thing and even if we go back if there's any error at this in the 2nd
tier or any this here the wrong way for whatever reason in humans humans make mistakes when we have to essentially go back to that tear and gets read start all over so that's the that's the only way that so we're trying to improve in and get across that barrier of taking so long to deliver the weather systems the uh and so this
is an we attack about growth rolling downhill and this is where we the really see it in an 2nd team leader there at the the 2012 times and it by 2013 we had 8 people on the team we have handle over 900 systems and and 4 years later we we've appearance doubled in size on the server account and we've added an I at the time and the 11th FTE which is a misnomer I don't have time to do that anymore so I'm like negative help from the perspective of the 10 and that's that it's a higher ratio at a higher ratio and on the environment in particularly the closer we got to a thousand servers it in alarms server going after we need a better way and on and on an axially we were managing the whole thing with dish and in here that's that's great if it works if it doesn't work at least it doesn't work consistently on thousands of that the balance point as the ad people on to piggyback on the shift the media markets in the kitchen in your brain and your brain and senior people and you say this is how we do things and maybe they do it that way by the letter and probably they don't because they think you will this has the same net effect it's a little different and so we would find that in and the jerk you know when when were patchiness system doesn't behave 100 % the way we think it should and so we really needed and a way make sure that when we were managing all they extends the way they needed to be managed and all the Linux things it's in bands in was really important to me to keep both teams together because they think everyone would rather a 10 week on call rotation a 5 week on collocations that's kind fact that our fair that there should and so this is in
some the so we have we don't do w cars that my bosses would religiously against the beam pipes but by using the that actually come up and portable we talk about some of the technical things you you the work around and so they are not to that point we are and where about 97 % virtual so and heavy on and the area of a penny on the wire but and nowadays we have a smattering of Linux on power as well the outside conversation the so about 2014 we in look now down 3 things we when for that class this kind of the big corporate tall right handed me it's pretty we did think is a good fit for a lot of things we're doing it was quite customizable and at the top there the shaft the that was 1 of the big factors for shaft was that stated partnership between Scheffer 90 m IBM is all gung ho and end that's 50 cent the environment so that was a kind of a when here and it in with our project management cycle and if we take the product and then we we got to wait a year before we can do it so as can we take advantage of that time to start looking at our business processes that what are we doing the via scripts for the via someone remembering to do it in and nothing but that in itself and the and since
since I get to play manager sometimes and when I I did was you have a in your review process and I assigned everyone as a goal and the online Ruby class and said OK you know we can really work in December and January that now this is something you can you can do it during business hours and I strongly strongly recommend you do that and tell me when you've done it on and I did 40 % of myself so that was cool is the the In so once everyone
can finish their the Ruby class in license product and both of us have and how do we get people actually end up in what we did was we occurred to everyone into pairs and may use the word curmudgeon with affection fit and at the end the EU kind of the understanding that people tend to think of X people as a little and rigid in their ways and I think that's universal free people that and say we we had a curmudgeon and evangelist someone was a little reluctant or a little hesitant about all the process and general and someone that was really excited about it and if those dividing lines tended to be someone on a exercise and someone on side then so be it so they were paired together and they were assigned please simple cookbook and and supervise those sections from and we also had Allen that kind of leader weekly Brownback session come in here and should that be the team is remote and we have the permanent teleworkers we have 1 in Florida his combined so we have to 1 with 3 different buildings in Virginia where people Her located so there's not a lot of face time everyone gets together so that we had a video conference online things once a week where we and pick a topic and turn around 3 about staff and when I teleworkers renamed the median is the what the hell is on talking about meetings which chip was was
and so on the end of the day so what we did have going for of everyone liked the idea of shaft and everyone they you realized that you were at were on call for all the systems in numbers not getting smaller in the eccentricities of the systems are not getting if you were a ceremony at least agree that that the end state is going we want that we want everything all at the same time maybe not everyone was super excited about learning how to code a cookbook and that there's some acceptance on my part where I have to say you know what that's OK you know not all going to be OS encoder you were hired to be us encoders you're hired with you understood the DOS you understood the hypervisor in and that's you know your area better at the end of the day you you do have to manage assistance control and you at least have to be able to read a contract and see OK this is what someone wrote and and why it didn't do what what it was supposed to and for in the training at this point and I call it a little bit face to where we went a little outside of core operating system management to eat you as someone was assigned a project of building an environment for a new application OK you have to take that and you have to put these requirements into a book so were putting in model B then right except for greater building out what the Mongo DB file systems are gonna look like and here that's a lot of looking at another quick and doing a find-and-replace on the flesh whatever to such Mungle on but it gets them and it gets them familiar with that process
in it and so but we ended up with at the end of the day and was being nice and our are split is more 60 40 have on exercise and so we ended up at the end of the day is we wanted to make sure that we had a single pane of administration and so we're configuring in my favor go to because it's easy is a T P so we were configuring NTP we wrote obviously 1 cookbook that's going to be applied on both the X and Linux and the logic in there to the site had actually do that thing and so different service names of you know whatever had been different after all be taken taken care of with inside of 1 cookbook sometimes we had logic inside the default to you running AIX cookbook in order run Linux cookbook sometimes robbers and 1 going on how how difficult it was on some quick numbers and when the when you're looking at like chef managers see how many resources are being managed her run we ended up managing around 318 things on red hair of about 180 5 things on AIX that's not to say that we are managing Linux at a greater great right were managing are configurations 100 per cent through Shep just less things for some reason on air exercise because it doesn't have a one-to-one neither many things tweak just yeah so come to pass so bowl we ended up with the end is is we had that and we went from lab right so we have very which we have very strict change control process we would from lab we went into production and really broke for things and I think that was also right so we need to look at everything that was in lab and over a fairly short amount time showed that all into production so we're managing everything which is a whole bunch of stuff we found configuration dressed in every environment on most of the servers and we fix all that and you see running through and and you realize how much was kind of going behind in the backgrounds is being being wrongly configure but not broken necessarily wrongly configured but really really broke broke for things and I comes down to really those processes were taking advantage of bad configurations so they were doing things that they should have been doing as a staging is really something like that and so those broke and so now we have this sort of side project to get rid of those problems and bring those back into into submission ship so so I think it was really good a really good result from that and when things started breaking I decide to create staging environments so in our lab we have a staging environment and then I'm in production we also have a staging environments we have a subset of the servers and that makes everybody feel more comfortable about about so there you go so
now if you're a chef employees promised up listening but test kitchen was not a fit for us and and 1 of the things comes down to religiously against w bars I asked a question I was told absolutely no but it also comes down to our processes and how we manage we don't have access to the US we can just spin up ephemeral anything much less ephemeral and X and again but it's easy on the Linux side because we could just run any number of things that are running do you guys had why you use this but we want a single pane of administration that we don't want to have to do something differently for x and we have to do for Linux so really when it came down to it could I have made it work sure but you have to think about the level of effort that you can put in at any given time and still make decent of progress and so I banged away at it for a while sorta got a working but then I realized that to repeat this and the teach somebody else is going to take quite a lot about quite a lot of time and so it was like to 30 on wednesday and I got tired of doing things the hard way and I said forget it I'm going to write a simple shell script fish to fix this and that's what I did not so test kitchen wasn't if it's so I wrote a couple shell scripts to replace it using shift to longer the but Burchell really did make sense in the way we do things that maybe it does now that I have more more knowledge but at the time it was super confusing but again we have a fix that environment so and this is this is the true when we have administrators will be working on ships they have an axle Porter test 1 and they have a Linux server to test on there's not like go grab 1 like they have been assigned to because very very steady now we made some progress in time this will all be rearchitected but this is how from the super super old system we tried to get everything working as quickly as possible but we got infinite few months of work out pretty well and our servers obviously have 0 interconnectivity and if you're a member of new federal you're happy about that so I so that these are the things we had overcome yeah OK I was trying to think of the the muscles and all tools with which went on and smoking there so I will show later but the yeah and no well after she gets the thing that is to make a bigger so everybody can see it like it's I'm completely or things like that you not I can't use it was going back to and in the what but in so this is like a demo thing that obviously is not gonna work at all because to them there's this smoke
and mirrors on yeah to go to the game social it you do your laptop sources try to who good so that the public there so basically this is all that it will always happen but they went way way it's for the others right so 1 of the
cool things about the the the tools that chef provides to us is that they can be used with those the way chef intense interviews but they can also be used in really cool ways that maybe some of the but we're going to and I don't know they're all here and they can answer that afterwards them but like I said we had these things we had overcome and we can talk about why it a sketch of work for us but you know outside the room but have essentially when so I wrote this thing like I said it was like to 30 on wednesday cycle the test it don't the economy calling it tastes like like that so the system is going to a a thing here because I thought this would be a lot smoother than actually like I don't know what I was completely wrong about that but but I built functionality to it that essentially I think in most cases will look familiar as far as test kitchen goes on but I wanted as I wanted to combine the new tools that we had which was get have and share and I wanted to bring those into the development of the development thing that people would sort of be forced to use right and so on the testing methodology is you write to the cookbook right you make a change you push it up to get out of the new untested is going to go out to your configured server which you can figure in a test a configuration file with in your home directory and you actually run are based on your own you based on your options it'll run the things that you want to run the to really cool part 2 lycopods actually seen it work if I do that everything often that but did this is what was on our demo working among the it's a it's a equated to do so but essentially you did it gives a much information about what you're trying to do and going into a cookbook them and now look at book mn what branch and an electrical stuff so it's automatically picking that up but this is inspect running on a Linux server right but the really cool thing is in fact running on the axis so in order to not be able do this attached test kitchen I use the SSH functioning of of our inspect so no you know so it runs natively BSS agent was very easy to do so I was able to get that done but this isn't TP because I always use in TB now now let this run it may or may not be interesting to you but I thought it was pretty interesting and figured that's like wire on you can do low levels if you want you can pass in Jason and it handles everything comes from our get of repositories so what we'll see in a 2nd is 0 actually run the so it's the wire but 0 actually you can pass J. Sun in his like no data and it'll be transferred over in a 1 of weights the way you expected to do so it's going around now that converge everything's continue new as we expected because I made this video so that it would and so yeah goes up and tells you all things it's going to services connecting to and all that and then runs the converge and then you see the normal out that that you would normally see through like a kitchen converge right so all that stuff is there and then you can run the inspector Miss inspect and we see that it's actually passing the thing from so I don't what to do to so I'm going to guess that's what I was trying to show they so essentially although this whole thing because it's really not I can so everybody if if you really care about the point is that in order to get around some of the things that we of OK in order to get around the things that we had problems with we were able to just use the same so we were able to
use thing were able to use the tools a
chess present so the bias of running a problem like that because you're working with and those environments the source shut off and you can't make immediate changes for the changes you need to take effective to take quite a long time we can still use the same tools as well as trying to get to and you can code around those problems so then I really quickly if which is run through these we created also a code review process we called CPR because of of really cool shit peer review right so in order get cookbooks through you have to go through shaft peer review of this was also in the process so versioning Kodansha and peer review were in process force the and
this is the important part this is our new build process and everything centers around ship to ever coding it of obviously but everything centers around chapter we now have something or power and satellite still on the left hand side but we build both operating systems through the thing that builds the operating system the best but chef is reaching out and converging everything on 1 side so we have essentially automated builds so we start out this very hard process to get it done and we ended up with an automated build working backwards down the tree where the DNS in the Turing working backwards they're trying to to to include those things but we've gone from and maybe they do get a server built to about 10 times as long as it takes to to the lady operating system out and actually run version of it on the path is that the registration piece to shepherd is automated as well so it's how in partly cmp even in you is using by the and that that that actually the then a huge thing on X and the house is that is that the button and it doesn't agree with infobox so here this summer aware of the fact that he's where and we have permission from the network team work we can assign a on justice on that side which is like all square that goes away did going we voted down to discover buttons for now but really the same process for building a axle X was this to build Linux which is always super easy it's just as easy to build a X was shift the way that we that we market going in and I know and I mean the the things we do have a storage team under the same umbrella mind I was able to tell them hey we're going to do this give us a lot of stories felt so that's another square it's it's against phase 3
so we're going to call it a lot of word per year teams and to show so we're starting with our middleware teenagers were primarily of a web search up so sounds to do his whole thing again with period of doing that Brownback sessions except now the daily the the and the but not the so there and this and this is a new thing for an open so we're we're expecting in the next couple months that we it is not to literally practicing the cookbook skills hopefully that's the key thing happens I let them with the homework to discover so you see I guess when really comes down to it you can work inside of a really close environment that you can change then even with a AIX you know the commercially operating system shift for 2 perfectly and really but I'm agnostic no because I have shaft and I don't care what things running on because I figure them all through this thing and all the questions sees you change the we use the negative of the reasons that and everything that was built as used in Chaps so we have to we did were not the land and it isn't do we so we know in have of it when we found and differences that have to be corrected corrected them and we did not escape slide in there that is there as new there were there was a period of time the team where you buy process was new and so you know we we have to kind of restart some Brownback sessions on OK if you run into an issue with the separate deployment and that default response is not stop the chef-client let's figured this this out and put it where it needs to be and you keep it in check I think this is a would would the so yeah so question was we use satellites and we can have we decide what's like figures in what cookbooks configure the easy questions cookbooks in figures so the salary and now the remember whatever and their their sole responsibility is put the US on the bus so you had to how it is all for the in goal the so that if you're asking about how we're how we respond to the changes that we have to make for more security standards of you talk about packages right updates to the operating system that's deftly we handled by satellite it's it's going it's managing operating system packages as far as that goes the versions of packages and and actually patching them but it's not managing the configuration of those things so if it's a configuration change that has to be made for security standards there we go back to the cookbook we make that and we push it out at whatever speed is necessary for the those threat and she goes through and within you know 40 minutes entire I entire speakers patch the so if we don't have to do anything we may have a package we install the package range that there's a kind for and we there's this kind of event from the an analysis that anyway that the team has some salient about batch money eggs and that's about over the city how it's if it and so we had the supported versions and say 7 1 7 2 and we have some 6 1 that on his her spirit in there and those are the levels that were stuff clients are on and we do have some 5 3 as well extended support agreements factory and we we protest has an instance we don't have to climb on the side the the who all of all so the pattern we use as far as like registering servers now in in those but we do it what was probably consider the old way we have environments that are essentially the environment and the location and so we have a lot of the in the data center so B PC lab BDC prob those that's the main and environment a break that we have and then we actually have roles which who knows not necessarily everybody likes roles but we have roles in so we put on every server goes into our server role that so we we actually us and our our configurations and then they also have the apples we put and so it's environment role and cookbooks all applied to roles we don't we don't put anything on the you data that is responsible for configuration of the service the focus has been so that