Consuming Open Source Configuration

Video in TIB AV-Portal: Consuming Open Source Configuration

Formal Metadata

Consuming Open Source Configuration
Infrastructure and configuration is now code, and some of it is open source. What is it like to be downstream of one of these projects?
Alternative Title
Configuration Management - Consuming Foss Configuration
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Group action Open source Code Multiplication sign Inverse element Continuous integration Theory Element (mathematics) Number Revision control Data management Internet forum Term (mathematics) Robotics Operator (mathematics) Gastropod shell Cloning Diagram Configuration space Physical system Thumbnail Scripting language Email Key (cryptography) Inheritance (object-oriented programming) Feedback Open source Code Planning Bit 3 (number) Lattice (order) Multilateration Instance (computer science) Term (mathematics) Configuration management Open set Category of being Voting Software Tower Repository (publishing) Hard disk drive Self-organization Configuration space Hill differential equation Pressure Physical system
Implementation Building Service (economics) Confidence interval Multiplication sign Software developer Projective plane Open source Code Configuration management Product (business) Revision control Architecture Operator (mathematics) Configuration space Computer-assisted translation Resultant Computer architecture
Standard deviation Computer file Software developer Reflection (mathematics) Software developer Patch (Unix) Open source Code Bit Product (business) Arithmetic mean Goodness of fit Spherical cap Configuration space
Random number Software developer Java applet Software developer Multiplication sign Video tracking Open source Code Bit Software bug Voting Personal digital assistant Term (mathematics) Internet service provider Software testing Summierbarkeit Configuration space Endliche Modelltheorie
Trail Group action Implementation Server (computing) Service (economics) Open source Software developer Multiplication sign Firewall (computing) Distance IP address Rule of inference Element (mathematics) Bit rate Different (Kate Ryan album) File system Diagram Selectivity (electronic) Configuration space Physical system Social class Point cloud Service (economics) Projective plane Java applet Open source Code Streaming media Computer network Stack (abstract data type) Open set Connected space Software Network topology Data center Point cloud Software testing Musical ensemble Local ring
Point (geometry) Slide rule Context awareness Presentation of a group Code Multiplication sign Ordinary differential equation 2 (number) Hypothesis Template (C++) Mathematics Computer configuration Term (mathematics) Software testing Configuration space Proxy server Mathematical optimization Covering space Default (computer science) Email Validity (statistics) Consistency Open source Electronic mailing list Code Computer network Bit Uniform resource locator Computer configuration Software Integrated development environment Data center Software testing
Polar coordinate system Complex (psychology) Installation art Scripting language Code INTEGRAL View (database) Multiplication sign Decision theory Set (mathematics) Mereology Proper map Subset Mathematics Conjugacy class Different (Kate Ryan album) Repository (publishing) Software framework Endliche Modelltheorie Pole (complex analysis) Error message Physical system Social class Scripting language Software developer Open source Streaming media Menu (computing) Bit Maxima and minima Variable (mathematics) Flow separation Wave Repository (publishing) Software repository Website Moving average Right angle Modul <Datentyp> Physical system Resultant Directed graph Module (mathematics) Cone penetration test Divisor Open source Patch (Unix) Directory service Drop (liquid) Regular graph Rule of inference 2 (number) Product (business) Revision control Latent heat Goodness of fit String (computer science) Gastropod shell Energy level Software testing Configuration space Data structure Theory of everything Metropolitan area network Module (mathematics) Projective plane Physical law State of matter Code Directory service Line (geometry) Mathematics Word Integrated development environment Revision control Synchronization Social class Gastropod shell Tuple
Pulse (signal processing) Complex (psychology) Group action INTEGRAL Code System administrator Mereology Coefficient of determination Mathematics Different (Kate Ryan album) Endliche Modelltheorie Hill differential equation Physical system Vulnerability (computing) Scripting language Covering space Email Open source Data storage device Sound effect Maxima and minima Open set Type theory Process (computing) Software repository Repository (publishing) Internet service provider output Website Configuration space Software testing Right angle Modul <Datentyp> Figurate number Arithmetic progression Writing Resultant Point (geometry) Module (mathematics) Computer file Disintegration Virtual machine Inclined plane Streaming media Plastikkarte Rule of inference Product (business) Template (C++) Revision control Latent heat Natural number Operator (mathematics) Selectivity (electronic) Software testing Configuration space Computer architecture Module (mathematics) MIDI Inheritance (object-oriented programming) Projective plane Code Planning Plastikkarte Stack (abstract data type) Line (geometry) Uniform boundedness principle Word Film editing Software Synchronization Video game Pressure
Demon Code Multiplication sign Source code Execution unit Set (mathematics) Subset Web 2.0 Inference Mathematics Cuboid Cloning Office suite Endliche Modelltheorie Social class Physical system Logical constant Closed set Software developer Open source Data storage device Instance (computer science) Open set Twitter Category of being Website Self-organization Software testing Right angle Modul <Datentyp> Resultant Laptop Server (computing) Link (knot theory) Real number Computer-generated imagery Virtual machine Density of states Product (business) Googol Energy level Software testing Configuration space Game theory Form (programming) Information Physical law Code Stack (abstract data type) Cartesian coordinate system Uniform boundedness principle Synchronization Natural language Local ring Abstraction
but a thumbs up something but if he'll lateral I'll try to get you going ahead again this stock is called I can't vote for both doors of this theories use role in this article consuming open source infrastructure and so I will monomer Spencer across I work HP on my primary open source contributions are to the puppet ecosystem into the OpenStack ecosystem and what we do is we we run a CI system inside HP and it turns out that the clone of an upstream OpenStack CI system and lives when we talk about that directly after this talk yeah so running somebody else's infrastructure as your own it turned out is not super trivial I thought it would be a lot easier than ended up actually being so the stock is going to cover certain wire was what we've gone through and stuff like that so the target audience here there's is really for obser operations people on this 3 main people on a stock today from the 1st to people who are currently pulling down someone else's configuration consuming someone else's open-source configuration the SEC is people who are considering doing that may be making plans and the the 3rd group of people who were producing something new source and and want to learn how to develop better as somebody who is actually doing it I can give you some feedback I'm so this is very much a what are we doing talk and I will talk about where 0 is we can you across the river we actually are which does not mean the number to give you 10 steps to do it easily means only give you a few key takeaways and that's kind what you're going to know the stock and I'll try to slow the narrative down a little bit when we get those takeaways so that we can we can focus on and also I were going to be fine on time so feel free ask honest so it's only to get loud and ask a question immediately when you think it I so we always define our terms and we start of that so open source open source to me means something that has an OSI-approved license on I but also means all that more than that so when I 1st learned about open source I'll start by Bob Massey is a professor he assume what he said is that you can write a piece of open source software and put on a hard drive and then put that hard over the basement and it's just as open source as it was before and so this is element of being able to to share and that is kind so if you're so I consider it only really open source of pressure if you're actively doing your best to share that code with people not just licensing and hiding somewhere and so that means you put some more visible as the concept called Open development and so what that means is that it's really open source unless it's developed in the open so for instance of a company give you know every goes the company 5 and they have meetings inside the company their internal e-mails in the company all the code abuse internal to the company and the crimes of runs every Friday that pushes it to get out of that's not openly developed it is not forum which anybody can join clinic alright so infrastructure infrastructure well this is kind of the beginning of structured some kind of bland grape and that we can take that plan that diagram of things connected to each other and using configuration management tools like property we can put that in code and that code can be committed to some kind of version control repository and that can be used to share and now that it's coding the OSI licensed what you start talking about it infrastructures code so consuming was consuming consuming literally means to eat but a base using something you did you yourself do not write your organization did not right someplace you get started also use words like upstream and downstream ominous you me us then we I'm waiting that is that I consider myself a downstream we HP weary downstream we consume from upstream and upstream is OpenStack's continuous integration and when I say we I generally mean downstream see they generally mean upstream will talk that the OK so a lot it's not just OpenStack it's doing and not by a long shot on the Wikimedia Foundation has an open source infrastructure in you know all my bad but Jenkins are towers around here he's always of people help with the the Jenkins infrastructure and Mozilla's infrastructures also open source and so this a takeaways immediately on why this is good and is in the the simplest thing to say is that we benefit from consuming infrastructure because we don't have to do all the work ourselves is right away the fact there a timeline shell script we get to use it means we have to do significantly less the the classic not invented here is the inverse it rather robot team in my college career we're building a quadcopter to fly and 1 of our experience that said you know you should just start with the parent AR Drone and then add your features on top of that we like no we need run around in and 10 months later we finally had something it would fly 0 to start working on real projects and so we don't have lower
results this is a good thing but I think is more important lesson we benefit from consuming the
infrastructure because we have confidence that the architecture is viable so that means is when you set out a team of architecture to have DevOps people to go build something they you spend some time planning on going spend some time mapping out things and sometimes building and all that time is is essentially riskless project there waiting for them to complete was developer time spending on and if it doesn't come to something that's useful to you that's all lost but if you know someone else's use this implementation successfully then that gives you some confidence that you're not going to end up losing that much which basically reduces the risk and so the 1st thing we said was an advantage was that we didn't have to write everything ourselves as a corollary which is that any departure we take from the reference implementation is immediately technical that as something we have to think about when we consume someone else's infrastructure the the the so let me Our Team at HP is called the Ghostbusters and our product is called go there and I would consume the OpenStack CI infrastructure and it this cat with rubber meets the road right so there's some tools they use these public for configuration management they responsible for orchestration these get for version control and they run all the infrastructure OpenStack and so they have to open that up there Rackspace Cloud and H column make us but the service so this is how operators do
we get these days the and and the the but they did on everything that was a good thing you know we have a bad because it gives you the opposing diversity we used to tell the people that this is managed by get by touching a file called all caps use begin blue so it's a good thing it's getting means everything to get which means it's Cheryl which is actually so if you look at the polls for both of these
teams and you know both teams of building CI Infrastructure for OpenStack developers upstream believer overstepped their upstream developers and downstream were building a for HP-developed so each Piazza OpenStack product of coherent you buy it on and when we we take the standard of step we apply a little bit of HP internal patches and we test those age being called that is with the to be internal CIA and its attempted to be a pretty good mirror reflection of what happens after so
I mean you look like this the but but if you take a little bit further work all models are
actually somewhat divergent so the upstream OpenStack team has to run a lot of Damon's that we don't have to they don't have to run a bunch of infrastructure for publishing packages to paideia after on a lot of infrastructure for voting in other technical over governance needs of the 2nd they have to run a bug trackers and we have to run those things is in many cases of Don applied in terms of tracking Jira so we don't have to pick the but they also such downstream because downstream we started out as a as a team that would provide CI infrastructure to open sekulovic an HP but they get so excited about the idea that now we have developers all over each becoming a works and run our test so we have . net test so we conducted tests and we have you know more tests with Java tests and some of those can map to the models that we learned from upstream the sum of those k and every time we roll some custom solution ourselves we incur technical that so really don't I don't know how
to make a band diagram of open source software this is like a like 45 minutes of my time but this is reasonable the 1st of 3 bad so what is the
take away so for a downstream what you can do is add service project with early in the project look at what the remember the diagram the think that everything else what do not need to replicate right resist the urge to replicate all the things because it's just work using maybe you don't have to do it for upstream keep track of what your coupling to what and where your assumptions are obviously it's really hard to figure out which assumptions of here's some ground so don't implied co-location so if you have to services that are co located on the same host talk over a local Hosaka don't just go directly file system that way downstream but I don't assume user rating group memberships of the SSH is 1 user to another user on that on a different host to assume you have a group permission to write permissions and some director somewhere and discovered a firewall rules and network proximity slashed layer it's really easy to think a lot of work is sort of like this and maybe you can change that maybe that's just the way it is you can document which will onstage for you to do so let's start with the network so upstream the kind of reference implementation of the CI system has to teach people how to 0 % of their work 1 is Rackspace 1 is HP class and so every server the spin-up has a public IP address and the firewall it all the way up and in white list select and so what that means is that any host on the network or any host in the infrastructure we have to make a connection in any other host can do so as long as it's plus 1 body administered when we went to do that at each PU we made a mistake of not respecting that impression that that network topology and so we built it out with 2 data centers will 1 data center in a physical datacenter 4 columns a Colorado which begin center we put somewhere researcher each class and that represented a different network topologies because connections from the data center to the cloud were fine at that works but the cloud can make connections in the dataset and so the reference implementation has push push push push push push push we had a hack it to be push push pull push push for what is hard enough to begin with but we have to somehow trigger poles and so it I wouldn't this in
the and the and there was another issue as i anybody's ever worked in a corporate datacenter knows about like HTTP underscore proxy as a variable and that a so that is that anywhere inside each piece network if you want to make an outgoing network request you're allowed to do that as long as you could use the proxy and what that was we had to change a little we had to making small little changes to an HTTP proxy validation add that to the environment as it deems remember and so that meant then we have a go the
start about upstream contribution so downstream so must contribute option is like a private central thesis of my presentation on every single member of the Ghostbusters team contributes upstream and we don't do 50 % of our time but there some days at 100 % of our time in Sunday's attempts at the summit without From practical and social reasons why you do that the practical reasons the stuff like this which is the that is the public templating but you just have to sneak in there not a little it's HCV proxy do something in this is if you if you're downstream this is your bread and butter that you're gonna submit this stuff optimal the little the small change change behavior slightly better that way it's off by default and you could turn on consumed downstream the but there's also but also social and social reasons are important can think of the social reasons as an exercise in such social capital and what you how do you get social cover while social capital to trust the essentially just a trust exercise so these are the things that we get from upstream we get testing the test our stuff for us we they they write about a code that we consume from that they review your code they support us when we have issues so we get to read the docks and so when I 1st started writing this slide I have a different list I have grown long list of things we provide that but then I realized it's the exact same list in terms of what we push up here and the reason we have to do all of those things it is because when we have an issue with 1 of them we need it to be important to the upstream quarter velopers the origin becomes all right and if you show up in an IRC channel a mailing list and say it's all broken familiar New year someone who is not known to the community they will spend 10 seconds trying out you and again on the line but if you are a consistent contributor you're solving problems for them they're going to go that extra mile for you and it becomes a 2 way street it's a trust relationship and so the takeaways context matters from we got bit by the network and I think after this presentation if any of you set and try to emulate something you would really hard them with there might be other assumptions that you're not thinking and you can see and so 1 of the things I have in my mind that could be true is geographic location on network latency maybe virtualization verses bare-metal but those are just examples but this this is really really and upstream contribution is not an option you have to contribute upstream you don't you read upstream you might as well just stay at home there's really no point to so there's different
levels of different ways to contribute upstream different levels of upstream contributions on you can kind of describe into conjugacy successful string unsuccessful contribution upstream and the use of the same rules for contributing to any public open source projects is that this 1 has to be informed happens to be infrastructure-related the simple things it's it's being consistent it's being reliable infrequent in your reviews the it's submitting small patches instead of big giant don't patches of that's having a regular cadence it's showing up to the need and it's it's doing things layers to like active listening who and I think we all know what an unsuccessful contribution looks like right it's like somebody showing up with a 3 thousand line passing here solvable problems just murdered and were good the developers of the callable bond and we know and the developers they have all these technical complexity issue ending the downstream jurors as no man it works in Orange just merge it from and that and then they get hostile and they say no it really works this doesn't work bottle lovable and that's that's the kind of person you don't want to be in a law that kind of person you're not very successful in this model of consuming and and giving back in fact consuming is really the wrong word because it situation OK so now the title of a more technical and this is the integration in the room of the puppet were a talk a little bit of a public situation in oversight inferring so when I 1st got there about 7 or 8 months ago this was the situation from upstream you Tyra 1st secrets only on their poverty . 7 which is really bad even for 3 or 4 even for 7 months ago on the other side of the 711 Yancy and they have a monolithic repository which meant all of the kind of module the code they wrote was in 1 big git repository and had a modules during their and then 6 years of my 1 they were stuck on Apache module version 0 0 0 4 which is from like 2012 and and that you In the view you bring that up and so Apache model 0 . 0 up 4 is very old and with that does more so than because you can get anything you want have you model for the courts I guess features missing from that but because you can't do this they can change you can upgrade like a modern Apache version that means you can use any module that depends on a modern patches so that only send you further down in the system of consuming old crap and writing your own they have fought BCS Rico module in fact we still do on which means that so this OpenStack CI Dasher PCs through the problems that you see a which is lovely models of ambiguities and I'm had decent role system only talk about that a minute what type of in which turned out to be really good and they have a shell script for installing problem so what that means is a rather than using arts 10 K something this is the shell script that wraps a proper model stuff so for the roles they have a pretty good roll system so there was a module an entire model called OpenStack underscore project and that had classes like OpenStack underscore project called Jenkins and actually tried it almost the OpenStack specific stuff and then they have modeled and they wrote themselves that was more or less free of Harken variables courses Harcourt variables all of the place but at least is a way forward here right the same zeal which is a dangerous player will prior to the n so when we bootstrapping it's where 1st getting workers the 1st getting things ready to go with deep copying of OpenStack project to a new module called HP and so we in the finer place several set of that project were replaced it the so now we have our own each Pecola cone Jenkins they can go use because public Jenkins model for we actually did his with is to get repository so we we put we pull down a lot formal our configu repo which basically just had the HP model in in and there's which had OpenStack project as well as a bunch of other models we put both of those directories and model path and so the 2nd cortical project world the 2nd for product was still available but was referencing when the site of the peak so that she worked pretty well on and that give us the opportunity to like do some really neat stuff like I wouldn't change HP anything we actually change without touching upstream AN downstream we can just run public cherry-picked tuple changes from so it's a really great wave describes something that actually were pre terribly from so we bought every repo we use which you can argue that that decision novel we need to be able to control what their environment so economy to control its we Canada working on and we very rarely sink that kind of public from upstream we're a thousand to it's behind you know Jim on which is actually bad as it sounds but the worst part is that we have this several patches ahead so there's places were we needed changes and now that we have changes on our own we can just pull anymore it can merge conflicts and 3 and even though from OpenStack and further upstream has moved to 3 . 7 we're still stuck on at 7 you can fast enough to to call this so the takeaways here on find your intermediary between upstream and downstream and I'm not saying that you make a module called over seconds were project them say that you find some place the boat you did as downstream and they as upstream can hard code variables without stepping on each other's toes and that can I think the traditional way if you were your Greenfield unless you probably do that today with high and there's an interesting story about that which is that the 1st time that I tried to do it I suggested higher and I wrote some patches and I sent them up and they were merged any going away what was having roughly why I think it was because I was a relatively new contributors who had not build that trust relationships back-and-forth trust co-dependent relationship with the upstream chorus and I was proposing this patch that was very complex and ripping out a lot of things changing things that were coded hardcoded into variables doing higher lookups and so if you look at it you consider those 2 factors together it's a new contributor we don't trust and a big bite of complexity and that's obvious enough I think today now that I have a fair amount of social capital these people I can't I could say Look guys I think we need to go this higher I know a lot of them I think I know how to address the concerns I know that I had a talk to these people and as a result like a colleague that merge now and is just 1 example of how you have to be serious about tripping upstream otherwise this model can work the thing is they use small cents and and it's so if you go upstream right and you drop of 3 thousand 1 patch of you're going to be told them you can be sent away their last year of other similarly if you wait 3 months before using from upstream you have you called of 3 thousand light patch they try to dump in here structure you know what I wanted you added so the only way forward is to sink often with small little sinks what that means is you need some kind of testing for what are the errors we made is we didn't set up some minimal acceptance test framework at the beginning and that has put us very far behind this and that's what what I mean by reflects the lottery is that it you wouldn't accept upstream why would they accepted or they would accept upstream why would you accepted the so this is my
model for keeping up of upstream and has a familiar with the story of of the Our pulses correctly Sisyphus was agreed with pissed off of God and so he was sentenced for life to push a boulder up a hill and it as soon as he posted up the hill would roll down again and heat up the pressure where he be a kitten on the sled the the those talk about what
has happened since upstream and and I'm pretty serious I said at the Ghostbusters Irving part of these changes that operated from properties of and above 3 also note that I use the word we I totally consider myself part of the OpenStack upstream the and I think they consider me a part of the and so you should let your goal we put the data into a separate the and interestingly enough we not actually higher so there was a lot of hard coded data there's some if involves the play will tell you about the but the these giant blobs of data like the Emil which jobs to run an army project started love merge what where and as a kind of a a public employee lane my inclination is to put them all in higher but with the team figured out to do is they can put these flat in the file that configures the minute you get repository and tell public BCS repo just make sure it's over there and points you will at a point that the minute you could fit file and you're done you don't actually need the complexity that higher provides which turned out to be a really cool result it also means that there is no when someone pulls down a configuration tries to downstream they just have to add their own git repository for can take 5 reflect on this but the model of the repo into repos for model selection yesterday or something Jeremy sent out an e-mail that said we announced that we have 61 new modules and they're all pretty OpenStack specific that's totally true of but now they're all split out individually and it can be published the forge or opening it folks like you are involved in maintaining and using on the machinery to release models to for achieving cover blacksmith more or less the same way release type I model right now so if we want release a new version of say OpenStack Nova was somebody does the score a nova is they make assigned did she that did tag and they push that the tagging work Overview system and machinery fires and ends up on high to now you push assigned the tag of a puppet module as the machinery fires and a new versions of forward which is great the see you said that the Apache 0 . 0 . 4 models for each I think of you said OK well somebody In our team is really excited about the public 0 zero-zero for model that's because the public Apache 0 0 4 model is a weekly so modern Apache model you have defined times over for the mom rewrite rules for example and so you have to not only understand your Apache configuration but you have to translate that into poverty itself with the Apache 0 0 4 module that you do is just say here's a template put that MID host pools and so that means you as an administrator can write native Apache code and public figure out what what have any visibility into the resources but it can get the job done and so some people think that that's feature complete weekly model versus strongly model that's just as good as so that she does we've dug deep copy another fine replace we've created OpenStack each CV model which means that anybody who's a stickler and wants to use the old 1 can use that 1 but it can coexist in the same module packed with a modern pub allows a parking lot which means that we can pull in 1 and there's a natural upgrade path it also means that anybody who wants a weekly model Apache module can you lots and we've we have made are in store that estate script super smart on it now has 3 different types of models that has models to install from the forge models to install from get and models tools sold during integration and that ties in the last thing which is we built this integration testing system upstream so when you propose a change to the Jenkins module that test that with all the other modules and the site of PP in upstream to make sure that everything works but now it's not the best testing it kind of we testing is all really does is make sure that public compiles but that's better than we were before so what's planned upstream 1 of the so the plan of streams were input summer in all models Morgan price which out of using a forged angular straight gage dependence of the class because we're tired of the transitive dependency problem in public and we're figure out some kind of a fixer VCS you know there's a couple ways we can go on the ad are we can change our module to provide a new provider for the Puppet Labs ECS repel we can also try to merge the behaviors of hours into the provider of the Hubble lab CCS people model them a figure that out yeah the what you the people of the so when you use problem model installed you have a transitive dependency problems as a transitive NC problem means is the the question was why do we wanna go to get instead of using that the models and the answer is that there a transitive dependency problem where you ask for Puppet Labs of PostScript's right and a dependency that is the concat module and the NCP much and so it'll pull down the NCP module but if you already have an interview module installed it will fail to install the NTP model and version mismatch and maybe the post that model works it's all the pub model tools also kind of bad at identifying this problem as occurred and reporting that the user in a way it's detectable scripts is that into your question the the so as planned downstream as we're gonna start dog the OpenStack product that HPV develops which means that when a change from network that does not well reflect the reference architecture to a network that does well from the worker that's a huge on your planning upgraded 3 that some progress but it's more were operators right so we're fixing machine walls driving and so that's scary but we have testing with have minimal cut weak testing the effect of testing of strain that we want bring downstream and
we're gonna now that all the model or sink out individually were thousand commits behind across 61 models right but remember were only using a subset of OpenStack inference to it so we're not actually a thousand miles from its behind were more like the and eighties doable especially when you can do the commits that went to will not when the commits that went to Jenkins born we knew that 1 the the it's also worth noting that while the 2nd 4 sets up a lot of but have you ever been so mad at Google Hangouts that you set up an asterisk server these people where the perfect but I love them and so some closing thoughts found I was hanging out the Nido person from other labs yesterday and he was talking about the days of public 0 . 2 4 and he said that time everybody wrote code and what he meant by that was that everybody who is using the product at that time was a reading the source code and changing it to get results that they will and so that means it's you yeah treated like in it so that that means there's no there's no users who are reading code likely were office for instance a user without reading the source code can install and running play and but that's kind where these these these open-source infrastructures right you have to consider yourself a developer you then have to go in there twiddle that there isn't clone DOS it's not very yet maybe 1 day you'll become polished enough it'll be infrastructures in a box and you can download them and run them and I think some organizations like 4 J. store is 1 of them are trying to build that all right now with all the 2nd prior and probably the other that I mentioned at the beginning we could have to roll up your sleeves and get in there and write some code and this is also want to links that you can use when you look inside the mirror and that's my contact information are there any questions we OK thank you very yes yeah the development 1 of yeah mentioned mentioned so that we can refer right to this and much you the 1 thing that you do not need to the yeah that is absolutely 1 of the paths we wanna development aid which is that the models right now despite best efforts have crossed dependencies right the hidden crossed dependencies the worst kind right so we wanna do over the next several weeks is identify where those are and endoderm and the but if we to have a clear dependency we need to call out the documentation so yeah the all right yes so usually there's like you need to know yes the unit kind of acceptance yeah this the the with that I've done some work that is the bigger were brought up to do some work right now I know that improve that I thought I saw spearheaded so that we can use in your 2nd for noble system spin of a machine and and run beta test on that machine that was actually kind of difficult to do an active and the because speakers opinionated tool bigger wants to be in charge it wants to identify when the VM spends up when spins down and wants to tried reported results and where you have a lot of tooling for that stuff it's like sir we better so to Damon's that 1 of the in charge of this or 1 1 set of demons another like run command tools can be done a laptop don't really along the way we have a site that offers after this the the you find change in the law of the land and they don't have the properties you have to do with this so much in the so what do you want to know what the right of the the you the and the year and what and I would like the and you next then you have no way of to make a the the you want to do with data and want to is this 1 but the the the only thing that about this you write you the question of the day the like so 1 thing that is our model called of right so that there was a question about was how long but the the the the the the idea was that you don't have real abstraction layers and pop you still have to ask for what you want even if you're a high level as you can ask for web server you have basically be specific about Apache and right on so what we did in the pub aboard model was we we tried to got tired of forcing dependencies on people and so there's a class you include the set up the public were with the application and then you include either in in the form of a worker local Apache or a puppet for Colon Colon genetics so so that's provided by the public modular newspaper that's that's better than not and ideas that the 2nd thing this is we will begin retention right time the the the