FreeBSD Unified Deployment and Configuration Management

Video thumbnail (Frame 0) Video thumbnail (Frame 1190) Video thumbnail (Frame 10742) Video thumbnail (Frame 20294) Video thumbnail (Frame 24690) Video thumbnail (Frame 32915) Video thumbnail (Frame 44132) Video thumbnail (Frame 55349) Video thumbnail (Frame 66565) Video thumbnail (Frame 79190)
Video in TIB AV-Portal: FreeBSD Unified Deployment and Configuration Management

Formal Metadata

FreeBSD Unified Deployment and Configuration Management
A practical approach to managing highly heterogeneous installations
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
When we needed dozens of storage, processing and front-end machines for a prototype of a new cloud media service, we developed a cost-effective, but technically challenging hybrid strategy of purchased, rented dedicated and rented virtual servers. FreeBSD was an easy choice thanks to its performance, reliability, and unparalleled ease of management on a per–node level. However, while the number of infrastructure–level tasks kept growing and we needed to scale through beta and release stages, there was an obvious need to reduce complexity. After a year of tentative design and experimenting with partial solutions, we started implementing in November 2011, the result-in-progress being something we call unified configuration management (and deployment), bringing immediate returns on time invested. The talk focuses on a new unified approach to deploying and managing modern versions of FreeBSD across a wide variety of technical and administrative circumstances: different countries, data centers, hardware, access policies, boot methods, networking, support contracts, machine roles, etc. While avoiding any popular Linux-centric CM systems, such as Puppet, Chef, and CFEngine, we achieve very low complexity by leveraging rc(8), loader(8), glabel(8) and other existing instruments, such as pkgng, to their potential as necessary. The cornerstone is keeping configuration and deployment versioned and unified — same across all cases, with no duplication of common parts and very simple specification of per-role/per-case peculiarities. The approach spans everything from installation and booting to managing third-party and custom site-specific software. The method is being actively developed and applied in production environment of a popular online music service.
Multiplication sign Network topology System programming Configuration space Videoconferencing Extension (kinesiology) Open set Physical system Systems engineering
Complex (psychology) Information management Digital media Local area network Code Multiplication sign System administrator Range (statistics) Design by contract Water vapor Client (computing) Mereology Facebook Different (Kate Ryan album) Hypermedia File system Cuboid Series (mathematics) Position operator Physical system Scripting language Enterprise architecture Channel capacity Kolmogorov complexity Structural load Digitizing Data storage device Electronic mailing list Bit Thermal expansion FLOPS Trigonometric functions Flow separation Disk read-and-write head Type theory Arithmetic mean Root Process (computing) Internet service provider System programming Hard disk drive Compilation album Configuration space Quicksort Metric system Physical system Purchasing Statistics Service (economics) Open source Divisor Virtual machine Heat transfer Streaming media Field (computer science) Number Product (business) Root Operator (mathematics) Computer hardware Operating system Energy level Task (computing) Systems engineering Information management Standard deviation Scaling (geometry) Inheritance (object-oriented programming) Forcing (mathematics) Projective plane Computer network Database Directory service Information Technology Infrastructure Library Uniform resource locator Word Voting Software Personal digital assistant Data center Vertex (graph theory) Point cloud Video game Musical ensemble Routing Local ring
Windows Registry Enterprise architecture Complex (psychology) Service (economics) Computer file Kolmogorov complexity Image resolution Multiplication sign Digitizing Branch (computer science) Directory service Open set Line (geometry) Product (business) Revision control Type theory Different (Kate Ryan album) Single-precision floating-point format Normed vector space File system Configuration space Video game
Group action Context awareness Scripting language Code Multiplication sign Execution unit Set (mathematics) Water vapor Parameter (computer programming) Mereology IP address Web 2.0 Medical imaging Web service Different (Kate Ryan album) Semiconductor memory Single-precision floating-point format Cuboid Flag Physical system Scripting language Area Mapping Structural load Sound effect Flow separation Sequence 10 (number) Type theory Digital photography Message passing Process (computing) Oval Repository (publishing) Hard disk drive Configuration space Right angle Knowledge-based configuration Cycle (graph theory) Quicksort Escape character Flux Spacetime Booting Slide rule Functional (mathematics) Server (computing) Service (economics) Computer file Video game Virtual machine Mass Rule of inference Theory Product (business) Supercomputer Computer hardware Gastropod shell Software testing Booting Metropolitan area network Address space Task (computing) Default (computer science) Standard deviation Dependent and independent variables Scaling (geometry) Model theory Line (geometry) Cartesian coordinate system System call Subject indexing Maize Software Personal digital assistant Password Hybrid computer Mixed reality Statement (computer science) Routing Local ring
Torus Satellite Demon Group action Euler angles Multiplication sign View (database) Execution unit Numbering scheme Real-time operating system Water vapor Mereology Order of magnitude Computer programming Medical imaging Direct numerical simulation Sign (mathematics) Synchronization Different (Kate Ryan album) Single-precision floating-point format File system Entropie <Informationstheorie> Cuboid Damping Office suite Logic gate Information security Partition (number theory) Fiber (mathematics) Physical system Scripting language Boss Corporation Structural load Software developer Moment (mathematics) Data storage device Trigonometric functions Flow separation Connected space 10 (number) Partition (number theory) Type theory Root Repository (publishing) Computer cluster Order (biology) Configuration space Quicksort Resultant Booting Point (geometry) Functional (mathematics) Computer file Computer-generated imagery Flash memory Virtual machine Login Rule of inference Pivot element Number Revision control Latent heat Root Term (mathematics) Operator (mathematics) Computer hardware Gastropod shell Authorization Energy level Software testing Computer-assisted translation Address space Compilation album Systems engineering Task (computing) Information Directory service Line (geometry) Cartesian coordinate system Scalability System call Database normalization Kernel (computing) Software Integrated development environment Password Video game Finite-state machine
Complex (psychology) Pixel State of matter Code Multiplication sign Execution unit Source code Sheaf (mathematics) Mereology Neuroinformatik Different (Kate Ryan album) Single-precision floating-point format Cuboid Office suite Information security Physical system Area Theory of relativity Moment (mathematics) Shared memory Bit Staff (military) Repository (publishing) Order (biology) Configuration space Right angle Procedural programming Figurate number Quicksort Point (geometry) Slide rule Server (computing) Functional (mathematics) Computer file Patch (Unix) Virtual machine Theory Attribute grammar Revision control Operator (mathematics) Operating system Booting Noise (electronics) Scaling (geometry) Interactive television Planning Generic programming Line (geometry) Software Personal digital assistant Data center Chief information officer Point cloud Routing
the we can the eyes when you're all done that our courts an extension cords Vanguard's you come on in this I a guys my name is Andrew I x systems engineer of so many years ago was quite active in the ports tree as any a girdle systems previewed a systems engineer is but I yeah let's happened and the idea of kind of of as time passed boy
I go tentative array of interesting situation of being the the you so systems engineer on sort of large scale project of birds have in like a zillion of other tasks along with it and having just like 5 to 10 per cent of my time dedicated to all you would do this the sort of tasks I'm used to doing was used to doing and the that's the phone one-handed ever unfortunate position like you or whatever whatever you know to do best on designing systems architect in I make imports by a list of I you just don't have very kind of small portion of the time to do that but on the other hand you are you don't have any what excuse to to do the routine stuff to waste your time to to do some of that can be automated so that when you forced when you really have a little time when you forced by the way when you forced to do stuff efficiently really forced then I find myself in a very fortunate situation of being forced to come up with an efficient solution so this story stars like many others it's a currently in a medium-size company that would have of the kind of moderately sized private cloud which so these tasks were tasked with in in in a large assortment of the tasks associated with me online media services so ingesting processing streaming large amounts of it but currently most of the computed always spent on a on a music service so that's ingest in coping encoded in extraction ordered features are and then of course stream into lots and lots of a lots and lots of clients so it's said he it's by no means anywhere in in this sort of scale yokel Facebook or any that kind of company accompanies but it's it's a fair amount of processing currently petabytes of storage and almost 100 Gigabit of aggregated transfer capacity Endre teraflops of processes so to give you just just a bit more stats and it grew up is grew to 4 countries now Western Europe eastern Europe North America and in 10 cities of more than 10 data centers we have to deal with with a number of very different service providers are different support contracts SLA levels so ranging from a very very jile and hands-on support to no support at all and still it's it's just around 200 machines so that they're fairly powerful many of them carried a lot of storage attached to them but it's a ball 20 really distinct hardware configurations and just about of of mostly large-capacity hard writers so that those where all the parents but of lots of data Allah cases and judge it was onto the several dozen local networks and different network types depending on the service provider with we only it's from a lot of 7 about 7 types of art of band of consoles currently 1 operating system I can a you know I'm sure you can guess at but the sort of this sort of task would be would dealing with these will probably warrants a potentially more operating systems we're still it's currently about 5 types so local hard drive local USB flash a network with several types of metric ruge types and this is a this is what really of forces us to do stuff efficiently there's just 1 systems engineer 1 network engineer who of course is 90 % in an African tasks and 1 field engineer Our so initially when we just said was starting out we've we've sort of solved all the problems the usual usual way so the machines we owned by that weren't nicer collocated in just 1 data center were there and then we basically made a closer out of them it was initially and that would would it with NFS fruit and that will thanks to the work of Brooks Davis and many others it roast quite nicely right of the out of the box you don't have to go to a dual and notice that note that the shed rude configuration where anything you edit on any of our books is immediately shared with all other books is what really annoys so that's that's a topic that is it was it was kind of a well can for us not lots of demanding too much attention as for the least so this week I went with the usual route of that's a up once and kind of forget for this for some time because it becomes increasingly difficult to to update everything in time to change configuration
unless you employ some kind of external automation so when things started scaling we briefly considered our pockets and all the configuration management systems like it and I kind of try to deal with them but before that book they always seemed like maybe maybe due to a guy who didn't really I spent a lot of time with them they they almost seemed like an unnecessary level of complexity so that there was also kind of the usual way of using a lot of in-house scripts to do a to generate basically all the configuration of farmers and everything else I was custom-made scrapes and deliver them to all the machines but obviously that well if if if that food is going to be considered seriously you better go with a standardized solution like pocket so at Taylor's series were quite so this me where cast men hallway and there was still needed extremely high the reliability and performance higher than a lot of a lot of out of the box ready-made solutions provided so of course a life if you want to store a few petabytes of data you can just purchase solution from NASA for the is a long assume and that that was sort of a lot of management problems of problems so we will but personally dealt with that of appliances and they are ever really easy to to deal with that but for example we we needed streaming processing and storage so net of gives you just storage on gives you a storage in streaming but no process in and that's when we started we wanted to be cost-effective that's not so critical now if we wanted to be cost-effective and slipstreaming all these 3 major tasks into 1 cost freebies database solution seemed very logical at the time and in hindsight is still a looks that way so there was a period of of agony so we had were had to maintain configuration to deploy a mode of prosodic on more and more waters in different locations differences Narragansett Tops different different as the lady pipes whatever Cavium types so there was there was that and we we needed to scale internal processing so a load of different factors and the needed to be 1 answer to all of that so how can we the is there a solution cost more existed the to these everything at once and start talks to a of systems engineers very vertices and very experienced and so when you lay out the whole range of problems and look for 1 single answer to while in part to all of them than most of them look at you so it will buy you media to the solution to that is a solution to that and so you need popular you need an enterprise or open source well what whatever out of band management solution you need kind of this solution for deployments you know well in fact freebies do doesn't do well with automated deployments as much as these these Linux's and so have the the agony was who is there an answer to that and don't believe we kind of found all own maybe my personal holy grail and the heat to balance down to just a just a few simple of just a few simple methodologies a very simple all edges so if some of your expansion rocket science it would be a lot of buzz words I am afraid you will be disappointed this varietal code very little actually very little stuff to talk about but a few simple methods of doing stuff so I call it unify unified configuration management and unified deployment and is basically the same thing and so on so just describe the current status as opposed to the long road that led to the West Unified it's so we have exactly the same root file system of and basically all the configuration everywhere and this this kind of it it sounds difficult to to do production but then again looking back at the end NFS so could common root NFS-based wurde vote works really nicely that's see that ceases to be a fantastical so I know it it requires so surprised be of sorcery as so we decided to go with kid the and currently there are this on Mainau the digit was to recovering the whole root file system this some cost customs proves kept elsewhere and to use on local of project named delegate and actually every at every home directory every user that every administrator or external user that wishes that he is a home directory and is distributed through the whole probably cloud needs to convert it to do it and and from there it's a really a when you achieve that than it really
becomes a a straightforward solution
as straight for fully distributed solution our beauty is really that 1 of the nicest to to do a true master master saying yes it's manual work that gives you unparalleled flexibility it it has voted bowlful conflict resolution and when you think about it think about it when you think about if any of you had experiences with open or an enterprise solutions which kind of do master master but you know they have to spend a lot of time setting them up and when something goes wrong and you'll spend a lot of wild sleepless nights body but then I G H and I won't give based file system based on a registry of configuration doesn't seem like a like a crazy solution and the way you where you didn't deploy in assembling copying of file hoping solution based on this indeed a lot of people are using those like you keep your data-based a checkout or its version based across someone or of the some work to the side and to that so that you know the you so that it's it's kind of more it seems more orderly but 1 you have to do you need to really is disable show contact files and then you can really kind of keep keep the the digit college the in the production of life to directory so what they what they gave us is really when when we slipstreamed of doesn't so dozens of different very different configuration types are different different types of seed of qualifiers different types of all configurations of all the services we employed issued a appear and everything else so ashamed that into a single receiver is single reposting no single branch that's really concentrated on the complexity in a single place is so instead of when you want to point when you want to find out what kind of you know how that machine differs from it this 1 you don't have to log in on to due to different missions or what Look optical out to different branches to different eateries they just look at a single on the 2nd configuration depending on how exactly is done you either see it all in a in a single file or into follows the line just next to each other what so
but but of course the question is if you have a hundreds of thousands of machines a lot of different files assigned to each of them in so it's not in a it's being a HPC cluster of where everyone is sort of doing the same job but it's is really a productive private lot productions at all so depending on application I can a hardware configuration if half of books is dedicated to ingestion of dedicated to processing at a streaming in all or local collection so how do we keep it toll on our photo give the rules in 1 place without infringing on each other so 1st things 1st I I thought how to store roles as a rule is a very simple concept its used was basically in units the as synonym is group and sewing research would have roles in in Solaris there separate concept of roles it's a very widespread concept and so I needed set and it did not just for users but for machines and I thought about you know implementing them just just keeping it in a separate file mapping forced names to rules and all that but then it struck me that I had no tell a scale of hundreds thousands and maybe tens of and even maybe of hundreds of thousands of machines we could really use password just passed within Group follows the standard uses UNIX infrastructure for users and groups and that works really nicely so you know this place every Spain there and every cost is assigned and and at the numerous group and then you have groups of of who would signify their rules so that the web server products will would be assigned to the group named Tao whatever where crews just 1 1 file that that is kind of separates and needed for machines to learn who they are when they would all user called where Apple's awareness is an when you have when each machine has exactly the same route you have to ask yourself how does it know what was going to a test yourself watch address if it's not they should be booted or they should be enabled to mass but what's IP address to assign to itself and so we decided to go with just 1 man will just 1 file that ties whose famous which is the same as role names because every has a unique role apart from all this society names tied to 1 or more MAC addresses so when you good news do you want to remind our and you look forward for you Mac as and when you find as well basically where map just look into where and you instantly know which which close to you all which basic food role you have and all the other rules on you get from it is the group and all the other files configuration files so how do we get away with having for example 1 policy don't on a hybrid of very differently tasks the toaster the boxes so with Aussie don't call it's really a breeze because it's basically a shell script so I I call the the type of can the configuration files that are very easy to and to convert to roll whereas the top 1 convict file for a hundred different it dust motorcycle cycle them role where so It's shall stroke stroking the the 1 a problem is that you need to know how it said in a village it but because it's not just sourced 1 time you have to really understand our especially at boot how exactly it is sourced and what happens there but when you do that that was done to of do solution somewhat like this so there's a common parts and oversee 81 90 percent of configuration I will you dependent on the dependent on the system is common to all waters so basically want PGE and some other parameters and every 1 of your books but then comes the interesting part when you want your web service I do have a specific parameter enabled and what the what we currently do this in positive conference just define a function with the name role built name and place said anything specific you want the them so if you want a specific hack to be enabled on just 1 post you can do that too just at the
end of the policy don't qualify while you can actually do that not just been isotope compote anywhere else where where you know so that it would be sourced just after all seed of of we do use this so for for each of your own you you look for a functional named like that and if it exists exists you just executed so that's kind of half the flux capacitor the kind of of the all the complicated code that went into enabling this kind of setup while engine next call is not so for example 1 not sure how many of you have ever used tension but it's sort of the the configurations index is sort of reminiscent of 4 it's a deeper D and lighted a pretty and of the lives of those so it's not lower in that it can do really crazy stuff within it you can't from within the single complete file you can read it ask yourself if I'm that close to do that but to what you can do is define so all these servers all the types of service you need I just put in 1 file and use the effects of is the fact that the on each of the colonies each of your course only the relevant parts of your configuration file will be invoked needed of of that our well actually another example that I didn't put a month in my slides is the new of than You has the day month of the year however let go to store a statement about what I like about it is that it is configurations system configurations syntax is specifically supports our model to cruise figuration so if you ever tried setting up Hastie they know that you can put our configuration of or 2 and a half of your cost of course in metafile and there will be involved each part of the configuration file will be invoked of exactly where where standard the most configuration files that are problematic for example syslog look on theories of wrote something I call unaware it's very difficult if you using specially from years and the stalk so small growers day month it's very difficult to use a single configuration follows across the whole infrastructure especially if you if you have a dedicated to war collection service so so what we need 1 reduced just a in this case are just keep keep to configuration files 1 is common to most nodes which just send most configuring all or most of the a log messages to the local actor and a local like for milk collector would have a 2nd file they both of them are kept in the same ditch repository so you can always look for that and we have on raised look around so our food for all the boxes by default us is logo don't is used while if if the machine is assigned the old name what collection of whatever then it we just had inositol were just assign a different configuration that if for I put this the the flag here was that I just or both from memory the hard case is the 1st time so obviously with the F. Estevez needed as good so you can do all sequence based work around and what we settled on is just keep it empty escape attempts here and there of reviews were basically specify where to load were to boot from in loaded of on how you can you can obviously use you can always the use of the unloader parameter named of the small true to form something that has and what we discovered well do and that is that while we would have what what's what's interesting about the top is that machines don't care they would've from network of locally but refuted the configuration files moderately insane could be the just don't care so what would have is when you machines arrive or when they would drive from any 1 of them goes a gross toast then they just started which and from that work and it is you what we discovered if you have a Vespa remedy in the if in load of corn but that says food from local hard drive if it doesn't find that local hard drive then adjust to a white and is rooted from network from an affair as they just you know the line so it's kind of a very useful fallback so if you booted from a friend and a fast than this is the local hard drive his present then you want food from as if it's not then you you know for all the other petitions botch the response we just use a single strip that does a very simple thing it sort of looks for what's what's available and just answered and work when needed so yeah here we come to actually how how to do deployment efficiently with and it's in most cases it's not a problem when you have like a single type of infrastructure just just rented boxes of just anaphors quoted Borcherds would just anything and it's it's not a big problem with money have a lot of difference in areas at once you have to come up with something the kind of different and so were what what I wanted to do is to find some kind of set of that kind of a mix of mimics what appliances do but like all the while all beloved Juniper devices that half of all the all the freebies Z and LinuxONE the cost of always based appliances they're very straight forward in that you know there's some kind of a single emission to them and they just work you have to tweak configuration but they just work if you want to operate a sort of just below the new configuration and and you image and they just continue to work you reboot they just work I so so why work came to know of is where would wouldn't do embedded so all all machines basically have of wells at least several tens of gigabytes of space right so it go down to an image the size of the image
the size of our attitude 10 gigabytes of petition and 1 for gigabytes will petition so what you do is actually of funded the drive that's suitable for bush in whether it's flesh USB flash or as dealership D and you petition that other drive the reason we use GPT for that it petitioned that that the drive through at least 3 petitions with 2 of them is fruit so want to basically for redundancy and operates but 1 of them goes to the small and anything that's left on the drive it might be a 4 terabyte tries goes so just what is partitioned into your 1st year and we use the scheme of their view of us and serial number the of of and so yeah that's th in order coffee just specified basically dev you 1st rude and it and then if you booted from an anaphors and you 1st is not does not have that petition at the at the moment of truth just it just falls back to an office if it does find that petitioned ponds if you not booted from an affairs then you probably have that will repetition and it's it's all of very good that's how we keep melodic on the single loaded Conf are across the whole infrastructure both Net anaphors boarded part of it and look at important part of it while so if you should get a in your also sort of hardware adjustment or you have to do is a is a just a lamb forget in a sign in MAC addresses 2 articles for you you place so the new the new crossing into your password and group files assigned whatever rules you need them to be I just your G should be and what you basically units of network infrastructure tell a DNS and then I you just well why the for the new books are however you got to be whether it's a fast affairs BSD-based rescue environmental has all your view it yourself a self-made custom anaphors would environment a 1 year in that walks you just find a suitable hard-drive petition adjective according to the script is just a 5 lines of or shell script and Europe on torn recent image from any other books you onto our recent image of a room partitioned into it so you just well you can use ssh for the Asus agent or if you have a talk All radio then you just a cat is viewed as a stationed on toric life also quality foreseen indeed deed extended active users and all the other stuff nicely then you would probably use that but it's it's extremely straightforward so when you have a disorder that becomes a question of operates so we have 3 levels of crop operates the most disruptive warmest full operate it's not completely of sinister the moment so the idea is just a you well you have a current complemented will drive just on top of a new the complete a complete new image to the 2nd 1 and the new fibers you assign you change your 1st labels of these drives or well maybe due to you what other we would change your 1st labels recognizer for us so I'm just not sure what happens the what what where the care kernel would be taken from a but it's just it's it's there for us it's a kind of the beach and of any Kapitsa top part mostly it puts it is to use land will I will always be fresh this way so we obviously there's a there's a desire to use all seen all while maybe package and G if we come to the point where all be system is our sort of represented the ball represents to bow in packages in for most of us packages or maybe you are the car cost in on inside configure the freebies updates of something like that but for now up while actually it's not it's not 24 gigabytes of information the it the image is just about 2 comma decimal 5 gigabytes and when compressed it's all like 7 100 megabytes so if you have like a hundred megabit connection then that's really easy into a load it's a loaded fully on tori fully each time this so the second-level less destructive level of great use package operates obviously where you operate all the lectures you use and that's fairly straight forward thanks to the action OK we don't have to keep our ports on on any of the waters or compiling a anything of that everything is done just once all the compilation what you can do is a dynamical sign any books to 2 was and so just check out yourself ports building once and then you have you basically have the new image and the new package repository so the environment is really it was all for us to make it fully distributed so that all obviously it's it's never perfect but mostly it is it is the fully distributed and then the least disruptive way to operate and don't want to be the 1 that goes on continuously is just get pulled of the other root partition the Customs traits and home directories that's that's now on the of well semi automatically of about it can be I believe it can be automated you know very close to real time so that would constitute a a very nice well no data very consistent very reliable distributed file system if you were if you now me to call that so we have all that you edit any configuration on any of your bosses wherever you confortable login name whichever box is closest to your implication just looking him and his Commission pollution and you don't need to think about you know in a centralized template-based pop it's a top you just you work with your with Europe and configuration files with a common configuration files you don't need to think how so how how they'd
be how they will be generated and stuff you just Edgerton committed bush and the deployed verb attempt to or you're on the boxes and if you have a conflict situation obviously due has very powerful instrumentation to resolve the and so that heaven the beneficial for having the whole system almost perfectly distributed so you are a kind of experience Avery it kind of have a very scalable solution where it doesn't really matter so that untreated confident the skin and to up to a tens of hundreds of thousands of of boxes will be pretty sameness because you don't have to do is to tweak how basically the Guinea-based configuration files will be distributed you don't need to do it from from a satellite place you just in was making a hierarchical or maybe a random randomly of a distributed infrastructure in a check your check your neighbors and transfer to them of whole from them if the new sort of of and it's it's scary it's also scalable in terms of human resources so we just we we have very few people we will be glad to hire a few dozen more even in today but while we limited we're confident that it's is scalable even know because of the load on operations is very low at the moment it's mostly the mostly known routine stuff so the sort of a new type of problem you have to accommodate in in constant to whatever and you type of hardware here you have to think about that and accommodate them but to attend problems are really really it's well at that point started demanding several orders of magnitude less of our attention you don't have this like hundreds of different configuration files just have everything at all in 1 place the of out of the problems of course 1 of them is gage it is really kind of beautiful and until you really know it's so when you when you start working with the leveraging its are kind of obscure parts of its its functionality well you you start hated and is also not really designed to support of file-system file system version and it's it is designed for called version and it would be trivial to as light emissions and the and more to that part of this leave whoever is behind cognitive development is really are against that the well that might be understandable but I think it should be really reinvented it's a very nice most a most synchronization for the solution our it opened of it in May nearest respects gates opened of the eyes of many developers and systems engineers to hold our how things could be done but you know where that experience it can be reinvented the thing from scratch at a very a renewed solutions also because obviously does not support many of freedom is the specific and basically any specific features is just there universal tool and will like there would be using it much more if it and better support for all the of attributes and of the persistence specific stuff and then the the the the I think the most important problem is I would say Amy Damon author any application office interests or think that his application is not a will longer run on a single machine if it's a good if it's a good 1 of them so people will eat will use it in companies where they don't have just 1 boss his company's firm for really simple tasks 1 what is not enough time will choose a mountain of these days usually have in for moderate tests you have to have dozens of hundreds of boxes at least started from that and so if you if you design a software to it with that in mind then you have to think about how well how will that work of that pool guy with systems engineer badge will be managing your the configuration of your demon or whatever and to just to accommodate to a commodity sort of role where the stuff into your configuration also sort of like of all the beat with Hastie where you from from the start he knew that has the can of on a single machine its storage synchronization they it doesn't make sense to run it on a single machine so this 1 on several machines why don't we have to have the opportunity to keep all the the configuration of all the machines in 1 configuration file that sort of stuff is really the the sort of my set a it's really something that I think many so while software authors should awaken to but also also st the the result I think it's pretty simple there's no as I promised not rocket science just just a couple of simple tricks and that will couple of obscure our obscure issues that you just need to stumble upon once and learn them but it's pretty foolproof you can know you can have and well you can recall about really easily whatever you do you have everything in in 1 place is a so you don't have we do have our out of sync issues and you can have 2 different 2 different versions of your GigE repository on different machines in fact I do think it's it's useful to you know to have slightly different versions of configurations are on less that security related to to to just have a you know when you scale them now of your programs your machines will crash and when you have the data of you know this configuration precious machines or more often than this 1 whether it's also don't call for a particular kernel configurational kernel version then it really helps if it has more than 1 way when you have just 1 single configuration across the whole infrastructure you know and so can I think the
units so if you have questions should if you don't have questions for those of you who manages a lot of machines and thinks that most of this what we're doing is pretty I nonsensical then just I'm really I'd like to hear that from you and may it maybe we have a chance to respond to that yeah so the problem with NFS is that we got from a single data center to to a lot of them and well you can't keep a single and 1st of route in the center because of an office is very sensitive to latency so even larger 20-kilometre 20 kilometers really messes things up but when we got to several hundred kilometers obviously you can have an office room is everywhere but that that's a layer of complexity but well NFS basically it purchased a really nicely I just wanted to have a single solution but also works for rented costs of that that we don't have an opportunity to boot from an and that the only way I could do that is come up with a local of which solution which is compatible with an office and another thing with NFS is it's really unstable or it manifests phosphorus work nicely but when you saturate your network operating obviously you have problems when you were basically when you do a lot of access to manifest phone systems your system I think crime become unresponsive place and you know and double that I think it would only exacerbate issues so yeah work it's it's a nice solution my or yes security yeah I kind of 1 approach to security is if you if you don't have your infrastructure during anything useful efficiently that there's no security to speak about so this whole thing is kind of it just gets in this area is to doing something useful well you don't have to work 24 hours honors from there currently just started to the look into security I don't think it should be it should be put before functionality all just when you when you when you don't have any functional which but yes it is good to actually cloud security is really kind of messed off topic when you kind of have to have machines accessing data and functions some other machines without European involved hint of that interaction and the and a yeah but we're were working on that I have some ideas may default next of these 2 can the planning yeah yeah someone those White while I want at that point obviously not familiar with that kind of scale but at that point I would ask why the configuration has to change so from my right right yeah I would just separated the really dynamic a high-volume staff into something all something supper rather than OK yeah he was called call the but yeah that's all but theory or and you won't get it just we would have liked to have blood 3 different roles or while 3 different servers with different configuration files each of them had sections of so well with server name so the name of Wiles so the name to something and 3 and then wait where we wanted to convert it to a cloud of all sort of single configuration files were just to put those sections into 1 figuration file and then boarded unleash wants Sony plants under the section that matters to it is really involved a full in many cases where may even managed to do well to condense it even further so somewhere well some several was caught in a merged so when the they don't the from young men actually we wouldn't have also with the problem with Asus SSL keys that some books have have them some most on so obviously young we in that case will also use that the ORCID of Connes around where we have and you know a generic scores rappers would just include all the androgen that's all configuration files and that's also a little have it's it's a bit of a yeah it's it's not that nice but it it it works quite simply what we love yes this is currently or a kind of natal state were is semiautomatic and there's a discipline have of a command that in evokes it's just it's just on the order of hundreds of machines have a command that invokes the 100 SSH commands on on all the machines if I want them all updated so it's it's in past in that audience is currently mania what what I'm working on is in a in relation to security to this the security topic is the Khot of a way for machines to talk to each other securely where issue while each machine has its own host a call and so I want to leverage that into that a secure secure remote procedure calls but it is not fully automated here I think
I think that's pretty of what what doesn't scale in the simultaneous no you would have doubling now in the year that the Sutherland at its 3rd attributes I think that 6 to 4 yeah and there's most patch hashing them but obviously there's just of a way to do it simply without writing anything custom when you don't have time to we don't we don't detect wouldn't have compromised grows at least yeah all this would do I mean everyone does have compromise yeah so do you have compromised those yeah OK so you do have compromised was that his own detect because some of this theory of problem due to no we don't we don't while the will will do you deal with security later but when I the the idea well it's just worked lecturing I agree while it's it's easier for it would be easier for you there we were just log pushed or for people who really break into boxes I think it doesn't matter for the mission is it also if you really I you know I think it was only it's important to to think about security as well as as a kind of important because the kind of an issue that doesn't need to get in your way before you have something useful because if you start with security then you better not touch computers at all a theory of the beginning of yeah on the other but how does a conflict I mean I don't I don't require having the opportunity to adios anything anywhere I just it's is a common feature that you can disable and then you just have to work you can use I guess you can use freebies updates instead of and SSH it where well we use package and G. instead of quartermaster which is also package induced as solutions and we just there were just I just didn't so do you want so fractional because you can you can really do very simple very well with them if you are if you don't need if you don't need that security compromise the situation where you can push anywhere from when you use a sable at I don't know how I would be worried about reinventing if I really had something complicated at the moment this slide 5 lines of code and the K. I'm fine if I invented something of them is just so simple that if I followed up through Python whatever's creates and thousands of lines then yeah I would be awarded did I waste my a month of my time doing that what partly it's too simple to more about that you know 140 you know well what 1 would like to see is the lack of operating systems and cooperation at least part of the solutions so that it won't actually reinventing the wheel is what bothers me I will personally as I see it every company every set up for is whether using operation last they still have a lot of cost and stuff and sort of push on some of that back to the operating system where I think it's post to be because of the world the will present systems or is not because of it's it's repository of shares profit some of so what would be nice to rollerblade configuration files with the noise source into something completely else the just makes all all your solutions easier or and notes I think it's is the final session and CIO wrong pixel common the if