Beyond NixOps: Compute Fleet Operations with NixOS

Video thumbnail (Frame 0) Video thumbnail (Frame 14043) Video thumbnail (Frame 14757) Video thumbnail (Frame 21796) Video thumbnail (Frame 22264) Video thumbnail (Frame 23082) Video thumbnail (Frame 25373) Video thumbnail (Frame 34241)
Video in TIB AV-Portal: Beyond NixOps: Compute Fleet Operations with NixOS

Formal Metadata

Beyond NixOps: Compute Fleet Operations with NixOS
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
For my day job, I operate a compute fleet on a mixture of private and public cloud. We started out with NixOps, but ultimately found ourselves constrained by its limitations and "pet"-focused approach, so I ended up learning a lot by building our own cattle-focused deployment tool.--- Bio: Hi! I'm edef, and I build software and services for other people to build software and services with
Axiom of choice Group action Run time (program lifecycle phase) State of matter INTEGRAL Multiplication sign Neuroinformatik Medical imaging Mathematics Hooking Computer configuration Single-precision floating-point format Information security Physical system Scripting language Touchscreen Structural load Data storage device Entire function Type theory Data management Configuration space Right angle Video game console Laptop Game controller Service (economics) Variety (linguistics) Patch (Unix) Virtual machine Ultraviolet photoelectron spectroscopy Number Goodness of fit Term (mathematics) Operator (mathematics) String (computer science) Ideal (ethics) Computing platform Module (mathematics) Addition Inheritance (object-oriented programming) Forcing (mathematics) Planning Machine code Limit (category theory) Kernel (computing) Software Personal digital assistant Point cloud
Server (computing) Aliasing Repository (publishing) Client (computing) Right angle Function (mathematics) Communications protocol IBM Client Access Statistics Abstraction
Point (geometry) Building Computer file State of matter INTEGRAL Virtual machine Ultraviolet photoelectron spectroscopy Theory Medical imaging Latent heat Single-precision floating-point format Booting Physical system Software development kit Default (computer science) Server (computing) Weight Keyboard shortcut Binary code Data storage device Client (computing) Bit Entire function Independent set (graph theory) Cache (computing) Subject indexing Googol Algebraic closure Internet service provider POKE Vertex (graph theory) Configuration space MiniDisc Musical ensemble Communications protocol
Asynchronous Transfer Mode Function (mathematics) Kernel (computing) State of matter Cursor (computers) Core dump Modul <Datentyp>
Kernel (computing)
Point (geometry) Service (economics) Algebraic closure Bit rate Function (mathematics) Binary code Virtual machine Cursor (computers) Machine code Musical ensemble Local ring Physical system
Laptop Ocean current Context awareness Building Scheduling (computing) Service (economics) State of matter Multiplication sign File format Ultraviolet photoelectron spectroscopy Virtual machine Gene cluster Similarity (geometry) Field (computer science) Derivation (linguistics) Medical imaging Mechanism design Different (Kate Ryan album) Repository (publishing) Energy level Booting Physical system Module (mathematics) Area Demo (music) Structural load Planning Kernel (computing) Process (computing) Algebraic closure Software Quicksort Figurate number Video game console Fingerprint
all righty so on to our next our next speaker which is EDF and on happy half I'd like to ask you to not take pictures ever and and treat them in the world and also I'd like you to to pay note that you should refer to as she and her so thank you very much for that and apart from that she is she's here today to talk to us about mix-ups it's a limitation that she has run into using it and the ways that she has found to do things better right off we go [Applause] I'm not quite going to talk about Nexus in terms of how I've worked around but for starters what do I actually do I run a small platform as a service company called mutable we could we put the cloud experience in places where you wouldn't otherwise get them and what everyone's now calling edge cloud but that means we have to manage a lot of systems and a lot of places across diverse infrastructure and the cloud never sleeps so we ought to be able to patch stuff up well it's running we can't really have non-atomic updates to things because that's an operational nightmare so somewhere around two years ago we switched everything over to Nick's OS we use in immutable infrastructure practices to build everything outside the compute but we have a constantly evolving stack so we have to manage those as well we try to solve problems at the lowest layer it makes sense for so we have to integrate custom patches to your kinds of software kernel patches kernel modules we have to integrate our system like as closely or at least our own we have to integrate our own software as closely with everything else as third-party software that we pull in like we use console and things like that we use a heavily modified docker currently but that's not such a huge problem if we run XS we run on diverse underlying infrastructure we run on ec2 we're moving our control plane to Google compute currently but most of our targets are bare metal provided by telcos and places like that so and we have a lot of like machines we're deploying to you so we can't have like something where we're doing some like significant amount of per machine work so I was aiming to deploy our compute nodes which run basically a docker daemon a bunch of magic of ours and our own services because our system runs on itself so Nick's UPS was my first choice when like ok I've to deploy a bunch of machines with Nick's but next subsys neat if you're deploying like a couple dozen machines it has it's like slightly more pet than cattle approach to deployment where individual machines are things you're aware of it sets networking hostname for each machines or each machine already has a unique configuration it sets EGC hosts to contain every other machine so every time you add a single machine or remove one you're touching every machine you already have you can like do networking to extra hosts is make force empty string that works out but really what I want to do is always deploy a one-size-fits-all image to every single machine of a certain type and then mutate it after the fact because we have things like custom kernels nick slips likes to deploy the pre-built a.m. eyes that get built with every nexus release we have to reboot those right away if we deploy them and there's no real automation for that could be done by like matching booted system to parent system etc but it's no ideal we also in the our bare metal targets want so differ support there isn't much of that but that was a sensibly patchable then there's support for things beyond role compute so on AWS and Google Cloud that's things like security groups storage buckets there's some support for those but it's not very comprehensive but the real kicker was not being able to use it in a distributed fashion easily we have a variety of people deploying we're working on automating deploying our entire stack but also just we can't have that state exist on a single machine everything about our system is designed to account for the failure of so in addition to that we want to target a lot of places and next ups has been slow to integrate targets we are not super up for hacking on a Python code base so what we ended up doing was handling the role compute resources when they accepts with a couple workarounds for the issues we covered a bunch of bash scripts around it to take care of things like adding a number of new machines to cluster and we ended up covering the we ended up running into the managing of non computing infrastructure we use terraform for that in the past already and terraform is a tool for managing immutable infrastructure generally built from like pre-built images so you the normal approach when you use terraform and like the entire action corp stack is you pre build a mais running like debian or whatever and you apt-get and do horrible things and all that to make your thing work and you bake that into an image and forget about all the horrible things involved in making it and if you ever want to change the running machine you throw it away and replace it with a new one which sure but in our case we have a customer containers running on those machines and we do need to actually do you hooked fixes from time to time some of them are physical machines so we can't just throw them away but terraform has a load of good sides to it it has great operator control ahead of like deploying a change you can run terraform plan which will tell you like hey we're recreating this resource because it because it changed in a way that we can't do at runtime it has a good story for using it in a distributed fashion you can tell it to back at state store with you CD or an s3 bucket though that one's not super clean but there are a variety of options that mean everyone can just run terraform on their laptop and not clash with anyone else it has rather pleasant integration with other tools so I ended up using them together that is wrong screen
so that is not
so terribly terraform looks like this basically every resource abstraction and you have outputs which is all that matters to the
next layer really so I just do some
horrible things and poke inside its state files and get all of those bits out and I use that to deploy everything that isn't compute that isn't like actual machines I used this next expression from inside my NEX ops config and then everything works with that it's a little manual but it's fairly workable but we still have the issue of like when we're using next ups we're doing an X copy closure to every machine there's a configuration that differs per machine by default I maybe there's something I'm missing but I haven't seen any like significant integration with like managing a binary cash but ultimately what makes ups boils down to is kind of a fancy way of doing mixes rebuild switch - target host and that's all we're really using it for at this point so what I'd like to do and what I'm building is we have excellent support for building disk images index packages Linux OS so I build a single disk image per node type you use that to be fresh nodes when I'm like powering up new machines or just like anytime I'm running a machine with a clean disc so that gets turned into a mis for AC to Google compute like machine images and for bare metal and a boot image that we do to do not necessarily boot over PXE but the net boot builder builds your entire everything in TV and it Rama fest so that's quite convenient once we have those machines up we want to be able to change them without shutting them down so every system image we built gets pushed to a binary cash and then we just connect to the machines and tell them to run that so or that we'd in theory have to interface with a bunch of next bits there wasn't a convenient google's conclude storage c++ sdk there's a deprecated one and one that doesn't have the right api is that's google's thing I guess so I decided to most of our stack is go and I'd have to bind go to the C++ stuff if I wanted to really integrate it closely with our tools that was not something I was excited about so I ended up implementing most of the NYX store protocol and building myself something that roughly re-implement snicks Poosh but you can easily back it with arbitrary stores without having to write C++ [Music] and something that pushes images builds an image using the the meek disk image stuff takes care of the per provider specific things so now we have binary caches and like images to boot machines from actually
so this little thingy called nib which takes like roughly any anything you can execute commands on and like will take care of if you give it a next or path for a system of invoking that rate it's a ton of code because go is terrible but it's pretty easy to use it's roughly equivalent to a programmatic Knicks us rebuild switch - closure - target host so the binary like has you exposes the built-in support for using goes SSH doing it on a local machine of course it also supports Nix OS in an illumise own because visit onto a some point played around with that [Music]
or stack on top of all this is a bunch of ghost services running in Ducker
those get stuffed into a docker
container with this little thing called next to docker mostly built because the docker tools and expect jizz are very very flexible but aren't very fast this just basically sticks the Tarble like takes a closure stick Siddhant docker image terrible does the minimal work necessary for that our service derivations are built with something called go X which I would very much appreciate the nix the next stuff for because we use import from derivation to do dependency discovery and build every go package as an individual next derivation which I'm not sure if it's a waste of time yet but it's been working for a good two years now so I think that it's alright I'm served my future plans are like to like release that and make it into a polished thing and for deploying our clusters weird like working on more stuff that like has more awareness of cluster state is aware of the subsystems it's deploying like we currently deploy ruffed clusters like etcd and console with Knicks ups and we there's sort of the automatic joining stuff to build a cluster like a logical cluster out of that at the layer of that software that doesn't work out quite as well as having like your deployment software be aware of what it is deploying and what its state is in general we want like a more programmatic approach to it we specifically want it to work in a distributed fashion so there's no state that's on any single machine and it's some level we need to eat full approach because we can't reboot running stuff freely so I'm sort of working on a plan where we I look at booted system versus current system figure out whether I can upgrade the system on a given machine without like losing the ability to load kernel modules and I'm still working on like a plan for judging CI for all this but I'm hoping that's easy enough and sort of our long-term plans are probably sort of changing how we use NYX OS because system D has been giving us a lot of problems in container context so I've been playing with building something on top of the s6 stack which is an approach to service supervision that is a little more UNIX II and I hope will be very nice with Nix but yeah I think that was most of what I wanted to talk about I'd have had more demos but my laptop meseta your cable field so yeah questions time I guess [Applause] all righty I don't you have mentioned that you're trying to move NYX West to a different process provision mechanism is it right or like do experimenter in this area yeah I am experimenting with that because like system these job scheduling system is like very surprising at times and duplicates a lot of like what we do on the next to s modules end of things what is the name of the other one you're trying to so s6 is it is this Garnett service supervision sweet it's one of the many like DG beware things that like solve this in a similar way it's very worth reading about and I recommend digging in I can't really give like a full overview of it more questions nope then thank you again very much for your talkative thank you so next up
again 30 minutes of