okay there we go so the first speaker for today is record and he's gonna tell us about deployment about setting up a deployment system from scratch and how to make use of NICs those modules even outside of an exercise so take it away and draw everyone and warm round of applause for regard [Applause] so good morning Ricard I am here to talk about the NYX module system and how you can use it for more than just configuring mix OS so short inter me I'm a freelancing developer and DevOps guy I've been building stuff based on Nick's professionally for the last five years so I run the company images mutable solutions and I'm proud to sponsored every Nick's con so far so during this talk I want to show you the way I've used Nick's modules to implement operations infrastructure for a client of mine and the stuff up sent today have gone through a few iterations since I first started about two years ago and I still work for this client and we keep using this in production and we keep improving it and in the end I think I've come up with a system with some pretty interesting properties and that's it's what's I that is what I hope I will be able to demonstrate today this is no I have no product to show you this is not something that you can take and drop into your own projects unfortunately we have some parts small parts of it open and out on github very poorly document though so it would make sense to open up more of it but that's always it it takes time and energy to to do that so what I hope you will take with you and after my talk is that you can use the NYX module system for more things than configuring mix OS and that's modular declarative configurations be it using NYX modules or anything similar it's a very nice way of configuring deployments and probably lots of other stuff too so these are some core principles you might say that I've had when that I've had in back in my mind when building deployments tools so I want to be able to bootstrap from nothing except maybe nix and get I want to configure as much as possible in a declarative fashion and there should be minimal differences between production development staging and so on configurations and it's nice to have all Ops related tasks available in some kind of shell so all of those things makes it makes it easy for new recruits to get started with projects and it makes it easy to make changes in places you're not that familiar with and of course to try out new things and experiments so when you save you implemented a deployment system based on Nix one of the first question is why didn't we I didn't use Nick's ops and I think it's been brought up during a few talks yesterday and I there's some issues that I have had with it so it's not declarative enough you have this state file you have to manage anyway in some way and that's not really compatible with my bootstrap from nothing approach [Music] it's not modular or extensible enough you can extend it by using the Python API so you can write new backends and implement new functionality but hi I had a more much more sort of flexible extensibility in mind and when I started working on my system and I hope to be able to to show that today also my client is using the Google cloud platform and when we started Nick stuffs back-end for that was not as much sure today as it is today and I think there's still things missing that that we need so I knew I had to implement some of the provisioning stuff anyway yeah so now I'm going to jump directly to how production deployment might look like for one a my for my team and this is simplified of course but the concepts are same and if you configure Nix OS you recognize this syntax this format we use the exact same kind of modules that Nix always does so in the top we import two other modules one of the hello world module defines the applications application where we deploy and the

Google cloud platform defines everything that is needed to provision and deploy an application on Google cloud platform and depending on what modules import you get different configuration options available so the greeting phrase option comes from the hello world module and the GCP settings comes from the google cloud platform and then we have base modules that define things that's used for all our deployments like name and domain settings and environments are things so what I wanted to show you is that we can have a very clean top level deployment configuration and of course this is a silly application and we don't configure much here but that the concept is the same we can have things that make sense that you can read and write that makes sense even if you don't have detail no like knowledge about how everything is setup and this is exactly how NICs OS works - we can sort of hide details in lower level modules and have a sort of a layered configuration so here we have a staging setup for for the same deployment and I just want to show you that this is also deployed to Google Cloud so the differences are very small here and here we have a development set up we still import the exact same application module but now we use another back-end the development back-end that we have and we import an additional application module the monitoring module and this is exactly the exact same module that is that we using in production so it's said that sets up all servers needed for log aggregation metric visualizations alerting and all that and we deployed that to in production we deploy that to a dedicated deployment and all other deployments will send their monitoring data to that deployment but in during development we don't want to send things off to a central aggregation point so in yes instead we just bring in all of that infrastructure inside our deployed development deployment so the developer will get all those services locally on his computer and that's very useful to make sure there's no like deviations between development and production and you can set up things like alerts log queries and everything and try it out locally and then just directly move things to production when you're when you're satisfied okay so I showed you three variants of the same application deployment and it all looks very nice but probably a bit abstract so how do we use this we have a thin we have a script that is a thin wrapper around the next tools that will evaluate this top level module and all the important modules and in the end it will produce deployment shell and this deployment shell welcome we give you all the commands needed to work on that particular deployment so it includes commands for like provisioning the cloud resources deploying and activating the applications performing maintenance tasks so in in a way each deployment configuration will result in its own unique deployment tool with all the features that's needed for that particular deployment and that is what that was that isn't what I meant with a more like flexible extensibility to the NICS ops so we can we can plug in new things support for new cloud resources or cloud providers or anything by just implementing unix modules and drop them into our deployments so i'm going to show you how this works in practice using this development configuration so here we have our development deployment and the top level configuration is exactly the same that I showed you on the slides and we have our shell builder that will that will take this configuration and build a deployment shell and then enter that so now you can see what that we're inside the hello world deployment shell and we can start things off by running the provision command and this will start up

since this is that use the development pack and it will start up systemd containers for all the servers that are defined in our configurations and it will also configure DNS locally on my laptop here for the the endpoints that was that we have defined in our applications so now we can actually open up a web

page and see our application running here and we can

we can change the config we can change this greeting phrase

and then we reload the config and this

will actually rebuild the shell and now we can run the deploy command which in this case is a little shortcut since we can reuse the existing system D containers so we will just the deployed command we'll just activate the new

configurations and those errors are

expected here and we have the new the

new greeting place in deployed and I can also show you that for example we have

agra fauna installation here with some

dashboards yeah there won't be much data here we can

refresh this sometimes maybe and then we

yes or

oh yeah waxing we can say we've got six six requests here now and it's yeah and this is yeah so those dashboards are the

same that will be deployed in in production and then we can experiment with setting those things up during deployment and have the actual exactly the same sort of chain of flow of metric data locally so we know

and finally here's a test deployment for the same application and now we import the test backend instead and it's it's a very it behaves very similarly to the development back and actually it uses systemd containers and brings brings the whole deployment up but it does that inside Nick's build so we can run this on Hydra for example and we can now add a test script to our configuration and as you can see inside this test script we can use the complete deployment configuration so we we don't have to hard code a greeting praise the phrase in this in this small test so we can pick up the things we need from the configuration and we write this general assertion in our test script so as you seen depending on what modules we import to our deployment we get things like different shell commands different deployment tasks we also defined Hydra jobs in this way so application can sort of plug in there are tests and bills that they need and it will they will appear in our Hydra instance and the monitoring dashboards also depends on like what back-end you use what applications use and and all of this you can also define in a top-level configuration if you want to so if you need some specific command for a specific deployment you can just add it to that deployment and get that available in your deployment shell so this is the the complete contents of the hello world application module so as you can see we have more levels of modularity here so the application module imports another HTTP endpoints module and that will take care of setting up nginx configuring request logging metric collection DNS setup as I saw and you can even have it can't even handle load balancing and stuff like that so in our application module we just have to drop in a small piece of nginx config and in normal case you would probably proxy that request off to some real application but here our complete application implementation is inside nginx so this way as an Operations team we can sort of define a suitable abstraction abstraction for for the applications developers this is one of the strengths of the next module system that you easily can define those abstractions that makes sense in the specific context but you can still allow for for uses to sort of drop down to lower levels and use stuff below your abstractions so as I mentioned the modules we are using are are the exact same same things as there as in Nix OS so there's good documentation on how to implement modules in the official Nick service manual I'm going to save just a few words about how we evaluate configurations and build the deployment shell so as an example consider those two very simple modules the customers module just define a customers option which is a list of strings and my customers module import this module and sets the customers configuration to ABC so next modules are evaluated by using the eval modules defined in Nix packages and we evaluate we give this this month the function takes list of modules as arguments and it will compose those modules and everything they import and evaluate and produce a final configuration out of it and we can simply ask Nix for the customers attributes and then we will get back ABC as expected and for our shell builder this is a simplified version of it but but the concept is the same so we just evaluate the top-level configuration module and then we ask for the bash RC if I build and launch a new bash shell with it with this and so in lower level modules we we have things for defining commands and they will build up this bash RC file that will make those files those commands available inside your shell so after having showed you love some nice and happy stuff for in this talk I just wanted to admit that the next module system is it's not perfect so the main issues I have with it is that the namespace in in next modules is global and everything is public so any module you import can change the configuration or even option the declarations of any other module in your composition and this can get messy and it would be nice with some additional level of control maybe elko has some something for managing that and it's not new new configuration proposal - I don't know but some way of sort of typing the module interfaces more strictly or having private interfaces or something would be beneficial and evaluating modules is quite slow and in our setup we make things worse by by having to evaluate several Nix OS configuration inside for each deployment so that is also a bad thing and our Hydra setup is sort of spending lots of time of just evaluating all of those configurations for for the test builds finally since the modules are evaluated in a recursive fashion it's easy to get into a situation where you make a configuration depend on its self somewhere and you get those infinite recursions and it actually can be quite tricky to find out exactly what caused it and the huge stack traces that Nix will provide right it's not always helpful because they can sort of the they often point to library functions within X packages and not do not reveal reveal daesh's and then my method of debugging that our fixing that is often just to remove stuff until the recursion the infinite recursions go away and then add back one thing at a time so that's not ideal yeah that's all for me for me thank you for listening [Applause] all righty so ok I was about to say who has questions and the first hand is already in the air so here we go hi thanks for the talk I see that the way you set this up basically does all the infrastructure set up DNS machines everything and as he I'm sure you know there is a tool around called terraform that does most of this stuff so what was for you a compelling reason to go and develop your solution with Nix rather than using terraform yeah so so terraform would be a great back-end to our system I mean it would be very useful to to generate terraform configuration from formal confirmation and use that to provision stuff so so that is absolutely and we do use for for the cloud resources we do use Google's deployment manager which was also a declarative way of specifying the resources we generate a description of the resources we want from our configuration but it would make perfectly sense to use terraformers also the backend and you could probably use Nix ops as a back-end - something like this - like generate mix-ups configuration to make use of provisioning code more questions yes what does the dev backend look like in the code yeah so we have we have implemented this small daemon for running multiple NIC so as containers basically so the dev back-end we basically just generate basically as a map of machine names to Nick so as configurations and hand this description to our demon and it will bring up those NIC's always containers and keep them running until one of them stops or something like that so we used that demon both for for development and in in our test scripts so so Hydra will actually we do sort of impure Hydra bills where it will use make use of the local Nick OS containers demon thing that we have so so the dev backend is is actually quite simple it will just build that description and fire it up more questions yes hey so we've talked to us a little bit one of the issues I've run into with the module system is the global nature makes it very hard to reuse one module in multiple places because there's only one namespace have you run into that or what do you do when you we haven't had any big issues with this but that is mainly because we as a team and as a companies can sort of control our module suit so we don't have to to share them with sort of the public or we don't use modules provided by someone else but it would definitely be nice to have better control in this area so so that we would allow some sort of libraries of modules and stuff like that because that's entirely you have to sort of just come up with some convention now and hope that people adhere to it does the set up require me running mixer so can this be used in a different Linux distribution and do i needs privileges for running this kind of setup or can I run it as an unprivileged user so I think that you only need Nick's working Nick's installation to run this but I haven't actually tried it on on anything non Nick source and we have sort of deployment machine that our developers log into and and run their deployment shells from when when they doing production deployments and so on but III do deployments for my local computer since I have any services everything set up and the the small daemon that I was talking about that brings up system D containers needs to run as root on your computer but you can but you can then communicate it over a small over homegrown protocol - one more question nope okay so then thank you very much for your interesting talk record and I foolishly forgot to mention that record is also sponsoring ours company material solutions is actually sponsoring ex-con so when I ask you for a warm thank you and applause I expect at least five additional claps per person extra so thank you very much regards [Applause]