How to make Nix ./pleasing to use at work

Video thumbnail (Frame 0) Video thumbnail (Frame 8300) Video thumbnail (Frame 16600) Video thumbnail (Frame 22716) Video thumbnail (Frame 25857) Video thumbnail (Frame 27997) Video thumbnail (Frame 35074) Video thumbnail (Frame 40537) Video thumbnail (Frame 42692) Video thumbnail (Frame 51879)
Video in TIB AV-Portal: How to make Nix ./pleasing to use at work

Formal Metadata

Title
How to make Nix ./pleasing to use at work
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English

Content Metadata

Subject Area
Abstract
At Release Engineering within Mozilla, we develop and support Firefox release pipelines that help Firefox developers test their patches and Release Management plan and execute Firefox releases. Release Engineering manage a number of services where each one aids in achieving these pipelines. e.g. kicking off and tracking a new release, analyzing the source code, and providing external dependencies for building and testing. Currently we use Nix to manage deployment and development of these services.. For this purpose, we wrote a small wrapper around Nix, to make interaction with Nix easier, and help with on-boarding. In this talk, I would like to share my experience of Nix adoption at Mozilla, especially within the Release Engineering / Management team. How we overcome some shortcoming of Nix-the-tool and what is still waiting for us to improve. What worked well and where Nix failed us. The main goal of this talk is to help others who are looking to introduce Nix at work, to know how to set expectations and what an such effort looks like. --- Bio: Rok is a Release Engineer at Mozilla, in Berlin. He is part of the Release Engineering team which helps ship Firefox and other Mozilla products. He is the author of pypi2nix tool, which tries to package python packages in Nix. In his free time … he hasn't had free time since his son was born.
Building Service (economics) Channel capacity Multiplication sign Bit Limit (category theory) Number Product (business) Revision control Process (computing) Roundness (object) Order (biology) Chain Software testing Quicksort Position operator Physical system Computer architecture Relief Task (computing)
Axiom of choice Thread (computing) Beta function User interface Open source Code Graph (mathematics) Multiplication sign Range (statistics) 1 (number) Similarity (geometry) Branch (computer science) Product (business) Formal language Wave packet Number Revision control Term (mathematics) Repository (publishing) Gastropod shell Software testing Physical system Area Installation art Domain name Beta function Scaling (geometry) Channel capacity Wrapper (data mining) Closed set Projective plane Bit Process (computing) Repository (publishing) Network topology Fiber bundle Cycle (graph theory) Task (computing)
Ocean current Axiom of choice Service (economics) Computer file Open source State of matter Multiplication sign Gastropod shell Energy level Circle Data structure Logic gate Multiplication Computing platform Stability theory Installation art Scripting language Multiplication Binary code Moment (mathematics) Projective plane Sound effect Incidence algebra Software maintenance Cache (computing) Category of being Data management Integrated development environment Repository (publishing) Software repository Network topology Buffer solution Quicksort Figurate number Window Asynchronous Transfer Mode
Email Googol Information Projective plane Electronic mailing list Data logger Function (mathematics) Resultant Formal language
Point (geometry) Functional programming Slide rule Presentation of a group Computer file INTEGRAL Code Multiplication sign Mehrplatzsystem 1 (number) Parameter (computer programming) Mereology Tracing (software) Metadata Formal language Revision control Medical imaging Object-oriented programming Synchronization Cuboid Website Overlay-Netz Installation art Scripting language Default (computer science) Boss Corporation Shift operator Inheritance (object-oriented programming) Software developer Moment (mathematics) Projective plane Electronic mailing list Maxima and minima Data management Exterior algebra Friction Software Integrated development environment Personal digital assistant Mixed reality Website Musical ensemble Table (information) Window Resultant Asynchronous Transfer Mode
Data management Inheritance (object-oriented programming) Link (knot theory) Multiplication sign Mixed reality Expression Content (media) Website
Point (geometry) Building Service (economics) Link (knot theory) Computer file Parity (mathematics) Mehrplatzsystem Multiplication sign Formal language Medical imaging Root Bit rate Different (Kate Ryan album) Internetworking Gastropod shell Software testing Data conversion Data structure Domain name Email Information Inheritance (object-oriented programming) Projective plane Expression Electronic mailing list Word Software Personal digital assistant Mixed reality Phase transition Website Right angle Asynchronous Transfer Mode
Web page Wiki Goodness of fit Roundness (object) Different (Kate Ryan album) Internetworking Mixed reality Right angle Resultant Social class
all right on with the show I'm really happy to I'm really happy to announce our next speaker Rock who is working at Mozilla and I'm especially happy because Rock actually happens to be the person that sort of gave me a kick start to mix quite some years ago when he was he doesn't even remember when he was running a hackathon in Berlin so that's pretty cool but today he's of course not here to talk about that but he is here to talk about his experiences introducing Nix at Mozilla and especially things that we should consider to make the experience of using and getting into Nix better so give him a small round of applause rock [Applause] November 9 2004 who can give me a guess what this date stands for I am the hint I was not born at that time exactly so firefox 1.0 what this tells us right so Firefox codebase is old I mean it gets updated it gets modernized but there are traits of old architecture in design and so goes with all the relief services release practices and release tools that come with it so there there is a bit of you have to listen to the elders when you come at the position I am and learn from them 6,300,000 and a bit more it's a number of tasks which we from our Firefox CI that we ran in September previous month to kind of put it differently which is this is it's just a number how many tasks we ran but to put it in compute hours this is 227 years of compute hours so this says that we are doing a lot of building this means also running the tests verifying that the test ran successfully the chain of trust and all of this that we verify that the end product end that 80 megabytes that you download depends which version but the the final product that you download it's as safe as possible itself is we can make it so we are quite busy 265 is the number now don't hold me by the exact number but it's roughly around there the number of releases which happened this year which are the human assisted human assisted releases where humans were involved QA was involved and this is only Firefox this is excluding Knightley's these are automated it just whatever lens it goes out the same day or twice a day I think even so there is a lot of we have in place processes and you can see that we are releasing every day so if this is true we will reach 300 which means almost every day we ship a release even on weekends but it's not true cos we ship multiple a day so I'm regardless I would like you I'm released engineer at Mozilla today I'd like to show you or try to explain how make how to make Nix a bit more pleasant experience to use at least the tricks I think it worked and also things which I tried and didn't work or we could improve upon right so which I'm certainly I will write RFC's after this so it released engineering we use Nick's in not in a big capacity we have some surrounding services of our build system and our build pipeline there are few surrounding services which we Nixa fide i was the main protagonist when I joined Mozilla so I I'm quite I was quite excited that I was given the opportunity to do so and that's this will be more lessons learned while doing so and in order to do so I would like to first start with the bit short introduction just what we do who is the T what kind of team it is what are our fears the desires and so did you know where we're coming from what are our tools as well so that you will have a better understanding of our limitations ours thought process so with that in mind let's start we are a team release engineering is a team of 13 people and release engineering you would say a lot of people don't understand what we
actually do in as we are scattered all over the place of the release pipeline which means once the code lens is it did it landed correctly do we need to verify we don't do actually QA but we need to communicate with QA that it fits the pipeline the release pipeline afterwards once the code lands so we are in every place but we are not expert in particular subject so being 13 people is not it's a lot but it's not a lot in terms when when you have so many products and so quick release cycles we do our process we do a train releases so every six weeks roughly six weeks we do a release this means every six weeks we have three branches or repositories we have mozilla-central beta and release every tree every six weeks merge happens and then all the versions shift down so if beta was with I was just yesterday 63 and now is 64 because the merge day was yesterday so and this means that once something lens in the mozilla-central it takes two times six weeks to be in release if it passes the QA and everything so the tool we use the tool we use the most and it's kind of which I want to mention is the test cluster test cluster is our CI developed in Mozilla before we were using Bill but they are reasons why we start why we wrote test cluster just the story is too long for this short talk it's open source and soon it's you will be even able to install it apart from us no it's it's just a matter of priorities so we need firstly something running that we can use that we can but then it's intended to be used also by others our main language we we use all our infrastructure and all the glue code and our even build system is Python so this is our main choice so you can see that we those six million tests come from our test cluster so if you are in that range of capacity you might start considering looking into that area especially once the the bundles which can can easily install test cluster will be available I'll start with number one of the things I almost observe now that I been here for the whole day it's one common thread that everybody is employing and the ones who don't they should when they start working on a project with Nix and they try to introduce it to their colleagues or maybe even a lander scale and this is not the kebabs but it's wrapping Nick's is the tool in itself is completely works completely fine but it takes a bit of time to get used to it what you want to do is bring the problem the the the domain problem as close as possible to your colleagues this means that while Nick's shell in Nick's build and all these tools they do do correct job you still want to fetch some secrets from your company repositories place where you store secrets you have your own CI so there are these little things which it's very easy to abstract and wrap and have a tiny wrapper I'm not talking about implementing the whole makes a tiny wrapper between executing three comments in the bash and just making a nice UI and it really takes a long way it goes a long way so next time coming to Berlin will have kpop Knicks but how could this look like and this is actually a tool we also wrote so the tool is called peace for the lack of better name so when when you want to start developing you basically say please shell and you will enter the name of the project usually there are multiple projects so you enter a shell you run a project and you run the tests of a project and you deploy a project and what happens behind those commands it's up to you now you don't have to call please it can be your company name or something similar but the idea is this is quite common scenario and the onboarding when your colleagues come is going to be much smoother because once they know how to deploy one project they know how to deploy all the projects next important thing I think this is one of
the this is one of the key reasons why Nick's by my opinion why Nick's is not so widely adopted I think this is like the first I mean there are many reasons but this is the first big one which I think it when I started introducing Nick's the first thing was what do you do when you finally convince somebody and they they they have time to use Nick's they try to install it it's not so easy there are multiple ways how to install it while choices are good giving beginners choices it's not good once you install Nick's and let's say you install it you figure it out that in our setup we require in multi-user mode and it has to have a sandbox enabled there is no way I could enforce this in our in my NYX file to kind of say like these are the properties they might be actually you can read the NYX conf not sure but yeah saying there is more to come to configuring NYX than just installing it might be your custom binary caches and things like that so while this can be all wrapped in the script like the first experience in while installing this is not that great for but we must remember that this is going to years back and looking at how it's now it's much better like I can see already the big big big improvement on the stability of the Installer but both of my managers which are kind of I'll see a buffer in my tree structure they both failed to install NYX let's say the last incident was I think it was in May when for two three days there was actually mixed install script was broken so the curl wasn't working I know it was like bad propaganda but it happened so I lost that moment of I finally I could get somebody else on next but I'm almost ready to so why what I did to kind of circum to kind of go around this problem is the darker stuff things in docker so the nice thing about docker is that somebody else put the effort of documenting how to install it it works on all platforms and you can run Nix in it easily so that's what we're doing so whenever you've run please shell it will displease is actually a best script which runs itself in a docker container and then based on the environment because we detected it will do what it's supposed to do so it execute on the deployment or enter the shell when you are in when you run it from the inside the doctor environment so this way we also configure the whole darker environment so which with secrets you have access to to which not secrets binary caches you have access to so this kind of sorts all of these problems it even makes it possible to use it in in Windows so that's while being and quite having an open source policy kind of being open first at Mozilla this is important because a lot of our contributors are actually coming from Windows platforms so we put the next inside and everything is it's okay we are happy with the current solution next feature which is
quite often forgotten and I think Nick's here is where Nick's really shines is Mona versus multi repo I think Nick's was made to solve this problem whenever because then there is almost no debate whether it's a Mona or multi repo because both sides have cons and pros right so what the good side of the Mona Reaper is if you have let's say three projects which closely collaborate and they're closely connected it's quite easy to have them in one project and quickly iterate right but then once they are kinda more kind of reach the maintenance level it's nice to just let them to be on their own and whenever the update is needed or some feature is missing you can only pull that repository so there are pluses and minuses right with Nicks you can actually have this top repository from which you deploy you know exactly the state you are in and then you refer to another repository so in that sense the you are you are doing a mono repo but you have you're using actually multiple you can use multiple repositories when project is in a maintenance mode when needed or when certain team prefers to work in their separate repository so you can have both with Nix yeah and having top repository which includes everything it's a nice way where you can check between services basically run gate blame git bisect and figure it out where those tricky problems occur in the whole deployment so it's also one thing one nice side effect one thing the solve the next thing which you should basically check first before you even
start considering using NYX is is your language supported in NYX so how do you go about this so we open the google your brow doctor go you browse your language tunics and hopefully you'll get the result don't look for this information on Nixon's dot org it's something maybe there isn't a mailing list but we see that if we kind of hurt ourselves by not exposing these tools because when somebody comes with a project they usually come from a background JavaScript Haskell and there's like okay how do i bootstrap my project and if your language does not have this it's going to be quite rough road you can partially Nick safai your project the rest it's more of a manual work from there on like like before so luckily we have a lot of - Nick's tools some work better some work less I'd like to maybe just shout out to yarn - Nick's by for the design it has where you don't have to run a command so to actually produce the output but you'll
use existing log files which yarn
produces and then the integration then the trick ssin friction between development team which ones to use yarn is there is much less friction between the dev team which wants to use yarn and release management which needs to package everything because nix and yarn lock file are already in sync and there also other tunic's tools we just do this I'm just trying to kind of bring out that this is the nicest way to work with these tunics tools and through not all languages also support this lock files or have enough metadata in those lock files so that we can create this kind of tunics tools but this is really important if you don't have this you will be in a lot of trouble or you have time to write them which is also nice one nice thing is if this is quickly you can overdo it the next topic I'm trying to explain is overlays overlays are nice but don't overdo them like me you can quickly get excited and we'll put this in a box here in this in a box here and this is how it's going to be connected but you there is I would say in your company you should have a layer for your company which builds certain tools certain version of tools which are used in your company and every developer should kind of have access to that overlay then you have per project and I think that's where it kind of ends you don't have to go Raeleen more in the detail of building more and more overlays otherwise I mean it's a nice picture but it gets crazy yeah the next one introducing mix I you probably notice I asked many questions in today how did you introduce Nicks what was the reaction what was the experience what I noticed is that it's a cultural shock especially because our team is used to Python and object-oriented design when we combine code we use inheritance in Python mostly over composition so just this shift is a big kind of alert for somebody trying to learn and as whenever you will try to introduce a functional language you will face this this basically means going to somebody which is which spend the last 10 20 years learning about object-oriented or object orientated programming and saying yeah no this is better right it's going to be not accepted well so what I suggest here is be prepared there is going to be chances where people are frustrated with their current tools and you have to be ready then to step in and ask them this is an alternative this is how I came to Nix I was frustrated with Python package management still am but I came to nix and it solves most of it right a lot of stories I heard were I was struggling there is nothing better and I came so but it's important that when that moment comes the tools which are using a company need to be packaged in mix the the at least the bootstrapping the first default Nick's or shelled out Nick's should be already there in place so they can get a quick feel of it they won't switch in the first day but you need to get them but you need to be prepared and this moment will never happen during the week but it's going to be weekend I guarantee you because then that's when we have time to play around and test new things so if anybody knows this band awesome just been in the concert so kind
of I added this slide I mean it's coming but based on the alcohol presentation this morning to kind of add on the list of the RFC which we need to do so when I was preparing this slide I thought what are the least amount of effort that we could put in to get the most of the results so how to make in a year when somebody in a year comes to their boss and says like let's use Nick's there is gonna be a higher chance that this succeeds this is come not completely everything of course because they're only three points but I think these are at least in my experience I encountered them a lot of these arguments so first one is Windows support I know this is in the world where everything is Linux it's not important but this is not the case a lot of times Nicks is a build tool and if we want to use it as a build tool is going to be compared against does it has a window support and sadly we don't have the good part why I say why I put it here is because there is a Windows whatever the name is subsystem yeah okay exactly that might give us an easier way to reach that goal and be able to compile by Windows executables using necks it's I know there is a poor request and it was revived just today because it was Nick's con so whoever is working on that I'd love to help I can only give moral support in beer or preferred but this is because of other reasons not because I don't want to help time the next one is which is I already kind of hinted it is dr. Enix this should be this is how people use software today like it or not this is how people try software today doesn't matter if Nick's can be removed without traces and you don't even know it's there people don't know that people need to read that but they know darker you know how to install darker so we don't have to teach them how to install Nick's mixes is going to be there inside the docker so while we work on our story of how to install Nick's we can give them a really good story of the first impression and they can start playing in the environment they're used to this is already happening from the work from tweak recently this was this is all just amazing but I think this is a with the minimal amount of effort so how as we create our table which is then picked up by the our curl install script we could as easily just push docker image with Nick's inside in a sandbox mode in multi-user installations of the hardest of all right this would save a lot of man-hours of pee beginners right and the third one it's not yet their website I'm not talking about documentation I think this is this is actually the most important one maybe I shouldn't put it at the end but I'm not talking about it we need a new design of the website when I'm away on
parent to leave soon and my manager again tries to or somebody from our team or your friend tries to learn how to use next it's gonna have hard time finding this resources I know where the
documentation czar I can send you the link discoverability on the website is very poor a lot of content is not there as it should be it should not take more than a minute to be at the with the next installed and with few with an example expression written and start playing from there I'm talking about getting started manual this is not tutorial or short tutorial so this is not something that it should be in the manual in such but it should be something to get people excited that this actually works because the worst feeling is when things fail and then you're like okay this mix doesn't work I don't have much time but I'm really willing to work on this personally and there is no easy solution because there are you know as many as we are here there will be different opinions what should be on the website where but I
think we should start changing it and including more links exposing the content doing some testing it's you know show somebody in your company find me how to install Nix when you come to nix OS network nobody knows currently I would like to change that thank you [Applause] all righty time for questions yes all right yeah my question is about the darker and Nicky support so I used to maintain the darker image on the hub which was mostly when somebody sent me an email I would update it and now it's it's kind of automated if you go to Nick so it's slash talker you you basically just do a pull request and you can update it but I'm not sure what's really missing right I mean okay sandbox support is it is it really that useful if it's within the docker if if there is a reason we can add that probably very easily I agree multi-user yeah doesn't really fit so what do you think is missing there okay starting docker it shouldn't be docker file we should build Nix with Nick's so they are support it's possible already to have Nick's docker image built with Nick's because then we can actually just have it in Hydra and have it ins an artifact of the next build every time we commit multi-use sandbox mode I don't know who would use it without sandbox mode we just have personally if I have a project I want sandbox mode i explicitly would like to disable somebody using without sandbox mode if possible like that would be an assertion multi-user wise multi-user in our case maybe this is not really needed but in our case we need it because we have to run we installed PostgreSQL and if you want to run PostgreSQL you need to run it as a different user than root so and reusing the same image kind of makes sense so that's where I'm coming from yeah okay more questions for the third problem the website can something like a github curated list like an awesome list but more structured can that solve the problem it's a nice idea I don't think so reasons imagine a docker calm would be a list of awesome websites would you use docker right it's like I'm not saying we should be docker but I want that feeling when you come you have a getting started in five minutes you know you have a successful mix built and you're in Nick shell so you have something tangible that you can go after fifteen minutes and say like ok I will leave this I will come this to later when I have more time so having these early early adopters I like what domain is doing where where it comes these services with Kashuk Hercules this is awesome but we are still stuck with the first phase somebody actually considering Nix and I think this is where our conversion rates well I use the big word actually we lose a lot more questions yes yeah so to the same point for me like when I when I made a PR against the manual I don't remember which one it was Nick so as next packages when I realized that it would only show on the website after six months that for me that was like I was like wow okay I'm not gonna go through the trouble again so for me that was like really something that I felt like that doesn't really help contributors contribute I think this got improved already that there is a kind of unstable not yet I saw some pull requests or some discussions that surrounding this where the master documentation it will be available somewhere in the Nick so as that work might be completely wrong but it should be possible I mean this is a lot of like work we should do agree but while I'm what I want to point out with the website is not that there has to be a structure like deep nested structure just the front plate to include all the information which language tunics tools do we support list them there this is important give examples how to use those tools that's an extra bonus getting started manual in like five minutes docker image I think like that would make it easier a lot for my a lot for a lot of my co-workers and friends who I try to convince the snicks and they gave up after five minutes of doesn't work ok maybe you've got time for one more question we've got a question from the internet that's fantastic do we have telephones as well aren't you afraid that your co-workers will find another solution while you're on parental leave especially if nobody can help them debug their Nix expressions from NVP this is the fear I'm living with putting no putting this aside like the joke aside it's a valid question the next my parity leave there is going to be problems so sure I'm going to
teach them so I'm going to show them but it's hard to do this when they're stuck
with installing mix right so for me this is the first goal and just having like the documentation if you read it it's actually good right but you need to find it right you need to you need to know that there are three different documentation manuals I after a year of using NYX in mixed class I didn't know there is an expected is manual right I mean I'm not saying like the everything here is done with a good intention right I think for a little effort putting in the front page and gathering these resources and pointing to wiki pages to the oldest tools we could achieve a lot of better results and being a lot more friendly okay so after this nice question from the internet let's give Rock another round of applause for this very nice talk [Applause]
Feedback