Sneaking Nix at work and become a hero, hopefully

Video thumbnail (Frame 0) Video thumbnail (Frame 2001) Video thumbnail (Frame 2874) Video thumbnail (Frame 3531) Video thumbnail (Frame 5576) Video thumbnail (Frame 7923) Video thumbnail (Frame 8606) Video thumbnail (Frame 15175) Video thumbnail (Frame 15820) Video thumbnail (Frame 19203) Video thumbnail (Frame 20239) Video thumbnail (Frame 29148) Video thumbnail (Frame 36083) Video thumbnail (Frame 36851) Video thumbnail (Frame 39250) Video thumbnail (Frame 41048)
Video in TIB AV-Portal: Sneaking Nix at work and become a hero, hopefully

Formal Metadata

Title
Sneaking Nix at work and become a hero, hopefully
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2017
Language
English
Production Year
2017

Content Metadata

Subject Area
Abstract
This talk explores ways to introduce Nix into an existing infrastructure (at work), based on real-life experience.
Goodness of fit
Source code Digital filter Code Building Channel capacity Projective plane Source code Computer program Client (computing) Binary file Scalability Computer programming Product (business) Cache (computing) Phase transition Computer science Gastropod shell
Point (geometry) Slide rule Computer file Java applet Connectivity (graph theory) Source code Price index Mereology Front and back ends Revision control Data model Derivation (linguistics) Gastropod shell Data structure output Physical system Scripting language Source code Debugger Projective plane Bit Stack (abstract data type) Variable (mathematics) Derivation (linguistics) Hooking Integrated development environment Software repository Personal digital assistant Right angle Gastropod shell
Source code Building Distribution (mathematics) Computer file Projective plane Line (geometry) Mereology Power (physics) Derivation (linguistics) Hooking Process (computing) Gastropod shell Configuration space Gastropod shell Asynchronous Transfer Mode
Slide rule Digital filter Building Computer file INTEGRAL Multiplication sign Disintegration Source code Function (mathematics) Mereology Formal language Attribute grammar Revision control Derivation (linguistics) Different (Kate Ryan album) Phase transition Operator (mathematics) Software testing Vertex (graph theory) Acoustic shadow Source code Scaling (geometry) Projective plane Bit Stack (abstract data type) Derivation (linguistics) Type theory Hash function Mixed reality Phase transition Revision control output Software testing Data logger Physical system Spacetime Asynchronous Transfer Mode
Module (mathematics) Source code Digital filter Building Phase transition Multiplication sign Source code output
Filter <Stochastik> Boolean algebra Source code Rule of inference Digital filter Functional (mathematics) Inheritance (object-oriented programming) Computer file File format Computer file Source code Projective plane Infinity Directory service Loop (music) Resultant Library (computing) Condition number
Scripting language Euclidean vector Multiplication sign Source code Set (mathematics) Derivation (linguistics) Computer configuration Information security Physical system Overlay-Netz Scripting language Structural load Feedback Infinity Bit Derivation (linguistics) Software repository Phase transition Configuration space Right angle Physical system Data structure Resultant Point (geometry) Slide rule Inheritance (object-oriented programming) Computer file Connectivity (graph theory) Control flow Directory service Attribute grammar Revision control Latent heat Goodness of fit Root Hacker (term) Software repository Gastropod shell Utility software Configuration space Default (computer science) Default (computer science) Slide rule Projective plane Line (geometry) System call Loop (music) Personal digital assistant Revision control Gastropod shell
Dataflow Scheduling (computing) Game controller Scripting language Module (mathematics) Computer file Multiplication sign Source code Function (mathematics) Revision control Goodness of fit Cache (computing) Configuration space Aerodynamics Traffic reporting Logic gate Physical system Electric generator Projective plane Open source Data storage device Electronic mailing list Bit Declarative programming Cache (computing) Data management Process (computing) Voting Hash function Circle Logic Personal digital assistant Mixed reality Scheduling (computing) Physical system Reading (process)
Cache (computing) Personal digital assistant Multiplication sign Binary file Number
Asynchronous Transfer Mode Scripting language Computer file Mehrplatzsystem Multiplication sign Data storage device Client (computing) Cache (computing) Different (Kate Ryan album) Single-precision floating-point format Damping Physical system Point cloud Link (knot theory) Wrapper (data mining) Optimization problem Binary code Data storage device Client (computing) Binary file Personal digital assistant Synchronization Configuration space Key (cryptography) Asynchronous Transfer Mode
Multiplication sign Content (media)
Digital filter Server (computing) Run time (program lifecycle phase) Multiplication sign Computer-generated imagery Content (media) Maxima and minima Modulare Programmierung Content (media) Existence Product (business) Web 2.0 User profile Derivation (linguistics) Subject indexing Personal digital assistant Integrated development environment Configuration space Modul <Datentyp> Physical system Booting Point cloud
Slide rule Computer file State of matter Multiplication sign Source code Mereology Declarative programming Revision control Cache (computing) Gastropod shell File system Plug-in (computing) Physical system Module (mathematics) Default (computer science) Distribution (mathematics) Projective plane Binary file Hand fan Derivation (linguistics) Internetworking Software Software repository Configuration space Gastropod shell
all right our next talk is by Jonas he's been a contributor since 2015 he organizes the Knicks London meet up with Thomas hunger and is now working as a Knicks contractor thank you thank you all right so today I want to talk to you about using Knicks at work the title is a bit facetious but basically I want to show how you take a good ways from work or customer and next to fights and there's different stages that you will see that you have to go through so just for a quick history so September last year I was tired of hacking Knicks on the side and so I decided to quit my job and become a next contractor but there's not a lot of work around yet and after a while I met with tweak and they have really cool customers and I was able to do a lot of next stuff with them and now I joined tweak and they're doing sort of
aren't applied computer science so it's kind of orangey loud before other customers alright so the program is
really to show you the different stages there is gonna be first phases next shell then you want to package things and finally you set up your CI ok so I
can't really show the customers code so I built a little up I actually took some of the source code from a project code to do MVC and then I next it fired it
and you can find all of the source code over there so if I'm going a bit quickly through the slides you can always have a look on the repo and you can see the file structure is basically you have one back-end one front end and the back end is a skill project and it has two components and the front-end is just a bunch of Java scripts and yeah first thing is NIC
shell so you know you you just drop a shell that Nixon the project and your and your colleagues asked you what is this file and you say don't worry about it and then at some point suddenly they have problem with system dependencies and then you're like oh yeah you could try to fix it or you can install Nix and just run next shell and it's going to be fine and basically the shell in the next file would look something like that so you import next packages and then you create the derivation and the important part is the build in puts where you put all the system dependencies that you need and then a last feature of an excel is you can run some bash scripts that in this case with source dot n file that contains typically environment variables for the project right so that's kind of the version zero and so one thing you might notice is you have to put a name you have to put a source which is always null or some path but it's not really relevant so I think we should introduce MK shell that just simplifies the
process and also that's pulls in all of the building parts from your different packages that you're going to define later on all right so here I'm using an overlay and so I can pretend it's already in the expected but it's in your project and my attention is to submit a PR at some points or if you want to do it you're working the other thing that's important or that can trick you quite easily is if you run Nick shell in - - pure mode then lots of the tools that you have available are not going to be there and usually if you customize your bash RC then suddenly you're gonna have failures and so that's why this first line is quite common you find it on most Debian distributions and then I recommend to add this second line which is something that basically escapes the configure of your bash RC if you're in the pure mode or just any next trial actually alright so you can even start running things in the next shell without really stepping into it by using the - - run command and now any you can even plug this in to UCI and now you build with the next shell and it's not really pure but it's kind of does a job okay so that's the first
stage and it works but it's not really pure and I would say the main issue is that it's not composable because one of the really cool thing about Nix is that you can take make a derivation and then compose it and make another derivation and that's the really the power of Nix and something you don't find with darker files for example and so you would build
packages or I'm not really going to read into how to make packages the main recommendation I have is read mix packages there is lots of examples out there but basically it looks a little bit like that you have a package name then you specify the source and then you have different types of inputs which you should probably learn because not everyone knows so the native build inputs are tools that you used to build but are not gonna be part of the outputs and this is important especially for people who do cross competition then there's a propagated build inputs which are tools that you might use for build but you also want to install afterwards so they kind of come with like if you have a binary dependencies and then the building puts that probably everyone knows you so want to learn about the different types of phases there are different types of outputs and yeah so for this project we have some high scale project packages and I'm not going to go too much into details but right now in this space you have multiple tools so you have cable tunics stockage tunics you can also try to use the just the Haskell packages that are in the next packages you can also use the stock tool which is a tool that's in the Haskell community and it has a - - next mode but basically all it does it puts GHC from the next door but that's it or you can swap things around and create a derivation that invokes the stack and it's maybe impure but you can maybe control the dependencies a little bit better and for the node parts with Martin or is Martin art it is we worked on this project that's called yarn tunics and what it it's does is Soyoung is a face book project that tries to replace NPM and the good thing about it is it generates a log file that contains the hashes of all of the packages that should depend on sorry after download the package from NPM okay I have a question okay can you okay so the cool thing about this is that's actually what we do is so the first phase was to do like any other languages where you do a yarn to next projects that generates the next file from your lock file but then we realize that actually this operation was pure because all the inputs is the hashes from the other file so now we can import from the outputs another we can import the next file that's generated and we don't need to compute any checksums which means that basically your yarn package doesn't have any shadows that's the magic so what it does it takes the name from the package to chasten the version from the package to jason so you can see the name attributes is missing here and you don't have a shot for the rest of the dependencies okay so this is a slide that I wanted to finish earlier but I didn't get the time I was actually taking on one of the talks were seeing earlier where what I do is I instead of making one derivation y run the tests I tend to create multiple derivation one that has the build outputs and another that has the tests and because next is composable you can do that and it's really nice because sometimes you just want to build your project and you don't want to run the test again because maybe there are integration tests and they take a long time to build but if you change the do check attributes then you're forced to rebuild so it's kind of a way to be more dynamic ok so what one thing that's
missing here is you see the icer SRC inputs is actually pointing to your current folder which is where you have all your node modules and all of your source code is going to be inserted into the next door at Build time but you don't want to have the node modules
further being inserted into the next door so there's a built-in that's called
built-in that filter source and you pass a function you pass an absolute path with which is where you have your source code and it gives you a next door and the function itself gets the absolute path of each of the files so it's it goes through the files of your projects and then invokes the function and if it returns true it keeps the the file and so you get the absolute path to the file and the file type which is like a file or a directory or maybe a symlink and in the next packages there is one tool that exists that's called lib that clean source which basically removes the result file that you would help you get from our next build so you don't if you don't do that you're gonna run next build and you run next build again and it builds again because it's inserted the results from the previous build and so you kind of are in an infinite loop and it's nice but it doesn't really compose so typically you would have to rewrite it or make a function so what I propose is that we change a couple of things to filter source first I think the function should return true if we should remove it so it inverts the boolean logic because it's more natural if you look in the I think in next packages oh you're not gonna see it here okay just trust me but the clean source function is actually doing exactly that for lots of conditions and then adding notes in front so I think it's more natural to have it this way and then I have a second function that allows you to compose these filters and that way you can take the clean source from next packages and then add your own special cleaning functions all right so now we
have we have packages and there's a last thing we need to do to make it really nice and it's painting next packages so
what's happening right now is you have your project with all your derivation that are being built but each colleague might have might be on a different Channel and so they might actually get different results which kind of yeah the point of Nexus to have reproducible builds while one of them actually so I went through different different phases of how what is the best way to pin next packages so the main like the trivial
version a version 0 is you use the built in called fetch tarball and you say fetch the next packages from this show on github and that's it and you get back the source code and then you can import it and it works pretty well I mean the only issues that every time you invoke next build or next shell is going to try to read unloaded and one solution is to upgrade to next 1.12 and then add the sha-256 but then your colleagues who are still on next 1.11 it's not going to work anymore because the Shia attribute is not supported in that case so then the next idea was okay maybe we can import that you know fetch from github that we love and use and all over next packages and just fetch the source like that and it works also really well until you set the next path through this file because then you're importing next packages from itself and you have an infinite loop ok so then the next idea was so this one you're not really supposed to read it but it's a bit crazy and is it's been invented by a guy called dr. which I don't think is there today but basically he reimplemented fetcher I'll fetch star ball by in this line he finds the config of NYX and it's a secret file well I didn't know it exists and it contains the reference to all of the built-ins that are used to build the next utility and so you can actually without importing NYX packages already have access to gzip and tar ball and you wrap this in the derivation and you put the sha-256 and you're good until you do next build the option sandbox true and then because these paths are actually they come from the next star but there are not their strengths and they're not really pass and that's breaks the system so I was a bit disappointed cuz I really like how convoluted and there's a special place in my heart for this kind of hacks so in the end I think we should just have a compatibility layer for vegetable that switches on the next version and that's it that's the best way I found currently alright so now we have the source of next packages ok offer a question from Duman alright so domin made it work and it's gonna show us clever ok cool so that's one of the reason I wanted to give the talk is to get this kind of feedback okay so one more to this slide basically you should really have our data scripts because now you're probably tracking 1709 and you want to get the latest security updates and to make it easy you need the scripts that you can invoke either by hand or by CI but that's really the last step that we need to standardize then make it easy because otherwise all the good work you know that's been done by the security people is kind of wasted okay so we have the source but there is a last step that you have to do and is when you import so we are familiar with this import brackets next packages and then you have empty attribute sets and actually you're supposed to set some stuff in there you're supposed to set the empty config and overlays because if you don't do the empty config then each user can have its own config and then it's not pure anymore and the same folder overlays plus overlays are cool so you should really use them alright so now we have our next nexif ID repo each of the different component has a default next and then in the old packages you kind of call packaged into old packages the overlay and you call packaged to load of these components the default dot next is the one that ties everything together the next packages SRC is the version four and the release that next is just gonna react sport everything from default annex that you are interested to build for your project because default done next contains all of next packages plus your project packages and that's it alright this is just a like nifty trick I found out just like last week is usually you have a scripts folder where you have tons of scripts right specific to the project and one of the thing that you can do is if you set the - I next packages equal next and next is the folder that we have here right and so now you're reusing all of the same setup that you were using before and you're nuts again diverging with the packages that you're using only downside is you have to invoke your scripts from the roots of your repo alright so
last step set up the CI so the general
approach is just going to be next build the next release that next file and that's it and I think most of the logic should go into the next files and then maybe later on you're gonna add some impure stuff for example pushing darker packages or talking to Cuban ETS or something new that's so the other thing is you're coming at the existing place where they already have the CR in place so one thing you're gonna see from this list is Hager is missing and I actually tried all of these sea ice and basically I just want to go through quickly through all of them and show you the advantages and disadvantages so Travis Travis is the first shot at Todd mixed reports after Hydra and is is doing a good job and it's probably working better on smaller projects but it's bit hard to debug and it's sometimes it's a bit slow and but it works alright and then there's a circular sea ice 2.0 it's darker based and you can restart your builds and you can get SSH into the container so it's kind of handy sometimes to debug things the the only issues that they have an immutable cache so if you want to store the next door for the next build the problem is you need to give the unique hash and the unique hash is gonna be I don't know something but if it changes then you have to redownload all your next store so it's not I don't know I'm a bit annoyed by this actually I don't know if there's a better way to do it but basically it's kind of they're working one against each other and it's actually quite common with the next store and other caching system is that if they're not perfectly aligned so I would say if they don't have the exact some notion of hashing I would say I'm not really sure but it makes things a bit difficult in the Travis case I think they just load latest stuff from s3 and then read um bits back and so most of the time it works alright so it's actually an advantage for us that they are not too pure alright so github is also really cool it's a agent-based so you can run the agent on Nexus and then have get lab managers schedules all of the jobs and you have good control on the targets and it's already in the next size next packages the only downside is that you need to move all your source code to github so if you have already your own work flow around github or some other source control it's not very handy but overall the gate lab on on his own is also pretty good then there's Jenkins I don't know if I need to talk about it but it's actually they're making a lot of efforts to make it work but I tried even last month I was still trying to make it work and I spend a lot of time just fixing things so maybe it was because I tried to make it work on Cuban --'tis I don't know ok so last one is build kites and it's a commercial thing but the agent is open-source and it's already in the next packages and then they control the dashboard and the job scheduler and it seems to work pretty well it's configured as well with the pipeline that you can configure it with a yeoman file but you can also dynamically generate Yaman file so the next thing I want to explore is see how I can split up the bills by invoking next build finding all of the outputs that's all gonna be built and then maybe generate a youngin that and then like she'll vote the bills I don't know so if you have any ideas let me know after the talk all right so we have a CI
now or one of these but don't forget about the binary cash especially if you're doing high-skilled evidence it would be really nice if you could take most of the time the bills from the binary cash and you would just save a lot of time also in some of these cases
when you're scaling the number of nodes is really nice to be able to download from the binary cash because otherwise you're just gonna have one node that's
rebuilding from scratch and it's takes two hours so to do that what you do is
so I didn't know about next 1.12 features which looks really really cool but so I had to build my own little wrapper that's basically invokes next push and then use in this case it's cents to Google Cloud storage and it works pretty well the only issues when the nar files are big files like darker containers or something like that it can take a lot of time to build so I don't know if this is a optimization problem or something but it would be nice sorry don't push the current is yes and but that's not enough so that's the first side and then the second side is setting up the clients and why now you have to change the system config to allow your to allow to fetch from the binary cash and unless you're in a single user mode I think so it's actually difference if you're in single or multi user mode so now you have to ask your colleagues you know are you using the single user or something so it makes things a little bit complicated but as we learned recently with next 1.12 gonna be served all right so maybe last
bit how am I on time all right I feel like I spoke already enough but all right so darker content I think one of the cool thing about Nexus the composability which has mentioned before and one of the thing you can do is take
your package so in that sense the in that one is the content and just put it into this other derivation that's gonna produce a darker container and your darker container is gonna contain just a minimal amount of stuff that you need to ship into production just the runtime dependencies so in that case you have the the the web the index dot HTML some JavaScript and then caddy which is a web server alright and that's a nice hug but I'm running out of time so this is how you can rebuild next you can reduce the next rest module system to build a container and that's just
pushing the containers so that's it we
started with the next shell then we did the the revisions and then we set up the CI and now we're very good thank you
[Applause] is there time for questions yeah all right yeah yes a small question about this filter source example and the node modules folder did you never run into any problems with that because for example some packages may also declare bundled node modules and they may use slightly different packages than the upstream versions and I know if you've ever run into trouble with that not every corner ok because I've seen a couple of packages that that really required the bundled node modules folders to work all right so yeah you may want to make that optional because in general it's a good thing that you clear you clear out the mess but sometimes you actually need the the node modules folder that is in your project okay thanks Krishna Hey so I can't quite believe that I'm about to defend Jenkins and I feel dirty already for for for doing that we we I'm not a fan of trinkets but we do use it at RNA check because we have an exposed which aren't which aren't fewer and it's actually working pretty well for us so we've got everything like is this is defined in a declarative way and we have all the packet all the plugins pinned and it's working it's working really well like I'm not loving Jenkins but it's pretty easy to set up and to use or for Nick stuff so ok that's my experience so I think I agree with so what Jenkins did is they introduced Jenkins file that you can add to your repo and this part is very declarative now and they also did a lot of work on cleaning up so the default set up now is gonna integrate with github more properly because there was lots of work to do for example just make sure that it pushed build States to the github PR stuff that so that's much better but you still have a snowflake problem where the config on its own of Jenkins is not declarative okay maybe it's declarative all right so have you had to think much about distribution of channel and what I mean by that is those source references with pathway you know dot slash Pat doc become very painful at least in my experience in the similar problem when you when you actually want to distribute your software to other people using NYX and I was wondering if you've encountered that or you had any thoughts of that or you want me to clarify and actually explain what I mean by that so in general what I have is a self-contained repo where I don't the the artifacts I'm pushing out our darker containers and things like that so maybe I'm not in contrary and there's a last question yeah sure I'm gonna publish the slides online all right so it generates the next file system but you don't have system D in the container so this thing it would be nice if we could solve it actually and angry yep thank you [Applause]
Feedback