Using Nix in production for the last two years

Video thumbnail (Frame 0) Video thumbnail (Frame 9133) Video thumbnail (Frame 15641) Video thumbnail (Frame 18655) Video thumbnail (Frame 27806) Video thumbnail (Frame 31541) Video thumbnail (Frame 41501) Video thumbnail (Frame 43628)
Video in TIB AV-Portal: Using Nix in production for the last two years

Formal Metadata

Using Nix in production for the last two years
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Talk with go through lessons learned using Nix in last two years to automate software distribution. What are the common technical and process obstacles using Nix to ship software in production - mean that customers depend on the pipeline. What supportive tools and Nix features were used, what Nix features aren't useful, where do I see improvements are needed. What do I see as something that will boost Nix usage in companies and what's my plan to get there.
Point (geometry) Trail Functional programming Group action Multiplication sign Online help Mereology Information technology consulting Formal language Heegaard splitting Mathematics Sign (mathematics) Goodness of fit Universal product code Thermal fluctuations Information security Lambda calculus Capability Maturity Model Source code Distribution (mathematics) Email Stapeldatei Graph (mathematics) Trail Block (periodic table) Forcing (mathematics) Uniqueness quantification Projective plane Bit Software maintenance Type theory Number Process (computing) Kernel (computing) Network topology Blog Order (biology) Self-organization Right angle
Functional (mathematics) Inheritance (object-oriented programming) Scripting language Computer file Multiplication sign Source code Virtual machine 1 (number) Ultraviolet photoelectron spectroscopy Expert system Mereology Software bug Revision control Derivation (linguistics) Different (Kate Ryan album) Software repository Gastropod shell Configuration space Extension (kinesiology) Logic gate Stability theory Module (mathematics) Personal identification number Scripting language Graphics tablet Default (computer science) Multiplication Theory of relativity Inheritance (object-oriented programming) Computer file Content (media) Data storage device Directory service Flow separation Individualsoftware Derivation (linguistics) Bootstrap aggregating Hash function Software Software repository Mixed reality output Right angle Gastropod shell Physical system Local ring
Context awareness Scripting language Code Multiplication sign Parsing Mereology Semantics (computer science) Perspective (visual) Proper map Derivation (linguistics) Mathematics Different (Kate Ryan album) Hash function Encryption Error message Information security Physical system Graphics tablet Scripting language File format Computer file Moment (mathematics) Data storage device Sound effect Bit Mereology Type theory Hash function Ring (mathematics) Order (biology) Configuration space Right angle Pattern language Spacetime Point (geometry) Functional (mathematics) Server (computing) Implementation Link (knot theory) Computer file Virtual machine Control flow Content (media) Attribute grammar Power (physics) Revision control Goodness of fit Hacker (term) String (computer science) Gastropod shell Computer-assisted translation Installation art Key (cryptography) Validity (statistics) Chemical equation Projective plane Content (media) Line (geometry) Leak Uniform resource locator Error message Personal digital assistant Function (mathematics) String (computer science) Mixed reality Charge carrier Buffer overflow
Building System call State of matter Multiplication sign Source code Price index Client (computing) Parameter (computer programming) Mereology Medical imaging Derivation (linguistics) Mathematics Benchmark Very-high-bit-rate digital subscriber line Different (Kate Ryan album) Vertex (graph theory) Physical system Mapping Wrapper (data mining) Computer file Electronic mailing list Shared memory Maxima and minima Bit Benchmark User profile Type theory output Right angle Modul <Datentyp> Quicksort Information security Metric system Physical system Point (geometry) Functional (mathematics) Game controller Inheritance (object-oriented programming) Computer file Link (knot theory) Patch (Unix) Virtual machine Ultraviolet photoelectron spectroscopy Amsterdam Ordnance Datum Streaming media Differenz <Mathematik> Content (media) Virtual LAN Declarative programming Attribute grammar 2 (number) Number Revision control Software testing Configuration space Booting Mathematical optimization Module (mathematics) Key (cryptography) Tape drive Interface (computing) Counting Line (geometry) Device driver Group action System call Performance appraisal Kernel (computing) Loop (music) Software Integrated development environment Personal digital assistant Function (mathematics) String (computer science) Network socket Communications protocol Local ring
Demon Concurrency (computer science) Code Multiplication sign Time zone Set (mathematics) Insertion loss Open set Function (mathematics) Mereology Stack (abstract data type) IP address Formal language Front and back ends Web 2.0 Software framework Descriptive statistics Physical system Overlay-Netz Programming language Channel capacity Mapping Concurrency (computer science) Software developer Binary code Electronic mailing list Bit Maxima and minima Hecke operator Benchmark Connected space Type theory Befehlsprozessor Process (computing) Configuration space Right angle Problemorientierte Programmiersprache Modul <Datentyp> Point (geometry) Functional (mathematics) Computer file Link (knot theory) Virtual machine Electronic program guide Ultraviolet photoelectron spectroscopy Control flow Attribute grammar Revision control Differenz <Mathematik> Crash (computing) Profil (magazine) String (computer science) Touch typing Software testing Traffic reporting Metropolitan area network Plug-in (computing) Module (mathematics) Default (computer science) Multiplication Key (cryptography) Forcing (mathematics) Debugger Line (geometry) Calculus Performance appraisal Cache (computing) Bootstrap aggregating Software Integrated development environment Algebraic closure Personal digital assistant Mixed reality Blog
and he's also known for being the only one who has access on the github organization apart from me two years ago he decided to quit his job to become a next consultant and he created the e/m lembaga and then he was managing to nexuses releases in 2016 and by the end of 2016 he joined force with I wish everyone so first of all of course I lost my hair because of doing too much necks no I've started and lambda XC is it's exactly one year now and the only thing I would like to say I get about every two weeks an email for help for consulting so I do encourage everyone to start doing Nix consulting if that's what you feel you would like to try and do and you can come to me and talk and ask me any questions if you need somehow one of the things that Robin actually talked about and I would like to add a bit on top of that is this graph well first of all besides that it's it's kind of you know growing from this small group of people that you know they heck out something on there three times on weekends it's obviously becoming a distribution which people use every day right and you can see that of course this big fluctuations between releases are not just security issues but I think I would I would claim it's also people you know that have deadlines and they want to push things upstream but but of course they come at the very last minute and I think that's a sign of maturity although we have to rethink how to offload this less time released fixes like we talked about before at previous talks and and with that in mind I think one of the major things to reckon reconsider is how we do contributions and there is a very good blog post why github can house the Linux kernel community where it talks about when somebody maintains a subfolder for example in Ex packages let's say Haskell you would have a tree of folks that then you contribute for example to Peter assignments and and then he maintains the Haskell changes and then those get pushed as a batch to Nick's packages right and then instead of like putting everything into master you have a fork of master and then Haskell changes this is basically how kernel is developed and we have this master workflow right now in the pull request workflow and they're completely different and I think most of confusion comes from there so why does it matter if you're using Nick's as a company you like splitting up everything into a coherent full request is not that easy right you care about a subsystem and you do a lot of changes and then you you want to push this to upstream to a maintainer but we currently say hey you you know you're pushing a lot of things right now and we this will never get reviewed right but we already know for example for for Haskell that nobody upstream is one package right we for example commit everything at one side so there is a bit of contradiction there and I think this is really important if we want to scale in the future to decide what are we going to do so this talk is going to be mostly about the major pain points and some tips that I've gathered over the last few years part of the talk that Jana said was how do you how do you bring NICs to a company but I think one of the tiny bits missing and unfold is that probably more than anyone else is how do you first track someone to use necks right because when you have deadlines when you have you know production code out there you cannot just say hey you know what let's just go and you know to a nice Island and talk about Nix for two weeks and then we're going back to real world and you know everything's going to be fine right and so there needs to be some incremental changes how do you get there and I found actually this block flows to be very very nice for people and and one disclaimer I always work with people that have been previously exposed with functional programming so that doesn't that didn't need to be explained the concepts behind it but they did they still need to know how NICs language works but they're usually used to work with types and so on but besides going from you know the language itself how do you how do you contribute uniques packages and a whole ecosystem that there is still some pieces missing there and I mean interested what's your what's your do you have some suggestions of you do you have so your own approach then so if yeah I'm interested to talk about that if somebody has something to add so I think that's one of the things that I would like to work on when I have finally have some more free time and and and this is actually part of this gradual process of Nixa fiying right you go from these impure packaging slowly to to completely Nick safai your company or whatever project that you have right and this is a process and and you cannot say again yeah we're just gonna do one month and it's going to be done and then everything will be Nick's based so and this was the talk that Yanis that's what Tiana's was speaking about and I think that there are some things that we need to fix in order to get this process a bit nicer
so okay in enix itself we have a couple of functions how do you get a source but the main problem is although you might have the very same source that's you know can't the its content is is always the same you will you will get completely different hashes and this is this is quite painful once you move away from Nick's packages and you're using mixed packages right or and it's in any disguise it's just custom software living in any separate river rhine and so for example Hedra Nick's preface Gates has a different kind of behavior that Nick's packages okay we can merge the two and you know that's gonna get fixed but still you have fed good and fetch terrible and dot slash dot right and the main problem is that all of these have a different derivation name at it is part of the inputs right so because these are kind of nice names for fur fur you know a human then that means that you have to rebuild your whole package because this this is an input to your derivation and it will change the hash and particularly the last one is is the worst because who knows where does this name come from if you use a dot slash dot sorry yeah it's a parent directory so what does that mean if you build this package on hit right it's going to have a completely different parent directory then on your local machine or someone else will have a different parent directory they might call the dev they might call it you know temp or whatever right so it to some extent it breaks referential transparency because for example mix-ups default dot nix uses dot slash dot and if you build that on on Hedra you will never be able to fetch that binary because it will have a different name and and this is extremely extremely painful especially if you have big builds right so okay so how do we fix that right well we have to sacrifice probably the human readable name and just fix to skip you know stick to something but there is as far as I know and correct me if I'm wrong there is no primitive Linux which would allow you to switch the name but but to keep everything else and I I think that's probably we could add some kind of a rename derivation name but then you would have to for example wrap that using the dead slash dot dot slash dot and you would have a new primitive for accessing a current folder and I think that's without sacrificing backwards compatibility that's probably the only sane way and we have to talk about this because it is it is in practice really painful for example right now we use a fork of mix-ups but we always rebuild it because it doesn't get the package from from the binary cash okay we talked about pinning Knicks packages and you know as you as you as you create your your custom repo with all the fixes you know which custom dick size modules with custom packages and so on you start to put you start to put the the the pinning or but first you don't spin at all right and then sand it breaks okay then you start pinning and then you have these different places like in shells you have you pin and you painting Knicks and you pin in in shell scripts and you paint on Hedra and so on right so then you have this then you have to maintain the spins Knicks packages at six places right and then you're like okay now we have to clear this up right and you know and another problem that happens is even if you pin when you say Knicks packages these will always you can always leak the hosts if you don't set it correctly right and then you have a problem of using two different ones and you know we have different concepts how the Knicks packages is is the search pad is populated but we have found that the
only way to really make this reliable is basically to force the Knicks path to be empty and then use and dependence packages locally so so as docto it is was this was actually a joint effort from you know multiple people I don't take any credit for him or actually most of the things I talk about it's it's not my work it's just how do you get things done right and and and so this is this works right so this is a small snippet you copy paste to every repo because I mean it solves the problem of bootstrapping mix packages so you cannot pull from ups from somewhere and then bootstrap because that that's kind of what you want to solve right so what it does is it uses the built-in derivation and the the fetch URL and some local paths and actually the trick is to use the store pad so that you can actually make this available in sandboxes and and then you build this and you get mix packages at the relation time and you can just reuse that right and we store the mix packages source in JSON file and then we have a small command that just bombs that you say you know set revision to something and it will update that and it's very nice and and and the best thing about this is you can have multiple of those files right or you could generalize it that it would consume multiple JSON files so you could have multiple mix packages and use it you know for example if there is a bug you could have two different ones and one packages build with one expected use version and the other one is with the first one right and actually we do that for for for the Haskell software itself we use the different mixed packages and then for the machines we use a more stable version and that's mainly because of Darwin we try to use the stable mixes release for the servers but unstable so that we can get the darwin packages although that's about to change
okay so so how do you use that okay first you set the name cleaning spat of course it can be populated but the best way to make sure that none of the other stuff leaks in it's very good to it and to enforce that and then if you want to use it in mixed scripts you just say you know export mix pad and you can do it like that although preferably if you're just using makes you import it directly from the links file one one morning though if people are using for example Gentoo installed makes the the fetcher the the config that needs cats will have impure parts and this won't work and the workaround is of course to install mix with nix and then you get the the one that has the next balance so this this works pretty reliably and in one point well hopefully this is all gone because you can add the shah attribute of vegetable and i actually think that's probably we might just use fetch URL and add ability to unpack but yeah it's just an api thingy all right so there is there is a special type in well in Enix where you can actually have a URL without the string without the quotes right but a side effect has due to the error of see a you know column a is considered as an URL and it's converted into a string right so III think that we should we should probably this allows such things because they're not so useful and I think I don't think we're gonna break anyone's code maybe we're gonna make it work and this is really painful why is it painful right well because we don't have types right and then you get this kind of error when something inspected a function and well it's really a string right I mean it's not specifically this one it would be a probably a bit different right and and and this is this is really painful because it's not easy to debug right and and this one one-line whitespace changes are you know a waste of time right another example is is multi-line multi-line string so so who who has an idea what's wrong with this example yes they dentition is wrong so to put it in perspective our monitoring system didn't work because somebody did something like that and it was offset of course the positon past yama fall and say oh I don't see any contents and the monitoring system just didn't have anything configure right this is the effect of these things and you look up because we kind of rely on indentation in X and we assume it works this is actually one carrier that we have found it is extremely painful right and of course now if you want to fix that it's it's not easy because you would essentially break compatibility right if if you would change the semantics there would be a white space change and you will get completely different stop right so by using different nix versions you would have different hashes right so if we ought to fix this it's gotta be you know Nix 2.0 although talk to Python guys what happens when you when you change backwards compatibility right there is lessons to be learned but it's not impossible yeah that's and there's actually a pull request by byelarus as he was speaking before to fix that so alright so just to be clear I'm not I'm not trying to say Nick is bad we have done great things with ningke's I'm really happy I'm trying to point out what are the anti pattern cynics that you have to be careful about right so okay then we have two types one is string and one is path and one pulls in a folder to Mixter and one just reference it as a string right and and okay this is how you do it if you want to go from path to string you just to string function right oh and then you're like okay but let's go from string to path back right and there is a built-in called two paths as well but it doesn't do anything useful really you still get a string so okay so how do you actually do that well if you first assign something and concatenate with a path then you get nothing useful but actually the two part function is is the second example where you first take a path and then Kinkaid in the string and then you can actually you know do that alright it's painful but it's doable and even more strings are not as simple as strings there is actually strings and strings with context so what does that mean a context means that these strings refers to a derivation and if you if you use a string summer than it will the that derivation will depend on that derivation which is in the context of the string right so you can discard the string context or not and and for example next right doesn't make a difference between the two strings right so and if you use two string and unsafe discard string contains a different order you get different things and they're both string right so there is actually a stack overflow question I think that Niklas answered it probably needs a bit more work that it's going to be complete but I would really like to have a really good explanation of how this works so that it's clear for people I've just presented the problem I didn't go and actually try to make total sense
of it so okay we have some you know bad parts and inks that's fine javascript has those as well and it's widely used right so one of the one of the approaches how to fix that is is by Gabriel and I really like his work because he's trying to be really pragmatic of you know how do we fix things and he he did a Munich hack project to basically write a Nick's format which would then basically some kind of a static analyzer which would say ah you're doing crazy things like don't kind of concatenate multi-line strings because you know that's gonna end not end well right it's not it's not completely working because the powers of Enochs is is not very well defined or it has some special cases with anti quoting needs work and their needs there needs to be a special moment at implementer to handle that with the current puzzle so if somebody wants to work with that please let me know I would I would really love to to basically Bend these things in inks and just say don't use to pad and so on and of course build the tool that tells you some kind of like a shell check or or H lint or something like that right and then actually we could just you know get a habit of using this tool and we're pretty good right of course we can fix all of those things with just static analyzing but I think that most of those could be fine well then another thing is the the next strings are terminated by no so if you use random data and you put it in X strings you will get well you can see the iron but Oh basically says it's not the Velux it's not a valid strength right and that's because it conveys a no and you can represent secret so this is this is kind of painful okay if you if you put a secret interest ring and you import it into your store then it's insecure right but sometimes that's okay for example if you have server deployment where you know you you there is it's not just such a security threat or something like that so we added for example key file tunics ops which instead of taking a string that's then copy to a machine it takes a pad so it doesn't go to Nick strings at all and it's just copied over but in general if we ever want to support proper encrypted string lines if we want to represent them in in Nick's as a text and this will probably have to be solved
okay enough of ningke's let's go to new shops so one of the things that that I got bitten by is that actually who knows how long it takes to evaluate this it's about half a second right and one thing to be aware of that if you if you evaluate an eClass system it is one thing like all the modules are evaluated and if you have multiple machines there is no sharing between that you evaluate one machine one exercise machine and and another one and so on so it's basically completely linear so okay so for one of the clients basically I misused or abused or whatever you want to call that next to to run benchmarks right so we have this thing in Nix we can in a derivation we can say this derivation has a feature of KVM and that means it needs kayvyun to run and then Hedra is able to say oh this machine actually can run KVM or only this machine can only run KTN so I used that basically because we were doing snap high-speed networking protocol and and we would use and we would use and Hedra basically 2gc boot benchmarks one per machine and with specialized software in a control team isolated environment run those benchmark so i'm and what we wanted to know is that if we take different kind of kernel version different kind of QE MO and different kind of our software and so on build bigger build metrics of things and and see if there's any regressions and and we also had support to like patch those so you could like for example we had some patches so we could compare them patch software versus other software and this is just what snippet so basically it's you know it's a flat map some sort and then in the middle it selects benchmarks and the benchmark name is a list of benchmarks you want to run and then it runs and then it puts all the inputs that those differ from right and then it basically builds one and then just basically fat naps everything into one big attribute itself but you can also do a list so the problem here was that inside there I was building I was actually building a VM image that would then be used as a kind of a fixture right and the problem was if we actually built 20,000 packages like that and it ever evaluated Nick so as 20000 times right and that that took actually the evaluation took like 20 minutes so with this change it only took 10 seconds so because because Nick's doesn't do any sharing or optimization when you move out a creation of the next size of this fixture only to be in the the second loop right then it doesn't have to do so much work and there is actually a paper bye-bye kudos or maximum maximum laziness and and and it's basically you can you can make Nick to share things but then GC becomes a problem because it basically keeps track of of all the inputs and everything and it's not there and and it used to be part of links actually a couple of years ago so how do you debug that there is a there is a Knicks count calls which basically prints out how many times Nick called a preamp and how many times nee called something else but what you don't get is how long those calls took and you know evaluating in in my case the the Nick saw is evil one call too many times right so it was not useful so I couldn't find a better way and you know probably probably there is a better way and I'm interested to hear that but I I basically just did bisect so it pulled code out until I got to the point where I found out oh it's it's the it's the fixtures and then I figured out that what's the problem so there is some tooling probably to be developed around this or we can just identify what are the things that are really painful here one of the anti-patterns is that you you when you use Nick's OS modules you wrap them into you know other functions and this is terrible don't do this it's very tempted there's some faulty of this so guilty as charged but what what happens then is that when you actually specify inputs it's it's not a file by this thing but it's actually a function that you I mean an actual module that you pass and that messes up with ningke's OS because it doesn't know that it it's a few things it does are basically gone for example you might lose the line numbers and then your MMS is going to be anonymous function that unknown file failed right and then you have no idea what happened so no don't do that this is this is one of the worst things and and another trick is we have Nick saw as tests right and and then it suddenly fails and you want to know what's going on but there is no way to go into that machine and inspect it say it so I have this local PI from time to time and I know how it's really upstream it is that you tell the VDS which to basically expose the network in excess on tab zero interface and then if you if you just add you know you have to start SSH and add your some key to your Nix OS machines but then you can actually go and ssh into those Nix OS tests and you can you know I just create these big sleep thingies sleep for two hours and then is is a chin to the chest and and I can inspect what's going on there and and I think this is this is really useful because otherwise it's like Travis you know it runs there and you cannot access it and and why do that if we are able to run the DBMS locally right so another thing that probably needs to be up streams and done
properly how am i doing on time ten minutes oh okay that might do all right so NIC stops my major problem with Nick sobs it's it's really imperative right you you have to run steps you have to run set arguments you have to run this thing and other thing and so on so in our H K we created this small wrapper in Haskell around it and then make it a bit declarative it's not general purpose maybe one day it will be when we have time but the idea is that you you you can specify what arguments are passed and they're tagged by by Haskell type and you know for example what files you want the deployment to be so for example in Ex ups if you if you add a file to the deployment you have to call mix-ups modify - - to modify that state and so on and this does everything in the background so every time you run you know deploy it basically sets the arguments it modifies things so it's pretty much declarative and then what we also can do then is like we can say oh if you have for example Explorer in there we're going to do something a bit different right and this is actually from our source code
another very nice undocumented feature of mix-ups is that you have this default stinging and this is an excise module which is then included into all machines in your NICs webs mix-ups configuration so so basically I encourage everyone that in here actually you would have a list of all your custom modules and then they would be available to all of the machines right and you can set your your for example deployment of holes and so on otherwise you always have to add for each machine and usually you import these global mix OS module or something like that to share the global or common configuration but that's what defaults does and I've actually documented this and in this release it's going to be there in the NICS of documentation another nice feature that Nix UPS has it you can actually get a note attribute and this is actually the whole network that you define so you can go like notes back and to that config and then you know get anything and and for example if you wanted to create a big cluster that and get all the IPS in your network you this you can just basically map through all the attributes and notes and then get the config networking and public ID before and you know you cannot it Duty and you can access those and then filter for example and stuff like that that's another feature that's not too well-known actually encourage everyone that use Nick substitue to go through examples because they do include this but usually you look at one but for example there are gems hidden in like you know random files that are unrelated I mean they're they don't say we we we you know this file basically tries to explain defaults but it says so here's an example about ec2 right and so it's a good idea to go to all of them so we have been deploying many Nix ops machines and and the defaults once you go below ten are a bit not not right for example if if you if you copy everything from your deployment machine to the machine - to the deployment machines deployed machines and if you have 100 for example by default it will only copy four and then another four and another four and this will take a lot of time right so you have to set the max concurrent copy to like 20 if you said it too much then it's gonna you know hang everything right but even better is that there is now this has fast connection and there is a description of what does that mean so it basically switches between does it copy from the deploy machine to all machines that the the closures or does it download it from from the cache and then try to do the rest with just what the binary cash doesn't have then it tries to copy that so that there is then a smaller diff of things that need to be copy and then Hydra are so Hedra it thanks mostly to Shelli nowadays we have declarative configuration we have actually we can test github pull requests in a very happy way and there is a way to update the standards and and I think only the first one is documented and not even that is totally clear so that's something that if I would maybe like to work with I oh my eye I run my own Hydra for a couple of customers but I would really like to document this though it would be easier for everyone to do that right but again in a company you always you as I've said before you almost never have completely pure Nix built right there's always some part it's impure and parts that are pure and the problem is Hedra doesn't do impure stuff right it has this perl plugins system where you can write some of the impure stuff right but that's pretty much it so then you have to kind of integrate so basically you end up with two CI systems at the end right one one that you know what what people do is either use Jenkins for everything and they skip Hedra completely alright but I think it has really nice features or use hid run then you end up with another CI so so I died which K for example we used Travis from Hedra but then the problem is you make a pull request and Hedra start building binaries but Travis as well so it doesn't have the binary cash right and and then you have these two competing who is gonna be first and and stuff like that right so I've researched a bit and you know to be honest I'm really disappointed by Travis because they do all kinds of scary stuff that I don't think it's even considering morale because they claim that you get you know this man this much CPU but then their ability becomes really big and they just throttle that right so you're you're a bill that took previously 30 minutes takes an hour because they have just reduced your capacity for the CPU and then it times out so I call that unfair because I mean Europe I am you are paying for that for example right so don't do that right so ok we want to control the CI machines that's pretty clear by now and as Jana said we have found a build guide so just just to be clear we haven't implemented it yet but we have a pretty clear idea and what it basically does it runs a agent on your machine a kind of a Bashi agent so you can actually run this on your Hydra slaves so the idea is you would have hid rather than then first builds the package and then three girls the build guide to to run on the same machine the impure stuff so you don't have to copy stuff anymore from Hydra and it has also things like you can secure these environments and so on and these nice pipelines and so on so I think in the next couple of months we're gonna try this and replace Travis and then you know hopefully have time to write a blog blog post and report back and I think Yanis will we'll keep in touch how does this work but I have based on of what I've seen so far I have a very good hopes and there is Hercule CI that I've started which is spiked and written in Haskell and Fontan in Elm unfortunately I don't have time to work on it but I do have a budget for that so if somebody would be interested please let me know we would estimate there is about two months of to get something like what Hitler does and we will basically reuse the C++ demons that are there but mainly rewrite the web part so we can make the API a bit better right because one of the nice things that we have in inks that other CIS don't have is you could just say you know Hercules build me this local thing it would conduct a calculus and then that would build your thing and bring it back so it would be basically a bill just it's true Hydra right and then everyone else can already benefit from the binary cash and other things like that I have one minute right okay so Hesco we're doing mostly Hesco and there is a bunch of front-end JavaScript stuff I think has Co infrastructure is amazing I have no complaints thank you peter simons we added two things one is multiple output support let's that's that that is really nice for example if you depend on e key G package and Haskell it has static files and that then references the whole eqg package and then you have everything up to ghz in there for example if you aesthetically now that out multiple outputs are there the static files are a reference separately and you don't bring the GHC in the whole closure anymore anyway we have built stack tunics they're actually stack Aegina cycads tunics and stack tunics they're two different approaches to this we'll see which one is better but it follows this philosophy let's use developer stack which is the development tool for Haskell but let's generate package set which use exactly the same version as developers did and then deploy that and that's what we actually use and it works pretty well there there are a few things to be fixed like mac OS support and so on but I think it's it's it's a shown to be pretty good and the last thing this is actually what I would probably like to work on hackathon tomorrow is take all of this knowledge and create like a bootstrap II example of how do you basically run your company and how do you start from and you know use these Phenix packages overlays and these tips and tricks and then you could just start using links OS and nicks and nicks ups as well and of course you start with she'll like Genesis but then the next thing is probably bringing this things in and I think there is a couple a couple of things we have learned so far so if we join forces and we bring this into one maintained thing of course we will it's always opinionated but we think we can work with that it's going to be really useful for everyone all right so that's it I am two questions one questions their question is I've already been bitten by the fact that two paths does not create a path but a string is there any reason why not to fix this just to heck two paths in the sauce - um well that's probably more a question for a locker than me but I I don't I I think that two paths was added at one point and nobody's really using it right so I I don't think there is too much higher or too much I don't think there's a problem actually fixing it right but if somebody is using it it could of course break backwards-compatibility in that case I mean probably a better way would be just to deprecate two paths and create something similar that would then see people would start to use pinning yeah I think Gabrielle Gonzales and then you know their team actually created a poor request and they plan to move it to wiki but then it also depends if do we need that if we're gonna get Nick's 1.12 you know there's yeah so but yeah there is a pull request open for that already but the main problem is you have to always copy paste this through to every place right because it's a bootstrap process not really question I would just wanted to make a remark it seems like that in the beginning you always think ah we're just building like a simple language it runs only at like evaluation time so like features and there aren't really that important and then the couple years later you find that like any simple language eventually turns into like a fully featured programming language then you have one guy who suddenly really need something like a profiler right or maybe debugger on these kind of things and I think that is an an interesting insight and another thing is that from the same idea you say like okay in the beginning because it just runs an evaluation time before deployment machines or evaluate my OS and so on you don't really need like types and if it's crashes this is not so bad because the crashes before something bad happens right but then you end up with like a million lines of code in mixed packages and then you have multi hour time loss when you get exactly what you said with the anonymous function in the unknown file right so I think those are interesting insights yeah I I think that well first of all Elco always says you know Nix is not the general-purpose programming language right so the fact that I we created this whole framework of doing benchmark is probably you know our fault but but yeah I think that we can fix most of this bumps and and and and make this language actually good enough and live with it with the fact but but this whitespace thing is and so on don't make it too nice when when you you know and yeah but in general I agree the more you use it the more tools you need around it right and I think that at the end we will end up with with all of that at some point [Applause]