We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Nix roadmap

00:00

Formal Metadata

Title
Nix roadmap
Title of Series
Number of Parts
27
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Our BDFL will be talking about the Nix roadmap --- Bio: Eelco is a senior software engineer at Tweag I/O. He obtained a PhD in Computer Science from Utrecht University in 2006 and was a postdoctoral researcher at Utrecht University and Delft University. As part of his PhD research project at Utrecht University, he developed Nix, the purely functional package manager, which forms the basis of the NixOS Linux distribution. He previously worked at LogicBlox and Infor.
Statement (computer science)Chemical equationProduct (business)Multiplication signHand fanQuicksortMetropolitan area networkCore dumpPlanningProcess (computing)CompilerProjective planeoutputCycle (graph theory)Intrusion detection systemDirection (geometry)Software developerComputer programmingPoint (geometry)Self-organizationRight angleLevel (video gaming)Mixed realityLeakComputer animation
Computer configurationPerformance appraisalPasswordStatement (computer science)Algebraic closureData storage deviceKey (cryptography)AbstractionAlgebraic closureLevel (video gaming)PasswordQuicksortPerformance appraisalMultiplication signComputer configurationFunctional (mathematics)Cache (computing)Configuration spaceSource codeMoment (mathematics)Functional programmingOcean currentFormal languageComputer filePerfect groupStatement (computer science)System administratorUsabilityParameter (computer programming)Lecture/ConferenceMeeting/Interview
Computer configurationPerformance appraisalPasswordStatement (computer science)Algebraic closureAddress spaceContent (media)Internet service providerBinary fileDerivation (linguistics)Radio-frequency identificationHash functionGraph (mathematics)Independence (probability theory)MathematicsData storage deviceElectronic signatureTrojanisches Pferd <Informatik>Cache (computing)ExpressionLie groupDerivation (linguistics)Hash functionBinary codeContent (media)Projective planeFunctional (mathematics)Core dumpRandomizationExtension (kinesiology)Internet service providerLevel (video gaming)Source codeMultiplication signStatement (computer science)Computer fileAddress spaceRecursionTimestampRevision controlCryptographyCategory of beingPhysical systemDemonAlgebraic closureMechanism designPerformance appraisalConfiguration spaceFunction (mathematics)outputRight angleComputer animation
Content (media)Binary fileDerivation (linguistics)Radio-frequency identificationHash functionIndependence (probability theory)Graph (mathematics)MathematicsAlgebraic closureRun time (program lifecycle phase)Attribute grammarPattern languageFunction (mathematics)Function (mathematics)Algebraic closureCompilerExpressionRun time (program lifecycle phase)Error messageAttribute grammarUniform resource locatorConfiguration spaceCASE <Informatik>Data miningBinary codeSpacetimeCache (computing)Software developerINTEGRALProof theoryContent (media)Multiplication signRule of inferenceRandomizationData storage devicePattern matchingPerfect groupAddress spaceVapor barrierQuicksortHash functionInstance (computer science)MathematicsGraph (mathematics)Installation artProjective planeSource codeGaussian eliminationPhysical systemBuildingTimestampResultantMeeting/Interview
Performance appraisalRead-only memoryComputer configurationConfiguration spaceMechanism designPlug-in (computing)Physical systemModule (mathematics)Attribute grammarType theoryFunction (mathematics)outputProblemorientierte ProgrammierspracheGraph (mathematics)Parameter (computer programming)Modulare ProgrammierungFunctional (mathematics)Run time (program lifecycle phase)Performance appraisalSemiconductor memoryDerivation (linguistics)SpeicherbereinigungConfiguration spaceQuicksortComputer configurationMereologyAttribute grammarFormal languageProcess (computing)Physical systemGraph (mathematics)Revision controlBuildingMeta elementPlug-in (computing)Descriptive statisticsSet (mathematics)Type theoryComputer programmingGoodness of fitComputer animation
Module (mathematics)Field (computer science)Computer configurationFirewall (computing)Type theoryDerivation (linguistics)Revision controlInstallation artPhase transitionGeneric programmingInheritance (object-oriented programming)Home pageConfiguration spaceScripting languageCodeGastropod shellError messageGraphical user interfaceQuery languageModulare ProgrammierungSource codeHome pageDescriptive statisticsIntegrated development environmentStandard deviationQuicksortDefault (computer science)AbstractionNatural numberModule (mathematics)Derivation (linguistics)Type theoryLevel (video gaming)Keyboard shortcutMultiplication signRevision controlConfiguration spaceRight angleAttribute grammarGoodness of fitMechanism designFunction (mathematics)Functional (mathematics)Phase transitionComputer configurationMathematicsParameter (computer programming)Greatest elementScripting languageGeneric programmingoutputBitPhysical systemArc (geometry)Variable (mathematics)Field (computer science)Poisson-KlammerFormal languageBuildingRecursionInstallation artComputer animation
Goodness of fitPhysical systemFunction (mathematics)Attribute grammarAlgebraic closureoutputSoftware testingDefault (computer science)Extension (kinesiology)Overlay-NetzType theoryConfiguration spaceRevision controlModulare ProgrammierungIntrusion detection systemComputer configurationQuicksortPerformance appraisalSemiconductor memoryPoint (geometry)Level (video gaming)Category of beingSimilarity (geometry)Slide ruleMoment (mathematics)BitSubject indexingRight angleDerivation (linguistics)BuildingWechselseitige InformationDecision tree learningDifferent (Kate Ryan album)Key (cryptography)Service (economics)Lecture/Conference
Transcript: English(auto-generated)
Okay, all right so first up is the is the man who doesn't need an introduction, but gets one anyway So first up we've got we've got echo And for those who maybe don't know yet because I did see a couple of fans of people attending an excon for the first time
Echo is the person that we have to thank for for for the initiation of nix basically because he bricked on that during his PhD research and Today is however not the time to look back, but today echo is going to tell us about what lies ahead For nix the roadmap of the future of nix basically right, so please give him a hand echo. Thank you
So I guess does this work? Can you hear me okay great? Okay, so first of all thanks to the organizers and apart from that. Let me say that
talk about External monitors being hard to configure on a nix OS is fake news. I just plugged it in and it worked its magic Yeah, so this this talk. It was originally called a nix roadmap, but There is no nix roadmap. It's towards a nix roadmap
Because this should really be a community effort, so this is sort of a starting point towards a roadmap So people have been saying for years that we should have a roadmap So lately I've been doing a lot of rust programming. I've been really drinking the rust kool-aid so
Whenever I have a problem now I ask myself what would rust to do so it turns out to have rust a they have an RFC for everything They have a beautiful process for everything So yeah, so why should you have a roadmap to begin with well, so they answer that so it's
So the main thing is that it allows the world to see what are sort of the long-term plans and the strategic Priorities, and it allows all the developers to hopefully get behind that and so everybody is hopefully
So hopefully then everybody is kind of pointing in the same direction And another thing they mentioned is that So they have a rapid release cycle But it it turns out that that kind of meant that sort of bigger
features as sort of long-term Projects were sort of falling by the waysides because they don't really fit into that rapid release cycle, so Establishing annual goals so in 2018 where we're going to make the compiler fast That's a that's a way to really get people behind it and
Ensure that they Spend time on that so that at the end of the year. They don't have to say We fail to reach our goals So the process that they have and I'm not necessarily saying that we should follow that but so they have an annual
roadmap and so they have a process for for creating that roadmap, so they have a They write a an RFC where they get our problems that the community has and then from that they extract a bunch of goals, so
That sounds like a reasonable thing to do and and and then they have a whole Plan for the year so in say February they start planning how to reach those goals And then they start implementing them, so I'm not saying that that necessarily makes a lot of sense for us, but At least yeah having a roadmap
For say 2019 saying these are the the goals that we want for Nix That sounds like a very valuable thing to have So in the rest of the talk I'm just going to do a brain dump of some things that I think are problems with Nix and
And and and from that it follows that there are some goals that that we should implement but this is just sort of my IDs and so so I would like to Do kind of this RFC process and get everybody's inputs and from that hopefully?
We can Yeah, get a roadmap for 2019 Okay All Right so some problem statements, so So these are just some things that are currently problematic with Nix so for example
Or things that I would like to do that I can't at the moment so one is I would like to use Nix as a make replacement or a basal replacement so And Nix is all these nice features a purely functional language Reproducible builds isolation and we do this for sort of large things like packages and for very small things like
configuration files in Nix OS So it seems like it's should be a perfect build tool as well so for building things like C source files or whatever other language you want to build, but there are a bunch of reasons that Currently you can't really do that at the moment, so I'll come back to that
So that's one problem another Nix package options are not easily discoverable or configurable so right now Nix packages have all sorts of options so for example a Package function might have an enable foo
Arguments, but this is completely undiscoverable except by reading the Nix packages source code and And it's also not configurable via Nix and for any other tool, so this is not very good UX
Another problem is that This is and this is an increasingly big problem is that Nix packages and Nix OS evaluation is slow, and it's getting slower all the time because the Sort of the level of abstractions that are used in Nix packages are increasing And it uses a lot of RAM so
And actually it turns out that this problem is kind of related to the previous problem Another one that Comes up a lot is that Nix currently has no way to handle secrets so things like passwords or keys You don't want to store those in the next store because then they're world readable which is bad
So you need to some way to to deal with them Or so another problem as an unprivileged so right now if you want to pull something from a binary cache from an arbitrary binary cache, so you cannot do that as an unprivileged user how things need to be signed by a
Key configured by the administrator So if you just want to pull some something from some arbitrary cache you cannot do that as an arbitrary user Another one so Closure bloat, so this is a fairly big issue so in Nix
It's very easy to end up with a package that has way more runtime dependencies Than it than it actually needs So I'll also give an example of that later So yeah, these are just some random problems, so
Probably many of you have other problems, so we're interested to hear them But so here are some goals Skip something yeah, so here are some goals that that you can extract from those problem statements So these are more at a technical level so by the way if you look at the rust goals. There are also a lot of
non-technical Goals there like improve documentation Eventualize in certain communities make the community more diverse, so that's all great as well
Yeah, but but here. I'm more focusing on on technical stuff, but We should definitely not restrict ourselves to that in the roadmap Yeah, so make Nix a compelling build tool so compelling replacement for Nix or basil for make or basil
Or something that can actually Complement those tools Make the next or content addressable, so that's a very technical goal, but it's it's kind of related to all the others Make Nix packages discoverable and configurable Improve the evaluation efficiency
Provide mechanisms to prevent closure bloat prevent provide a way to store secrets in the next door So yeah, these are just some Goals that I would like to Work on in the next year And to some extent I have been working on them
So the rest of this talk is just Some random brain dump on on how these goals could be achieved So Yeah, so first the gold Nix as a build tool, so So what do we need to get to that?
So in a way you can already do this in fact you could do that ten years ago In fact there is a Nix make repository somewhere, which has a bunch of functions for building C or C plus plus projects and and that works fine, but the problem is
So now you have your project and you're using these Nix make functions So you can run Nix builds to Have incremental builds for your project, and that's all great So now you want to package this thing and put it in next packages So you make a tar ball containing your source code and your next expression
And now you want to write a next expression in next packages that Extracts this tar ball and builds it and There you run into the problem that you need to be able to call Nix from inside a Nix build Because you're using a Nix expression to to build your project so instead of a make file so previously you would call make
right But now you have a Nix expression that builds your project so you need to be able to call Nix build but you're inside a Nix build already and So so and and and Nix derivation doesn't actually have arbitrary right access to the next door in fact
it only has right access to its outputs, so So this doesn't work, so now you have a very embarrassing situation so you have a Package that's written that has a build system written in Nix, but you can't actually put it in the next packages You could put it in Debian probably, but you can't put it in next packages
So so this is not good So you need need recursive Nix So that's kind of a required feature Another Not essential, but very nice to have feature is content address ability
which I'll come back to and caching of copying files to the store so a tool like So if you have your project which might consist of thousands of source files So now every time you run Nix build it has to read all those source files and and and copy them to the next store
Or at least check whether they already are in the next store, so that's a lot of IO, and that's slow so a tool like make prevent avoids that by Only checking timestamps and even that can get slow for very large projects
But yeah, Nix needs to hash all these files, so it actually needs to read all of them So you you want to have some kind of caching for that and maybe in something like an I notify demon to efficiently notice when files change But this is in the nice to have category
Right so the content address ability this is has kind of been a Could say a holy grail for many years So this is the property that? Should step back for a second so if you remember So a Nick store path that contains a cryptographic hash
But that cryptographic hash is a hash of the derivation that built that path It's not actually a hash of the content of that path And and and this is why you need signatures on binary caches Because you need to trust that Some store path was actually produced by the derivation that it claims to be built by
And that could be a lie so somebody could set up a binary cache Where and so you you get a legitimate Nick's expression So that's for example builds Firefox And then you pull a binder from the cache that actually contains something completely different like a trojan version of Firefox
so That's why you need signatures So in a content reversible store the store path So the hash in the store path is actually a hash of the contents of that path
So you no longer need to trust anything so you can just verify that for example in the path like this Nick stores hash you just check that the cryptographic hash of the contents of this path Is this so a path basically contains its own proof of integrity
So if you have this then yeah unprivileged users can install things from arbitrary binary caches Another very big advantage is that you get deduplication so for example if you
So So you make an irrelevant change to something in the dependency graph like you You make a white space change to glib C. So currently that causes the entire system to be rebuilt Which is bad and and actually yeah, not just rebuilt but
Had duplicated in the Nick store, so you need twice the storage space now Whereas with a content addressable store So because this change is irrelevant. It doesn't actually change anything to the output of a build It ends up being stored in the same location So that's that's much nicer
And in fact it it does prevent rebuilds because if for instance So you make that change to glib C You still need to recompile glib C to discover that that change doesn't matter But after that you don't need to rebuild anything that depends on it Because you've already discovered that
Yeah, this glib C. Is actually the same so so so it sort of acts as a barrier in the in the dependency graph Yeah, so this is why content addressability would be a great feature to have So a few interesting things about content addressability it to make this work properly it really needs
Perfect binary reproducibility, so that's currently not the case with next packages So if you build a package twice you might actually end up with Slightly different results so for example if a binary stores a timestamp somewhere
But yeah a lot of people are at work To improve that so for example. There's a whole reproducible builds dot org project That's that's yeah basically improving all sorts of packages and build systems to
to eliminate sources of Binary impurity okay, so for the other thing so yeah prevent Sorry Preventing closure bloat, so this is something of a obsession of mine. Sorry about that
So in X because of the way it detects it finds dependencies It's very easy to have an accidental dependency So this is not the case in say had Debian where you specify Dependency so you don't end up with an accidental runtime dependency on say the C compiler
So here for example. There was a situation where Thunderbird was storing its build Configuration so you can do about config in the URL bar and it will show you the path to the C compiler used to compile it
Which is of course a kind of a useless thing But it does yeah adds 1,200 megabytes of blow to the closure and Yeah, so this can happen very Accidentally you don't get any errors if you do that So yeah, we need better tools to to yeah
Detect when this happens, so we already have some attributes that you can specify in in next expressions for example You can say disallowed requisites To say that something should not have a runtime reference to say the C compiler But this is very limited for example you cannot do any pattern matching and you would like to say
This thing should not have any references to developer outputs And it should be per output because for example your death output Probably should be allowed to have references to other death outputs
And you might want to have a size check so for example if say the Nixos ISO suddenly Gets a gigabyte bigger than who would like to get some error so What I recently implemented partially so not all of this works yet is that you can specify
Per output check so for example you can say The outputs the out output should not have a closure bigger than 256 megabytes It should not be should not reference the C compiler or any death output But the the death output itself can reference anything
But it should not be larger than 128 kilobytes As a random example So Yeah, so we can start putting these things in next packages It could even be a generic thing so for example the rule that things should not allow
Should not be allowed to reference Death outputs is something you could actually put in standard ends so as a general policy So yeah, that would be very nice How am I doing on time actually I can just check okay, ah great
So yeah now I come to the really wild and vaporware part of the talk So So yeah really a big issue is discoverability and efficiency like I mentioned so
Next packages have basically no discoverability. Well. I mean you can discover that they exist sometimes I mean Nixon doesn't necessarily recurse into everything, but So you can see the packages exist, but you can see what options they have and Customizing Packages is also very ad hoc it sort of
Evolved it's not really a properly designed thing so if these things like dot overwrite dot overwrite derivation config And And in fact this this whole dot overwrite thing is kind of disastrous for for performance, so
It's really one of them. It's one of the two main reasons. Why Nick's evaluation takes a lot of memory So so dot overwrite Basically destroys the ability of the garbage collector at runtime to actually collect any garbage
Because you call a function so you pass it some arguments which can be very big because there are arbitrary dependencies large graph And And then the output just contains the inputs so the inputs can never be garbage collected, so This is this was you know kind of a bad idea, but we don't really have anything better
So There's kind of a meta issue here of all these things So somewhere along the way we forgot that Nixon's intended as a domain specific language for specifying
build graphs and configurations like like Nick's OS systems But as a DSL it's not really doing a great job so for example it has no concept of a package or an option or a configuration or things like plugins so any sort of features you or concepts you might
expect in a DSL intended for doing these things so So maybe we need to get back to Well we need to improve Nick's as a DSL so so one thing that I've been thinking about is
So essentially Turning the Nixon's module system or an improved version of it into a language feature Into something called a configuration which you can really think of as an an attribute set an extensible attribute set Which is really what a Nick's OS configuration is it's a it's a bunch of edge attributes that you can change and
So if you change one attribute it can trigger other attributes to change Yeah, so So a configuration is attribute set which contains attributes called options that can be set
And they can be overwritten later But unlike attribute sets and like Nick's OS options they can have types and Documentations and merge functions, and and that's the thing that gives you discoverability so things like package options can be
expressed in this way and because they have things like a description and a type tools could Can discover them and present them to the user and then allow them to be changed programmatic programmatically So so the
sketchy design for this language feature is a bit like this so a Configuration looks a lot like an attribute set only it uses angled brackets My change don't get too angry or or enthusiastic about the syntax But yeah, so it's you could think of it as an attribute set
So we have an attribute foo and attribute bar and an attribute ABC that actually refers to foo and bar So it's a recursive. It's like a rec attribute set So if from this thing you select the ABC Attributes you would get a value one two three because bar is true, so if if true then one two three
Right, but what you can now do is take that configuration module and apply an a new module to it that sets bar to false and So now if from this module you select ABC, then it will return
a false then It goes to One two three times two, so it will return Two four six so this is pretty much exactly the behavior of the nixos module system
Yeah, and then the idea is that you can have some sort of syntax Which I'm not sure about but to to attach fields or annotations to those to those options like documentation a type default value and so on Merge functions
priorities Yeah, all that sort of thing basically all the things you have in the nixos module system Right, but now the idea is that we can apply this to Building packages so rather than having packages as functions
Which have the problem that? Well, there's no override mechanism, I mean no good override mechanism There's no documentation and so on We can basically treat packages in the same way as the nixos module system treats
system configurations, so you you build a package in sort of a modular way by combining a bunch of modules so for example you could have a very Sort of at the bottom you have a module that Captures the concept of a derivation, so what is a derivation?
Well a derivation nix has a name and a version, and it has a builder and it has arguments, and it has an environment And if you set those things then you can evaluate a derivation attribute Which produces a low-level derivation?
So so this is very low level Now you can build higher level modules on top of that so for example this thing basically expresses the basic standard environment so the concept of phases And dependencies on our packages so this thing adds a an option called built inputs and an option called phases
and it's it implements this on top of the Lower level derivation module by Setting builder and arcs and nth and that's causes
a derivation to be computed Yeah that uses these things So just to continue this a bit so you could have a module that Captures the concept of a package so package has a description and the home page and so on And all these things have had descriptions, and they have types, so they're they're discoverable
And and you get error messages so for example if if you use this previous So currently with derivations if for example you you misspell build inputs You're not going to get an error Because a nix derivation is basically just a bunch of environments variable bindings, so
Yeah, so there's no checking whatsoever there, but here if you set an option that Hasn't been declared then you get an error message. Just like in the in the module system
Yeah, so you can build higher and higher level abstractions on this so for example you can extend the Sort of generic standard environment with the concept of a Unix package which for example has a configure phase Because it runs a configure script Yeah And then finally you can define a package so a package is something that extends the Unix package module with
something that sets a name description source but it also Has its own option namely enable GUI So in this fictional example hello world has a gtk support
So now you can say build inputs is if enable gt GUI then use gtk And and this thing is now discoverable so You could have say a nix query package command which will show that this thing has an enable GUI Option and they could have a nix install command
That that allows you to set that option So Yeah, and you could override things using the exact same module system so that just like in nix OS So that's about it so there
Lots of other things you could imagine for the roadmap. I'll skip that So yeah, so what I should do is create a sort of a roadmap issue and where everybody can go wild with IDs and suggestions, and then we should try to synthesize something workable from that
So yeah That's it. Thank you
So for the configurations idea use it doing overrides. How does that solve the memory issue?
Don't you still need to hold on to the references to all the inputs no because as soon as you Evaluate the dot DRV attribute You don't need anything else anymore you can that after that you can just discard
Everything that went into it, so I sort of like in nix OS if you evaluate system dot build top level you get a derivation out of that and And at that point you can garbage collects all the inputs to that thing Oh, so like the when when you're passing something as an input to something else you're passing to the dot DRV
Not the composable, right? So Presumably this is all fiction so so here this GTK thing would trigger an evaluation of GTK dot DRV Implicitly on the slide about the nix as a build tool you have been talking about avoiding rebuilds and
similar properties Have you looked in to the recent paper? Built systems a la carta by Neil Mitchell Simon Pena Johnson and the remark off
They're analyzing various build systems there and nix is one of them and nix ticks of almost all features of the ultimate build system Was that the ice Fp paper? Yes, okay? Yeah, it's on my to-do list, okay? Please take a look at this it gives very good names good glossary to talk about those properties. It's a great paper. Thank you
one more so the configuration Options it excites the type system in me
I won't go there Is there also? So the nix os module system also supports the notion of overlays Is this should is there a version of this also in this or is this then? extension yeah, probably so That's sort of a high-level thing that I haven't figured out yet, so how you actually put these things together
so I but you need some way to do that and Yeah, so I don't know yet, okay? Yeah, I think we've got a move maybe one more
last one It's more of a policy question. I've seen a lot of commits like fixing Thunderbirds like you showed and There has been commits for every separate packages Why don't you enforce this kind of like do not reference GCC for all the package by default and then?
If some package really needs GCC then you could enable it right well That's kind of the point so we don't have a way really to enforce that yet So right now. It's really only if people sort of notice That's only a closure has become much bigger
So you you can use these disallowed Requisites attributes, but very few packages do that But you could use that by default like that to empty the revision and then by default just do not reference GCC There is no reason for most packages to do so right and so probably for GC that would work, but
You really want to do say things like it should not reference any death output And that doesn't work at the moment because death outputs should be allowed to reference death output So you can't use the existing attributes for that and But yeah, so I would definitely like to have in Nixos for example that we say that all the Nixos
VM tests could just check that their closures don't have any Death outputs in them and and no see GCC or clang So that that would already help quite a bit, okay, all righty, so thank you again
For your talk