Nix roadmap
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 27 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39608 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NixCon20187 / 27
3
5
6
7
10
13
16
19
20
22
24
25
00:00
Statement (computer science)Chemical equationProduct (business)Multiplication signHand fanQuicksortMetropolitan area networkCore dumpPlanningProcess (computing)CompilerProjective planeoutputCycle (graph theory)Intrusion detection systemDirection (geometry)Software developerComputer programmingPoint (geometry)Self-organizationRight angleLevel (video gaming)Mixed realityLeakComputer animation
04:52
Computer configurationPerformance appraisalPasswordStatement (computer science)Algebraic closureData storage deviceKey (cryptography)AbstractionAlgebraic closureLevel (video gaming)PasswordQuicksortPerformance appraisalMultiplication signComputer configurationFunctional (mathematics)Cache (computing)Configuration spaceSource codeMoment (mathematics)Functional programmingOcean currentFormal languageComputer filePerfect groupStatement (computer science)System administratorUsabilityParameter (computer programming)Lecture/ConferenceMeeting/Interview
08:00
Computer configurationPerformance appraisalPasswordStatement (computer science)Algebraic closureAddress spaceContent (media)Internet service providerBinary fileDerivation (linguistics)Radio-frequency identificationHash functionGraph (mathematics)Independence (probability theory)MathematicsData storage deviceElectronic signatureTrojanisches Pferd <Informatik>Cache (computing)ExpressionLie groupDerivation (linguistics)Hash functionBinary codeContent (media)Projective planeFunctional (mathematics)Core dumpRandomizationExtension (kinesiology)Internet service providerLevel (video gaming)Source codeMultiplication signStatement (computer science)Computer fileAddress spaceRecursionTimestampRevision controlCryptographyCategory of beingPhysical systemDemonAlgebraic closureMechanism designPerformance appraisalConfiguration spaceFunction (mathematics)outputRight angleComputer animation
14:14
Content (media)Binary fileDerivation (linguistics)Radio-frequency identificationHash functionIndependence (probability theory)Graph (mathematics)MathematicsAlgebraic closureRun time (program lifecycle phase)Attribute grammarPattern languageFunction (mathematics)Function (mathematics)Algebraic closureCompilerExpressionRun time (program lifecycle phase)Error messageAttribute grammarUniform resource locatorConfiguration spaceCASE <Informatik>Data miningBinary codeSpacetimeCache (computing)Software developerINTEGRALProof theoryContent (media)Multiplication signRule of inferenceRandomizationData storage devicePattern matchingPerfect groupAddress spaceVapor barrierQuicksortHash functionInstance (computer science)MathematicsGraph (mathematics)Installation artProjective planeSource codeGaussian eliminationPhysical systemBuildingTimestampResultantMeeting/Interview
20:20
Performance appraisalRead-only memoryComputer configurationConfiguration spaceMechanism designPlug-in (computing)Physical systemModule (mathematics)Attribute grammarType theoryFunction (mathematics)outputProblemorientierte ProgrammierspracheGraph (mathematics)Parameter (computer programming)Modulare ProgrammierungFunctional (mathematics)Run time (program lifecycle phase)Performance appraisalSemiconductor memoryDerivation (linguistics)SpeicherbereinigungConfiguration spaceQuicksortComputer configurationMereologyAttribute grammarFormal languageProcess (computing)Physical systemGraph (mathematics)Revision controlBuildingMeta elementPlug-in (computing)Descriptive statisticsSet (mathematics)Type theoryComputer programmingGoodness of fitComputer animation
24:16
Module (mathematics)Field (computer science)Computer configurationFirewall (computing)Type theoryDerivation (linguistics)Revision controlInstallation artPhase transitionGeneric programmingInheritance (object-oriented programming)Home pageConfiguration spaceScripting languageCodeGastropod shellError messageGraphical user interfaceQuery languageModulare ProgrammierungSource codeHome pageDescriptive statisticsIntegrated development environmentStandard deviationQuicksortDefault (computer science)AbstractionNatural numberModule (mathematics)Derivation (linguistics)Type theoryLevel (video gaming)Keyboard shortcutMultiplication signRevision controlConfiguration spaceRight angleAttribute grammarGoodness of fitMechanism designFunction (mathematics)Functional (mathematics)Phase transitionComputer configurationMathematicsParameter (computer programming)Greatest elementScripting languageGeneric programmingoutputBitPhysical systemArc (geometry)Variable (mathematics)Field (computer science)Poisson-KlammerFormal languageBuildingRecursionInstallation artComputer animation
30:03
Goodness of fitPhysical systemFunction (mathematics)Attribute grammarAlgebraic closureoutputSoftware testingDefault (computer science)Extension (kinesiology)Overlay-NetzType theoryConfiguration spaceRevision controlModulare ProgrammierungIntrusion detection systemComputer configurationQuicksortPerformance appraisalSemiconductor memoryPoint (geometry)Level (video gaming)Category of beingSimilarity (geometry)Slide ruleMoment (mathematics)BitSubject indexingRight angleDerivation (linguistics)BuildingWechselseitige InformationDecision tree learningDifferent (Kate Ryan album)Key (cryptography)Service (economics)Lecture/Conference
Transcript: English(auto-generated)
00:03
Okay, all right so first up is the is the man who doesn't need an introduction, but gets one anyway So first up we've got we've got echo And for those who maybe don't know yet because I did see a couple of fans of people attending an excon for the first time
00:22
Echo is the person that we have to thank for for for the initiation of nix basically because he bricked on that during his PhD research and Today is however not the time to look back, but today echo is going to tell us about what lies ahead For nix the roadmap of the future of nix basically right, so please give him a hand echo. Thank you
00:51
So I guess does this work? Can you hear me okay great? Okay, so first of all thanks to the organizers and apart from that. Let me say that
01:04
talk about External monitors being hard to configure on a nix OS is fake news. I just plugged it in and it worked its magic Yeah, so this this talk. It was originally called a nix roadmap, but There is no nix roadmap. It's towards a nix roadmap
01:25
Because this should really be a community effort, so this is sort of a starting point towards a roadmap So people have been saying for years that we should have a roadmap So lately I've been doing a lot of rust programming. I've been really drinking the rust kool-aid so
01:44
Whenever I have a problem now I ask myself what would rust to do so it turns out to have rust a they have an RFC for everything They have a beautiful process for everything So yeah, so why should you have a roadmap to begin with well, so they answer that so it's
02:01
So the main thing is that it allows the world to see what are sort of the long-term plans and the strategic Priorities, and it allows all the developers to hopefully get behind that and so everybody is hopefully
02:21
So hopefully then everybody is kind of pointing in the same direction And another thing they mentioned is that So they have a rapid release cycle But it it turns out that that kind of meant that sort of bigger
02:40
features as sort of long-term Projects were sort of falling by the waysides because they don't really fit into that rapid release cycle, so Establishing annual goals so in 2018 where we're going to make the compiler fast That's a that's a way to really get people behind it and
03:04
Ensure that they Spend time on that so that at the end of the year. They don't have to say We fail to reach our goals So the process that they have and I'm not necessarily saying that we should follow that but so they have an annual
03:21
roadmap and so they have a process for for creating that roadmap, so they have a They write a an RFC where they get our problems that the community has and then from that they extract a bunch of goals, so
03:40
That sounds like a reasonable thing to do and and and then they have a whole Plan for the year so in say February they start planning how to reach those goals And then they start implementing them, so I'm not saying that that necessarily makes a lot of sense for us, but At least yeah having a roadmap
04:03
For say 2019 saying these are the the goals that we want for Nix That sounds like a very valuable thing to have So in the rest of the talk I'm just going to do a brain dump of some things that I think are problems with Nix and
04:23
And and and from that it follows that there are some goals that that we should implement but this is just sort of my IDs and so so I would like to Do kind of this RFC process and get everybody's inputs and from that hopefully?
04:42
We can Yeah, get a roadmap for 2019 Okay All Right so some problem statements, so So these are just some things that are currently problematic with Nix so for example
05:02
Or things that I would like to do that I can't at the moment so one is I would like to use Nix as a make replacement or a basal replacement so And Nix is all these nice features a purely functional language Reproducible builds isolation and we do this for sort of large things like packages and for very small things like
05:23
configuration files in Nix OS So it seems like it's should be a perfect build tool as well so for building things like C source files or whatever other language you want to build, but there are a bunch of reasons that Currently you can't really do that at the moment, so I'll come back to that
05:42
So that's one problem another Nix package options are not easily discoverable or configurable so right now Nix packages have all sorts of options so for example a Package function might have an enable foo
06:03
Arguments, but this is completely undiscoverable except by reading the Nix packages source code and And it's also not configurable via Nix and for any other tool, so this is not very good UX
06:20
Another problem is that This is and this is an increasingly big problem is that Nix packages and Nix OS evaluation is slow, and it's getting slower all the time because the Sort of the level of abstractions that are used in Nix packages are increasing And it uses a lot of RAM so
06:42
And actually it turns out that this problem is kind of related to the previous problem Another one that Comes up a lot is that Nix currently has no way to handle secrets so things like passwords or keys You don't want to store those in the next store because then they're world readable which is bad
07:03
So you need to some way to to deal with them Or so another problem as an unprivileged so right now if you want to pull something from a binary cache from an arbitrary binary cache, so you cannot do that as an unprivileged user how things need to be signed by a
07:23
Key configured by the administrator So if you just want to pull some something from some arbitrary cache you cannot do that as an arbitrary user Another one so Closure bloat, so this is a fairly big issue so in Nix
07:44
It's very easy to end up with a package that has way more runtime dependencies Than it than it actually needs So I'll also give an example of that later So yeah, these are just some random problems, so
08:03
Probably many of you have other problems, so we're interested to hear them But so here are some goals Skip something yeah, so here are some goals that that you can extract from those problem statements So these are more at a technical level so by the way if you look at the rust goals. There are also a lot of
08:24
non-technical Goals there like improve documentation Eventualize in certain communities make the community more diverse, so that's all great as well
08:41
Yeah, but but here. I'm more focusing on on technical stuff, but We should definitely not restrict ourselves to that in the roadmap Yeah, so make Nix a compelling build tool so compelling replacement for Nix or basil for make or basil
09:01
Or something that can actually Complement those tools Make the next or content addressable, so that's a very technical goal, but it's it's kind of related to all the others Make Nix packages discoverable and configurable Improve the evaluation efficiency
09:22
Provide mechanisms to prevent closure bloat prevent provide a way to store secrets in the next door So yeah, these are just some Goals that I would like to Work on in the next year And to some extent I have been working on them
09:42
So the rest of this talk is just Some random brain dump on on how these goals could be achieved So Yeah, so first the gold Nix as a build tool, so So what do we need to get to that?
10:03
So in a way you can already do this in fact you could do that ten years ago In fact there is a Nix make repository somewhere, which has a bunch of functions for building C or C plus plus projects and and that works fine, but the problem is
10:23
So now you have your project and you're using these Nix make functions So you can run Nix builds to Have incremental builds for your project, and that's all great So now you want to package this thing and put it in next packages So you make a tar ball containing your source code and your next expression
10:43
And now you want to write a next expression in next packages that Extracts this tar ball and builds it and There you run into the problem that you need to be able to call Nix from inside a Nix build Because you're using a Nix expression to to build your project so instead of a make file so previously you would call make
11:04
right But now you have a Nix expression that builds your project so you need to be able to call Nix build but you're inside a Nix build already and So so and and and Nix derivation doesn't actually have arbitrary right access to the next door in fact
11:21
it only has right access to its outputs, so So this doesn't work, so now you have a very embarrassing situation so you have a Package that's written that has a build system written in Nix, but you can't actually put it in the next packages You could put it in Debian probably, but you can't put it in next packages
11:43
So so this is not good So you need need recursive Nix So that's kind of a required feature Another Not essential, but very nice to have feature is content address ability
12:02
which I'll come back to and caching of copying files to the store so a tool like So if you have your project which might consist of thousands of source files So now every time you run Nix build it has to read all those source files and and and copy them to the next store
12:23
Or at least check whether they already are in the next store, so that's a lot of IO, and that's slow so a tool like make prevent avoids that by Only checking timestamps and even that can get slow for very large projects
12:41
But yeah, Nix needs to hash all these files, so it actually needs to read all of them So you you want to have some kind of caching for that and maybe in something like an I notify demon to efficiently notice when files change But this is in the nice to have category
13:03
Right so the content address ability this is has kind of been a Could say a holy grail for many years So this is the property that? Should step back for a second so if you remember So a Nick store path that contains a cryptographic hash
13:23
But that cryptographic hash is a hash of the derivation that built that path It's not actually a hash of the content of that path And and and this is why you need signatures on binary caches Because you need to trust that Some store path was actually produced by the derivation that it claims to be built by
13:46
And that could be a lie so somebody could set up a binary cache Where and so you you get a legitimate Nick's expression So that's for example builds Firefox And then you pull a binder from the cache that actually contains something completely different like a trojan version of Firefox
14:05
so That's why you need signatures So in a content reversible store the store path So the hash in the store path is actually a hash of the contents of that path
14:21
So you no longer need to trust anything so you can just verify that for example in the path like this Nick stores hash you just check that the cryptographic hash of the contents of this path Is this so a path basically contains its own proof of integrity
14:41
So if you have this then yeah unprivileged users can install things from arbitrary binary caches Another very big advantage is that you get deduplication so for example if you
15:00
So So you make an irrelevant change to something in the dependency graph like you You make a white space change to glib C. So currently that causes the entire system to be rebuilt Which is bad and and actually yeah, not just rebuilt but
15:20
Had duplicated in the Nick store, so you need twice the storage space now Whereas with a content addressable store So because this change is irrelevant. It doesn't actually change anything to the output of a build It ends up being stored in the same location So that's that's much nicer
15:41
And in fact it it does prevent rebuilds because if for instance So you make that change to glib C You still need to recompile glib C to discover that that change doesn't matter But after that you don't need to rebuild anything that depends on it Because you've already discovered that
16:01
Yeah, this glib C. Is actually the same so so so it sort of acts as a barrier in the in the dependency graph Yeah, so this is why content addressability would be a great feature to have So a few interesting things about content addressability it to make this work properly it really needs
16:28
Perfect binary reproducibility, so that's currently not the case with next packages So if you build a package twice you might actually end up with Slightly different results so for example if a binary stores a timestamp somewhere
16:44
But yeah a lot of people are at work To improve that so for example. There's a whole reproducible builds dot org project That's that's yeah basically improving all sorts of packages and build systems to
17:04
to eliminate sources of Binary impurity okay, so for the other thing so yeah prevent Sorry Preventing closure bloat, so this is something of a obsession of mine. Sorry about that
17:26
So in X because of the way it detects it finds dependencies It's very easy to have an accidental dependency So this is not the case in say had Debian where you specify Dependency so you don't end up with an accidental runtime dependency on say the C compiler
17:44
So here for example. There was a situation where Thunderbird was storing its build Configuration so you can do about config in the URL bar and it will show you the path to the C compiler used to compile it
18:00
Which is of course a kind of a useless thing But it does yeah adds 1,200 megabytes of blow to the closure and Yeah, so this can happen very Accidentally you don't get any errors if you do that So yeah, we need better tools to to yeah
18:23
Detect when this happens, so we already have some attributes that you can specify in in next expressions for example You can say disallowed requisites To say that something should not have a runtime reference to say the C compiler But this is very limited for example you cannot do any pattern matching and you would like to say
18:45
This thing should not have any references to developer outputs And it should be per output because for example your death output Probably should be allowed to have references to other death outputs
19:02
And you might want to have a size check so for example if say the Nixos ISO suddenly Gets a gigabyte bigger than who would like to get some error so What I recently implemented partially so not all of this works yet is that you can specify
19:23
Per output check so for example you can say The outputs the out output should not have a closure bigger than 256 megabytes It should not be should not reference the C compiler or any death output But the the death output itself can reference anything
19:42
But it should not be larger than 128 kilobytes As a random example So Yeah, so we can start putting these things in next packages It could even be a generic thing so for example the rule that things should not allow
20:00
Should not be allowed to reference Death outputs is something you could actually put in standard ends so as a general policy So yeah, that would be very nice How am I doing on time actually I can just check okay, ah great
20:27
So yeah now I come to the really wild and vaporware part of the talk So So yeah really a big issue is discoverability and efficiency like I mentioned so
20:42
Next packages have basically no discoverability. Well. I mean you can discover that they exist sometimes I mean Nixon doesn't necessarily recurse into everything, but So you can see the packages exist, but you can see what options they have and Customizing Packages is also very ad hoc it sort of
21:03
Evolved it's not really a properly designed thing so if these things like dot overwrite dot overwrite derivation config And And in fact this this whole dot overwrite thing is kind of disastrous for for performance, so
21:23
It's really one of them. It's one of the two main reasons. Why Nick's evaluation takes a lot of memory So so dot overwrite Basically destroys the ability of the garbage collector at runtime to actually collect any garbage
21:40
Because you call a function so you pass it some arguments which can be very big because there are arbitrary dependencies large graph And And then the output just contains the inputs so the inputs can never be garbage collected, so This is this was you know kind of a bad idea, but we don't really have anything better
22:07
So There's kind of a meta issue here of all these things So somewhere along the way we forgot that Nixon's intended as a domain specific language for specifying
22:23
build graphs and configurations like like Nick's OS systems But as a DSL it's not really doing a great job so for example it has no concept of a package or an option or a configuration or things like plugins so any sort of features you or concepts you might
22:43
expect in a DSL intended for doing these things so So maybe we need to get back to Well we need to improve Nick's as a DSL so so one thing that I've been thinking about is
23:01
So essentially Turning the Nixon's module system or an improved version of it into a language feature Into something called a configuration which you can really think of as an an attribute set an extensible attribute set Which is really what a Nick's OS configuration is it's a it's a bunch of edge attributes that you can change and
23:25
So if you change one attribute it can trigger other attributes to change Yeah, so So a configuration is attribute set which contains attributes called options that can be set
23:40
And they can be overwritten later But unlike attribute sets and like Nick's OS options they can have types and Documentations and merge functions, and and that's the thing that gives you discoverability so things like package options can be
24:00
expressed in this way and because they have things like a description and a type tools could Can discover them and present them to the user and then allow them to be changed programmatic programmatically So so the
24:20
sketchy design for this language feature is a bit like this so a Configuration looks a lot like an attribute set only it uses angled brackets My change don't get too angry or or enthusiastic about the syntax But yeah, so it's you could think of it as an attribute set
24:43
So we have an attribute foo and attribute bar and an attribute ABC that actually refers to foo and bar So it's a recursive. It's like a rec attribute set So if from this thing you select the ABC Attributes you would get a value one two three because bar is true, so if if true then one two three
25:05
Right, but what you can now do is take that configuration module and apply an a new module to it that sets bar to false and So now if from this module you select ABC, then it will return
25:24
a false then It goes to One two three times two, so it will return Two four six so this is pretty much exactly the behavior of the nixos module system
25:41
Yeah, and then the idea is that you can have some sort of syntax Which I'm not sure about but to to attach fields or annotations to those to those options like documentation a type default value and so on Merge functions
26:02
priorities Yeah, all that sort of thing basically all the things you have in the nixos module system Right, but now the idea is that we can apply this to Building packages so rather than having packages as functions
26:23
Which have the problem that? Well, there's no override mechanism, I mean no good override mechanism There's no documentation and so on We can basically treat packages in the same way as the nixos module system treats
26:43
system configurations, so you you build a package in sort of a modular way by combining a bunch of modules so for example you could have a very Sort of at the bottom you have a module that Captures the concept of a derivation, so what is a derivation?
27:02
Well a derivation nix has a name and a version, and it has a builder and it has arguments, and it has an environment And if you set those things then you can evaluate a derivation attribute Which produces a low-level derivation?
27:22
So so this is very low level Now you can build higher level modules on top of that so for example this thing basically expresses the basic standard environment so the concept of phases And dependencies on our packages so this thing adds a an option called built inputs and an option called phases
27:47
and it's it implements this on top of the Lower level derivation module by Setting builder and arcs and nth and that's causes
28:02
a derivation to be computed Yeah that uses these things So just to continue this a bit so you could have a module that Captures the concept of a package so package has a description and the home page and so on And all these things have had descriptions, and they have types, so they're they're discoverable
28:25
And and you get error messages so for example if if you use this previous So currently with derivations if for example you you misspell build inputs You're not going to get an error Because a nix derivation is basically just a bunch of environments variable bindings, so
28:45
Yeah, so there's no checking whatsoever there, but here if you set an option that Hasn't been declared then you get an error message. Just like in the in the module system
29:00
Yeah, so you can build higher and higher level abstractions on this so for example you can extend the Sort of generic standard environment with the concept of a Unix package which for example has a configure phase Because it runs a configure script Yeah And then finally you can define a package so a package is something that extends the Unix package module with
29:25
something that sets a name description source but it also Has its own option namely enable GUI So in this fictional example hello world has a gtk support
29:42
So now you can say build inputs is if enable gt GUI then use gtk And and this thing is now discoverable so You could have say a nix query package command which will show that this thing has an enable GUI Option and they could have a nix install command
30:03
That that allows you to set that option So Yeah, and you could override things using the exact same module system so that just like in nix OS So that's about it so there
30:22
Lots of other things you could imagine for the roadmap. I'll skip that So yeah, so what I should do is create a sort of a roadmap issue and where everybody can go wild with IDs and suggestions, and then we should try to synthesize something workable from that
30:46
So yeah That's it. Thank you
31:18
So for the configurations idea use it doing overrides. How does that solve the memory issue?
31:24
Don't you still need to hold on to the references to all the inputs no because as soon as you Evaluate the dot DRV attribute You don't need anything else anymore you can that after that you can just discard
31:42
Everything that went into it, so I sort of like in nix OS if you evaluate system dot build top level you get a derivation out of that and And at that point you can garbage collects all the inputs to that thing Oh, so like the when when you're passing something as an input to something else you're passing to the dot DRV
32:01
Not the composable, right? So Presumably this is all fiction so so here this GTK thing would trigger an evaluation of GTK dot DRV Implicitly on the slide about the nix as a build tool you have been talking about avoiding rebuilds and
32:30
similar properties Have you looked in to the recent paper? Built systems a la carta by Neil Mitchell Simon Pena Johnson and the remark off
32:40
They're analyzing various build systems there and nix is one of them and nix ticks of almost all features of the ultimate build system Was that the ice Fp paper? Yes, okay? Yeah, it's on my to-do list, okay? Please take a look at this it gives very good names good glossary to talk about those properties. It's a great paper. Thank you
33:05
one more so the configuration Options it excites the type system in me
33:24
I won't go there Is there also? So the nix os module system also supports the notion of overlays Is this should is there a version of this also in this or is this then? extension yeah, probably so That's sort of a high-level thing that I haven't figured out yet, so how you actually put these things together
33:47
so I but you need some way to do that and Yeah, so I don't know yet, okay? Yeah, I think we've got a move maybe one more
34:03
last one It's more of a policy question. I've seen a lot of commits like fixing Thunderbirds like you showed and There has been commits for every separate packages Why don't you enforce this kind of like do not reference GCC for all the package by default and then?
34:26
If some package really needs GCC then you could enable it right well That's kind of the point so we don't have a way really to enforce that yet So right now. It's really only if people sort of notice That's only a closure has become much bigger
34:45
So you you can use these disallowed Requisites attributes, but very few packages do that But you could use that by default like that to empty the revision and then by default just do not reference GCC There is no reason for most packages to do so right and so probably for GC that would work, but
35:03
You really want to do say things like it should not reference any death output And that doesn't work at the moment because death outputs should be allowed to reference death output So you can't use the existing attributes for that and But yeah, so I would definitely like to have in Nixos for example that we say that all the Nixos
35:25
VM tests could just check that their closures don't have any Death outputs in them and and no see GCC or clang So that that would already help quite a bit, okay, all righty, so thank you again
35:45
For your talk