Nixpkgs Overlays – A place for all excluded packages
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 14 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39629 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2017 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
NixCon 20171 / 14
3
5
6
8
9
00:00
Letterpress printingBitPoint (geometry)Just-in-Time-CompilerOverlay-NetzModulare ProgrammierungPrisoner's dilemmaPresentation of a groupProjective planeSlide ruleMeeting/Interview
01:19
Presentation of a groupDeclarative programmingElectronic mailing listInheritance (object-oriented programming)Google ChromeProxy serverOverlay-NetzPresentation of a groupSet (mathematics)Mixed realityModulare ProgrammierungFunction (mathematics)Functional (mathematics)Computer fileInternetworkingProxy serverOverlay-NetzModule (mathematics)Connected spaceParameter (computer programming)Computer networkWeb 2.0Directory serviceInternet forum8 (number)Term (mathematics)Ferry CorstenPhysical systemShared memoryLevel (video gaming)Revision controlSlide rulePoint (geometry)Different (Kate Ryan album)Control flowSuspension (chemistry)Extension (kinesiology)Projective planeConfiguration spaceExistenceComputer configurationGraphical user interfaceComputer animation
07:46
Inheritance (object-oriented programming)Directory serviceLevel (video gaming)RootData storage deviceCompilerIntrusion detection systemComputer animation
08:21
Overlay-NetzVideoconferencingInheritance (object-oriented programming)Derivation (linguistics)Function (mathematics)Algebraic closureConfiguration spacePoint (geometry)outputLibrary (computing)BitProgrammschleifeWeb pageWritingOverlay-NetzFunctional (mathematics)Inheritance (object-oriented programming)Algebraic closureControl flowLoop (music)Time zoneLevel (video gaming)Key (cryptography)Cellular automatonDerivation (linguistics)HoaxSocial classFunction (mathematics)Module (mathematics)Lecture/Conference
12:23
Inheritance (object-oriented programming)Derivation (linguistics)outputLatent heatVideo gameParameter (computer programming)Derivation (linguistics)Inheritance (object-oriented programming)CASE <Informatik>Overlay-NetzFunction (mathematics)Attribute grammarComputer animation
13:40
Overlay-NetzInheritance (object-oriented programming)Physical systemCircleComputer fileRevision controlParameter (computer programming)Overlay-NetzSet (mathematics)Heegaard splittingInheritance (object-oriented programming)CodeComputer animation
15:28
Inheritance (object-oriented programming)Default (computer science)Overlay-NetzMixed realityOverlay-NetzTerm (mathematics)Computer fileCodeInheritance (object-oriented programming)Default (computer science)Heegaard splittingRule of inferenceDerivation (linguistics)Functional (mathematics)Parameter (computer programming)System callComputer animationLecture/Conference
16:50
Function (mathematics)Inheritance (object-oriented programming)Overlay-NetzElectronic mailing listParameter (computer programming)BitRight angleSimilarity (geometry)LogicMixed realityOverlay-NetzSet (mathematics)WeightFunctional (mathematics)Level (video gaming)Proof theoryOperator (mathematics)Extension (kinesiology)Library (computing)Electronic mailing listMereologyPoint (geometry)CASE <Informatik>Inheritance (object-oriented programming)ResultantComputer animation
19:35
Inheritance (object-oriented programming)Beta functionFunctional (mathematics)Operator (mathematics)Library (computing)Overlay-NetzRevision controlAttribute grammarComputer animation
20:27
Overlay-NetzInheritance (object-oriented programming)Beta functionFunction (mathematics)Utility softwareDerivation (linguistics)Integrated development environmentContent (media)Computer networkParsingRevision controlProduct (business)Root10 (number)Sample (statistics)Hash functionEuclidean vectorDigital filterElectronic mailing listString (computer science)Token ringError messageAttribute grammarFormal languageToken ringRow (database)QuicksortMereologyPositional notationKey (cryptography)Heegaard splittingString (computer science)Computer fileOverlay-NetzBasis <Mathematik>Functional (mathematics)Revision controlElectronic mailing listOrder (biology)ExpressionParsingLine (geometry)Term (mathematics)Source codeIntegrated development environmentUtility softwareLibrary (computing)Level (video gaming)Moisture2 (number)Software testingReverse engineeringDirectory serviceServer (computing)Repository (publishing)Hash functionBeta functionFitness functionNumberRotationSimilarity (geometry)Set (mathematics)CASE <Informatik>Computer networkGastropod shellPresentation of a groupPhase transitionMetadataRight angleFluid staticsUniform resource locatorFile formatSlide ruleFreewarePoisson-KlammerCompilerAddress space1 (number)Derivation (linguistics)InformationMatching (graph theory)Regulärer Ausdruck <Textverarbeitung>Stability theoryDefault (computer science)BitGoogolBinary codeOnline helpComputer animation
28:17
String (computer science)ParsingPhysical systemOverlay-NetzState of matterContext awarenessMultiplication signLevel (video gaming)Boolean algebraSpacetimeLimit (category theory)Element (mathematics)EmailForcing (mathematics)Recursive descent parserRecursionDesign by contractSet (mathematics)Pole (complex analysis)ImplementationPhysical systemLibrary (computing)InformationNetwork topologyBellman equationData conversionAttribute grammarElectronic mailing listString (computer science)ParsingOperator (mathematics)Stiff equationDefault (computer science)SequenceToken ringFunctional (mathematics)Computer fileStack (abstract data type)Data storage deviceCodeComputer animation
31:39
Operator (mathematics)Data typeStrategy gameFunction (mathematics)Functional (mathematics)Point (geometry)Derivation (linguistics)Strategy gameEndliche ModelltheorieParameter (computer programming)Insertion lossLibrary (computing)Overlay-NetzComputer animation
32:57
Derivation (linguistics)Computer networkParsingInheritance (object-oriented programming)Beta functionFunction (mathematics)Overlay-NetzTerm (mathematics)Revision controlFlow separationRight angleOrder (biology)Level (video gaming)Strategy gameControl flowFunctional (mathematics)Performance appraisalBitDefault (computer science)Module (mathematics)Complex (psychology)Process (computing)Set (mathematics)State of matterParameter (computer programming)WordPoint (geometry)Musical ensembleAttribute grammarLipschitz-StetigkeitLibrary (computing)Loop (music)CodeRecursion1 (number)Modulare ProgrammierungDerivation (linguistics)QuicksortEndliche ModelltheorieMultiplication signLatent heatCellular automatonFunction (mathematics)Physical systemComputer configurationDirac delta functionRegulärer Ausdruck <Textverarbeitung>Numbering schemeMixed realityWeightMereologyRootIterationResultantExpressionFault-tolerant systemNetwork topologySocial classBuildingSubject indexingProjective planeError messageInformationInheritance (object-oriented programming)Field (computer science)Computing platformSystem callGraph (mathematics)Perfect groupSubsetComputer animationLecture/Conference
Transcript: English(auto-generated)
00:03
Our next speaker is Nicola, who you probably know from the module system. The one thing I definitely know about Nicola is if something annoys him strong enough, we'll do something like implement the module system. He works day-to-day at Mozilla, recently working
00:25
on a new project called Holy JIT, which is a just-in-time compiler for JavaScript written in Rust. So, one thing I was looking for a logo for, a first
00:48
slide for this presentation, and I was trying something with overlays, and I figured out that we already had overlays, we just need a bit of transparency to see them. And adding transparency in the logo, you can see that we already have
01:03
overlays, and we also had a fixed point. The logo had a way to fix the depth of printing it, and if you recall some old t-shirts, it's not always easy to understand fixed points, especially if you are a printer. So, the goal of this
01:21
presentation is to make sure you understand overlays, and make sure you understand them correctly. I gave to the NIT community a gift for Christmas. I gave you overlays, and almost a year after, I came back to and went to the
01:40
web and looked for things where people used overlays, and realized that, oh crap, people have used it, but there are things that are in the documentation which might be not well explained, and I hope this presentation will clarify that. So, I will go and describe what they are, how they work,
02:01
and what you can do with them. So, NIT2S is awesome. We have the module system, which gives us the ability to be declarative, and this module system also gives us the ability to compose different modules from all over the place, and make that into one set. Except that, okay, that's nice, but not
02:22
everything is awesome. Sorry for the Lego movie. But, NIT2S packages is like, you have these functions, and they return something, it's not that much declarative, and it was not composable before overlays, and we will see how
02:42
overlays make that composable. So, before going into overlays, let's look at how we used to extend NIT packages before that. Before that, we had these three functions, and these three functions were like, okay, there is a package that everybody, almost everybody uses in config.nits, and it's nice, you can
03:03
extend things, you can change the sources, okay, great. But you cannot share that easily, because you have some custom stuff, and it's like there is this one file, and if you want to share that, you have to share a portion of it, and we're back to the problems that I had before making the NIT2S modules, which were, okay, how do we avoid getting into forums when we want to have
03:24
a solution, and how can I just share a file, and just pull that down. Then we have this other function, which I'm glad I removed it, which is override packages, and if you don't know about it, don't go dig. Don't dig further, it's like awful, and it's no longer there. It's not, not even try to use it, it won't
03:44
work. Then there is this other style, which is like, okay, I'm from an external file in another project, and I want to import NIT packages, and it extends this versions that I imported, and adds these new packages inside it. That works
04:01
well, unless, until the point where you want to add multiple, and then you realize, okay, then I want to import something else that needs packages, which already extends it, and it's becoming hell. So all of this, all of these methods are replaced by overlays, and the second one no longer exists. Great.
04:22
Soon, maybe, I hope to break all of you, if you are still using that. So composable, we can compose overlays in a way where we have the same syntax for all of them. We have a simple syntax, which is made to copy and paste, and that's basically all, and then you can do all the things that you could
04:45
do with the overlays before, which is like, you can add packages, you can replace some of them by tuning them, you can change the recipe, and you can remove some of them, which, for example, say, oh, it sleeps, people, no, no more
05:01
it sleeps, which will break tons of stuff, such as emites, I guess. And the way overlay works is that you have one directory, or an option in NixOS, or a single file, which got added recently, and this one directory lets you add files at a coarse-grained granularity, and you add files into this
05:25
directory, and this will be overlays, which will be used one after the other, and will be combined into Nix packages. So I will go through multiple examples, and after a while I will ask you to solve some of them, so I
05:44
will go with the easy one. So this is the simplest overlay that you can make. It takes two arguments, one is self, the other is super, and you give it an empty set, and you extended Nix packages with nothing. Great. So overlays
06:01
are not just some things that are made up, it's actually the internal of Nix packages. Nix packages is using the overlay system, except that it's doing a mess, which is currently elided in this slide, and it's using the overlay system to basically stage the different levels of Nix packages that we currently have. So this, basically, the things that I did was like, I was
06:28
trying to do the grafting work again, if you recall the presentation from two years ago, and I realized that, huh, there is this function override packages, which gives me tons of trouble, and I can replace it and
06:43
just add overlays at the end. And that's basically all it is. Overlays are just adding something to the internal of Nix packages, and you get to extend all of Nix packages. And I can remove one of the side features that was there and
07:00
was awful in terms of performances, and, hey, no longer here. So, no, we have overlays, yes. So a simple example that I found on the internet, it's quite a full, it's resourceful, the internet. You find tons of things, and sometimes you find good examples. And this one is just adding a command line to, a command line argument to Google Chrome, which is saying, hey, use this
07:25
proxy to redirect all my network connections through this proxy, which is really nice, especially if you can set it on the command line and get all the nice features and protections that you get with a proxy.
07:41
So, okay, that's a good way to get an overlay, and you get to recompile Chrome once more. So, while we are discussing about recompilation, some other people wanted to to get Nix, but in a different directory, and basically this is interesting if you are stuck in your home
08:04
and you have no root access for adding the Nix directory at the top level. So this is a simplified example, but this has the same ID, which is that you want to configure Nix to have its Nix store at a different directory, and that's an interesting one, as well. So, then you have other examples where you just have
08:25
ordinary packages, as we do in Nix packages, and just one of the tools that we use within Mozilla for discussing. So, we saw this self and super, and it's not clear when you see them.
08:43
So, self is basically, in Nix packages you have a fixed point, you have all these stages, and you have a fixed point which basically takes the output and gives it back at the input of all of them. And super is, you have all these stages, and it's the previous one. You take the
09:03
next stage, it's the previous one, and so on. So that's basically all it is. But that doesn't tell you how to use them. Self is made to basically find all the dependencies. So if you have a package, and your package depends on sed, on bizon, on firefox, then you will use
09:28
self. You will say self.sed, self.bizon, self.firefox. And that's all. If you want to use self for anything else, that's wrong. That's the only bullet point here. That's the bullet.
09:46
Super is basically all the rest. If you have functions called packages, or library functions, or write text, or run command, if you have functions, it comes from super.
10:02
If you want to override the recipe, it comes from super. It comes from super for the following reason. Let's say I have sed, I want to override something in sed. And I want to, after overriding something in sed, I want to define it at sed. You get an infinite loop, because you say,
10:25
I want the recipe of the thing that I just defined. And that's why you have to look for recipes for making packages in the previous ones, until you find one. And that's why also overlays are ordered, as opposed to next class modules.
10:45
So, okay, we saw self and super. Now, I will ask you to raise your hand as soon as you find the issues in these examples. So, if you find one issue, raise your hand that way. If you find two issues, raise your hand that way. And if you are watching this talk remotely, then you
11:02
have the end key to see the answers. Okay, I see a few ends. So, the first issue is like
11:26
write text. Write text is basically, it's a function. It generates a derivation, but it's a function. As a function, it comes from super, not self. And the other one is a bit
11:40
simple to see. It's the fake closure. The fake closure is coming from above, because of the right keyword. And it's not nice to make an overlay which is using the right keyword, because if you are using the right keyword, basically that means that you're looping within your overlay, and not from the self, from the mixed packages fixed point, which means that
12:06
an overlay which comes after has no opportunity to change that. So, the fake closure loops inside instead of looping around, and if you want to give more opportunities to your users, you
12:21
should control yourself instead of write. So, let's look at another. And this is one issue, and raise your hand as soon as you see it. It's just above. The question was,
12:51
where does Python come from? So, you're almost on the thing. So, this is not an overlay.
13:01
And the problem with this one is that there is this Python argument, and this Python argument, it has to come, and if you want to make an overlay, it has to come from self or super. In this case, you're using the override derivation, so it should probably come from super.
13:21
And this Python is like, if you want to make something which depends on a specific name, or a specific attribute within the overlay, then you should probably use it as a name of the attribute, and make it that way instead of using Python as a package. Okay, another example.
14:04
Yeah, where is he by the way? So, if you are watching remotely, Garibas, I hope that you will face your code. So, this code has multiple issues. One of the issues is like, why do we import needs packages
14:27
twice? You first import it, add the argument, and then use the path which is inside it to import it again, which is terrible for people who want to do, for example, cross-compilations,
14:42
because then you got a different system, and if you were to import a package set which was configured for a different system, then you import it again, and you lost all this system customization with this one. The other thing is that you have this overlay which is defined within this file, and within this file, you have like, inherit packages, which is the one
15:07
which is that argument, and that's completely wrong, you are basically, you have three versions of packages, that's just, and if you're cross-compiling that's even worse, you might even get the wrong one. So, what you want here is basically first split it into two files, and
15:25
then you want to remove the package and probably use self or super. So, that's how it should look like, you have one file which is, okay, get the path of next packages, and only import next packages once, if you want to do that. Then you have this overlay, which is probably
15:43
a default.next or whatever, and this overlay doesn't depend on packages, it depends on self or super, and gives that to the requirements.next. And that's all, split the files, it will be nicer and smaller in terms of code. So, I have again an example, and same thing, raise your hand if
16:06
you see the issues. So, here's the issue, is basically, you have packages that you depend on,
16:26
and you want the derivation out of them. So, call package is a function that you get from super, okay, but the things that you give as argument, which are dependencies, such as the vte package or lua5, are derivation, and you want to get them from self, such as somebody
16:47
else can override them. So, let's look at how overlays work, and how we compose them. The logic of the composition is this update operator that is index. So, we have two sets,
17:03
and we will make this example a bit more complex. These two sets, we use the update operator, and that's fine, it returns the set. Then we go one step further, we add super. Super is the previous one. So, we give as argument of bar, the super argument, and the super argument will
17:26
be the left hand side of the update operator, which is the foo, even super argument, which is an empty set. So, in this case, we have z, there is a missing part here, but that's just for simplification. It does not give anything. Then we generalize the update operator and
17:45
give z the name x10, which is not exactly the same as the one which is in the library, but probably we shall rewrite the one in the library. So, the way it works is exactly the same thing. It's like you have the left hand side, and you have the right hand side,
18:02
and you call the right hand side with the left hand side as argument, and then you have the update operator. And with that, you can use them and compose them already with starting, okay, I have foo, which takes the empty set, and the result of that, I give it to bar, and so on and so on. So, we can make that for list, and we can fold that and say,
18:28
use this expand function, you have the empty set, and so on, and you iterate over the list, and that's basically how these packages work. And you can go even further, because you want to
18:40
have a fixed point, and your fixed point is basically saying, okay, now I have this self argument to all of them, so you have all your overlays in the list, and you have the fixed point, which is basically giving the self argument to all of them. So, this is what the fixed function is doing, and the map function is doing here. The map function is here to forward
19:04
the self argument provided by fixed to all of them. So, here we can always take the previous example and make that into, hey, I have this overlay, which takes self and super,
19:22
and compose all other overlays which are in the same directory, such as foo.overlay.nates and bar.overlay.nates, and that's really great, and we have a similar example in the next packages module. So, let's see if you can spot the issue here. Notice that you know how
19:43
it works. It's supposed to add packages and be a friendly overlay. So, I see a few ends,
20:05
and the issue here is that you have lib, and you have latest, and these two are attributes, and the way the update operator works is basically it will erase the previous one with a new value. So, basically what you're doing here is erasing all the library functions by
20:23
this Firefox version one, and that's not friendly, that's you're erasing all the work for all others. So, you want to make sure that whatever you do, you at least extend what was already in this attribute before. So, it's not yet settled, but we are trying to go into making a convention,
20:52
which is like, if you are making an overlay which has reproducible packages, then that's fine, you go to the top level. If you have utility functions such as, oh, I want to parse
21:03
some things, then you go into lib. If you have packages which are updated automatically, without doing anything in terms of changing the sources, then please don't go as a reproducible packages because that will be a help for anybody who wants to make a reproducible environment.
21:20
So, go into latest and add your stuff into that, and if you want to make a shell environment which is not dedicated for building anything, then probably go into devenv or some similar names. So, these are conventions. It's not enforced yet, but that's just playing nicely and I think
21:44
of everybody having the same conventions. On the same side, it might be nice if we added similar naming for googling or searching this set of packages. So, currently at Mozilla, we have neat packages Mozilla, which is Firefox, Rust, and some other tools.
22:02
And if you are making an overlay for some specific package, then you can probably name it after the name of the program and say, dash overlay.next. This way, somebody can say, oh, I want to pick only this one and then link it to the .config-nist-packages.overlay directory.
22:21
And if you are making a repository which is containing only overlays, then provide a default .next which aggregates all of them if you have multiple ones. So, what can we do with overlays? Overlays are quite powerful in terms of, yeah, we can compose and that's nice. And we can do something such as fetch from the
22:45
network because we have a new built-in for fetch URL. And as soon as you fetch something, you have another problem which is parsing it. And then you have to generate derivations. So, I will go for a few examples. Sorry if it's a tiny one.
23:02
So, fetching from the network is simple. You have the fetch URL. It will take a URL and put that into some file somewhere. And then you can go and parse, okay, I have a JSON file. So, in case of Firefox, we have a JSON file which gives us the version number of the various branches such as the nightly is 58, the beta is 57, and the stable is 56.
23:31
This version gives us, basically, we can derive from this version the location of some JSON file. And this JSON file has a specific layout which is the hash, then some other
23:46
information and the name of the binary. So, you can use the built-in match function to basically use the brackets and filter the parts that you are interested in and get, for example, I know what is the file names that I'm looking for, give me the hash,
24:04
and then I can generate the derivation for that. So, the way the Firefox overlay works is looking at this file, extracting the hash and the binary name, and then generating a derivation which is like, okay, your sources are fetched from a fetch URL with a SHA and a name, and
24:25
that's all. So, I got a remark from the slides that you cannot parse context-free languages from regexes. Yes. But they are clever, guys. So, if you look at the Rust overlay,
24:52
it's a bit more complex. So, there is first the fetch part, which is basically reverse engineering Rust up, which is a common tool in the Rust community which is like, okay,
25:03
I want to update the Rust compiler to this version for everything which comes next. So, this basically is a reverse engineer of Rust up, which is, okay, how do you get the address of the 2ml file, which is depending on the channel and which you are or the date
25:26
and the location of where the files are distributed, and that's basically how the fetching of the metadata is made. And it goes through HTTPS, such as, at least we know and trust the static wrestling.org, and that's how we fetch. Now, let's look at what we get. So,
25:49
we get some nice language which is harder to parse with regexes. And this language is 2ml. It's basically a HINI file which has the same syntax as we have in
26:03
X for having the dot notation for attributes which are under multiple layers, and then it gives us like, okay, you have this package which has this hash and a bunch of stuff, and it even has lists if you have two braces on both sides. So, okay, that's a nice format
26:25
to look at, and we can make a parser for that. So, for making a parser, it might be nice to split the phases into two and first make a tokenizer. So, how do you go and parse this
26:43
2ml file? You basically make one big regex which basically expresses the list of tokens that you expect. And you also express the layout, and that gives you not yet a tokenizer, but
27:00
the basis for it. Then you have a tokenizer function which is using a new feature of Nix 1.12. You can look at the old version which is in Nix packages for Nix 1.11. It's
27:20
more stable as this one, but it's more challenging to understand as well. So, this one is working by using this split function. This split function is basically taking a string, huge string, and looking at the token and split your string with, okay, I have something which is not a token, I have something which is a token, I have something which is not,
27:42
and alternating between the two. Then, as we go and alternate between token and non-tokens, then we have to filter out all the things that we are not interested in, and if there is something which is not layout as part of the non-token part, then we want to throw something, okay, I don't understand your 2ml file.
28:02
And that's basically what the filter layout function is doing. So, now we tokenize as we isolated things which are tokens and the things which are not token, it's that we still have states. But that's good enough for us. So, now let's make a parser. So, a parser is kind of simple.
28:27
You basically got some text, and you're in a state. So, instead of making your parser where you go and do a recursive descent through the information that you got,
28:41
we have to mutate the state. The reason why we cannot do a recursive descent into a parser is because Nix has some recursion limit which are not directly made into Nix, but made into the binary which is interpreting Nix. And we cannot use a recursive descent because of that,
29:06
and because it will be way too huge as soon as you go deeper into this level. So, the trick here is basically if you think of it as an LR parser where you have state and each time you move from one node to another, you change your state. Then you have to remember
29:22
what is the state. So, we want to have one operation which is applied every time in a sequence and which does not consume stack. There is a built-in which is foldL and with the apostrophe. This is a built-in which is kind of special compared to the foldL
29:43
implementation in the library. This basically avoids consuming stack. It will call every time the function but will not increase the stack space for every element in the list. So, we have an initial state. This state is like, okay, we are in state zero and we have
30:09
some context. When we iterate through the tokens, we basically, okay, we are in state zero then do the match, for example, the name and store the name into the state. When we are in, and
30:28
in state one, we expect the token to be always equal. So, that's what we are in this code. And so, we say, okay, no, it's a token equal, so let's go to state two. And in state two,
30:42
we match, for example, a value which is a string or a Boolean or whatever. And we have this token to value function which will give us the value as the next value interpreted from the true ML file and we will set it to the name. So, that's really simple, honestly. And it's just
31:01
going through state and saying, when I hit this token and I'm in this state, then I go to this other state. So, we made a parser and now we got to deal with the fact that the system is not represented the same way as the system that we have in the manifest file. And so, we implement a switch, which is like, if you give me the system in Nix, I can give you the system
31:25
for the tarballs that you are looking for. And the switch in Nix is just an attribute set, and at the end, you say, oh, I want this attribute. And if it does not exist, then you complain about it. So, we went through the overlay, we saw that we can make it composed-wise, inserted into
31:47
the fixed point. That's really nice. It's not yet declarative. And we can pick that. But that will be the subject for another talk, probably, because there is an RFC which is in
32:00
the pipe, which is Simple Override Strategies, which gives us the ability to have a single syntax for all the override functions. Like, override derivation, override, and whatever you can think of. It basically should be expressed with a single syntax with this model. So, that
32:23
basically is the end of my talk, and if you have any questions, please raise them and I will try to answer them. Thank you. I have a question about when to use the self and the super
32:50
argument in your overlay. Basically, you're saying when you want to use a library function, you should use super, right? Yes. What if you have an overlay, you want to extend
33:03
lib with your own library functions, and in the same overlay you want to use these library functions inside your packages? Shouldn't you do that? Should you then have two overlays? Or should you, because you can't yourself in your... When you're
33:22
a library function, you are supposed to use super. When you are defining the function, that's... you have already access to it. So, you could make it to... I won't recommend it, because you can just use the ones that you have. So, just use the ones that you have, it won't matter that much. Oh yeah, if you have them in a lab binding, then you call them
33:41
directly, right? But exposing them is nicer, because that means that older can work on top of the words that you did. One quick follow-up question. So, in NixOS you also have this lib argument that your module gets. I found that when you override the lib from
34:01
Nix packages, those overrides are not accessible in the lib argument of the NixOS module. So, I always use packages.lib in my modules, but I've read somewhere that that is... that's not the right thing to do. You shouldn't call... but how do you... basically, how do I make it so that my own functions are accessible from lib
34:25
from a NixOS module? So, packages.lib should not be used, indeed. The problem that you have is that the lib that is given as argument is basically used for the evaluation of modules.
34:42
So, unless you're modifying the module system yourself, then you should be perfect to use packages.lib. And the separation of the two was basically made for that, to split the module system part from all others. And the problem is that people
35:01
tend to use packages.lib to get make options and make default and others. And this is where we had to say, no, you shall not use packages.lib. So, if you have custom functions like the Firefox versions that I saw before... That's safe. That's safe. As long as you
35:21
make sure that you don't use module system functions from another library than the ones that you have for evaluating the module system. Okay, perfectly. Okay, yeah, we don't do that
35:41
I do self.call package all the time and it works great. So, I will be happy to break your code soon. Challenge accepted.
36:02
So, when you use self, it's one iteration slower because you have to go through the fix point. It's not a big deal in terms of performance. It's a big deal in terms of concepts because my goal is aiming for graphing and for the simple override strategies.
36:23
And the problems that you got by not using this super for the call package is basically that you're adding an external op going through the fix point. So, call package, if you use self for call package, you will say, okay, I'm here. I'm going through
36:42
the fix point to find call package. And then call package itself is using the fix point to go to the dependencies that you have. Or the dependencies that you have. And you are going twice through the fix point. And this is a problem for implementing graphing later. Because the way graphing works is by peeling off the fix point and making sure we can have
37:05
another version of these packages to make the deltas and make sure that we can do the graphing. I just wanted to make a quick warning. The RegEx syntax in Nix 112 has changed. And so,
37:24
so be very careful. And also in Nix 111, there were two RegEx syntaxes, depending on which platform you're on. Yes, I had this problem too. Actually, I think fetch run also requires a SHA-256, no?
37:43
The build thing doesn't. Mic-swering is a dangerous process. So, do you think that, well, I've sometimes had issues with overriding packages because,
38:04
also using override actions, because they, rather than having some sort of fix point, they just use rec. And I'd like to be able to override stuff like the version, which is then referred to by the expression itself. Do you think an overlays-like system
38:25
would be helpful for that as well, or reasonable, or possible? You mean within each of the packages? Yes. So, the simple override strategy should provide the answer to that, because it's basically giving
38:40
a decorative way of writing packages, and you no longer have a function, which is like SCM make-derivation. So, you get just an attribute set and no rec, and you basically can get all the information out of it. It might be a bit complex in terms of concepts, but the syntax
39:03
is still to be discussed, and we should aim for making it simple to read. And it should solve the problem of having the version, because the version will just be one field in the set that you can override without going through override derivation. Thank you.
39:30
One first simple question is, why is it called self and super? You could call them packages and functions, for example. That would be much more simple. So, there is one thing which is called
39:43
legacy. And yes, we shall probably rename them at one point. But anyway, it's in my file, so I can name them like that? Yes. And does it make sense? It totally makes sense. And if it's so easy to fact things up and use the wrong one,
40:01
is it possible to enforce that we are using the wrong one, the right one? Why are packages available in both? If you should not use packages from super, then just disable the packages in super. You cannot disable the packages because they are the result of the make derivation function.
40:21
But you can disable the functions from going through the fix point by putting no on them. That would be one solution to avoid using functions from self. That would really help to have just an error going up saying, okay, not possible, do not use this one, use the other one. So, it is possible for functions, it's harder for packages unless we are going to use a
40:46
stage where you can only get derivations if you go through the fix point, which is like we will have to have a post process at the end of the overlays and other stages, which will basically convert every recipe into its derivation from the part.
41:07
Is that something that we could implement in like two days in a hackathon, for example? Maybe if we'll break some code.
41:23
All right, last question. So, you go through a fix point of self and what is it you do with the fix point? So, the fix point is like a super will just descend until the empty attribute set is reached and check for attributes inside like the innermost loop.
41:47
Have you read into recursion schemes? Maybe because this reminds me of recursion schemes.
42:01
And maybe there's like a term already for that because you have different kinds of... So, we don't have a recursion problem when going through super because super will just go back until you find basically the attribute which is the empty set. When you find the empty set which is at the root, it does not go back but you make the
42:25
attributes which are becoming larger and the last one. You have recursion issues when you go through self because self is basically you can define yourself. Yeah, you're going through your own definition.