The Nix package manager development process
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61605 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2023151 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
Revision controlProduct (business)MathematicsSoftware maintenanceLine (geometry)Projektiver ModulNumberOpen sourceFrustrationMultiplication signProcess (computing)StatisticsElectronic program guideBitView (database)Projective planeBand matrixPoint (geometry)Data managementSingle-precision floating-point formatDifferenz <Mathematik>Slide ruleComputer fileImplementationComputer animation
05:05
Core dumpDistribution (mathematics)Local GroupTime evolutionFormal grammarProcess (computing)Function (mathematics)Software bugTask (computing)Vector potentialGroup actionGraph (mathematics)Multiplication signSoftware maintenanceDependent and independent variablesCodeStudent's t-testCore dumpProjective planeField (computer science)Source codePixelDecision theoryEndliche ModelltheorieBitShape (magazine)NumberRandomizationRow (database)Point (geometry)Scaling (geometry)Computer animation
10:56
System programmingElectric currentWhiteboardState observerArchitectureDependent and independent variablesBitProcess (computing)Scheduling (computing)Disk read-and-write headRule of inferenceCASE <Informatik>Standard deviationAdditionLevel (video gaming)Network topologyOpen sourcePrice indexWhiteboardProjective planeMultiplication signArithmetic meanDifferent (Kate Ryan album)CodeExpandierender GraphPlastikkarteDomain nameExpected valueFundamental theorem of algebraMereologyMathematicsSoftware repositoryElectronic program guideSoftware maintenancePoint (geometry)
16:46
ArchitectureWeb pageMultiplication signDefault (computer science)MereologySemantics (computer science)Projective planeImplementation
18:48
Normed vector spaceCASE <Informatik>Direction (geometry)ImplementationSemantics (computer science)CodeSoftware bugPoint (geometry)Bit
20:23
Program flowchart
Transcript: English(auto-generated)
00:09
Hey everyone, and so yes Brian introduced me. I'm too fun I'm So I'm a member of the next West foundation board, but I'm not going to talk about that today Because I'm also a nice maintainers
00:22
I have been a nice maintainer for the last few months and I'm gonna talk about the the packet the next package manager team Officially called the next team, but it's very confusing. So I won't call it like that at least not in the first slide Which is a team that has spawned a few months ago To actually expand a bit on the the maintenance of Nix. The the reason why we created this team is that
00:47
Nix in in some ways had a bad reputation People both for contributors and for users with this idea that the maintenance of Nix didn't care about
01:00
contributors and Users, which is not great for an well, it's not great for a product and it's not great for an open source product So to be fair That's not true Slightly less wrong amended version would be that the single Nix maintainer Doesn't have the bandwidth to care about all that because it's one guy
01:21
Hacking on a fairly big project as for a bit of history If you don't know Nix originally that was a PhD project that Elko dolstra started And then it has grown up Progressively the community has expanded But until recently that was still only maintained by Elko dolstra as one person
01:45
Which was definitely not enough for something of that size But like regardless of the fact that like yeah, it's not someone not caring about anything It's really more of an organizational problem
02:00
Well that that's not great That's not great for a number of reasons so as I said Contributors didn't really feel like they were as appreciated as they should have been A lot of pull requests stayed unanswered for a long time or forever Even when they were answered
02:22
There was no guarantee that you could get an actual Yes, or no answer to is this pull request something that's going to be matched eventually There was also no way for contributors to know before the fact whether What they were doing could fit the next time guide because there was one and there still is one
02:45
But only one person knew it and so as a contributor that was Really, it was really tough to contribute to Nix because of all this kind of little things and And
03:00
Another another problem, which is more something for the users is That the release process of Nix was a bit chaotic So the The 2.3 release of Nix was somewhere around 2019-18 anyway don't remember the exact dates
03:21
But then Nix went for two years without a release and when it when the 2.4 release came I just made some quick quick stats and I'm gonna show up the interesting numbers. So Nix was 55,000 lines of C++ at that time Well only counting the implementation files and the diff between the 2.3 and 2.4 release was
03:44
32 new lines or lines changed. So essentially half of Nix had been rewritten between the two releases and Yeah, as you can guess that came with a number of breakages Honestly very few breakages given the size of the diff, but still way enough to annoy users
04:04
a lot of regardless of the breakages new features had been introduced but not necessarily properly documented or Experimental features had been added and were like officially experimental, but that was not properly communicated
04:20
So people started relying on them as if they were actually Stuff that you could use for production and then they got changed because they were just experimental and people got angry about that I mean rightfully from their point of view Unrightfully from the point of view of the maintenance who applied for which these were experimental. You should just not rely on them
04:43
But anyway that led to a lot of frustration and that has been going on for for a long time I think we could say that Nix has starting being really a Big enough project since the early 2010s
05:01
And it has kept growing ever since so the frustration has kept growing with that and so in 2018 a group of people just sat together and decided that this just couldn't keep going and so they decided to create the Nix core team whose
05:20
responsibility was exactly This like improve the maintenance of Nix make it clear what design decisions are where the project is going what? Contributors can expect from them and what we expect from contributions so that we could smooth and all that and so this core team was a great idea, but
05:44
At least to me at that time. I wasn't that much involved in Nix at that time I was only a user and very casual contributor the main contribution for the Nix core team that I saw was one one year later Another announcement that the core team was disbanded because it hadn't done anything meaningful in that year
06:06
And so yeah, that was a unfortunately a bit of a failure Now And well then in the meantime I've came to contribute more and more to Nix and
06:22
I also it also happened that Elko and I got colleagues for some time Which really helps moving things a lot for my contributions because I could just ping him and say hey, I'll go I have this for request. Can you please review it? And that was really helpful, but unfortunately
06:41
Well, maybe fortunately I guess that depends but not everyone got to be Elko's colleague so that didn't really that model couldn't really scale and Yeah, at some point we thought well we we need to be to do something and maybe try again what that Nix core team
07:01
Well didn't manage to do but then the big question is well How can we succeed if that core team failed and like for the record that team If you look at the core others here These were the the biggest members of the Nix community at that time That was not just a group of random people showing up and saying hey, we want to maintain Nix
07:23
That that was really a big community effort So how how could we make something that works and why maybe could we make it work now? And so I Maybe expand a bit later on the how the first question is where the first thing is that
07:41
the circumstances were actually very different and If you look so I dig the bit in the GitHub history stats though this is very blurry, but that's where you won't be tempted to read all the numbers and you will just The big shape of the graph. So this is the this are the first the main
08:02
Committers to Nix between 2003 and 2020 so that graph represents Elko's contributions and this very very very very tiny graph that Yeah, I can see that when it's not smaller than the pixels on the screen, but barely So these are the other main contributors
08:21
So as you can see Well, that's really a one-person project with some contributors If we look between 2021 and 2023 and I'm probably getting out of the camera field so I'm going to stay where I am Well, you can see that well, Elko is still the main contributor. That's no question about that But then things are much more well distributed after that
08:44
So that means that there's actually people who potentially Are already investing enough time in Nix and could really take on maintenance ship and Another key ingredient that also probably was missing at the time is that
09:02
between 2018 and today Well, the next community has kept growing and we have more and more industrial Players in that community and more and more people who are paid to work on Nix Like well Elko has the chance of
09:22
Always having been paid to work on Nix more or less first as a PhD student then getting hired by a company which gave him a lot of time to hack on it than another and so on but For a long time. He was mostly the only one at least for Nix itself But yeah, now things are changing. We're kind of growing up
09:43
we're a real community of people actually like handling money and having people who have time to do this kind of Boring and pay for things of all reviewing pull requests of code that Someone else wrote and obviously I didn't write it myself. So it's not good and I don't want to read it and
10:05
But now we can leverage that and so that's what we did So we gathered a group of people we sat together with Elko and the other well people who would be like who are now team members of that team and
10:21
Everyone agreed that this was a great idea. We had a lot of disagreements on the tiny details because obviously that's how things go But eventually we found that team we announced it officially saying hey Yeah, everyone agrees that there's problems. So we're just going to create a team to try and solve these problems
10:42
We set a simple but Ambitious goal for the team which was to basically take ownership of that Nix source code and lead it to both To better to higher standards both for contributors and for the end user so that contributors could know
11:03
That their contributions would be taken into account Oh, no that they wouldn't in case these weren't things that we thought should land into Nix Actually, they should have clarity and the users should know that Nix was something that they could rely on that was robust that fixes were merged in time that they had clear expectations and the support they could get and
11:30
Because Nix starts to be reasonably big we also thought that a single team of four or five people
11:41
Well, that's already better than a team of one person But that's still not enough really to manage the the size of of the next card base well, not so much the size the code base is not that big but the The vitality and the amount of pull requests and issues that come in so we decided that the first mean by which we would want to take ownership of that code was to actually
12:08
enable more maintainers and contributors by writing some clear contributing guides so that people knew what to expect by trying and grow more maintainers out of the existing contributors base so that
12:24
we would not be a bottleneck for someone having his pull request answered and And as part of that stewardship also we Oh
12:42
I forgot a bit of history So this this was the main the main step towards that but actually someone to Decided to buy the bullet on another topic some month earlier and Yeah, same thing people sat down together and decided well the next release process is not great we need to improve that
13:04
Well, let's just have a fixed schedule one release every six weeks that's arbitrary Probably half of this rule would say well, this is about this is a bad way of doing releases You could just send her another half would say well This is a bad way of doing things just do leave hat head and the other half might agree
13:24
But anyway, everyone agreed that having that was better than nothing. Oh, and I'm talking too much But then this release schedule also was something that was still in Elko's hands and that Like mean doing a release is not a big deal
13:41
But you have to think about it and if you're the only one who's tasked every six weeks to do that. Well Certainly, please don't go on vacation that the date of the release because otherwise you're going to annoy everything and Please don't forget about it because then you have you're gonna have 20 people yelling at you because you're three days late so that team was also
14:00
a way to to share a bit this responsibility and Well, the big question then is did we succeed and I hope so. I really really hope so and
14:22
So I don't think I gave you the date already, but we started all that last September which means which means that we are six months in and Six months and it's it's still a bit tough to see the main the main indicator I have to think that probably we've succeeded is that we've turned the next
14:43
commit tree into something that starts to look like a plausible metro map which is always a good thing when you're maintaining an open source project and But in addition to that for some less tangible factors, I think that the team is going reasonably well
15:03
and Nyx is starting to move forward at a more sustainable and pre visible pace Which makes me very very hopeful for the future and Just adding two minutes to expand a bit of that on that because this
15:20
Nyx team is not an isolated phenomenon it's part of and a growing movement within the next community to try and structure the The fundamentals of the ecosystem Starting with something that domain briefly mentioned which was the Nyx OS foundation board Which got expanded a few months ago to try and be more proactive within within the community and expand
15:45
On his previous role, which was mostly pay for the infrastructure And this also materialized with the emergence of a lot of different teams the marketing team which Was created a bit more than a year ago then a team dedicated on the documentation
16:04
Which I think nearly every speaker here at some point say that the documentation is a problem for Nyx so everyone agrees that this is a problem another team that got created around main the maintenance of Nyx packages, which is like Correct me, but I think that's the seventh more active github repo
16:23
And well when you have something of that side you'd better not just it not have it just be Something informally made maintained by whoever happens to pass by and want to make some changes And so all that is like part of a big community movement towards organizing these kind of things
16:42
Which makes me very happy and very hopeful for the future of the minix community in general And since I want to leave some time for questions I'm gonna stop right here And if you want to know more after that we can can always have a chat or contact me wherever
17:02
Thank you
17:23
That's a big question it's all the question was yeah Okay the question was when if ever will flakes be the default With a commenter by someone who might not name who say that it was in the embargo, and I couldn't answer that
17:40
and So the answer well one answer is when it's going to be ready and actually so Huge part of the next team discussions were about refining the semantics of flakes To make sure that all the little corners that we wanted to have cut before actually making it stable were cut and
18:05
So I would I would definitely wouldn't advance a date or a time But it has been it's progressing which doesn't answer your question I know
18:36
Okay, so the question was how we were considering projects like TVX which is a
18:43
implementation of Nix and whether we wanted to engage with them or whether we were already engaging with them So I think I could that in my screenshot here but actually one of the explicit goals of the team was to actually engage with Any third party that like that could be interesting to engage with that obviously included the TVX guys
19:04
Which we didn't do because Well people never do what they say you know that And that was which is definitely something that we want to do For like in the case of TVX in particular for two reasons one of them being that
19:21
TVX was really born out of the very frustrations. We talked to at the beginning so it would be interesting to have their feedback on whether they feel that things are moving in the right direction and also because Well let's be honest having another implementation of Nix Which is kind of a concurrent is it's a real of the pain like these people are doing what we want to do
19:44
but it's also great because They are like the TVX people came every once in a while for example with the opening an issue saying oh There's this very very weird piece of semantics in Nix when you trigger this specific in this specific code I don't know whether that's a bug or not
20:02
I'm probably going to re-implement it for TVX because I want to be bug by bug compatible But is this really what you intended? In general the answer is no But my point is that I think we would gain a lot from collaborating a bit more together
20:22
Thank you