We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

The Nix package manager development process

00:00

Formal Metadata

Title
The Nix package manager development process
Title of Series
Number of Parts
542
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Present the newly created Nix team, and give a glimpse at the Nix development process. Nix has for a long time been struggling with a very low (~1) bus factor and a lack of clear development process, leading to some frustration from both users and contributors. To remedy that, we've recently created the Nix team with the goal of collectively “tak(ing) ownership of the Nix source code”. This talk presents that Nix team, and gives a glimpse at the Nix development process.
14
15
43
87
Thumbnail
26:29
146
Thumbnail
18:05
199
207
Thumbnail
22:17
264
278
Thumbnail
30:52
293
Thumbnail
15:53
341
Thumbnail
31:01
354
359
410
Revision controlProduct (business)MathematicsSoftware maintenanceLine (geometry)Projektiver ModulNumberOpen sourceFrustrationMultiplication signProcess (computing)StatisticsElectronic program guideBitView (database)Projective planeBand matrixPoint (geometry)Data managementSingle-precision floating-point formatDifferenz <Mathematik>Slide ruleComputer fileImplementationComputer animation
Core dumpDistribution (mathematics)Local GroupTime evolutionFormal grammarProcess (computing)Function (mathematics)Software bugTask (computing)Vector potentialGroup actionGraph (mathematics)Multiplication signSoftware maintenanceDependent and independent variablesCodeStudent's t-testCore dumpProjective planeField (computer science)Source codePixelDecision theoryEndliche ModelltheorieBitShape (magazine)NumberRandomizationRow (database)Point (geometry)Scaling (geometry)Computer animation
System programmingElectric currentWhiteboardState observerArchitectureDependent and independent variablesBitProcess (computing)Scheduling (computing)Disk read-and-write headRule of inferenceCASE <Informatik>Standard deviationAdditionLevel (video gaming)Network topologyOpen sourcePrice indexWhiteboardProjective planeMultiplication signArithmetic meanDifferent (Kate Ryan album)CodeExpandierender GraphPlastikkarteDomain nameExpected valueFundamental theorem of algebraMereologyMathematicsSoftware repositoryElectronic program guideSoftware maintenancePoint (geometry)
ArchitectureWeb pageMultiplication signDefault (computer science)MereologySemantics (computer science)Projective planeImplementation
Normed vector spaceCASE <Informatik>Direction (geometry)ImplementationSemantics (computer science)CodeSoftware bugPoint (geometry)Bit
Program flowchart
Transcript: English(auto-generated)
Hey everyone, and so yes Brian introduced me. I'm too fun I'm So I'm a member of the next West foundation board, but I'm not going to talk about that today Because I'm also a nice maintainers
I have been a nice maintainer for the last few months and I'm gonna talk about the the packet the next package manager team Officially called the next team, but it's very confusing. So I won't call it like that at least not in the first slide Which is a team that has spawned a few months ago To actually expand a bit on the the maintenance of Nix. The the reason why we created this team is that
Nix in in some ways had a bad reputation People both for contributors and for users with this idea that the maintenance of Nix didn't care about
contributors and Users, which is not great for an well, it's not great for a product and it's not great for an open source product So to be fair That's not true Slightly less wrong amended version would be that the single Nix maintainer Doesn't have the bandwidth to care about all that because it's one guy
Hacking on a fairly big project as for a bit of history If you don't know Nix originally that was a PhD project that Elko dolstra started And then it has grown up Progressively the community has expanded But until recently that was still only maintained by Elko dolstra as one person
Which was definitely not enough for something of that size But like regardless of the fact that like yeah, it's not someone not caring about anything It's really more of an organizational problem
Well that that's not great That's not great for a number of reasons so as I said Contributors didn't really feel like they were as appreciated as they should have been A lot of pull requests stayed unanswered for a long time or forever Even when they were answered
There was no guarantee that you could get an actual Yes, or no answer to is this pull request something that's going to be matched eventually There was also no way for contributors to know before the fact whether What they were doing could fit the next time guide because there was one and there still is one
But only one person knew it and so as a contributor that was Really, it was really tough to contribute to Nix because of all this kind of little things and And
Another another problem, which is more something for the users is That the release process of Nix was a bit chaotic So the The 2.3 release of Nix was somewhere around 2019-18 anyway don't remember the exact dates
But then Nix went for two years without a release and when it when the 2.4 release came I just made some quick quick stats and I'm gonna show up the interesting numbers. So Nix was 55,000 lines of C++ at that time Well only counting the implementation files and the diff between the 2.3 and 2.4 release was
32 new lines or lines changed. So essentially half of Nix had been rewritten between the two releases and Yeah, as you can guess that came with a number of breakages Honestly very few breakages given the size of the diff, but still way enough to annoy users
a lot of regardless of the breakages new features had been introduced but not necessarily properly documented or Experimental features had been added and were like officially experimental, but that was not properly communicated
So people started relying on them as if they were actually Stuff that you could use for production and then they got changed because they were just experimental and people got angry about that I mean rightfully from their point of view Unrightfully from the point of view of the maintenance who applied for which these were experimental. You should just not rely on them
But anyway that led to a lot of frustration and that has been going on for for a long time I think we could say that Nix has starting being really a Big enough project since the early 2010s
And it has kept growing ever since so the frustration has kept growing with that and so in 2018 a group of people just sat together and decided that this just couldn't keep going and so they decided to create the Nix core team whose
responsibility was exactly This like improve the maintenance of Nix make it clear what design decisions are where the project is going what? Contributors can expect from them and what we expect from contributions so that we could smooth and all that and so this core team was a great idea, but
At least to me at that time. I wasn't that much involved in Nix at that time I was only a user and very casual contributor the main contribution for the Nix core team that I saw was one one year later Another announcement that the core team was disbanded because it hadn't done anything meaningful in that year
And so yeah, that was a unfortunately a bit of a failure Now And well then in the meantime I've came to contribute more and more to Nix and
I also it also happened that Elko and I got colleagues for some time Which really helps moving things a lot for my contributions because I could just ping him and say hey, I'll go I have this for request. Can you please review it? And that was really helpful, but unfortunately
Well, maybe fortunately I guess that depends but not everyone got to be Elko's colleague so that didn't really that model couldn't really scale and Yeah, at some point we thought well we we need to be to do something and maybe try again what that Nix core team
Well didn't manage to do but then the big question is well How can we succeed if that core team failed and like for the record that team If you look at the core others here These were the the biggest members of the Nix community at that time That was not just a group of random people showing up and saying hey, we want to maintain Nix
That that was really a big community effort So how how could we make something that works and why maybe could we make it work now? And so I Maybe expand a bit later on the how the first question is where the first thing is that
the circumstances were actually very different and If you look so I dig the bit in the GitHub history stats though this is very blurry, but that's where you won't be tempted to read all the numbers and you will just The big shape of the graph. So this is the this are the first the main
Committers to Nix between 2003 and 2020 so that graph represents Elko's contributions and this very very very very tiny graph that Yeah, I can see that when it's not smaller than the pixels on the screen, but barely So these are the other main contributors
So as you can see Well, that's really a one-person project with some contributors If we look between 2021 and 2023 and I'm probably getting out of the camera field so I'm going to stay where I am Well, you can see that well, Elko is still the main contributor. That's no question about that But then things are much more well distributed after that
So that means that there's actually people who potentially Are already investing enough time in Nix and could really take on maintenance ship and Another key ingredient that also probably was missing at the time is that
between 2018 and today Well, the next community has kept growing and we have more and more industrial Players in that community and more and more people who are paid to work on Nix Like well Elko has the chance of
Always having been paid to work on Nix more or less first as a PhD student then getting hired by a company which gave him a lot of time to hack on it than another and so on but For a long time. He was mostly the only one at least for Nix itself But yeah, now things are changing. We're kind of growing up
we're a real community of people actually like handling money and having people who have time to do this kind of Boring and pay for things of all reviewing pull requests of code that Someone else wrote and obviously I didn't write it myself. So it's not good and I don't want to read it and
But now we can leverage that and so that's what we did So we gathered a group of people we sat together with Elko and the other well people who would be like who are now team members of that team and
Everyone agreed that this was a great idea. We had a lot of disagreements on the tiny details because obviously that's how things go But eventually we found that team we announced it officially saying hey Yeah, everyone agrees that there's problems. So we're just going to create a team to try and solve these problems
We set a simple but Ambitious goal for the team which was to basically take ownership of that Nix source code and lead it to both To better to higher standards both for contributors and for the end user so that contributors could know
That their contributions would be taken into account Oh, no that they wouldn't in case these weren't things that we thought should land into Nix Actually, they should have clarity and the users should know that Nix was something that they could rely on that was robust that fixes were merged in time that they had clear expectations and the support they could get and
Because Nix starts to be reasonably big we also thought that a single team of four or five people
Well, that's already better than a team of one person But that's still not enough really to manage the the size of of the next card base well, not so much the size the code base is not that big but the The vitality and the amount of pull requests and issues that come in so we decided that the first mean by which we would want to take ownership of that code was to actually
enable more maintainers and contributors by writing some clear contributing guides so that people knew what to expect by trying and grow more maintainers out of the existing contributors base so that
we would not be a bottleneck for someone having his pull request answered and And as part of that stewardship also we Oh
I forgot a bit of history So this this was the main the main step towards that but actually someone to Decided to buy the bullet on another topic some month earlier and Yeah, same thing people sat down together and decided well the next release process is not great we need to improve that
Well, let's just have a fixed schedule one release every six weeks that's arbitrary Probably half of this rule would say well, this is about this is a bad way of doing releases You could just send her another half would say well This is a bad way of doing things just do leave hat head and the other half might agree
But anyway, everyone agreed that having that was better than nothing. Oh, and I'm talking too much But then this release schedule also was something that was still in Elko's hands and that Like mean doing a release is not a big deal
But you have to think about it and if you're the only one who's tasked every six weeks to do that. Well Certainly, please don't go on vacation that the date of the release because otherwise you're going to annoy everything and Please don't forget about it because then you have you're gonna have 20 people yelling at you because you're three days late so that team was also
a way to to share a bit this responsibility and Well, the big question then is did we succeed and I hope so. I really really hope so and
So I don't think I gave you the date already, but we started all that last September which means which means that we are six months in and Six months and it's it's still a bit tough to see the main the main indicator I have to think that probably we've succeeded is that we've turned the next
commit tree into something that starts to look like a plausible metro map which is always a good thing when you're maintaining an open source project and But in addition to that for some less tangible factors, I think that the team is going reasonably well
and Nyx is starting to move forward at a more sustainable and pre visible pace Which makes me very very hopeful for the future and Just adding two minutes to expand a bit of that on that because this
Nyx team is not an isolated phenomenon it's part of and a growing movement within the next community to try and structure the The fundamentals of the ecosystem Starting with something that domain briefly mentioned which was the Nyx OS foundation board Which got expanded a few months ago to try and be more proactive within within the community and expand
On his previous role, which was mostly pay for the infrastructure And this also materialized with the emergence of a lot of different teams the marketing team which Was created a bit more than a year ago then a team dedicated on the documentation
Which I think nearly every speaker here at some point say that the documentation is a problem for Nyx so everyone agrees that this is a problem another team that got created around main the maintenance of Nyx packages, which is like Correct me, but I think that's the seventh more active github repo
And well when you have something of that side you'd better not just it not have it just be Something informally made maintained by whoever happens to pass by and want to make some changes And so all that is like part of a big community movement towards organizing these kind of things
Which makes me very happy and very hopeful for the future of the minix community in general And since I want to leave some time for questions I'm gonna stop right here And if you want to know more after that we can can always have a chat or contact me wherever
Thank you
That's a big question it's all the question was yeah Okay the question was when if ever will flakes be the default With a commenter by someone who might not name who say that it was in the embargo, and I couldn't answer that
and So the answer well one answer is when it's going to be ready and actually so Huge part of the next team discussions were about refining the semantics of flakes To make sure that all the little corners that we wanted to have cut before actually making it stable were cut and
So I would I would definitely wouldn't advance a date or a time But it has been it's progressing which doesn't answer your question I know
Okay, so the question was how we were considering projects like TVX which is a
implementation of Nix and whether we wanted to engage with them or whether we were already engaging with them So I think I could that in my screenshot here but actually one of the explicit goals of the team was to actually engage with Any third party that like that could be interesting to engage with that obviously included the TVX guys
Which we didn't do because Well people never do what they say you know that And that was which is definitely something that we want to do For like in the case of TVX in particular for two reasons one of them being that
TVX was really born out of the very frustrations. We talked to at the beginning so it would be interesting to have their feedback on whether they feel that things are moving in the right direction and also because Well let's be honest having another implementation of Nix Which is kind of a concurrent is it's a real of the pain like these people are doing what we want to do
but it's also great because They are like the TVX people came every once in a while for example with the opening an issue saying oh There's this very very weird piece of semantics in Nix when you trigger this specific in this specific code I don't know whether that's a bug or not
I'm probably going to re-implement it for TVX because I want to be bug by bug compatible But is this really what you intended? In general the answer is no But my point is that I think we would gain a lot from collaborating a bit more together
Thank you