NixOS: declarative configuration Linux distribution
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32582 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2014149 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
SummierbarkeitFigurate numberNormal (geometry)RoutingPublic domainRing (mathematics)PlanningQuicksortGoodness of fitUniform resource locatorSoftware testingLecture/Conference
04:18
Electronic mailing listJSON
04:49
Multiplication signConfiguration managementBitHydraulic jumpRevision controlQuicksortEvent horizonData managementLecture/Conference
05:59
EmailBitMultilaterationMachine codeWikiOpen sourceFunctional programmingData managementWebsiteDistribution (mathematics)File systemMaxima and minimaComputer fileFunctional (mathematics)Programming languageFunction (mathematics)Distribution (mathematics)Category of beingEndliche ModelltheorieoutputSet (mathematics)ResultantMultiplication signPattern languageSoftware developerRight angleExpressionClosed setWeb pageSystem call
08:55
State of matterFunction (mathematics)Revision controlFile systemDistribution (mathematics)BuildingFunctional programmingAnalytic continuationDecision theoryPoint (geometry)StapeldateiRun time (program lifecycle phase)Graph coloringFunctional (mathematics)Distribution (mathematics)Sound effectBinary codeDirectory serviceoutputRight angleSoftwareRule of inferencePatch (Unix)Service (economics)CASE <Informatik>Configuration managementWordLevel (video gaming)Data storage deviceProgramming languageMathematicsSoftware testingMachine codeMultiplication signBinary fileError messageInterpreter (computing)Term (mathematics)Lecture/ConferenceSource code
12:21
SoftwareRight angleSource codeMaxima and minimaArithmetic meanoutputResultantSet (mathematics)Multiplication signLocal ringIntegrated development environmentCASE <Informatik>Software testingStandard deviationLecture/Conference
13:34
outputPatch (Unix)TimestampData structurePattern languageBlock (periodic table)Binary codeComputer fileDeterminismOpen sourceFunctional (mathematics)Network topologyRight angleFunction (mathematics)Hash functionQuicksortLecture/Conference
14:37
Distribution (mathematics)Functional (mathematics)outputTheory of relativityComponent-based software engineeringFunction (mathematics)Type theoryIntegrated development environmentHash functionNetwork topologyOpen sourceBinary codeRoutingData storage deviceParameter (computer programming)Multiplication signDescriptive statisticsHessian matrixFitness functionGroup actionRevision controlDerivation (linguistics)Uniform resource locatorEmailLibrary (computing)CryptographyRootPatch (Unix)Meta elementCompilerBuildingScripting languageSoftwareSource code
18:02
Integrated development environmentRight angleData storage deviceOpen sourceHash functionSoftwareFunctional (mathematics)Parameter (computer programming)Substitute goodSet (mathematics)Component-based software engineeringLecture/Conference
18:51
Point (geometry)Profil (magazine)User interfaceWeb page
19:21
State of matterExistential quantificationRollback (data management)Electric generatorProfil (magazine)Set (mathematics)Scripting languageSpeicherbereinigungProjective planeWeb pagePoint (geometry)Integrated development environmentDifferent (Kate Ryan album)Right angleRevision controlPhysical systemMultilaterationoutputComputer fileData managementPole (complex analysis)Uniqueness quantificationMultiplicationPower (physics)Form (programming)MathematicsLink (knot theory)Functional (mathematics)Operator (mathematics)Open setCellular automatonHierarchyBinary fileVirtual realityLevel (video gaming)VirtualizationSpacetimeLibrary (computing)Data storage deviceAtomic numberLecture/Conference
22:46
Integrated development environmentDistribution (mathematics)MetreWorkstation <Musikinstrument>Open sourceComputer fileProfil (magazine)Functional (mathematics)User profileElectric generatorSoftware testingPhysical systemFile formatMathematicsLine (geometry)Parameter (computer programming)Machine codeoutputFlagDemonCASE <Informatik>Programming languageDirectory serviceConfiguration managementState of matterScripting languageControl flow graphVirtual machineComputer configurationData storage deviceNatural languageRight angleBulletin board systemRevision controlStudent's t-testModule (mathematics)Theory of relativityFood energyService (economics)Solid geometryPosition operatorExecution unitEndliche ModelltheorieDeclarative programmingCuboidFile systemBootingLecture/ConferenceComputer animation
27:49
Configuration managementCuboidVideo gameWebsiteLecture/ConferenceComputer animation
28:16
2 (number)Web pageLecture/Conference
28:50
Software testingOrder (biology)Multiplication signIntegrated development environmentMereologyPhysicalismConfiguration managementFlow separationFront and back endsProduct (business)Functional (mathematics)Spring (hydrology)Virtual machineSlide ruleMessage passingArithmetic meanCohesion (computer science)Fiber bundleVirtualizationDoubling the cubeDiffuser (automotive)CuboidComputer fileRevision controlArmWebsitePoint cloudComputer animationLecture/ConferenceSource codeJSON
30:29
Spring (hydrology)Block (periodic table)Figurate numberBitRight angleNumberGodProduct (business)FingerprintComputer animationLecture/ConferenceJSONXMLUML
Transcript: English(auto-generated)
00:37
Is Domen in the room, by chance? Our next speaker, Domen? No? For those leaving also,
01:35
there's two more talks and after that at 7 o'clock we're meeting outside of the room here to go and grab dinner, drinks or whatever it is that we're planning on doing.
01:44
But if you want to join, feel free to meet up here at 7 o'clock just outside the room.
02:27
So, is Domen in the room? Ah, there he is. Speaker located. We have some empty seats
03:48
there in the middle. If you guys could move over to the middle so we fill up all the empty seats. I see an empty seat over there, two empty seats in the back. We have an empty seat
04:01
over here. Can you turn it off? Test, test. All good. Plug you in. Should we start? Alright,
04:58
so the second to last talk, Domen is going to talk about NixOS. I think it's a very
05:04
interesting topic, a bit more exotic than most of the topics that we've had today. Before he starts, I wanted to quickly let everyone know that there is a config management camp in Ghent two days after FOSDEM, so it's on Monday and Tuesday. Unfortunately, it's totally full,
05:21
so for this year you're out of luck if you don't have a seat reserves already. But it's going to be a great conference, a two-day conference with basically all the configuration management tools. All the community is going to be there. As someone said today, it'll probably be the only time or one of the very few times that you'll see both the
05:46
people who started CF Engine, Puppet and Chef, talk at the same conference, all keynoting. So yeah, it's going to be great. For now, Domen is going to talk about NixOS. I'm curious to see what he has to say, so take it away. Alright, let's
06:01
get started. So a little bit about my background. I'm a Python and JavaScript developer, and I've used Gentoo for about eight years, and I've participated in Google Summer of Code trying to auto-generate Python packages. And I'm a NixOS user for about a year now, and this will become relevant later. I have no prior experience with functional languages.
06:25
So before we go and talk about Linux distribution, we have to rethink the package management. So this is where Nix comes in. It's a standalone package manager, so you can install it
06:41
basically onto any distribution. It's a DSL. It's lazy-evaluated. That's a very important feature and property that you will see later. Untyped and purely functional language. Because there is a little bit of Nix the package manager and Nix the language, we call Nix
07:05
expressions, the files and the language that we write. And Nix is the package manager. It tries to be minimal, to have minimal dependencies, so it's really easy to compile it from source if you want to do so. It's portable, as I said. I don't know if that's
07:24
the first release, but that's the first one listed on the website in 2004. It's written by Elko Dosto, and there's dozens of scientific papers published on the Nix side that are the theoretic base for the Nix and what's built on top of it.
07:45
So we're thinking the package management and if you think about how currently our distributions work, they would go into the file system and try to find headers there in a very funny
08:01
way because it breaks a lot of times, right? They will go into directly user include and try to find the correct headers, and that will break when you update some of the packages. So there is a set of problems that are called dependency hell, and there is a wiki page
08:22
which lists all of those problems that arise from dependency hell that we have today. And our motivation is to have really reproducible builds, right? So if I copy the instructions to another machine and run it, we should get the exact same result. So the idea behind Nix is apply functional language model to the packaging, right? So
08:47
a package is actually an output of some function, and all of those functions, all of those inputs to the functions are the instructions how this package is affected.
09:00
So this is, for example, we have a slash Nix slash store with all the packages that you have in your Linux distributions are stored, and there is a SHA-1 prefixed and then the name and the version, and inside this directory, this is basically the prefix where you install the packages. So inside you have directories like bin, lib, share,
09:24
and so on. Nix store is immutable. It's mounted as read-only, so you cannot modify it. So the idea is that when it's actually the package is built and stored inside the Nix store, you cannot modify it anymore. That's the way we ensure that it is really
09:40
an output of the function that you specified in your Nix package. So for the Nix to work, there is a term called purity, and that basically means that we are sure that the output is affected only by the inputs of the packaging function.
10:03
So nothing else should affect this package, right? And at the language level, this is solved by the way that we are using a functional language, so there is no state, no side effects. But on the level of a Linux distribution, we have to take a lot of steps to make sure, and some of them can't even be solved, and some of them are ongoing effort, right?
10:24
So for example, there is no user, there is no bin, so there is no global folders on your Linux distribution, because we don't want that to affect those packages that we prefix into the Nix store, right? There is also, we build packages in change root,
10:44
so that we are sure that they don't have access to the file system. So the store is immutable. So for example, one of the impurities is a lot of times code will have hard-coded bin bash, right? So that's again global. We want to patch
11:03
it so it points out to the exactly bash version that we are building this package, and the bash is a dependency, right? And then network, for example, we only allow network access when you download software, and when you actually test and do other stuff, and
11:23
build and so on, there is no network configured, right? We have patch-elf that will patch proprietary software binaries, and it will set the interpreter and the error path, so we can also make sure that the binary software points to the
11:44
correct Nix store packages. And there is this quote for the documentation that will tell you how actually we figure out what are the build dependencies and the runtime dependencies. So actually the output is searched for the Nix path output stores,
12:05
and then we know on which packages of all the build inputs the package depends on the runtime, right? You will think that this is probably wrong, but it works very well. There are zero problems that I've seen at least in the practice with this approach.
12:23
And this is of course possible because of this idea of purity. So there are also impurities. For example, network, a lot of times people will write tests for their software that will need network access, so in those cases we just disable those tests because there
12:42
is no way to ensure that the network is a result of the input instructions to the build of the package, right? It's something that is not deterministic. And for example, sometimes it will happen that the parallel builds of something will fail because the source
13:04
code needs to be fixed, right? That's also something that Nix cannot fix. So we have this concept of standard environment, which are the very minimal set of tools like the GCC and GNU set and so on that you need always when you package software. That's
13:24
the minimal set of packages that you need. And on OSX we actually use the OSX tools because it's more convenient and there is actually a ticket now to improve this in the future.
13:41
So there is also this idea to have a deterministic binary, so you will see about this hash, what it means when we prefix the package name. But basically this is the hash of all the inputs to the function that instructs how to package this software, right?
14:01
And the source is not the input, and also the binary blob is not the input. But we could extend this if we actually remove the impurities in the binary output. Sometimes people would store timestamps in the binary blob. For example, in Python
14:27
we have Python compiled files and we patch Python to not include the timestamps. But this is an ongoing effort. We don't have deterministic binaries yet.
14:41
And of course, like I said before, we patch software that has a character-coded path to the root. So this is, I don't know if you actually see, but this is an example of an Nginx, how we package it. So at the top, this is a function, and the parameters to the functions
15:02
are the standard environment, the two fetch functions, and then the dependencies that we will pass on. And the main derivation on the standard environment is the function that does the heavy lifting. So that's actually the function that will do all these phases and execute and package and store it into the Nginx store.
15:28
So as I've said before, if you remember, there was this hash before the name and the version. And actually the hash is calculated from all of those inputs to this function, except from the meta parameter, which is irrelevant for the output.
15:45
Because if you change a description of GCC, it shouldn't recompile the whole tree of what depends on GCC. So inside there you see a fetch URL function, and that's another way,
16:02
another type of how Nginx does the hashing, and it's called fixed output. So we know the hash, which is the SHA up front, right? So we don't need to actually hash all the inputs, because we already know what is the crypto output,
16:24
the crypt hash of the package. And then the fetch URL will actually, the output of the fetch URL function is actually the path to the Nginx store where this tarball is actually downloaded. And if you can imagine, the build inputs specifies the dependencies,
16:42
and those provide headers and libs and so on. So if any of those will change, the inputs also, the Nginx hash will change, so it will go and recompile and it will get the new prefix.
17:04
So I just put there configure flex and so on, to see that we have, this function accepts quite a lot of parameters that you can affect. But there are two more that I would mention. One is builder, which is actually the bash script that takes care of all the building.
17:22
And in the bash script we have phases, like you normally use build phase and install phase. And you can actually override here at this function those phases, or add a pre-phase or post-phase and so on. So when purity actually works, so that means that we've removed all the impurities,
17:43
we can actually have a distribution that has source and binary packages. So if you think about it, we have this hash that actually uniquely identifies all the inputs to this package. And if we have a build farm, which we do for NixOS, and we build a package on the build farm,
18:04
then you could use this as a replacement for the build function. So before Nix actually builds something, it will go and try to find a substitute, so something that already provides a binary package for this hash. So if I go and install Nginx now, it will download the binary blob,
18:25
but if I go and modify any of the parameters to the Nginx, then it will go and compile it from source, because the hash will change. So then comes the question of how do we get an actual environment?
18:42
We have this package software stored in prefix in NixStore, and how do we actually get to the environment? So there you see a package called userenv, and that's where all the packages will link into when you install them.
19:05
And from that point on, the symlink from the Nix profiles will point to the user environment. And as you can see, we keep a history of what you change.
19:20
So if you uninstall a package, we keep a history, because the Nix design brings us this benefit that we can actually have a history, and Nix provides rollbacks, so you can actually rollback. If you install a package and something is broken or you update, you can rollback to a previous state.
19:44
And then those profiles, generations that we call them, a history points to the Nix profile, and every user can have a different profile, or actually you can even generate a profile on demand for your project. So this actually replaces user space tools,
20:02
like in Python we have virtual environments and so on, and this is like a package management level virtual environment, so you can actually put a set of packages that are only specific for your project. And if you upgrade any other profile or install a package in another profile,
20:22
this will not affect the other profiles, so they are safe. And then, if you think about it, Nix is also atomic, so that means in the middle of an upgrade or installing, if it shuts down, if your power shuts down,
20:44
the changes won't be affected, because the last step is that this symlink to the profile is meant, and symlink is in Unix atomic by itself, so that's when everything is built and configured and put into the Nix store, when that switch is done, then actually you have the new environment activated.
21:06
Then when you think about garbage collection and how we uninstall packages, when you actually uninstall a package, it will just remove the symlink from your profile, it will not actually go and remove it from the store.
21:21
So that means that we need the concept of garbage collection, and it will collect those packages that are no longer referenced anywhere. So you could, because imagine if you're changing the inputs to different Firefox builds, then you could have multiple Firefoxes in your Nix store,
21:41
but maybe you don't have them installed in your profile, so then the garbage collection will say, okay, it's not referenced by anything else, let's delete it. So of course the cost here is file system hierarchy standard,
22:05
we still have the bin and lib and so on, but we don't have the user folders. And this also solves the dependency hell, so you can have different packages that depend on different versions of OpenSSL,
22:22
it doesn't matter, they're stored in the Nix store, and they're just linked to different versions of OpenSSL. So this is a very bad way to advertise how to get started with Nix, but it's a really cool way to express it in three lines.
22:41
I don't recommend to actually run it, so download the script first and check out the contents, but then you source into the Nix environment, and you can actually go and install packages, and you can do this on whatever Linux distribution, or S6 or FreeBSD if you want.
23:03
And if you want to uninstall it, just remove the slash Nix and all the profiles that it created, which is normally just your user profile. So NixOS is basically a generalization of the Nix package manager, and if you think about it, we don't have to do that much to get a Linux distribution on top of that.
23:24
So if we store packages in the Nix store, we can also store the configuration files, that's actually a simplification of a package. So Nix has this system profile, which is the NixOS profile. So when you upgrade NixOS,
23:41
you will upgrade the system profile and so on. And of course, then we need some configuration for the system, and this is the declarative configuration that you will see. And if you think about how NixOS works, it's different to Puppet, for example. If you're saying Puppet ensures packages install, it will go into the machine and inspect if it's installed,
24:03
and if it's not, then it will install it. So that's stateful, that's code that needs to work, and it needs to cover the edge cases. But with Nix, this comes out of the box, out of the design of the language and the system that we're using. So this is an example of configuration of a system,
24:24
and you declare a grub device and file system, and then enable the SSH daemon. And here I show, for example, how to configure Moon in a few lines of code. So if you want to install it, basically you create a partition,
24:42
you format it, mount it, generate a config, edit, NixOS, install, reboot, and you have your system with whatever you put in your configuration. And on the grub menu, you will see a history of your profiles, like the profile generation. So if something goes wrong when the system boots,
25:03
you can actually go and take the old profile that actually worked. So just to quickly go how to write the NixOS module, this is the package.writeText is the function that will actually store the file into the Nix store.
25:22
So this is like a complement to the makeDerivation function. And it's lazy-evaluated, and this is not calls. As you see, we already referenced the cfg.config, which is not defined, so that will get evaluated actually when it's referenced. And this is how you just define what options
25:42
the NixOS module provides, and here you say nginx enable, which package we will use, and what config will be used for the nginx. And in NixOS, we use systemd, because it really suits the thinking and the way Nix works.
26:02
So there we have a pre-start script that will start everything, configure everything, and then the actual execution. So there, when you see nginx, that's actually the path to the prefix of the nginx where it's packaged, and then it will pass the config file and the state directory.
26:23
And at the top, you see that it makes nginx available, installed on the system. And then you would use it something like this. So just enable nginx. And what we do actually here is, this is some kind of similar to the use flags in Gentoo.
26:45
You can have optional behavior, and the nginx is actually here, a function that we saw at the beginning, and we can pass parameters to it, right? So we could have a parameter, enable mp,
27:03
and if you turn it to true, then it would actually download the module and configure it and build nginx with it. But that means that you won't use the binary package anymore because you changed the inputs to the function, right? And when you make changes to your configuration file,
27:24
you run nyxos rebuild switch, and that will activate the changes. There is also nyxos rebuild test that will actually activate it, but on the next reboot, it will go to the previous generation, so it will not actually persist.
27:44
And there's also build VM, so it will actually build a virtual machine, like a virtual box, and it will configure everything in the site so you can actually see if it's working right. So this is the build farm, all the packages are built.
28:29
Is it? Okay. So we have packages stored on GitHub, and 300 seconds or something is the interval,
28:41
and it will pull the changes and build the packages and provide the binary packages for you, right? So here is the history of those builds, and you can see how many new packages succeed or how many fail. And we also have tests, so in order that nyxos channel is actually updated,
29:03
all those tests need to pass to make sure that at least something is working. And then as a last step, we have NixOps, and this is like deploying NixOS to the cloud and provisioning. So we have currently some backends,
29:23
four backends implemented, but of course more could be added. And this is a quick introduction. So install it with Nix, and then you define a machine, which is a function of a config and packages, and you, for example, here we configure Apache,
29:41
and NixOps has this separation of the physical and logical part. So the physical is, for example, this time we will deploy to VirtualBox, and we just say the target environment is VirtualBox. And then we say create, pass those two files, and name it trivial, and then deploy, and this will actually create a VirtualBox machine,
30:03
configure everything, and fire it up, and you have activated VirtualBox and configured with the packages. I have two or three more slides. So then you could have like a production version,
30:21
which would deploy to actual machine on some IP, and you deploy it the same way, just provide a different name. And we had a NixOS spring in Slovenia this summer, and I hope, last summer, and I hope we will have it this year again. It was supported by LogicBlocks and Shadowflow.net.
30:42
And that's it. Thank you very much. Domen, what was your name? Domen, was that your name?
31:00
Domen, thank you very much. I have one quick question. Is anyone using NixOS in production? Do you know? Yeah. Oh, we have one. Okay. That was the main question that I had. It looks quite interesting. I'm going to check it out a little bit more.
31:21
We have one more talk left, and then it's... You can do a quick question. Yeah, go ahead.
31:52
All right, cool. All right.