Testing, one, two, three. Good morning, everybody. I gather everyone's awake from the reaction

to Brian's intro there. I'm Linus. I discovered Nix and Nix OS for myself in 2016 and quickly became convinced that that's the way I want to do all my computing. I joined the ranks of Nix packages contributors. Hang on, that's, something's wrong here.

Sorry. There we go. I joined the ranks of Nix packages contributors and have been following the call of Nix in my career since graduating from university. Along the way, I've managed two Nix OS releases and I've been on the RFC steering committee since 2021.

We're looking for new members, by the way, so get in touch if you're interested in helping. I work at determinate systems where we're working on making the Nix the best choice for working together. So what's Nix OS? Nix OS is an operating system

based on the Nix package manager and Linux. Linux is an operating system kernel, which means it runs on the hardware and mediates access to that hardware by the user space, providing abstractions to allow applications and services to care less about what exact hardware they're running on and allocating resources

to allow multiple processes to run at once. I'm gonna talk about how we get from a bootloader to a complete operating system and compare how traditional Linux distros and Nix OS fit the pieces together and then showcase some of the possibilities that Nix OS puts within easy reach.

When did that happen? Is that better? Tell me if that happens again. So when you power on your computer, a number of things happen.

The first that we're interested in is bootloader. It's what's responsible for loading both Linux itself and for giving it the startup parameters needed for getting the user space up and running. Some widespread bootloaders include grub and systemd boot, and especially in the non-desktop world, uboot. These startup parameters are usually the command line

and the initramfs. The command line tells the kernel some configuration that applies before the user space can take care of that, such as logging behavior, how to respond to crashes, or what user space to actually start. And the initramfs is a bit more interesting for our purposes.

The initramfs, which is an abbreviation of initial RAM file system, is where user space starts. It's a tree of files, traditionally stored as a disk image. Nowadays, usually it's an archive. It contains everything that the system needs to boot.

The kernel will run a program, often just called init, from the initramfs as the first user space process. The role of that process is to get everything ready for the real system. Modern desktop kernels are usually modular, meaning that they don't include drivers for every piece of hardware they support,

instead loading them as modules later on. This makes kernels significantly smaller and faster to load, but that also means that drivers like storage, for things like storage devices and file systems, which you need to be able to start the actual system, aren't included in the kernel itself.

That's why it's part of the initramfs's job to include and load these drivers. Other things it does is, if applicable, for example, asking the user for encryption keys or setting up networking in case the user wants to enter the encryption keys via the network

or the root file system is on the network, things like that. Once it's done those things, it'll usually mount the root file system and pass control on to the next init, that of the real system. So the init in a root file system

then starts up the rest of the user space. On a typical desktop system, this includes services like clock synchronization and network management. One of the services is also the login service, which provides login prompts, both on the text consoles and the display manager, which provides graphical login.

So let's look at how a traditional distro like Debian fits these pieces together. The kernel is a package managed by the system package manager. It includes both the kernel image itself, which is what the bootloader loads,

and the modules it needs to support a wide range of hardware. The package includes post-installation scripts, which the other parts of the boot chain can hook into in order to reconfigure themselves once you get a new kernel. The initramfstools package handles generation of the initramfs, installs a kernel post-installation hook

so that whenever you get a new kernel, it'll generate a new initramfs by collecting some information from the system, from config files and such. Config files like etcfs-tab, which defines the file systems that are supposed to be mounted, etc-crypt-tab, which defines encrypted file systems,

things like that. It then copies the contents, well, it puts all these pieces into a temporary directory, so these pieces being kernel modules, init scripts, the dependencies of the init scripts and the configuration files for them into a temporary directory and generates an archive from that temporary directory

which is the initramfs. The bootloader is also a package managed by the system package manager. The package contains both the bootloader itself and, again, scripts for installing it to the hard drive and for generating its configuration. The installation scripts are usually only run when the bootloader is installed or upgraded,

and the configuration scripts, those define the actual boot options, so the kernel command line initramfs combinations, so they also hook into the kernel post-installation scripts and run every time you install a new kernel. And both the initramfs tools and the bootloader

can be configured by a multitude of files in slash NC. Of course, you're all here for Nix and NixOS, so let's have a look at how NixOS does it. One key concept of NixOS is the system package or top level. The entire system configuration is built into a single package which depends on everything the system needs.

I've taken the liberty of illustrating this part visually. So, of course, the kernel is going to be a dependency of the system package. I think someone's seen this video before. The command line is part of the config,

so into the system package it goes. The initramfs, we can build that in Nix, into the system package. Now, the bootloader, we have a slight problem here. The system can only have one bootloader installed at a time.

Let me hand wave away the slight technical inaccuracy of that statement. But we still want to be able to install it, so we have an installation script. And where do we put that? In the system package. As for the bootloader config, we also have a problem there.

We only have one configuration for the bootloader. But we do want to have the ability to have multiple generations of the Nix OS system. That's what we love about Nix OS. But we also don't want our new system

to depend on previous systems. So let's not include a bootloader config as part of the system package. But we do need to generate a bootloader config at some point. So we have a generator, and I think you can guess where we put that. So let's have a look at a real system package.

In a running Nix OS system, there's a symlink that slash runs the current system, which points to the system package that's currently active. This package contains, amongst other things, the following. A symlink to the kernel image itself. Often the bootloader can't load it directly from the root file system

and needs it copied to a boot partition instead. The symlink makes that easy. A text file containing the kernel command line. I don't have the whole one because it's quite long. A symlink to the initramfs. Nix OS uses a historic but now technically inaccurate name, initrd. Like the kernel,

this might have to be copied to a boot partition. A text file containing the Nix OS version string. This is nice for decorative purposes. It's good to have the version in your boot entry name. The init script, this is the root file system one, which the initramfs hands over to.

It does a little setup before starting system D. Nix OS is service manager because system D won't start successfully if its config isn't in place yet. And the switch to configuration script, which takes some steps to apply a Nix OS configuration. This is what Nix OS Rebuild does under the hood.

And what exactly it does depends a bit on the argument given. If you give it switch or test, it'll measure the current state of the system and determine what needs to be done to move to the configuration defined by the system package and then takes those steps in order to activate the system. Another thing the switch action does,

which is also performed by the boot action, is to run the bootloader configuration script we talked about earlier, which looks at other generations of the system as well and generates bootloader config for all of them by looking at the files that are part of that system.

And the switch and boot actions may also install the bootloader, but that only happens when a particular environment variable is set or an argument is passed, which Nix OS installed does so that you have a bootloader, but it doesn't usually happen. So that concludes our whirlwind tour of how a classic desktop Nix OS installation boots.

But what sets Nix OS apart? Besides, of course, declarative configuration, parallel installation of multiple systems, parallel installation of multiple versions of the same package, safe per user package management, rollbacks, easy and fearless patching, reproducibility, and all that is its flexibility. So what if, for example,

we were to just put the entire Nix OS system in an initramfs? The module that implements this is called netboot because that is what it was originally implemented for. Instead of generating an initramfs which has the necessary bits and bobs to mount a local file system,

it generates an initramfs which contains the entire system and all of its dependencies. That's an easy way to boot the whole system from RAM and is really useful for netboot, where a bootloader like ipixie will download the kernel and initramfs by the network, then boot it without having to touch any local persistent storage.

It's quite heavy on RAM usage since the system and all its dependencies are loaded into RAM. The system on my laptop, for example, is 5.4 gigabytes in size, so I might not want to have it in RAM at all times. Of course, while the system doesn't need local persistent storage, it can still make use of it.

For example, a swap partition can be used to allow swapping out unused RAM, such as all the software you're not running right now, and make physical memory available for things that actually need it. Or you can use the initramfs as an installer. You can use it to install mixos to the local storage.

That can be really useful if you want to provision a whole fleet of desktop machines or servers, or if using a USB stick installer is just not enough fun. I've been in both situations. As an aside, nowadays the netboot image does not embed the entire system directly

into the initramfs. Instead, it embeds a squashfs image, which will compress the file system a bit more efficiently than the CPIO archive, and allows keeping it compressed in RAM, unlike the initramfs, which is unpacked during boot. So that's a bit more efficient, but still has the problem of using up all the RAM

as long as the system is loaded. There's also an open-source project by Determinate Systems called nix-netboot-serve. Instead of constructing one big initramfs image ahead of time within the nix derivation, it punts them together from the constituent store paths on demand.

The end result is very similar, but allows much speedier iteration, since each store path is cached separately, and only the parts that change need to be recompressed. So netboot is great and all, but why limit yourself to booting it via ipigsy? Linux has a nifty little piece of functionality

that allows it to act as a bootloader itself, called kexe. The netboot image can be loaded via kexe without any further adjustments. The most common use case I've seen for this is bootstrapping Nix OS support from other Linux distributions on VPS providers

that don't allow supplying a custom disk image or booting from custom ISOs. You build your netboot image on your workstation, I'll sync it to your Ubuntu machine, call kexe with the appropriate parameters, and hey presto, you have a Nix OS system from which you can format the disk and use nix OS install to install the operating system you actually want.

I've also used it in the past instead of an ISO installer to make file system changes which weren't possible in a running system which had the file systems mounted, which saves you the effort of formatting a USB stick and having to boot into that. But another trick with similar applications is switchroot.

So let's go back to our normal initramfs. The init script in there uses the pivotroot system call to tell the kernel to swap some mount points around and that allows replacing the current root file system with the new one. But again, while limit ourselves, we can't only do that from an initramfs.

systemd exposes a command that allows shutting down the running user space but instead of powering off the machine or rebooting, this command will do exactly what the initramfs did. I'm sure it was originally intended for systems where systemd is used in the initramfs, but why shouldn't we use it outside the initramfs as well, right?

So for example, you can build a nix OS system and copy it to a tempfs and switchroot into that tempfs and boot into nix OS from another distribution without leaving any traces besides shell history behind. I've also used it to put an Ubuntu installation

on my laptop without letting it touch the bootloader without having to repartition the disk and without having to mess around to get zfs support in Ubuntu since I was still running nix OS's kernel. The reuse of the kernel is what makes this approach applicable in some scenarios where kexec is infeasible, such as if the kernel you're using doesn't implement it

or if it doesn't work on the platform that you're working on. Your mileage may vary for anything that relies on the kernel here since the system you're switching into is very likely to have an incompatible set of kernel modules which won't be loadable. This means that hot-plugging devices

whose drivers weren't already loaded or which need firmware files not provided by the target system will fail, for instance. But it's a bit more versatile than anything purely initramfs-based since it can use real file systems. As mentioned, I put that Ubuntu on a zfs dataset. So let's move on to something you're probably more familiar with.

Customizers. I'm sure this is one of the use cases many of you have used before. Nix OS's own installer images are built from full Nix OS configurations. Like the NetBoot module, the ISO image compies the system package into a squashfs. It then takes the squashfs,

the kernel of specialized initramfs, a bootloader with appropriate config, and the necessary bits and bobs for both BIOS and UEFI to recognize it as bootable and bundles them together into an ISO image which you can write to a USB stick or if you're feeling nostalgic to optical media. Single-board computers often boot from SD cards.

Nix OS has the tooling for building SD images as well since that's the workflow users coming from other distributions have come to expect. Generating non-installer images is a bit of an anti-pattern for a number of reasons. Hit me up later if you want to hear me rant about it. But it can be very useful nonetheless. So if you're not working with hardware you can touch,

VM images can be very handy. You can generate a base image pre-configured with your SSH keys and standard tools to enable quickly spinning up new VMs. Just clone the image and boot the image, then deploy your configuration. Or you can configure the services you want the VM to provide, build an image from there,

and boot without any further configuration or deployment steps. So here I'm gonna show from my employer a bit more. Determinate Systems has been working on a tool called Ephemera which given a Nix OS configuration, instead of building a disk image inside a Nix derivation, builds a Nix OS system package into a disk image outside the sandbox.

This allows caching some more parts, similarly to how Nix netbook serve does it, and enables parallelizing it a bit more, and brings image build times to only a few seconds. The images are stateless, so it's expected that if the configuration

needs to be altered, the image will be replaced. But after all, the images are really cheap to build. Combined with the AWS snapshot upload tool cold snap, this allows going from a Nix OS configuration to an AMI faster than any other tool set for building AMIs known to us. We're pretty proud of that.

But Ephemera is by no means limited to AWS. It should be usable on any cloud provider that allows uploading custom disk images, and we've also tested it on local QEMU VMs and on hardware. We're very excited about it, but it's currently still in its early stages and we haven't published it currently. Feel free to ask us if you're interested.

Even with Ephemera, working with disk images isn't always the most convenient way. Since the Nix OS system package contains everything that you need to boot it, it's also pretty easy to run a Nix OS VM on a host that has Nix installed without any image generation.

QEMU has a built-in file sharing implementation that allows sharing the host's Nix store to the guest's read-only, and booting directly from that file share. Nix OS allows generating scripts which take care of the tedious parts of building the QEMU commands for this. This is really great for... This is really great for testing configuration changes

without deploying them to a persistent staging system or without building images. I use these VMs to try out Nix packages changes for reviews to test out software I'm not yet sure I want to use, and I've also experimented with using them for fully scripted Nix OS installs onto physical media,

but I don't trust myself to touch only the relevant physical media and not accidentally wipe my laptop in the process, so I don't in a VM. And being able to shut the VM down afterwards makes cleanup much simpler, so I don't have to unmount the file systems, close, correct setup volumes, et cetera, manually.

However, this VM infrastructure is also used for what I believe is one of Nix OS's wildest features, the full system VM-based integration testing. It runs QEMU VMs, or even networks of them, inside Nix builds to verify a huge range of functionality provided by Nix OS. We have tests for a wide variety of installation setups,

which boot the ISO to install on various combinations of LVM and LUX, as well as various file systems. The Nix OS channels won't update if these tests are broken, which allows us to make changes to the installers and file system support with confidence that we won't break it for everyone, or at least, we'll usually find out before being confronted with disgruntled users.

We also have tests for many Nix OS modules. The Nix cloud tests ensure that all three supported Nix cloud versions work with four different database configurations, so we have 12 different configurations which are tested. Our networking tests ensure that a client can talk to an external network through a router with NAT, also a Nix OS VM,

and that our firewall works correctly. We even have an open arena test, which pulls up a network of VMs, runs a game server on one, and bot clients on the others, and takes screenshots of the action. And all of this is within the Nix sandbox, so these tests are completely independent

of online services and won't break from one day to the next due to external circumstances. Nix OS makes heavy use of them, but I'm sure there are many use cases outside Nix packages that would be well served by the Nix OS test infrastructure. Now, Nix OS isn't suitable for all machines that can run Linux,

because some of them don't have enough RAM or enough storage, but that doesn't mean we can't give them Nix treatment. Thanks to Nix packages cross compilation support and some kernel patches I encrypt from OpenWrt, I run a small Linux system built with Nix on my Wi-Fi access point. It's a little kernel in initramfs,

which I currently boot via the network, and the init is just a shell script that starts the necessary bits to supply network connectivity and drop there for SSH access, but I haven't really needed that SSH access since setting it up, because it just works. The biggest problem I had with it was when I wanted to take this picture,

I accidentally unplugged it, and then my Wi-Fi was gone. But yeah, Daniel Barlow's gone a bit further with the concept, and his work predates mine and inspired it a bit as well, and he wrote Nix WRT, though I gather he's currently rewriting it as LimiNix, which supports a wider range of devices

and more use cases, including router setups and things like that, and in principle, it can run on anything that OpenWrt runs on. Might need some extra work, though, for some platforms. Samuel DR also has some interesting projects going on around building embedded Linux systems with Nix.

I believe he originally started this in order to create this Tobunt installer, which he builds with Nix, but isn't based on Nix OS, but he's been working on how to generalize that into a more universally applicable

embedded Linux system build framework, not dissimilar from Buildroot, for example, in use cases, I believe, called Selen. It's still in its early stages, but he's got both the Tobunt installer building with it, and he's built operating systems for a variety of obscure emulation consoles using it.

So I think there's a lot of space to be explored there, and I'm looking forward to what Samuel DR, Dan, and the rest of the community will dream up over the next few years. So let's talk about Netboot again. I'm gonna start talking about things that I haven't done before

and that might not have generated practical implementations yet, so take these with a grain of salt. Serving a full system in NetramFS is an easy way to get your Nix OS on a machine, providing it has enough RAM and or swap, but it's certainly not the most efficient because you're downloading the whole system before you can even start booting properly.

Classical Netboot setups only serve a small NetramFS containing the network drivers and configuration for mounting a network file system as the root. This is, of course, also possible with Nix OS. Certain properties of Nix make it possible to do this much more efficiently as well. For instance, the Nix store is practically append-only.

Once a path is there, it'll only change in very unusual circumstances, and even then, it should be functionally equivalent. This means that store paths can be cached by clients indefinitely. This will combine the flexibility and central manageability of classic Netboot setups with the performance of a local installation.

There's a lot of room for creativity here. Since store paths are immutable and can be signed, they can be obtained safely from untrusted sources. This makes peer-to-peer distribution possible, meaning that the Netboot server can be provisioned far more economically without impacting performance. Again, in theory, I haven't done this in practice.

So a lot of operating systems can boot in these ways. What I think is fairly unique to Nix OS is that because everything is pulled together from a declarative config into an immutable package, it's much easier to do all these things. You can take the Nix OS configuration

running on your laptop, factor out any bits that are specific to the hardware, and then get your laptop's configuration running in any of these environments in a matter of minutes. You can build installer images with pre-configured Wi-Fi and SSH so that you can install on hardware remotely. You can put Super Tuxcart and OpenArena in for geeky open-source LAN parties.

I generated Nix OS images with a patched version of another open-source game a couple of years ago to allow my Windows-using friends to play with some of the bugs fixed. The possibilities are endless. The Nix OS generators project pulls several of these options together into one convenient command, and it's a great way to get started

playing with some of these. So you can probably tell I'm quite excited about all this. If you're excited about it, too, I'd love to hear if you implement the peer-to-peer Super Cache Nix OS netbook setup. Or if you come up with and implement some even more exotic ideas. Find me at the hackathon,

or feel free to drop me a message on Matrix. Thank you. And I've got links to a number of things I've mentioned up on there. Is there time for questions? Yeah, thank you. I don't need to take any. Of course there's some sound.

Here we go again. There's probably no sound, because I'm not sending anything to you. Do people want to yell for questions? I can yell. Yeah. Go ahead, and I'll repeat the question so the people in the room can hear you. So thank you for the great talk.

I'm wondering how far can we go with something with Cryo, which is checkpoint and restore, in order to make live migration of actual Nix OS machine easy?

What kind of use case can this unlock in terms of everything we does? Was the stream able to hear that, or should I repeat the question? Okay, so Ryan asked if Cryo, what Cryo, checkpoint and restoring user space,

might enable with this. That's exactly one of the exotic use cases I was hoping to hear about, which I didn't think about myself. It sounds really cool. Yeah, that sounds exciting, and I don't really know. Thank you for your question.

Okay. So it's not really a question, but I think it should be mentioned that the Nix OS tests are even faster by now because of you, because you fixed the 9P file driver,

file system driver, and so the Nix OS tests are even greater, and I think that should be mentioned. Thank you. I considered putting some shameless self-promotion in there but decided not to. I have a blog post on that, if anyone's interested. You can find my blog right there.

Cool, I think that's it. Thank you.