All right. Hi everybody. I'm Chris. I seem to have about 30 minutes and I usually talk very slowly, so I'm going to try to get through it quicker than normal. I'm going to talk to you about firmware today, which is kind of strange for a Linux user space conference,

but I promise you there's Linux user space involved in firmware. And we'll get to that in a second. First of all, I want to thank we have a whole bunch of contributors from other companies that we collaborate with on this project. Unfortunately, none of them are here,

but you should know they exist. The first thing that I actually want to talk about is what in the x86 world firmware usually looks like and then what we're currently doing about it. So if you have worked with UEFI before,

which is basically x86 firmware in most cases, you're probably familiar with this picture, which is probably about 20 years old at this point. And the individual pieces are not too important, but what I want to point out is the the left two columns, the security and the PII phases,

they consist usually of Intel binary blobs or chipset like a vendor binary blobs or a motherboard vendor binary blobs. And these are things that you're not likely to see being open sourced in the near future. So this is stuff that we don't, that we don't really touch.

But on the right from there you'll notice that you're going to recognize a bunch of operating system like concepts. You have network drivers, you have disk drivers, you have a scheduler that dispatches those drivers and you can actually also run, this is kind of important for later,

you can run arbitrary P32 executables in ring zero. And on, on your normal UEFI system, what comes together here is usually about 20 different vendors that put together codes that ends up on your system like 20 to 30 different ones.

And if I look at some of these driver modules, which are by the way, called Dixie's on a five, six year old open compute platform server called Winterfell, you can see there's some outdated stuff here, like the microcom networking protocol, which by the way,

the server doesn't have a modem. It's a modem specific protocol. Some ISA bus drivers like that bus from the eighties that isn't used anymore. And in addition to that, you have a full networking stack like IPV4, UDP, TCP, all this stuff re implemented in firmware.

And some of the observations you make when you work with this ecosystem is most of this code is closed source. You don't really have any way of looking at it except for some of the reference implementation that Intel open source says EDK2 that tends to be a

small portion of the code actually running as the firmware on your X86 devices. Though, it's all stuff written in C and it all tends to be run in ring zero in one shared address space between all of these drivers applications and all these things that UFI runs. The other thing you realize from this ecosystem is that when someone like,

actually I don't want to name any names, when there's a company that builds a new server or laptop or something, they will tend to contract out firmware to another, what we usually call an independent BIOS vendor, an IBV. Those vendors get paid when they deliver firmware that works on that server

and it ships. So they're mostly incentivized to get something working that runs on the server based on the constraints that the manufacturer gave them and to ship it and forget about it. And the problem with that is as there are like 20 to 30 different vendors

involved in shipping firmware because you need firmware for your devices in there, you need to coordinate with the board manufacturer, you need to coordinate with the company actually selling this stuff, all these different companies that have to be involved. You realize that even for people that are involved in the process,

you're kind of wrapping black boxes around black boxes. Not everyone is going to know the entire system. And one question I want to ask you about that is what do you think happens with vulnerabilities? Let's say you are a network card vendor, you found some vulnerability in your driver. Where does the first fix for that vulnerability go?

Usually Linux or Windows, something large like that. Let's say you also found that vulnerability in your firmware driver. What's the likelihood that that fix will end up on your actual end user system for that network card driver, even if it's a security fix? And that likelihood is probably 0.001% or something.

Because you have all these questions like who wants fixing it, who actually gets paid to fix it, by the way? Is it the network card vendor? Is it the independent BIOS vendor that put it all together? Is it the OEM that you bought the system from?

Who is actually responsible for getting this done? How do you integrate it? And then actually how do you release it? So once you release an update, you either put it on the vendor's website and you hope that your customers download it, or you try to distribute it through Windows or some Linux distribution. But then you also have to deal with those distributions. So the likelihood that any of this happens is pretty low.

Unfortunately, besides the ecosystem not being incentivized to fix these issues, you also can't fix it yourself. Because as I said, most everything is closed source. You likely don't even know if your driver that's in firmware is vulnerable or not.

I don't think they don't necessarily disclose that. And just because you can't build firmware from scratch for your platform, that also means that you don't really have a way of deploying fixes yourself, if you want it to.

So with that, I'm going to take the observation that you can run arbitrary PE32 executables in ring 0 in UEFI and tell you that you can compile the Linux kernel as a PE32 executable. So what we started with is we took the UEFI shell,

we scooped out the code section, and we replaced it with a Linux kernel compiled with config efi stub. It turns out that this actually works pretty well, even if you remove most of the drivers on the system. Because Linux knows how to initialize most devices. Linux knows how to initialize your disk drivers.

Linux knows how to deal with your network cards. In most, if not all, cases, without any interaction that has to happen with firmware beforehand, except for ACPI and similar tables that have to be passed to Linux. However, those usually come from the driver dispatcher

and not from any of the individual drivers. So if you take, for example, the OCP Winterfell server I was talking about earlier, and you remove about 90 of the 120 drivers that were on the system, and you replace it with Linux, what you actually get is a system with drivers that are open,

which means to you, you can actually get fixes for vulnerabilities and deploy them yourself. You also get a system that's reproducible, and I think a lot of people here are familiar with the reproducible builds project. The same properties apply to firmware and are important there.

And for those of you that deal with TPMs in some of your hardware, you also know that Linux is measurable. You know what you deployed, right? I think I've heard way too many stories about new firmware updates somehow ending up on your platform,

and you found out about it through a TPM measurement that gets reported because it's not what you thought it was. And it's hard for you to tell if you got owned or if it's actually just an update that you didn't know about through some channel that you didn't know about. The other thing that you gain, at least when you're a larger company,

if you're Google or Facebook, you maybe have a handful of firmware engineers that can deal with UEFI. You probably have hundreds of kernel engineers that know the Linux kernel. So it's a way for you to leverage talent that you already have,

not having to hire people that are very hard to find. And a nice side effect of removing a whole lot of Dixie drivers is that your boot time goes down. The Dixie driver scheduler is not an especially useful scheduler in that sense of word. It's more of a trial and error process.

I'm going to just say it like this. But for the Winterfell platformer we did this, we went from 8 minutes to 20 seconds. So, OK, I already talked about this. The one thing I want to point out is, Linux has too many contributors to count compared to EDK2 and Grub,

which have a combined 248 based on a GitHub screenshot that I probably took quite a while ago. But the point is, there are a lot more eyes looking at Linux and fixing things in Linux than there are in your current firmware and your current bootloader.

And the idea of putting Linux in firmware is actually not really a new idea. This is something that the OpenPower platform has done for the last four years and been pretty happy with. And it's the original idea behind Coreboot. Which used to be called Linux BIOS.

And it's something that works with UEFI, it works with Coreboot, and it works with U-Boot. So this is not specific to UEFI and the Linux combination. Here, if you want to look it up later, the thing that I kind of want to point out here is that the build process is pretty easy.

If you've ever tried to build EDK2, you know that this is probably much easier. And by the way, it works on Linux too. The supported hardware that we currently have upstream is a bunch of OCP boards, an Intel reference board, and I believe the Dell R630.

These are things where you can actually build the system right now using these four or five commands, if you wanted to. So, now we have Linux in firmware. We have drivers we can leverage. What do you put in the Nitro MFS to actually do something with that kernel?

And this is where we get to a point where we say, as a Linux boot project, you can put whatever you want in there. There are plenty of Linux bootloaders that people have written, like Pettit boot on the open power platform, like systemd boot. Or you can, if you want, just put Busybox and Kexec in there.

It's kind of up to you. But what we've done is come up with a Go-based runtime that we use as our bootloader platform. And Google, Facebook, and a company called IT Renew currently collaborate on this effort,

and it's open source, it's all out there. And it comes from the realization that we now have the full toolset of Linux applications at our fingertips. We might as well use them. And we also started with a premise that we would like to start with a somewhat memory-safe language, something better than C,

and a language that makes concurrency easy. So U-root is a project that has a bunch of Busybox-like tools, like all the standard stuff you know of, like DDLS, CP, all that stuff, and a bunch of Kexec-based bootloaders that implement pixi-like or grub-like protocols and file formats.

And the why Go, besides the memory safety, is Go gives you easy cross-compilation. In the initial days of U-root, the project was actually kind of born out of a frustration with building a cross-toolchain for build-root

and for Busybox and all these projects. And one of the nice properties of Go is also that it is reproducible without having to patch GCC or do any crazy things in the C ecosystem. And then, of course, you get a bunch of nice static analysis tools

similar to the ones that you get in the C++ world. Those of you that are familiar with Go are going to say, OK, firmware storage tends to be pretty small usually. We usually get between 4 and 8 megs of space, where we can put a Linux kernel and a runtime.

And you probably know that a normal Hello World Go binary is something like 10 megs, 11 megs. So we've had to come up with a kind of strange system to take many different Go commands and compile them as one binary.

We do this by taking all the source of those commands and rewriting it in a pretty simple way, to be packages, basically. And then compiling, then generating a main.go file that imports all those packages and compiling that program.

What that gets us is a compressed binary that goes down to about 3 megs, 4 megs, depending on what programs you actually put in there. And it's a system where you can plug in any Go commands that you want to compile into your firmware as well. So this is not specific to U-root itself.

And it's pretty easy to do. You basically, there's a make bb command, since this is like busy box style and argv0 decides which program you actually run. And you just give it some Go import paths

to compile into a busy box. And from there, it should be pretty easy to use. I'm gonna skip over that. So now that we have a base of Go programs that we can use, let's talk about some bootloaders written in Go.

I don't know how many of you are familiar with pixie booting. And excuse the shitty picture that I drew this morning. But if you are familiar with it, you know that there are some strange complicated pieces in there. For example, the original UFI firmware

will talk to your NIC to find out if your NIC has what's called an option ROM. An option ROM is just a, I'm just gonna call it a piece of code. The NIC can say, yeah, I have an option ROM, I would like you to run it. So you go to the NIC, you get some piece of code,

you run it on the x86 firmware, and what that usually tends to do is pixie. The reason for this in the, I'm gonna say the olden days, is that you didn't really have drivers for network cards in your firmware. So the network card had to provide you a piece of code to do that for you.

That's not really necessary anymore today since I also said there are drivers in UEFI, but it still tends to be the way it works in a lot of systems. And then from there, some pixie implementation makes a DHCP request. From there it gets a file name, downloads a file over TFTP.

It will run that file and that tends to be another piece of code, another, I'm gonna call it a microkernel, pixilinux.0, that will actually go and interpret the configuration for the kernel that you want to download and actually go and download it. This is a somewhat complicated process for

what I don't think should be as complicated, and there's some other problems with it. First of all, pixie booting is, at least in many systems, still based on TFTP. TFTP is a file transfer protocol with which it can take easily

30 seconds to download a 10 megabyte binary. And option ROMs are a problem because there are devices that hand you a random piece of code and you just go and execute it. First of all, do you trust that piece of code that comes from the device? Yes, no, maybe, right?

The other thing is that option ROMs tend to be your real mode code, 16-bit, which can also pose a security problem for you in some cases. Now the other question is, do you go and re-implement newer protocols and newer standards like HTTP, TLS, SSL, IPv6,

do you go and implement all that in UEFI? And if you do go and implement that, do you get a mature or audited or well-tested implementation? And the answer is likely no, which is another good reason for us to do Linux and firmware. But even if you have all these things,

a lot of companies out there have different security models in their data centers now. A lot of companies out there have very different security models on their client machines like laptops and workstations. And even if I go and I say, I try to verify a signature on an option ROM

or I try to verify a signature on a kernel that I downloaded over Pixy, whose keys do I actually trust? In a current secure boot system, those tend to be Microsoft keys because they're the only ones willing to run the CA. But does that really give you ownership over your platform?

And the other thing is, what's your actual boot policy if you can't verify something that you're about to run? If you download a kernel and the signature check fails, what do you actually do? Maybe on a laptop, it's acceptable to throw up a scary screen and say,

this is not safe, you can't use your laptop because someone has compromised you. But on a server system, that presents a nice attack vector as a denial of service attack where you can take down a whole data center just by serving something that's one byte off.

So these are all questions that I don't have an answer for for you, but that I wanna give you the tools to answer and program for yourself. So if you have Linux as a bootloader, you can implement Pixy booting and go pretty easily. The protocols are out there as packages.

The configuration file is not that hard to parse. You don't have duplicate drivers from EFI anymore, but more importantly, it's trivial to use modern features. Go already has an HTTP implementation. The Linux kernel already has an IPv6 implementation. These are things that are somewhat well tested

and audited and that we can reuse now. Or if you're not something like Google or Facebook, you might just keep it simple. You bring up your device up to the Linux kernel in firmware, and then you just get a DHCP lease. You download a kernel with wget.

You verify it with GPGV and UK exec. It can be that easy for your own systems. However, in many cases, this is kind of infeasible for companies that have large infrastructure. For example, on our servers, if a kernel fails to verify,

there are secrets stored in a certain location on our servers that we have to say any subsequent stage after this doesn't get access to these secrets anymore. These are some kind of set of keys that gives you access to production services.

So what I'm trying to say is our security model is not the same as yours, and whatever we end up coming up with is not gonna be something that you can just take and deploy in your data centers or on your laptops. You have to answer some questions about where are your secrets stored?

Do you have secrets like that? What do you want to actually happen when kernel verification fails? And you also have to take into account your hardware security model when you design something like this. So I would like it if once you answer

all of those questions, you yourself can write a bootloader in Go for your use case in less than 500 lines of code. But if we give you all the packages to kexec to deal with kernels, maybe in some way deal with the crypto involved,

then it should be easy for you to compose these things and write your own bootloader. Now, we have already written some, and our projects for bootloaders a little bit in disarray, and that's another reason that I'm kind of here. I want to encourage people to do this on their own

and also to contribute back to us, because we, the users of this project, Google and Facebook, we're busy building bootloaders for our own data centers, but they're not the things that are gonna be out there the same way that we use them in the data center.

So it would be great if other people were to look at this stuff and see if it's useful to them and to make it more usable. So there are a couple different bootloaders that exist, for example, with TPM support, with DHCPv6, but then there's another one that implements Pixy,

as we know it, only via DHCPv4. So we're kind of in disarray and we would love some help. And the plan is to hopefully have a shared code base of packages that we can all use and then implement our own commands that are the actual bootloaders.

So I'm just gonna quickly mention the people involved, which are, you know, Facebook and Google are testing this on their servers at the moment. There's actually a French American reseller of decommissioned open compute platform hardware that sells servers with Linux boot and Uroot already on them.

So you could go spend $700 on a server right now to get it and then there's various other smaller people involved at the moment. And kind of to reiterate what I was saying before, we wanna provide you with tools to build the systems you need, not policies.

You should be the owner of your platform, you should get to decide how things actually work in firmware and how your bootloader works. And somewhat orthogonal to Linux boot is we would like it if these platforms give you security and user freedom at the same time.

It seems that a lot of platforms make trade-offs there that they don't actually have to make. So if you, for example, buy a server that has Intel boot guard enabled, Intel holds the keys to your, I'm gonna say firmware universe. We would like there to be server hardware that allows a change of ownership

of those keys for firmware. I don't care if that's actually an Intel boot guard or if that's an external chip that, for example, Google is working on right now and many other companies are too. And the third thing that we wanna kind of encourage users to do is have systems that instead of throwing up

a scary screen when something goes wrong and telling you like you can't boot, boot anyway but don't allow you to get access to secrets or boot anyway and tell you it's not safe but you can still use your computer, which is kind of the Chromebook model. With systems that are available on the market right now

like boot guard based server systems, what you'll just get is a server that doesn't boot. And that's not especially helpful to you if you're trying to debug things. One of the things that we're doing is we are engaging with the open compute community.

So there's an open systems firmware project where you have, where we have weekly calls, actually biweekly calls now, every two weeks, where a couple of vendors are listening in. It's our way of trying to hopefully encourage some vendors to enable the option for people to use this on their system.

And there are a lot of things in the future, but I have about four minutes so I wanna leave some time for questions. We would love to see more contributors and here are a bunch of links. And also by the way, I brought stickers. They'll be outside on that table.

But so time for questions. Yeah, I think. By putting entire Linux into the boot layer,

don't you increase the surface attack? Yes and no. So one thing you have to realize is that the UEFI you're running on your system is an incredibly large system that has multiple millions of lines of code of attack surface. The other thing that we've realized

is you can compile a pretty small Linux kernel with only the features you want. Start with config, tiny config and add some. You'll get a kernel that is no more than 800 kilobytes and consists of only a couple thousand or a couple hundred thousand lines of code. So the answer is it depends,

but we hope that Linux will be a smaller attack surface that has had more eyes on it than UEFI. Have you worked with the ONI or O-N-I-E project, the Open Network Install Environment, which is another Open Compute project? It's effectively a bootloader for white box routers

for sort of advanced pixie. They are slightly different from us. I haven't spent a lot of time with that project, but they are not a bootloader that actually lives in firmware. And I think the definition of bootloader is getting metaled somewhere in the middle there too,

so we should talk a little more with them, but I don't think it quite fits together. Okay, yeah. Oh, they're primarily a Linux distribution for what again?

Oh, for Facebook switches, FBOS. I realize that this is a silly question, but do you think that in our lifetime we will be able to see this on normal consumer laptops?

I really hope so. We do have some contributors doing it on normal consumer laptops, but if you wanna do it on modern consumer laptops, it either has to be a Chromebook or a system where you find a boot guard vulnerability. So we have to work with vendors to enable users to be able to do this or encourage vendors to do it themselves.

So I hope soon, but I don't wanna promise you anything. Mike back there. Okay, I have a question as well. So if I seem correctly on your slides, so you need some downstream patches for the Linux?

You needed one patch and we're getting rid of it pretty much in a couple of days. Okay, so you're working on upstreaming this? Yeah. Okay, thanks. Have you approached Intel on this?

Actually, Intel is on those open systems firmware calls and we also meet with them every once in a while. Intel is pretty supportive of this project because you can frame it as a, this is Linux as an EFI driver project. So they have no problem with this.

We're actually hoping that they will enable us to do this on some systems with blobs they provide. I don't know that they will, but I'm hopeful, yeah.

So if I wanna try Linux boot by itself or Uroot by itself, could I, so in the case of Linux boot, if I have access to my EFI shell and I can load Dixie modules, could I presumably take the kernel that I already have

and kind of like get it into the EFI shell mod, Dixie module that exists? Actually, what we usually start with is we replace the code section of the EFI shell with a Linux kernel. Right, so if I have access to my EFI shell, like presumably can extract it from my firmware so I can put my existing kernel into it

and try to boot it. What will I expect to see if I just loaded that module from the shell? Is that a way to test it? Load in that module from the shell. So just to execute the Dixie module in Linux boot, what would I see at that point? You should just get Linux actually. Like it should just work. One of the issues there is you might not have enough space

to actually do that without removing some of the drivers that you won't need anymore. Right, yeah. So that really depends on your system though. Yeah, so I'm pretty sure I can do that in my shell. So then in the case of Uroot, could I already now try that go based init ramfs on just my laptop already today?

I don't necessarily use it. Yeah, you can use it now. There's a QAMU environment too, so it, yeah. All right, I think I'm out of time, so thank you very much.