library operating systems
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 167 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/34818 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Physical systemDoubling the cubeContrast (vision)Arithmetic meanData managementRoundness (object)Core dumpComputer animationLecture/Conference
00:45
AbstractionSubstitute goodPhysical systemMultiplication signExecution unitDefault (computer science)Meeting/InterviewLecture/Conference
01:15
Axiom of choiceBuildingSoftwareGoodness of fitOperating systemProjective planePhysical systemCartesian coordinate systemCategory of beingExecution unitTrailComputer animation
02:10
NumberNumberBus (computing)Projective planeSound effectBoss CorporationComputer animation
02:47
NumberNetwork socketFunction (mathematics)Execution unitAddress spaceImplementationNetzwerkschichtException handlingDefault (computer science)Communications protocolError messageOperating systemContext awarenessGreatest elementSoftware developerWebsiteCASE <Informatik>CodeComputer fileType theoryWrapper (data mining)Spherical capInformationComputer programmingView (database)Category of beingModel theoryQuicksort1 (number)WritingException handlingDiagramPurchasingNumberSubsetRepository (publishing)Domain-specific languageProjective planeParticle systemSocket-SchnittstelleFunctional programmingFormal languageCode refactoringWordOrder (biology)Equivalence relationParameter (computer programming)Product (business)Musical ensembleTelecommunicationOnline helpElectronic signatureInterface (computing)Reading (process)Network socketFunctional (mathematics)Error messageCartesian coordinate systemSelectivity (electronic)Electronic mailing listAbstractionExecution unitDefault (computer science)SoftwareBitLevel (video gaming)Connected spaceCommunications protocolComputer animation
08:23
Exception handlingDefault (computer science)Communications protocolImplementationError messageNetzwerkschichtNetwork socketAddress spaceFunction (mathematics)Execution unitPatch (Unix)FeedbackKernel (computing)Formal languageComputer programmingConstraint (mathematics)CodeOpen sourceUniverse (mathematics)MathematicsPatch (Unix)Projective planeKernel (computing)Normal (geometry)Computer programmingInterface (computing)Software testingFormal languageProgramming languageProcess (computing)SubsetComputer animation
09:24
Formal languageKernel (computing)Patch (Unix)FeedbackConstraint (mathematics)Computer programmingUsabilityMemory managementScheduling (computing)System programmingComputer networkVideoconferencingRun time (program lifecycle phase)Socket-SchnittstelleSystem callPatch (Unix)Cartesian coordinate systemSoftware developerSpacetimeKernel (computing)QuicksortVirtualizationIntegrated development environmentMultiplication signInterface (computing)Data managementFile systemSoftwareView (database)Virtual machineScheduling (computing)Context awarenessComputer hardwareTelecommunicationBlock (periodic table)Level (video gaming)Operating systemPhysical systemVideoconferencingBitSystem callMemory managementFormal languageBit rateProjective planeScripting languageModule (mathematics)Process (computing)Right angleComputer programmingEmailWritingSeries (mathematics)Computer fileComputer animation
12:59
Physical systemOperations researchDataflowData storage devicePhysical systemLevel (video gaming)Module (mathematics)PlastikkarteMultiplication signEntire functionLogicOperating systemKernel (computing)SoftwareCodeFormal languageSet (mathematics)Term (mathematics)Different (Kate Ryan album)Interface (computing)ImplementationLatent heatFunctional (mathematics)File systemContext awarenessQuicksortBitComputer fileVirtual machineDevice driverType theoryRight angleDirect numerical simulationAssembly languageForestRun time (program lifecycle phase)Model theorySource codeComputer animation
15:11
Error messageBlogDataflowType theoryOperating systemConnected spaceBitState of matterOperating systemProgrammer (hardware)Context awarenessDemo (music)Interface (computing)Electronic mailing listCartesian coordinate systemSlide ruleWeb 2.0DiagramProcess (computing)Electronic signatureModule (mathematics)ImplementationError messageAddress spaceMultiplication signResultantBuffer solutionWeb pageComputer multitaskingVirtual machineFunctional (mathematics)Musical ensembleConcurrency (computer science)IP addressModel theoryWeb applicationAdditionTerm (mathematics)CompilerWeightKernel (computing)Computer animation
19:21
InformationStack (abstract data type)Default (computer science)Key (cryptography)AbstractionElectronic mailing listError messageFunction (mathematics)Server (computing)Module (mathematics)StrutElectronic signatureRun time (program lifecycle phase)Fluid staticsSequenceNetwork socketWeb pageFunktorString (computer science)Principal idealMusical ensembleType theoryRandom numberDirectory serviceInfinityBlogBootingDirected setClient (computing)Thread (computing)Predicate (grammar)BuildingDemo (music)WebsitePhysical systemSoftware developerOperator (mathematics)Web pageProjective planeKernel (computing)ImplementationComputer fileConfiguration spaceLogicMereologyFigurate numberConnected spacePublic key certificateExecution unitElectronic mailing listTransport Layer SecurityCartesian coordinate systemRandom number generationSoftwareContent (media)InformationFormal languageOperating systemState of matterFlow separationModule (mathematics)TrailGastropod shellSoftware frameworkSoftware testingBitBlock (periodic table)Normal (geometry)CodeIntegrated development environmentDirect numerical simulationGraphics tabletSource codeDevice driverUnit testingVirtual machineSpacetimeFile systemoutputQuicksortProcess (computing)Interface (computing)Control flowPoint cloudSemiconductor memoryBinary codeGoogolPoint (geometry)Traffic reportingMetropolitan area networkOperating systemComputer configurationModel theorySurfaceContext awarenessWebsiteMultiplication signTask (computing)Server (computing)BuildingWeb 2.0Reading (process)Stack (abstract data type)Computer animationSource codeLecture/Conference
26:41
Operating systemWater vaporMassState of matterSet (mathematics)Cartesian coordinate systemData storage deviceLecture/ConferenceMeeting/Interview
27:11
Mechanism designMessage passingInterface (computing)Read-only memoryMiniDiscGUI widgetFirewall (computing)Physical systemLink (knot theory)Inclusion mapMereologyMessage passingInformationCartesian coordinate systemConfiguration spaceVirtual machineDecision theoryHard disk driveDimensional analysisPhysical systemScheduling (computing)Data bufferTracing (software)CodeInterface (computing)Component-based software engineeringThread (computing)BitState of matterMathematicsMultiplication signOperating systemKernel (computing)SpacetimeOperating systemSlide ruleSemiconductor memoryShared memoryExecution unitTerm (mathematics)Firewall (computing)MereologyProjective planeFlow separationCubeDifferent (Kate Ryan album)SoftwareContext awarenessLink (knot theory)LaptopOffenes KommunikationssystemReal numberBuffer solutionOperator (mathematics)Visualization (computer graphics)Right angleQuicksortComputer animation
30:55
InternetworkingMultiplication signProjective planeProcess (computing)MereologyRevision controlCartesian coordinate systemFormal languageState of matterArmLecture/Conference
31:51
Roundness (object)Multiplication signWebsiteReal numberArithmetic meanHand fanLecture/ConferenceComputer animation
32:19
JSONXMLUMLComputer animation
Transcript: English(auto-generated)
00:15
A wonderful good morning to all of you on day four. I hope you made it through last night halfway intact.
00:24
Our next speaker, Mindy Preston, who's a core dev on the team for Mirage OS and release manager for its latest release, will be talking to you about library operating systems. The solution for, well, if you're dabbling in microservices and Docker just is too fat
00:42
of a thing. Give a warm round of applause to Mindy. Hello, thank you. If you can't read this, now is the time to shout. But it seems like it's fine. So today I'm going to talk to you about ways that we can use library operating systems
01:04
and unikernels to reject the default reality or abstractions and substitute our own. A quick little roadmap of what I'm going to talk about today. This talk is in the resiliency track, which is new at CCC this year, and I wanted to talk
01:22
about why I think that the concept of resiliency is very relevant to the work that we're doing in library operating systems and unikernels, and talk about some of the properties that a project might have that lead it to be resilient, especially as how that relates to whether a project is understandable.
01:43
I'm going to talk about what I think of as the ultimate bad dependency in your project, which is a traditional monolithic operating system. I'll talk about how you can replace that bad dependency with maybe a dependency that fits better into the way that you usually handle your dependencies. And I'll talk about how we can actually implement increased resiliency when we take this approach
02:02
by using good software tools that we use in applications for the libraries that we make this OS out of. So to me, a really important concept in whether a given project is resilient is how many humans can disappear before the project grinds to a halt.
02:22
We have a name for this, which you've probably heard, which is bus number, euphemistically defined by Wikipedia as a sudden disappearance. If we want to increase the bus number of our project, the simple answer is, well, okay, if I want more humans, if I want to have a higher number of humans who could go away and my project would still go, I should get more humans involved in the project,
02:44
and then the bus number will go up and that'll be fine. But in fact, it's not a simple thing to just say a human who's exposed to my project is a knowledgeable or competent person in the context of this project. And a way that we can help people be knowledgeable and competent people in the context of our
03:01
project is to make our project easier to understand. But our project isn't just our code, so we can go and we can do a ton of refactoring and we can say, okay, we're going to use the best practices that we know of and document the hell out of everything and use nice language features and all that stuff. But we also have to think about the language that our code is written in, how
03:21
we actually interact with our code from day to day, like how do we build it, where do we run it, and what things do we need in order to actually use our code in some kind of meaningful way. Our project isn't just like the code that sits in our repository, it's also the list of all the things that it depends on. And you might be saying, okay, well, my project doesn't have any dependencies or has a really
03:42
small list of dependencies. I understand all of them really well, they're all really simple. And I would say to you that's probably not true. It's very likely that you have, in fact, a really heavy dependency for your project, which is a conventional operating system.
04:01
So I've made a nice little ASCII art diagram of how I think about applications, their You think where you put all these things in the same category of stuff that I actually need. And in this diagram, I've cheated a little bit by just saying dependencies, don't worry about them, but this one's really big.
04:20
But in fact, it is really big. The operating system is really huge. It's almost always got more stuff than you need. And the way that you interact with it is different from the way that you interact with the rest of your dependencies. The documentation is probably not where you normally look for documentation. If you have a problem with it, you probably can't debug it in the way that you normally
04:41
debug your code. And when you're writing code that deals with it, you are likely accessing it with an API that doesn't look very much like the APIs that you like to use when you're doing application-level code. So to give you a concrete example of what I mean by that, this is a selection from
05:00
MAN2 in Linux. The signature for the function is here. So this is socket, which would be the first thing that we call if we want to do some network communication. And socket wants a magic number and another magic number and another magic number, and it'll give us back a magic number or another magic number to tell us that we got an error
05:22
and we should look somewhere else for another magic number to find out what the error was. If you program it in languages other than C, this is probably not what you're used to seeing for a nice interface for doing some external kind of stuff. And worse, so that magic number that we got back from socket, we have another function
05:44
that we need to call with it, which is also doing a whole bunch of stuff with magic numbers, giving us this weird out-of-band error communication. And you might say, well, okay, but if you're doing application development and you're complaining that the C interface is bad, that's because it's a C interface.
06:01
That's probably not what you're actually dealing with in your higher-level language. But in fact, often you are. So the language that I usually work in is OCaml. It's a functional programming language with a lot of nice features. It's considered to be a higher-level language than C is by most people, and I'm going
06:21
to give you a quick tour of how to read OCaml type signatures. So this is the equivalent of socket. It's also named socket, so that's helpful. This is a function. It's going to take some arguments, and these are their types. The type socket domain is also a magic number, but it's at least restricted to a small
06:43
subset of possible things. Same thing for socket type. We have a magic number for the protocol type, which in the documentation we at least are told zero for default and then a number for other stuff. And we get this type file descriptor back, which again is under the covers a magic
07:04
number, and on failure we get an exception. And if you've ever programmed in a language that has exceptions, you're usually not very pleased to see that there is something that's going to be throwing them as a matter of course when you're trying to do normal stuff.
07:20
Connect is even worse. Connect is going to take this file descriptor that we got back from socket. It's going to try to map a socket address to it. It doesn't even give us any... The return type is unit, which is kind of equivalent to void. I just went off and did something. It was side-effecting. Don't worry about it. Look somewhere else for errors.
07:41
In this case we might get an exception, which will just wrap the information from Erno from Unix. So this is really not very fun to program against in OCaml either. And we can see why this API is the way it is. It's just a wrapper for the C API. It's got a few nice language features from OCaml but not very many.
08:03
And really all you can do when you're in this situation trying to interface with the operating system like this is you can write a higher level API on top of this. You can build more and more and more abstractions to get to something that looks nice to work with as an application developer or in the context of your language.
08:20
But somewhere down on the bottom, you always have to talk like this. And what if you don't want to? So if we're in open source universe, you can say, okay, Mindy, if you really don't like that API so much, go change it your stupid self.
08:40
And I would say that that is non-trivial for most people who are trying to write code that they just want to run somewhere. So if I want to make a change to how I interface with the kernel when I want to do something, I first have to learn to program it in the language of the kernel, which is probably not the language I use day-to-day. I have to learn how to do that kind of programming in the kernel,
09:02
which is another skill set that I probably don't have. I have to make a test, sorry, make a patch, make a decent patch, and then somehow get that patch accepted. The community that I'll have to talk to to do that is a different community than I normally talk to. It's not the community of my project. It's not the community of my language.
09:21
It has different norms. It has different processes. And it's probably not very excited about my patch that's for me, that's for either my application or my language. Like, the kernel doesn't care about whatever stupid user space thing I want to make easier for myself, generally speaking. And if you think that I'm just whining about doing kernel development is hard,
09:42
I would say that it's not just me that thinks that doing kernel development is hard. You may have heard of the Udiptila Challenge, which was a series of programming exercises for the Linux kernel that was done entirely by mail. It started from writing a basic kernel module, and the challenges would be
10:00
gated by if you would basically send in a patch bomb that said, okay, I've solved this challenge. Please give me the next one. And then some scripts would sit and think about it for a little while and say, oh, yeah, okay, I think you did a good job. Here's the next challenge. So there are 20 steps to the Udiptila Challenge. There were around 19,000 people who started. I was one of them. Of them, 160 finished.
10:21
I was not one of them. And I would argue that a 1% success rate of people who start out specifically wanting to do this one thing is probably indicative that it is, in fact, difficult. And since we're talking about resilience, I wanted to bring up another topic around this.
10:43
So members of your project might want to limit their exposure to communities that are known for their hostility and toxicity, right? Like they've decided to be in your community. They haven't decided to be in every community that's touched by the dependencies of your project.
11:01
And a way that we can protect each other is to limit our exposure to these communities that we know might not be safe for people to be in. And projects that are famous for their toxicity do include some very famous monolithic kernel projects. So, okay, that all seems really bad and kind of a bummer.
11:21
So let's think a little bit more about what operating systems are actually doing for us. And I've kind of zoomed out from the view that I gave you before of our application, its dependencies, and an operating system. I've thrown a little bit more detail in here about how we're doing this communication. How we're doing this communication with the things that are sort of below us
11:40
in the way that we normally think about the stack. And crucially, I've put in this last little block here. Usually when we're deploying applications, in this context where we have some operating system that we're running on, the operating system is actually not usually running on what we consider bare metal or hardware.
12:00
It's normally virtualized. So there's a thing underneath it, the hypervisor, that is doing sort of a, depending on how you look at it, in this view of the stack, it would be a lower level management of system resources. And then farming these things out to individual virtual machines. So, well, the view that the application has is that the OS is taking care of
12:20
memory management and scheduling and doing all the network stuff and file systems, clock, video, all of that. It's actually the hypervisor that's responsible for most of this when you're deployed in this sort of environment. So what the operating system is actually doing a lot of times is just passing calls to the hypervisor for a lot of these things or getting more coarse-grained resources
12:43
and then doling them out in a more fine-grained manner to its individual applications. So what if we can talk to the hypervisor directly and get the same sort of easy virtualized interface that the operating system is getting when it's running in this virtual machine context?
13:03
And then use libraries to take care of the things that kernel code would be taking care of for us that are higher level than just interacting with the device. So things like networking isn't just, I need to push this data into a network card. It's, I need to assemble TCP flows. I need to know how to do DNS lookups.
13:22
Storage isn't just take all of this data and write it to the sector over here. There's an entire file system layer that we're usually interested in there. The logic of how we do timekeeping isn't just, okay, go get the time. But we can implement these things in libraries that are written in the language
13:41
that we are used to dealing with and that we know how to deal with. And then shim out the bits that we think are complicated in a traditional operating system because we've been told operating systems have lots of drivers. But in fact, in this hypervisor context or not, we can have the language run, we can write small shims for it
14:01
and then have the language runtime take care of for us. This, by the way, is a unikernel. So I'm gonna show you some examples of what these libraries actually look like and what it sort of feels like to interact with a system like this. In MirageOS, which is the library operating system
14:21
that I work on, we have a common set of interface definitions for these libraries. So we have a set of module types that tell us, okay, if you're a timekeeping device, you should have these functions that people can call. If you're a low-level network device, I should be able to send a packet.
14:41
If you're a TCP module, I should be able to interact with you in terms of flows. And then we have a whole bunch of implementations of those interfaces. So specific examples of file systems, specific examples of networks. This is also where we have specific ways of handling networks in different hypervisors
15:02
so that we can be general when we are writing our code and then just deploy to the target that we actually end up on. So let's take a look at what this function that I've been complaining about in our C API and in the OCaml interface to the C API looks like.
15:24
In MirageOS, we have a module type TCP that says, okay, I am going to give to you, I'm gonna expose to you this function createConnection. So this type signature is quite a bit more idiomatic OCaml, which unfortunately makes it a little bit more difficult to read
15:41
if you are not an OCaml programmer. But I will try to make it a little bit clear. So in OCaml, we usually carry around some context or some state for a thing in a thing that we often call T. So this is the current state of the way that we're going to send stuff over TCP.
16:00
We're going to take an IP address and a port, and we're going to try to make a connection. The return type is either a flow that you can then write to with other functions that are provided by some implementation of this module type TCP, or it's some error, which is returned in band.
16:20
And so the type of, this whole thing, this flow error result is saying, okay, you'll either get a flow or I will tell you what went wrong. And we have this parameterized over IO, which is a concurrency type. Since this might block, we expose some stuff for doing lightweight cooperative multitasking.
16:49
This particular type signature also uses something that was fairly, just recently merged into the OCaml standard library. This particular result type was something
17:02
that you had to implement yourself up until fairly recently in OCaml 4.03. It was pulled into the main compiler, which means, sorry, let me back up. We were able to change all of these module type signatures, which we had custom error types for all of them
17:22
all over Mirage OS. We were able to unify them with this result type within I think about eight months or so of that being released in the main standard library, which is a turnaround time that I think you probably won't see with most larger operating systems projects. The implementations also look like idiomatic OCaml.
17:43
So if we look under the hood of what CreateConnection is actually doing, it's not making any weird buffers. It's not digging around in global state. It is just taking, okay, give me your TCP connection, give me your destination address and port. I'll make a PCB for it. If it failed, I'll log that and I'll send you the error.
18:02
Otherwise, you'll get the flow that I got. So the way that we actually assemble these into things that you can really use. So what I've shown you so far is conceptually
18:23
how we deal with these lists of implementations and what they are and the list of dependencies and what they are. But what we do in Mirage OS is we take the application that the user wants to run, the list of dependencies, a list of implementations which they can pick when they're actually building their thing,
18:42
and then compose those into either a process that runs on a traditional operating system. So we have a whole bunch of shim layers in these libraries. That mean that when you're trying to develop your application and you just wanna run it locally, you can do that, it's fine. Or you can have a minimal virtual machine that will run in the hypervisor context that I showed you in the diagram just a few slides ago,
19:02
which can run on several different hypervisors now. So I'm gonna show you a quick demo of what it's like to interact with that. I have a web page. Well, I have a web application, actually. It is a Mirage OS Unikernel.
19:22
It has a list of things that it needs. This is the configuration document for the Unikernel, which includes a whole bunch of information about how I wanna get my networking. So this generic stack v4 is just give me some IPv4 stack, the user can decide when they're building the application, how they wanna get it.
19:40
Where my content is going to be, that I wanna build an HTTP server, I wanna use TLS. I can do some customization stuff for ports, where I'll get my certificates from for the TLS connection. Some more customization for the HTTP ports. These are my application-level package dependencies, so these are the things that you might have in a manifest in another language.
20:02
And some composition that says, okay, get all this stuff together. I also need, from the library operating system, I need a POSIX clock, I need two places for data. First off, the actual data that you want me to serve, and then something with the certificates. And then give me something that knows how to deal with HTTP.
20:21
And we're good to go. And the actual logic for my application is in this dispatch file here, which I won't show you, because it's a whole bunch of OCaml that is sort of beside the point, because the one I wanna show you.
20:41
So when I invoke mirage configure, it says, okay, I'm gonna go look at that configuration document and the list of things that you said you wanted, and I'll figure out how to go make that. So part of what it does is it makes me a makefile, so now I can just type make. It's gonna go off, make that application. It'll make a binary called main.native.
21:01
I can run it. And I've got a webpage. Well, thanks. Let's make a unit kernel out of it now.
21:21
So this is the thing that I just showed you when I deploy it. I send it off to Google Cloud. Google Cloud wants, the hypervisor on Google Cloud wants something that knows how to talk to virtio devices, so I'm going to say mirage, please make me a unit kernel that will thrive in this environment. Make me one of those.
21:47
And now what I have is this https.virtio, which I should show you more fully. So I have 16 megabytes, which represents the entire virtual machine
22:01
that I need to run that same website. And you might say, ah, 16 megabytes is kind of a lot for like, all you're really doing is serving web stuff. I have all of the content for that page rolled up in an in-memory file system there, which is part of the reason that it's large. And I should mention for fairness that the binary that we built for Unix,
22:22
which is actually a symlink, sorry, let me show you the size of the operating system, the binary that runs on top of the OS, it's actually bigger.
22:40
So I hope that I've maybe convinced you at least to look into whether library operating systems make it easier to do systems development. And I hope that you have a language that you like enough that you think that it might be nice to be able to talk to your, to be able to build some of these operating systems things in it.
23:00
And now I'd like to convince you that being able to use your language tools makes it easier to do a good job of building these modules that you might want to use for these systems, these sorts of tasks. So library operating systems, there's a really interesting implementation of the ideas behind library operating systems in the Rump kernel project,
23:20
which was a project in NetBSD to take the kernel drivers, basically to make NetBSD be more like a library operating system. And the person who put the time in for doing this tweeted fairly recently that his original motivation for doing this was to make it possible to actually do debugging and testing for these kernel drivers.
23:41
Because once you have these devices in isolation and you can maybe run them in user space or even run them just by themselves in some other kind of space, it's much easier to make conclusions about how they behave in the presence of certain kinds of input. So the tests that we write from MirageOS, they look just like really boring unit tests
24:01
for any other bit of OCaml code that we might write. This is a test of something that is wrong in a lot of implementations of TCP, which is options padding. It is very, very boring, a bit slinging code that you want to test a lot and not screw up, especially if you're in a memory unsafe language, which luckily in OCaml we're not.
24:22
A feature of this code is that it is boring. And the environment in which we run it is also very boring. We just run it in our normal test framework the way that we normally run tests. We don't have to set up a special test harness. We just make our tests and then execute them. No drama.
24:42
Another thing that we can do when all of these implementations are just individual libraries is we can decide, okay, there's the normal library for doing some specific thing that is gonna do its best to implement a spec or whatever. But for my testing purposes, I want one that's kind of broken. I want one that does something unexpected so I can see what my application does
25:03
when that unexpected thing happens. So in MirageOS, we've come up with several sort of fun, I guess, if you like to break stuff, custom implementations of various things like network interfaces that always have new packets waiting.
25:21
A really nice thing to be able to do is always be able to sub out the random number generator that something might be looking at so suddenly all your tests can be deterministic even if they're not cooperating with you and they wanna go read from some source that you haven't set up. In the same vein, you can imagine that having entropy sources that always block is interesting.
25:43
Application code often fails in lots of situations like when the file system is full or a block device is busy or it gets short reads from the network or maybe your DNS has been hijacked and you wanna make sure that you don't do something stupid in that situation. When the implementations that you have for these things
26:02
are just modules, it's really easy to drop in one that's broken in a way that you think is cool. And some more stuff that we can do that is maybe a little bit less obvious. So in a conventional operating system, you have a whole bunch of global state. And the way that we're used to interacting
26:20
with this whole bunch of global state about how the system is running is we open up a shell and we say, okay, let me poke around in here and see what's going on. In the context of a unikernel, we have access to the state that these modules are keeping track of. We have access to how that state is surfaced.
26:42
So instead of us having to go in and try to think of like, okay, what can we think of? The application is doing something weird in this way. What are all of the things that we can think of in the massive set of global state in our operating system that might be affecting how the application is behaving? We can have the operating system tell us what state it's in and even more interestingly,
27:02
why it's in that state. We did some experiments using the library ermine, which is a distributed data store that presents interfaces that look a lot like Git to sort of see, okay, what if every change we made
27:21
to the operating system state had a commit message? So everything that wanted to change anything about this state would have to tell us why and we'd get a log of why. And if we wanted to change it ourselves, we could commit to it and say why we did it and then see what changes in the running OS and in our running application.
27:41
Another interesting thing is, so I mentioned that we have schedulers that we can swap out. Well, what if our scheduler left traces of what it was doing? We had someone in our project, Thomas Leonard, wrote a really nice visualizer of all of the decisions that the scheduler, that the cooperative multitasking scheduler makes
28:00
in a Mirage OS application, saves it to a shared memory buffer and then you can examine it interactively in two dimensions to see, okay, this thread failed, why? And a couple more things. So you might say, okay, Mindy, you've convinced me that I don't need the operating system anymore, but I still need a hypervisor if I'm gonna do this thing.
28:21
And a hypervisor is also not a tiny bit of code, like that's a pretty big dependency as well. So what do you think about that, smart guy? And I say, well, we can do stuff with hypervisors too. There's a really interesting project called Solo5, which has a component UKVM, which takes the device requirement information that we surface
28:41
when we're writing these configuration documents for Mirage OS Unikernels and says, oh, okay, you said you don't need a network device. I'm not surfacing anything about network devices to you. As far as you know, this hypervisor doesn't know what a network is. I'll put a link, there's a link to that project on the end slide. I think it's a very interesting project and very cool.
29:01
And we have a little bit more ongoing work. We have a lot more ongoing work for opening up more hypervisors so that you can run more Unikernels on more desktops. Some work on taking these libraries and using them outside of the context of library operating systems. There's nothing that says that you have to use them only in this context. You can use them wherever you want to do whatever you want.
29:22
We've been doing a lot of interesting work with Cubes OS. Cubes OS is a desktop operating system. It's running on my laptop right now. It is powering these slides before you. Where the conceit is that you have a whole bunch of different virtual machines to enforce separation of concerns between the different things that you're doing. And a problem in Cubes OS is that if you do all of this
29:42
with conventional operating systems, you need a lot of hard drive space, which they solve in part by doing some sharing. And you need a lot of memory to run all those VMs. So it's really nice to be able to have something smaller, both in terms of size and in terms of memory consumption. So we've had a whole bunch of small projects to replace small bits of things in Cubes OS
30:03
that run on big fat Linux VMs with real tiny Unikernels. The most widely used and successful one has been Cubes Mirage Firewall, which I'm using right now. I replaced a Fedora VM that needed, I think, like two gigs of memory or something.
30:21
No, not that much, like 500 meg of memory with a Unikernel that needed 32. We have a whole bunch of other projects that are ongoing right now. If you're interested and you wanna see the slide later, I can pull it back up for you. But your idea can go here. There's plenty of interesting stuff to do with library operating systems and Unikernels.
30:40
There's plenty of space here. And I'm interested in hearing your ideas. I think we have like maybe enough time for a question. But thanks a lot for your time. Thank you very much, Mindy. We actually have time for one question
31:02
and we're going to pick the one from the Signal Angel via the internet because those guys can't catch up with you later on. Oh, no question from the internet? Then we can have one question at microphone one if you're quick. Have you explored using capability-based languages?
31:22
For example, there's a project called MLE, which is a capability-based version of OCaml that provides more isolation between processes. So you can reason more easily about what parts of your application can use what state.
31:41
I personally haven't. I had heard of MLE, but I didn't realize that it had that interesting feature, so I should for sure look at it. Thank you. Okay, that's all the time we have right now. If you have more questions, please catch up with Mindy outside the lecture hall. And thank you again, Mindy, and give her a warm round of applause.
32:00
Thank you very much.