Thank you for coming. Can you hear me clearly up there? Looks like it. So I'm going to talk a little bit about virtualization in Embedded. There'll be quite a bit about XEN as well,

but I'm going to look at other open source hypervisors as well. But first, actually, we have to ask ourselves, why do you want to embed virtual systems? And to a large degree, it's actually quite similar to server virtualization. It's to one degree about consolidation,

which primarily is about cost, size, weight, and interestingly, in some cases, also about heat and so on as well in an embedded context too. I think there's a few examples. Like in some car environments, like when

you have the little camera which helps you park and stuff like that, that can cause some problems in some areas with heat, because you have a few CPUs in very close proximity, which has to be somewhere in a car. So there's additional aspects to this whole thing as well. Another interesting thing, which

is particularly interesting in the embedded context, it's also about, to some degree, reducing the development cost, being able to port legacy applications or something which are written to a different environment without having to do all the base porting work, because you're working in a virtual environment.

It's also around security and safety can be big concerns in that space. So one thing which you typically want to do in an embedded environment is to support it. This is called mixed criticality compositions, which

means you want to be able to run applications in different containers, if you want so, with different safety, security, and real time requirements. And one of the key things in some markets is you have to have a safety certified hypervisor

if you use a hypervisor technology for isolation. And then you have a whole set of additional embedded requirements, such as low interrupt latency, low or zero scheduling overhead. And particularly, if you do something with rich I.O., which is quite often the case for, say, automotive applications,

you need to have a lot of drivers for specialized I.O., touch screens, all these kind of things which you normally have on a phone. And to support all this, you need to have a fairly flexible architecture to build widely varying systems, if you want so.

So let's just really do a quick reminder of hypervisor architectures. I'm going to focus on ARM only, because that's sort of the main player in this space. So if you look at ARM privilege layer levels, you have EL0, which is user mode, EL1, kernel mode. And then you have that EL2 space, which

is where a hypervisor would run in. And then underneath, you would have something like TrustZone as well. And there's some interactions between TrustZone and Hypervisor's too, but that's going to get a little bit complicated. So I'm not going to go into this. So how does a hypervisor, how would they typically

map into this model? So your classical Type 1 would run in EL2. You would have your drivers run as part of the hypervisor. Then you have all your virtual machines sitting on top of it. And you have this Type 2 kind of model, where you have fundamentally the host kernel and the hypervisor kind of run together.

Like if you look at KVM, the hypervisor's kind of KVMIO kernel module. Then you do have this whole user space box, which comes with the host. And then you have your VMs there. And what's interesting, if you look at it from a control plane perspective in a model on the right,

you can just use the user space of the host for control, whereas typically what you would do in a model on the left is you either have some kind of interface where you externally control the whole system, or you have some bit of code as a module in there, or you use one dedicated VM to do that. So let's just look at a few examples of hypervisors

today in that space. So this one, which is kind of really interesting, is called L4e. It kind of comes from a white heritage of hypervisors. There's a whole family called the L4 family, kind of originally were developed

for the whole mobile space. This one keeps on coming up in discussions about automotive. It's written specifically for mixed criticality compositions, a typical microkernel design. In a kernel space, really, they're just to address brace, implementation threads,

and IPC kind of functionality. Everything else is in user space. It's got a solid feature set. I think it is a little bit weak on hardware support. It's also kind of arguably arguable, particularly at this audience at Fosten when it's really open source.

It is GPL, but it's got a dual license, and you have to sign, you know, do copyright assignment and all those kind of stuff if you want to contribute. Also, there seems to be no real community around it. But you know, this could go somewhere. Another one which occasionally comes up here is jailhouse.

It's relatively lightweight, less than 10,000 lines of code. It's a petitioning hypervisor. It comes from Siemens. Community-wise, relatively small.

I think 80% of all the code seems to come from one developer. They use Linux to kind of bootstrap the system, and then after the bootstrap, it's kind of, you know, still partly used for control plane. The focus there is really on petitioning, but not so much on virtualization. So they don't have things like scheduler, you know, IO emulator and stuff like that.

It seems to have a really good basic feature set, but there's some areas which are, you know, still under development, I think particularly because we're talking about arm virtualization here. You know, the arm 64-bit support is fairly new.

There's also really interesting talk from the AGL which was given earlier this year, which you can get to. The previous slide has a similar more in-depth discussion here. Then we have XEN, fairly, you know, general-purpose hypervisor.

Most people know it from cloud computing and basic server virtualization, but as we will show, it's actually used in a lot of other market segments today, too. Again, you know, we have Linux and NetBSD to bootstrap. Got quite an expensive feature set. It's highly customizable, and we

have a fairly strong and diverse source community around it. And the following picture kind of gives you a sense of this. Well, this just shows, you know, like different vendors having good products and services around it. If you look at the top right, that's sort of traditional server work. Then we have cloud computing.

Then the top left bottom, that's kind of security-focused products. And we'll kind of go a little bit into this because security is a big aspect of the whole embedded story as well, and then we do have quite a few embedded products as well.

And actually, I started working with XEN around 2011, and really what happened is that whole security segment started to pick up around that time, and it's grown since. There's been a push which doubled the number of players pretty much in the last two or three years,

which was around that time when all this embedded stuff just started happening. We didn't plan for it. It just happened, you know, basically because there seems to be a good match for it. So let's draw this back on. We talked about hypervisor architectures before. It brings us back to what is actually XEN

because, you know, it's kind of a type one, but it's kind of not a traditional architecture. So one of the differences between the traditional type one and XEN is that the device drivers don't run in EL2 and not in a hypervisor. We basically reuse the drivers from DOM zero,

and then basically because we have the specialized, you know, DOM zero virtual machine, that's then also our control plane. Strong isolations of example drivers are running in the L1. As I said, in the L4E, they're running in user space, so that's obviously a little better.

And then we have these grant tables which kind of provide additional isolation between different VMs. From a control plane perspective, you know, on a server system, you know, the sysadmin will basically, you know, use DOM zero or some graphical tool, you know, which connects to it to control your system.

In embedded, you need a lot less. It's really, you know, primarily use the control plane for a config and setup, and then basically primarily for system health monitoring, you know, applying possibly software updates and those kind of things. So you need much less functionality there,

so you know, having a big distro running in DOM zero isn't really necessary. So the people in this space tend to do now, today is just use a basic Linux code, you know, with a root file system or something like that, but this could be made smaller, and there's some work going on in this area.

And that shows you that control plane. So let's talk about some of, so we talked about some of the specific requirements in embedded, and one of them was around PV drivers, which is IO and so on and so forth.

So there's been quite a lot of momentum in our community for a while, so we have all the standard stuff around disk and so on that's been around forever. There's also some GPU sharing, it's not traditionally a PV protocol and the way how it kind of works, it's a slightly different model, but it has the same kind of effect.

There's also work going on, and this is being driven by some automotive vendors around general coprocessor sharing, so that taking some of the ideas we used for GPU sharing and extending them to FPGAs and anything which might kind of look like a coprocessor because it's the same kind of principle.

Some prototypes around there right now that have made some design changes recently, and we'll probably have a fully baked framework come this year. Then there's been quite a few new ones, some 9PFS PV cores, some POSIX forward, a core forwarding

where you have your POSIX API in one VM and then you basically just have a shim and it roots cores through. It's actually quite interesting. There's been benchmarks as well and there's things like multi-touch, sound display, DRM. And what's really interesting, and I find there's a lot of the embedded companies,

they just tend to, embedded companies, they just take stuff, they hack it, and then they push it out. And so we found a lot of them are writing their own drivers and a couple of them are managed to get them to upstream it and standardize, which is kind of good,

but it's relatively easy to write those drivers, particularly if you don't care about standardizing the ABI, which you can do if you do this for your own product. So let's look at some security properties of XEN.

Security is kind of important. So apart from the typical hypervisor functionality, you have functionality which enables you to do further system partitioning. So we already touched on this. You can basically disaggregate your system in a sense. You can write, have run device drivers in a specific VM.

Your network drivers are typically a main route of attack for a system. And you can do this for other drivers as well. You can put other system components into separate VMs as well, and then kind of plumb it all up to make a distributed system.

There's also the capability of controlling the capabilities of a specific VM with the XEN of equivalent of SELinux. And that kind of enables you to build a multilayered security approach, and that's actually being used in some of those products,

which, some of the security products which started appearing a few years ago. There's other security features like trusted execution environment and stuff like that, virtual machine introspection, L2PM live patching, all this kind of stuff, but it's not quite clear

whether that's interesting for that, for that whole embedded segment. I mean, I could imagine that virtual machine introspection could be something interesting for health monitoring of a system, but it's not quite clear how this would fit in. As we're talking about security,

I think we do also have to talk about Meltdown and Spectre quite quickly. I don't know whether we have lots of talks about this yet today, but obviously Meltdown is the most dramatic one. From an extent perspective, it can't really be exploited in fully virtualized VMs.

So fundamentally on ARM, we don't use software virtualization, we just use hardware extensions. So from that viewpoint, we're actually fine. And on x86 with HVM and PVH, that's true as well. We do have mitigations, but I think for that embedded and automotive use case,

we don't have to deal with the performance overhead, some of those fixes will give us. Spectre is a lot more interesting. We started going through the code base at this stage manually, hopefully at some point, tools like Covarity and stuff will help us find some of the kind of gadgets and code patterns

which could be used to exploit those vulnerabilities. But it's a lot harder to kind of go down this route in XM based system because, you know, to exploit Spectre, you need something called a gadget.

So in a Linux kernel, there's this eBPF engine, which was, you know, by Google project zero used to on KVM to demonstrate how you get information out. We have gone through the code and, I mean, not line by line, but through the architecture,

and we don't think there's anything similar to that in the code base. But, you know, at the end of the day, we won't really be sure unless there's really tools which allow us to verify that. So hopefully that should be fine. Let's look at one of the other techniques. So this is whole sandboxing desegregation thing

where you can run drivers in different VMs. So how does this work? Well, you have a regular, you know, your DOM zero on the right, a regular application running in a VM, and then, you know, we have this thing called a storage domain, which would have a disk driver in it,

or a network domain, which has a network driver in it. And then basically, DOM zero kind of connects the topology and, you know, your application will talk to the backend, it will talk to the frontend, it will talk to the backend in the storage domain, it will talk to the real disk driver,

and it will then talk to the controller, and the same can be done for a network. And fundamentally, the same can be done for any IO, pretty much. Then there's this thing called XSM or Flask, which is like XSM's variant of SE minutes.

So here you have your VM, you write the policy, which controls, you know, what you can do with various interfaces, so you would typically, you know, define some classes of VMs, and then assign those classes to specific workloads in VMs when they come up. That gives extra protection, because then you can only do certain things,

certain operations from within this VM, everything else gets blocked. It's disabled by default, so you do have to enable it by kconfig, and it's very similar to the Linux security modules, it uses the same policy syntax, you can use all the same tools.

It's just different new objects, which are used on top of it, and fundamentally, the guys who do that use XSM together with SE Linux in their systems. If you want to know more, there's a little bit of documentation. The slides will be online via the Fosbin site,

and also, you know, on my slideshow channel. So, that's some of the terminology around security. So, kind of want to look a little bit into how this actually works. So we have a number of security-based products in XSM, so there's QubesOS, QubesOS, they have a boost here

at Fosbin, go find them if you're interested. There's OpenXT, which is, in some sense, similar to QubesOS, it's just an open source project, which allows you to build QubesOS-type of applications, and then there's a product called Crucible Defense from a company called Starlab,

and they're targeting military-type applications, it's kind of certified product, and so on and so forth. So, I wanted to briefly show how some of this stuff works in practice. So, QubesOS, Edward Snowden likes it, which, you know, is probably a good thing,

and how does this work? So you have your typical Fedora UI here, on the top right, you see kind of all the VMs are running and then on the left, you kind of, it's like an application starter, like on Windows, where you can basically define which application runs in which VM. Under the, behind the hood, you have Xend,

the DOM0 kind of does all this UI stuff, there's a network domain, they have a firewall VM, which controls all the firewall policies, there's something called a USB service domain, where they put a USB device in, that gets started in a VM, and then you have, you know, those defined different VMs where you run different applications in,

and if, for example, you then start, you know, an application, then, you know, like your network traffic would be routed to those VMs, and then your USB traffic would be routed through that, so you kind of get this extra isolation, which makes it harder for exploits to infect your entire system, because you have all these little sandboxes in place.

And they're going down some interesting routes of being able to host some of this in clouds as well. So go talk to them if you're interested in this. So embedded in automotive. One of the interesting things is being able to partition a system. So in XEN, we have a lot of different schedulers

with different properties, and you can assign these schedulers to, you know, different groups of, you know, CPUs on the system, which shows, there's like a, we have an amin scheduler, there's a hard real-time standard, there's a soft real-time scheduler, you have a regular VM scheduler,

and then there's the capability to pin virtual machines to CPUs without overhead using the entire management stack. So that kind of doesn't give you a lot, you know, it does give you still those virtualization boundaries and all the tools, but you know, it's a very attractive proposition.

Here's the same thing again in terms of overviews. I'm not gonna go through this in a lot of detail. Have a look at this later on if you're interested or talk to me afterwards. Another interesting property, this is around interrupt latency.

So we have this concept of interrupts where physical always follow virtual interrupts. So when an interrupt gets injected, it's always injected on a CPU, which is running a virtual CPU. If for whatever reason that is moved by the scheduler,

then at one point that gets routed back to the other virtual CPU and the next time around, it comes in the normal way. And what's interesting is that actually that whole approach really gives you fairly low

interrupt latency. So this was around 2,000 nanoseconds with the null scheduler. And that's the sort of maximum over a specific run and there's a whole blog post about that too. I think one of the things we do need to talk about,

particularly if you talk about automotive, is safety certification because that's a big, difficult issue in open source. So there's several levels to this. So one is around code quality, typically you would have to run special static analysis tools which are quite expensive, which where you have to comply

to the cert mis-re-coding standards. We've been running a project with a tools vendor who's been doing this for us for free and we're starting to work through all the issues. So that's kind of a good first step. Another thing around safety certification is you have to prepare a lot of artifacts all the way through the development process

from requirements all the way how it gets implemented in the code. This is kind of tough in an open source project because it's a lot about reverse engineering. We have had two companies who have done this so we know it's doable.

So at the end of the day it's just a cost thing. But then those guys of course have a commercial project and they pass it on to, they pass the cost on to their customers. And you kind of lose some of the sharing stuff around open source. Development process is SORNY.

Open source development process does not comply with safety standards. There is a way to do that but that means backfitting a lot of stuff so that's quite expensive. That brings us then to the potential about how could you do this better in open source?

So one idea is that you could have some sort of official code line or project supported code line somewhere and that you get the interested parties to kind of do, maybe create a consortium or some sort of structure to do this together and share some of the cost. There's a lot of discussion around this going on

for various projects and so on in Linaro and in the auto, AGL, you know, automated and so on. The biggest thing though is you need to have somebody to blame if something goes wrong. And I mean that's one of the main purposes of certification really.

So this is interesting, this is evolving. I don't know whether open source can play a role in there but we're gonna try. So let's just look at some examples of vendors who are using send and embed it as well as an automotive and a quick demo and then close. So one of the vendors we have is DonorWorks.

They started out as a small consulting company in the US. They're now maintaining three send distros. They used to have different names but they've kind of branded them now together. It's called as like Virtuosity. That's Virtuosity for NXP and for Xilinx. Their core Virtuosity product is actually

certified, safety certified to various different standards. It's quite a long list down there. There is a company called StarLab. They're also going through that whole certification process right now and then we have a few, you know, that means that we actually know

this stuff can be certified, right? It's doable at some reasonable cost. Those companies aren't huge so you can do it. And then there's a number of others who play in that space too. Automotive, it gets even more interesting. So there is a company called GlobeLogic. They're a tier two automotive OEM.

They have a product based on XEN. Done some contributions over the years. Right now they're very focused on bringing some of this stuff to market. They have a competitor called EPAM. It's quite similar what they're doing. I like EPAM more because they're a lot more actively engaged in the community

and they have some interesting features where they look at the sort of cloud automotive interface. I'm gonna show you a demo quickly. And then we know that LG Electronics, Bosch, and a couple of others are playing with this whole stuff too. So demos, so those who are not gonna be there or not here, they're gonna see that demo

by going to the link. I'm just gonna try and play this now and see what happens. The internet of transportation is becoming a reality and soon individual vehicles will become

a part of this internet. Vehicles need to keep up with the latest safety standards and regulations. But combining internet enabled software with mission critical functions on the same platform creates quite a challenge for the industry. EPAM's CloudFusion platform is a unique solution that solves this challenge.

It includes hardware isolation enabled by virtualization, rapid services deployment using containers, and end-to-end security for full cloud integration. With the EPAM CloudFusion platform, traditional in-car software stacks run on independent hardware using a special virtual machine dedicated to cloud managed

dynamically deployed services. The instrument cluster running a custom Linux instance implements some basic vehicle functions. In-vehicle infotainment based on AGL IBI provides user applications and media features. The fusion virtual machine is sandboxed from hardware

and can interact with the rest of the system using an OEM managed set of restricted APIs. Each cloud service is deployed to the virtual machine in a container. Deployed services cannot interfere with other processes. In this demo, the cloud service collects telematics data from the vehicle. Using a micro service based architecture,

part of the cloud application is deployed inside of the fusion container. Importantly, deployment of the app does not require any prior integration with the car OEM or tier one. To demonstrate the simplicity of the deployment, we'll enable some new features in the cloud application. Fuel consumption, engine RPM, and gear settings.

In this scenario, only some of the desired features are available from the in-car applications. Via a web application interface, the vehicle part of the application is updated separately and new in-vehicle functions are enabled. Notice that the rest of the system operation is not interrupted during the service update. Even if the update fails or crashes

or is otherwise unable to run, operation of the critical system is not affected due to the isolation of the fusion virtual machine. After the vehicle service finishes updating, we see that new functions have been enabled in the vehicle. Fusion is based on open source software technologies. With fusion, new automotive...

Okay, that's enough. Anyway, so you get the idea and it's actually a really interesting use case. So looking at this diagram, that was the diagram you kind of saw earlier, but what's interesting again is that they're actually using some of these driver.

Rather than having specific drivers in separate virtual machines, they kind of split them up. They have the drivers for one application in a different VM and they're sort of cutting some corners there. But it's basically the same concept. So, just to summarize,

you know, why is XEN really interesting in this space? As I said earlier, we're not really pushing it. Haven't really starting to push this very hard because at the end of the day, this just happened. But, you know, after really two years of watching this, you know, there's really something there

which is worth building on. So, you know, it's quite a flexible platform. Used in a lot of different market environments. It's apparently relatively easy to port to new environments, otherwise you wouldn't see an explosion of so many different products. It's highly customizable and part of it

is also relatively easy to develop new drivers. There's quite a lot of security and resilience functionality there. Obviously, if we ever want automotive to become interesting, we're going to have to deal with the functional safety aspect.

We're kind of looking at ways now how this could potentially be done. It's going to be a hard nut to crack. It's not going to be quick. But, you know, everybody, a lot of other projects are currently looking at the same thing. You know, Linux has to solve this somehow for automotive-grade Linux. There's a lot of other projects which have to kind of help solve this

and we're starting to work together. There's a couple of challenges, really, which we're looking at. So, we have a lot of different IO drivers, some of them not open source, and we've been working with some of the vendors to standardize on some of the protocols. The same is happening around GPU virtualization. We're looking at maybe a more minimal Don Zero.

You know, like, maybe we can use some sort of RTOS for that. That would be interesting. And testing has been a surprising challenge for Embedded as well. So, again, there we've been working, for example, we've been working with Renesas now to get some of our car gen boards

into a server chassis with some management functions such that we can actually deal with this in the same way as with other hardware. And hopefully, some of the hardware specs for that will go open source, and then we can do the same for some other similar kind of, you know,

boards in a similar form factors. And that's really it for me today. If you have questions, we do have some time, do we? I'm all right. You stressed me. I could have talked for a little bit longer. No, I'm all right. There was a question out there.

Was it you? Ah. Well, I think at the end of the day, it comes down to, you know, like,

I'm not telling you, you have, oh. So the question was, when to not use XEN? I think it ultimately depends on your use case and preferences, right? So if you look at XEN today, it's a lot more complicated in many ways

than, for example, KVM. We've been hurting from that a little bit in the traditional server virtualization space because KVM is a lot easier to use. So I think, you know, really, if you look at, if you want to have simplicity and kind of maybe more ease of use,

maybe there's better competitors. If you're looking at some more complex stuff where you want to blanch things together and you want to, you're actually okay with taking the overhead of the extra complexity then that's probably for you, right?

Okay, so the question was around Intel support and, you know, particularly maybe in some industrial areas, in industrial, and I guess you're, there's a lot of these,

how do I call them, plug computers, kind of this Intel in use. So there is, there is some momentum there. I have one, two startups who are working on a product in this space and are probably going to use Intel architecture first to push into that market.

It's probably gonna be a container element in it as well. It is a lot more challenging from, I think anything which will require safety certification, it's a no-go area for, at least for XEN because the code base is a lot bigger, the arm port is around,

depending on how much you use or not between 20 or 30,000 lines of code, the x86 stuff is a lot more and quite often you need to use Q new as well, right? So that's obviously a barrier but I think in some market segments,

it's definitely a possibility. So the question was, the companies who did the safety certification for,

who succeeded for, actually I think, so Derner Works, I believe, did it both for arm and x86 but the x86 configuration would have to be very cut down and constrained, right? Because otherwise it would be too cost intensive.

And particularly in that, you know, today we have like PD, PV, you have to forget, right? I think it might become, you know, HVM then you have to deal with the Q new aspect. I don't know, you know, how we deal with this, you will probably have to use a Q new

and pretty much take everything out which XEN doesn't need, it's actually a lot, there's only very little XEN actually needs but then you have to maintain that fork, right? Maybe PVH is actually, when this comes more widely used, that's sort of probably the interesting route to take.

Yep, yeah, so power management is a really interesting challenge in this space right now. So this is something, oh,

so the question was, if you have drivers in different VMs, there are, you know, properties like, you know, power management which could cause problems and you need to kind of address this. So, in fact, there's a really interesting story around this. So, you know, I met some of the Xilinx guys who have their, you know, distro

which have a XEN based distro. Actually, we talked to them, I think in 2015, they invited me to an event, wanted a request to the meeting, asked a lot of questions about XEN. Two years later, I didn't hear anything,

they just launched a product. And it was apparently quite successful, you know, a lot of their users use it, but one of the things they have come across is exactly this whole issue with power management and stuff like that. But not only in the context of, not only in the context of, you know,

disaggregated drivers, but also higher up when you look at workloads. On a driver side, you could, you know, you could, you could, you know, focus on a number of operating systems which you allow and you can plumb some functionality in there such that that kind of gets coordinated. But, you know, like,

what happens even further up the stack, right? If you have, so if I look at this Xilinx XEN based product, they allow, they allow you to have, they have something called bare metal VMs which are almost like traditional ROM images with a support library and you just can bang, you know, one of these old images into a VM and they offer them,

lots of people use it. And then they have support for different artists as well. But then you start, unless you have all the power management and all those sort of heterogeneous kind of coordination, it's not just about power management and other properties as well, then you're gonna start to get into trouble. And I see some momentum building in this area

to start resolving some of these problems. We're out of time apparently. So, all right. And then my colleague here will follow up afterwards. So find me afterwards or at XEN based if you have questions.