Hi, my name is Roger and I'm here today to present PVH and Also give what I will give a little bit of an overview about the PVH which is a new virtualization mode in Zen And I will also show Some information about how to port an OS to it This will be based on the work

We've done with FreeBSD and port of FreeBSD to PVH First of all, I would like to explain the goals of this presentation The main motivation were to understand PVH and Architecture and to provide some some hints about the implementation details. This will be done by presenting the new

previously PVH port As a starting point, I would like to explain a little bit design architecture probably If you have used Zen before, you know its architecture, but basically Zen runs on top of the hardware

It's a microkernel and it runs directly on top of the hardware and it has access to basically the CPU and the MMIU That's all it takes from the hardware and the rest is assigned to what we call the control domain or DOM 0 That can be either netBSD or Linux. This is done in order to reuse drivers already present in

Operating systems, so Zen doesn't need any device drivers. It doesn't need drivers for block or net or anything like that It's all done by the control domain Then we have this control domain and we have I put two examples one is a fully virtualized domain

Which we usually call HVM and the other one is a para virtualized domain, which we call PV Para virtualized domain can be either Linux or netBSD or free BSD 32 bits and then fully fully virtualized domain can be any kind of guest you can use Windows you can use open BSD. You can do you can use almost any kind of operating system

So we'll start by describing the main issues on PV One of the issues with PV is that it requires a PV MMIU

When PV was designed there were no hardware virtualization extensions So in order to have different guests running on the same hardware, you actually need to para virtualize the memory management unit This is a very intrusive piece of code In the guest OS because you need to actually modify the core of the guest in order to add support for the PV MMIU

NetBSD for example chooses to do this by having a completely separate Memory management Code and you actually have to compile a different kernel to run it under Zen that you have to do to run on bare-metal On Linux they choose to do something different called PV ops Which is that they basically used to choose different MMIU implementations

but this code is really intrusive and both the Linux maintainers are getting a little bit pissed about this because it's hard to maintain and Also, it's really easy to introduce bugs in Zen when you modify code that should not be related to Zen But if you touch code around

PV MMIU stuff you there's a chance that you may actually mess up without even noticing and the other problem with PV guest is that the port the performance of 64 64-bit guest iPhone syscalls, it's slow This is mainly due to the fact that on 32-bit we have three protection levels

We can isolate the guest user space from the guest kernel using page tables And then we can isolate the guest kernel from Zen using something that's called memory segmentation that was present in 32-bit hardware, but that when Doing the transition to 64-bit AMD decided to drop because it was mainly not used by anyone except Zen

So at that point at six they were running on 64-bit hardware Zen had to play a little bit a little trick in order to run 64-bit guest that's the kernel and the guest is Isolated using page tables and then the kernel the guest kernel and the guest user space

Run on the same protection level and Zen takes care of the context switch between user space and guest kernel This is actually quite as low because it's time you have to do a syscall you have to call Zen in order to change from user space to guest kernel space and

It's not really optimal Then we also have HVM guest, but it also has its own issues Mainly it's HVM guest it requires a QEMU instance in the DOM 0 or in a stop domain that takes up memory It it's also using what we call the legacy boot

Which is that we boot and then we call the emulating BIOS and then we jump into the kernel This is a slower than PV And then we also have a bunch of devices emulated inside of Zen in order to provide better speed We have the timers emulated inside of Zen the HP HPET we have the APEC also emulated inside of Zen. So it's actually a little bit slower

so What's PVH basically? is a PV guest run inside of an HVM container and We need to use the best Features of both PV and HVM guests. You don't need any emulation

It's either emulated by hardware or it's done using para-virtualization Even the guest also have native mmiu which is good because we can get rid of the PV mmiu operations and Has access to the same protection level as a normal guest

The code was co-written by Mukesh and Radora Oracle and George Dunlap at Citrix and now we have Table that describes the different kind of guests that you can run on Zen on the top We can see HVM guests which is using hardware virtualization for privilege instruction and page tables

And then it's using software virtualization for everything else This is quite a slow because it means that the access to the IO devices is done using emulation Qm, it handles that emulation, but it's a slow. So we have An improvement over HVM which is HVM with PV drivers, which is basically HVM

But it has para-virtualized drivers for the network R and the disk and then we also have PV HVM which is one step further of the previous one and it's using hardware virtualization for the page tables and for the privileged instructions, but it's using para-virtualization for the disk and

The network and the timers and the interrupts There is still a little problem with that mode is that it requires the legacy boot. You have to boot it Through an emulated BIOS and basically it requires Qm On the other side. We have a pure PV guest we can see that it's using PV for everything

And but we have the problem with 64-bit PV guests that are actually a slower to perform privileged instruction and page table manipulations and In the middle we have PVH that should be using hardware virtualization For the CPU and the privileged instructions and the page tables, but it's using PV for everything else

That means that we have the PV booth path Which is much faster than HVM and that means that we also don't require Qm or any emulation behind it so As we said before PV runs inside of an HVM container So it doesn't make use of the PV and miu it also runs at the normal privilege levels

This means we don't need to do any kind of hyper codes in order to perform privileged instructions And for PVH, we also disabled any kind of emulation that was done in HVM We disabled all the emulation that was done inside of Zen and we don't launch any Qemu or anything like that in order to do

Emulation and it has the nice thing that PVH uses the PV start sequence Which means that we jump into the guest kernel with some basic paging setup And it uses the PV path for several other operations that were that were not used on HVM

Like we use the PV path for vCPU bring up. We use the PV hyper calls and we get the memory map also using PV hyper calls and We also got something from HVM, which is the PVHVM callback mechanism We'll see a little bit more about this in the in further slides But it's basically the way Zen has to inject events into the guest that's done using the same mechanism

That's used on PVHVM Then we have some difference with PV as we saw on the slide the page tables are in full control by guest Page tables are no longer controlled by Zen. The guest is free to modify each page tables as as it wishes

We also use what we call the GPFN inside of the page tables That means that the page tables contains the address that the guest sees Which doesn't have to match to the hardware address underlining It doesn't have to match the physical address on the hardware The translation is done by Zen because it runs inside of an HVM container

Also, the interrupt description table is controlled by the guest it can add as many interrupts as it wants There's no PFN-MFN difference in Zen PV We used to have a difference between what we call a PFN and an MFN PFN was the address that the guest sees inside of the kernel which matched an MFN

Which is an actual real physical address on the hardware And in the case of PVH the guest is only aware of GPFN the translation is done by Zen We also have native Cisco's center And we have the native IO privilege level and we don't have any kind of heaven or fails of callbacks

As the difference we change with HVM If you want to boot a kernel with PVH you need to add what we call elf nodes These are nodes that are appended to the kernel and that Zen reads before booting the kernel It mainly tells Zen which version of the operating system you are trying to boot

The name of the operating system, the entry point into the kernel Which features the kernel expects from Zen in order to boot It's some nodes that you append to the kernel basically It boots with paging enable And there are some difference in the way the grant table and Zen story set up, but they are very minimal

It's maybe five lines of code that you have to add to your grant table or Zen story implementations, basically it also Doesn't have any emulated devices So that means that you don't have the emulated APIC or the emulated timer So you have to use the PV timer since the beginning of the kernel. You cannot use any kind of emulated devices

We also have quite a lot of things that are not yet working on PVH This is something very new that was introduced in this release. Well, it's gonna be new in the release that's gonna come out in maybe One month and we are still missing a lot of things

We are mainly missing super for AMD hardware and right now PVH requires that your hardware is Intel and that it has EPT This is something that we will try to work around for the next version. It's also limited to 64-bit guest It means that you cannot use 32-bit guest. You cannot make use of the virtual TSC

As I said before you actually need to have EPT in hardware because we don't make use of the shadow mode in Zen We also don't have super for vCPU hot plug and there's also no super for migration yet We expect to work around this soon in the next release probably at least most of them in order to have proper PVH support

So now I would like to speak a little bit about the status of previous D before importing PVH support previous D had full PVH VM support it has a working same store and grantable implementation

It has event channel support it has the disk and network front ends and back ends what we usually call block front and Block back and net front and net back. They were fully functional We also had the PVH VM vector callback in order to inject events to specific vCPUs. We also had the PV timer

We also had the PV APIs and we also had the PV suspend and resume sequence as I said in the in the previous slide this shows the difference between what we call the PVH VM vector callback and

previous Mechanism used to inject interrupts into the gas In the previous implementation, we can see that interrupts are injected using a PC interrupt This means that the interrupt is injected by Zen. It's delivered to the Zen PCI driver Then this driver calls the event channel code that ends up calling either

The Zen PV or the Zen PV disk or the Zen PV NIC If we introduce the vector callback the event is delivered directly to the event channel of call and it can be injected to any vCPU and Then you can deliver that interrupt to PV disk to PV NICs and you can also deliver it to you

You can also use it to deliver PV APIs and the PV timer So the implementation of the PV timer in previous D

Contains several components. We have an event timer which is in charge of Delivering events to the operating system. This is done using a single shot timer implemented by a NiPer call Then we also have a time counter that's in charge of keeping track of time. That's done

By reading the information in a shared memory region that's provided by Zen That's called vCPU time info You can search for that in the sources and you will find quite a lot of information about this extract and what it contains It basically contains some values that allow you to calculate the current time

And then we also provide a clock using the information found in that structure So we can get all of our time requirements from the PV interface We don't need to use any kind of emulated device. It's actually quite simple Then we also have the PV APIs

On bare metal APIs are delivered using the local APIC, but as we said before on PVH We don't have the local APIC. So we have to deliver interrupts using event channels instead This was also done for PV HVM because it turns out to be actually faster to use event channels than the look the

emulated local APIC in order to deliver APIs and And Once you deliver the event to to the guest the guest Has this wired into the API handlers So we can get rid of all the emulation overhead that we had when we were using the local APIC

The emulated local APIC. We also had the PV suspend and resume protocol This is actually not that difficult to get it mainly requires you that when you resume you have to rebind all of Your API event channels and you also have to rebind all of the BIRQ event channels that are used for the timers

You have to re-utilize the timers on each CPU. You have to actually Re-utilize the timer from the vCPU itself. You cannot do it from other vCPUs and Finally, you have to reconnect the front-end disk and network. This was also done before PV HVM

So we didn't actually need to touch any code inside of the disk or the network drivers in order to obtain this So what was missing in previous D in order to get PVH support? We were missing the PV entry point into the kernel Since we were using PV HVM, the kernel was booting using the normal boot sequence

So we need to add the PV entry point into the kernel and we need to wire that entry point into the previous debug sequence We also need to fetch the memory map using hyper codes instead of the data provided by the virus We also need to add well to port the PV console from the old PV port into PVH

We have to make it common code so it can be used on 64-bit also We also have to get rid of the usage of any emulated devices In fact the PV HVM on previous D when booting was using the emulated Timers Before the PV timers were able to set up now

We have got written of that and PVH is using the PV timers all along the way since it boots And we also need to add support for bring up of secondary vCPUs This is done using hyper codes and it's in fact much more simple than what you usually have to do in order to bring up

Secondary processors. It will be simple and it's not intrusive code in general And finally we had to cope with the fact that we didn't have a CPI in order to get the hardware description So all the hardware description in PVH comes from Zen store instead of a CPI or any other kind of

hardware description mechanism, so These are lights contain the resulting architecture of the PVH port We have the Zen Nexus, which is the root level device in previous D We actually have a specific Nexus for Zen and

Then we have the Zen PV bus that's Connected directly to the Nexus and the Zen PV bus takes care of attaching the devices needed in order to work on the Zen In this case for PVH the Zen PV bus attaches dummy PV CPU It also attaches a Zen store that in turn

initializes grantable and then it attaches the PV timer and the PV console in order to get output from the root process Then from Zen store as we said, it's the place where we get the hardware description actually We get the disk and the NIC we could have a lot more of them

I've only added two on this slide, but you can have as many disk and NICs as we want And then we also have the control interface. That's a very small driver. That's basically in charge of Reading power events from Zen store like a shut down suspend resume this kind of stuff you'd read from Zen store and it's then wired into the previous the power control mechanism and

Then we also have handling from the Nexus what we call the event channels the event channels are the things that we use to deliver events into the guest and It's directly wired into the Nexus Although the PV disk and the PV NIC use it directly by calling a specific PV

event-channel functions Then I decided to to run some benchmarks on this new implementation in order to find out the performance We expected the performance to be at least a seam At least as fast as PV HVM because we are mostly using the same mechanism that use inside of PV HVM

and well, there are still a lot of things that need more work in order to to perform better but I think that in general it's a description of where we are now Don't expect it to be something set in a stone. We expect to improve that results, but at least it's a starting point

the benchmark were run in a low level Hardware. Well, it's a Xeon but it's not actually server. It has eight ways. It's contains 5 gigabytes of RAM The storage is on LVM in order to obtain better performance and then we limit the number of

VCP used to DOM 0 to only one and the memory that the DOM 0 can see is set to 1 gigabyte For the test we were using Zen 4.4 release candidate 1 and for the DOM 0 kernel We were using a Linux 3.12 release candidate 7

Then on the guest side we run Linux PV 8 and Linux sorry Linux PV Linux PV HVM and Linux PV 8 and then we also run 3vsd PV HVM and previous the PV H in order to have a

performance comparison between both All the guests were using the same The same template each one had four. Sorry four gigabytes of RAM seven vCPUs one Para-virtualized disk and one para-virtualized network interface The same VM was used for all the tests. The only thing that changes the way in which the VM was booted

sometimes it was booted as PV PV H or PV HVM depending on the test that we wanted to run and The code used for the guest kernel can be found in the following G repositories the first one is the repository from Conrad that contains the latest

PV H Linux port and the second one is using my previous the G repository That also contains the latest PV H implementation for previously You can also find all this information on design wiki so you don't actually have to copy these

These addresses if you want to test it afterwards There are in fact wiki bases about how to test Linux PV H and Linux and previous the PV H on the same wiki So we start I started with a for benchmark the first benchmark what it basically does it does

it has a loop that the spines fork and then waits for the child thread to die and Here we count the loops per second. That means the number of times this loop was executed In this case the higher the result the better a throat would because we mean that

Using a specified time as lies we were able to form more times We can see that the Linux PV is really slow on those tests This is because the PV MMIU is quite as low at doing page table manipulations And on the other side we can see that both Linux PV H and Linux PV HVM are mostly equal

We expect them to be exactly equal or maybe PV H to be a little bit faster So it shows us that we still have some performance issues on the PV H implementation This is expected because the implementation is really new on Zen. So there are a lot of things to improve And on previous D. We can actually see that previous the PV H is a little bit faster than previous the PV HVM

But there are more or less equally Then I wanted to run Linux specific benchmark that's called Kermit. What it does is basically compiles the kernel and Outputs the amount of time it took to compile the kernel. This was done using seven jobs

I don't know if you can see on the top the title of the of the graph But it was done using seven parallel jobs We can see that the faster one is Linux PV HVM. Then we have Linux PV H that

And finally we have Linux PV We know that Linux PV is a slower because we saw on the previous four tests that it was a slower doing fork So since Kermit does a lot of forks and a sponge a lot of threads is used is expected that PV is actually slower on this test and Then we can see there's a difference between Linux PV H and Linux PV HVM

We saw that difference on the previous test also on the fork test. We saw that Linux PV HVM was faster So it was expected that it will also be faster on the kernbench benchmark Then I also run what will be the kernbench equivalent of free BSD, which is basically

Building the previous the kernel and we can see that the performance is more or less equal But free BSD PVH is a little bit slower than TV HVM and has higher deviation in this test More or less the lower previous DPV H value is

More or less the same as the previous DPV HVM performance Then I also run spec JVV Which is a Java benchmark? and I was able to run this test on all the different kind of guests But I would not like to make

Comparisons between Linux and previous D on this test because they were actually using different Java virtual machines So it may be due to that that one is actually faster than the other I don't know, but it's kind of hard to compare them if you use different JVMS and actually the previous the

JVM was compiled using C line and the Linux one was compiled using GCC. So it really hard to make comparisons here But we can see that Linux PV is faster on this on this graph This is due to the TLB the low TLB locality of spec JVV, which favors PV

Then we can see that the Linux PV HVM and Linux PVH are more or less equal in terms of performance And finally we can see something that I'm not sure how to explain I think it may be due to the fact that on PVH we don't export We don't export the correct flags in the CPU ID

For example, the PVH implant the Zen implementation I was using here was not reporting super page support to free BSD This is due to the fact that as I said before PVH is still in the works and it's something that we just need to properly report. It's there it supported but it was not being notified to the guest

Then I also run another test which is an ffmpeg encoding using 12 threads In this case, the performance is more or less similar across all guests But we can see that previously PVH is actually slower I don't know if that may be due to the use to the fact that we didn't export the super pages to PVH

I'm not sure because I think ffmpeg shouldn't be using super pages probably but It's hard to say so finally, I would like to Talk a little bit about the future work that we have on both Zen and free BSD Mainly We have now full

3bsd PVH guest support I'm also working on Dom0 support and in fact There are some patches on the mailing list in order to boot 3bsd PVH as Dom0 Don't get so excited because it's only able to boot We don't have any devices needed in order to interact with Zen so you cannot create guest you cannot do anything But you can boot as a Dom0 and 3bsd is able to recognize the hardware behind it, which is real hardware

it's no longer the PV devices because Dom0 has access to the real hardware and Finally, we also need to port the the tool stack. We need to port the Livixe We need to port Livix L We need to make sure that QEMU is working fine on the 3bsd and

That will be more or less what's needed for to get 3bsd Dom0 and on the Zen side as I said before we are missing quite a lot of super we are missing super for AMD CPUs For 32 bit gas for migration for PC pass-through, there's a lot of work to be done, but it's being done by More than what I mean, it's not only me that's doing that

There are quite a lot of other companies working behind PVH. So we expect to have it fully working sometime in the near future then for the conclusions We can see that PVH should provide Performance similar to PVHVM who are being more lightweight. We don't we no longer need to run QEMU in the Dom0

That's great because we are not sucking up memory from the Dom0 It's also much more easier to implement Because it doesn't require a PVMMIU. It doesn't require as much modifications as a pure PV port does and

It's also nice because it serves a lot of similar things with with PVHVM This provides a nice transaction path. Sorry transition pass You can start with pure HVM gas then you can add PVHVM support and finally you can add PVH support If you are going the PV way, you only have a guess that you need to port to PV. There's no

intermediate steps in that process and Finally, we can also make use of hardware assisted utilization, which is faster than what we were doing in PV So as I said before starting the goals of the presentation were to introduce PV to provide an overview and finally to provide some details about the previous the

implementation and I hope that this presentation has motivated people in order to collaborate with either The Linux PVH port, the FreeBSD PVH port or zen in general

Now if you have any questions Do we have a mic or well I can repeat the question in the meantime if you want The question is when we expect PVH to be to beat the PVHVM or to at least be as

performance as PVHVM I'm not really sure because we have identified some issues the one that I said about not Exporting the correct CPU ID flux is an important one. In fact, because it prevents us from making use of super basis

so the Wise answer will be to say that I will have to rerun the stairs with the right CPU ID flux and see what's going on There and then trying to track down what's missing or what's causing the performance penalty on PVH

Yeah The performance sorry From HV well, the question is when we expect companies to move from HVM to PVH that

Depends on What operating system the company is actually using if it's using Linux or previous D It will also be quite a straight forward to move from HVM or PV to PVH

If they are using another operating system that doesn't have PVH support It means you have to port it to PVH. So that's depend on the amount of people you have the amount of work You want to do? I Mean it's hard to say and also it depends on the adoption of PVH by a cloud providers If for example Amazon starts of adding a PVH solution, I expect it's gonna be come much more

Used but it's not under our hands, but thank you