We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

oVirt applying Nova scheduler concepts for data center virtualization

00:00

Formal Metadata

Title
oVirt applying Nova scheduler concepts for data center virtualization
Title of Series
Number of Parts
199
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
For several years now, the oVirt project has been leveraging KVM and relevant technologies (ksm, etc) in data center virtualizations. Being a mature and feature reach, oVirt takes another step forward with introducing a Pluggable Scheduling API. This presentation will review recent oVirt improvements in the areas of VM scheduling. The first part will discuss the architecture of the new scheduler. In the second part we will show samples of VM scheduling plug-ins, and integrate it to a live setup
65
Thumbnail
1:05:10
77
Thumbnail
22:24
78
Thumbnail
26:32
90
115
Thumbnail
41:20
139
Thumbnail
25:17
147
150
Thumbnail
26:18
154
158
161
Thumbnail
51:47
164
Thumbnail
17:38
168
Thumbnail
24:34
176
194
Thumbnail
32:39
195
Thumbnail
34:28
Scheduling (computing)State of matterNumberProjective planeBit
Video gameData storage deviceComputing platformConfiguration spaceNumberData managementHuman migrationElectronic mailing listBitScheduling (computing)Sampling (statistics)Multiplication signConnected spaceSoftwareComputer animation
AlgorithmGraph (mathematics)Distribution (mathematics)Multiplication signSampling (statistics)WeightSlide ruleFilter <Stochastik>Set (mathematics)BefehlsprozessorNumberLogicStructural loadNormal (geometry)Category of beingScheduling (computing)Single-precision floating-point formatPower (physics)Lecture/ConferenceComputer animation
Data centerWeightScheduling (computing)Process (computing)Human migrationBitDecision theoryDomain nameSocial classVirtualizationGraph coloringIterationLecture/ConferenceMeeting/Interview
Domain nameWeightScheduling (computing)Human migrationService (economics)Filter <Stochastik>Structural loadExtension (kinesiology)MereologyLastteilungComputer animation
Set (mathematics)Social classChainFilter <Stochastik>Endliche ModelltheorieResultantWeightTable (information)Category of beingStructural loadDiagramLecture/Conference
LogicKeyboard shortcutFilter <Stochastik>Validity (statistics)Scheduling (computing)Sampling (statistics)Social classField (computer science)BitServer (computing)Computer animationMeeting/InterviewLecture/Conference
Validity (statistics)Server (computing)Physical systemElectronic signatureWeightMultiplication signFilter <Stochastik>Category of beingField (computer science)Order (biology)Structural loadSet (mathematics)BefehlsprozessorMessage passingLetterpress printingJSONXML
WeightFilter <Stochastik>DivisorEndliche ModelltheorieSampling (statistics)Lecture/Conference
Roundness (object)WeightMathematical optimizationState of matterTrailSampling (statistics)CASE <Informatik>Stability theoryDistribution (mathematics)LastteilungConnected spaceLogicServer (computing)1 (number)Structural loadProcess (computing)Endliche ModelltheorieBefehlsprozessorGraph (mathematics)Filter <Stochastik>Chemical equationNumberFilm editingMultiplication signTheoremScheduling (computing)Field (computer science)FrequencySystem callElectronic mailing listHuman migrationCodeSet (mathematics)XML
Social classVariable (mathematics)LastteilungWeightWordScheduling (computing)Power (physics)Open setProcess (computing)Lecture/ConferenceMeeting/Interview
Power (physics)Social classFilter <Stochastik>WeightConfiguration spaceMathematical optimizationPhysical systemVirtuelles privates NetzwerkOpen setChemical equationComputer animation
Multiplication signStudent's t-testTouchscreen
Social classLecture/ConferenceMeeting/Interview
NumberUtility softwareEndliche ModelltheoriePlug-in (computing)CodeElectronic mailing listPhysical systemSystem callSemiconductor memoryExtension (kinesiology)Scheduling (computing)Filter <Stochastik>WeightLogicDefault (computer science)Server (computing)LastteilungDirectory serviceRevision controlService (economics)Group actionFormal languageLogic gateJava appletRight angleBound stateQuicksortExecution unitStaff (military)Pole (complex analysis)Roundness (object)Computer animation
Structural loadData storage devicePhysical lawInformationElectronic mailing listData managementStudent's t-testSemiconductor memoryMassBefehlsprozessorWordDialectScheduling (computing)Endliche ModelltheoriePhysical systemCASE <Informatik>Uniformer RaumVirtual machineAxiom of choiceHuman migrationHand fanInternet service providerSoftwarePower (physics)Parameter (computer programming)Open setMultitier architectureEnterprise architectureCrash (computing)Software frameworkLecture/Conference
BitFood energyJSONXMLUML
Computer animation
XMLComputer animation
Set (mathematics)Software testingPerfect groupComputer animationXML
Lecture/Conference
JSONXML
Multiplication signXML
Transcript: English(auto-generated)
Okay. Can you hear me? Okay. I want to start. Hi, my name is Gilad. I work for Red Hat. I'm a co-maintainer in the Ovid project and I'm working for the SLA and scheduling team. Today I'm going to talk about VM scheduling within Ovid, how it got evolved
and et cetera. Basically we'll talk about the problem of scheduling a little bit. Then we'll go ahead and talk about NOVA filter schedule concepts and then we'll deep dive into Ovid scheduling, giving a lot of samples. I think the best way to
understand stuff are through samples. Okay. Can you raise your hand if you are? It's a home crowd. So Ovid, for those of you who don't know what is Ovid,
Ovid is a management platform for VMs based on KVM hypervisors. It can handle thousands of VMs, has live snapshot, live storage migration, live VM migration, live, everything is live basically. And, sorry, we support also advanced network
configuration for those hosts and a lot of storage connections, Sun, NFS, Gluster, et cetera. Basically let's see what we've got. A long time ago we
got the questions from the user's list. I urge you to use that list if you have any questions about Ovid. It's quite active. How can I define a maximum number of running VMs per host? It's pretty trivial, but we didn't support that yet then. So we get back to it later. I just wanted to say
that you should use the user's list. Okay. What we had in Ovid, a long time ago, basically we had two distribution algorithms for running and migrating VMs. When we selected a VM, when we ran a VM, then we selected a
host based on CPU load policies, either even distribution CPU load or power saving, and then it's pretty limited. We have only two distribution
algorithms and we can't construct a user-defined one. Basically that is it. Then we take a look, we looked at the Nova scheduler, which brought us
filters and weights. Very simple and easy way to filter, to schedule VM on host. Basically the filters are very cut and clear logic that gets a
set of hosts on the left-hand side, then we run a filter on them, then we apply weights on top of them. Another cool slide from Nova scheduler. We collect a set of weights, then we sum them up, and then we
order the host that we got. Let's see a simple sample for Nova scheduler. This is a ROM filter written in Python. Basically it's a very simple method that gets a single host data and a set of properties, and then
run really simple code that either decide if this host is in the scheduling process or excluding it from the scheduling process
according to the requested ROM for that VM. When we looked at it and took it to the data center virtualization, which is over it, we saw that we have a little bit of problem with that concept, because
in Nova each filter and weight is applied on a single host, and we have a larger concept of clustering in over it, which is called migration domain. In the migration domain, each VM on a host can
be migrated to a separate host on that migration domain in that cluster, and we have another concept of load balancing for that cluster. And also a policy container, which a user can decide and create its own policy and apply that on that cluster.
So let's take a look what we had in over it. Basically we can create internal and external filter and weights. The internal part is used for better performance. We are within the server, so we get a quick access to the DB, and originally all the
filters and weights from the previous over back then when we didn't have filters and weights were migrated into that internal scheduler, and the external that basically all the
users can support and extend. We apply all the filters, we apply the filter and weights on all the hosts in the cluster for better performance, and we want to have a better grasp of how the cluster behaves. We have containers called
cluster policies. For each cluster we can define the set of filters, set of weights, and a single load balancing for each cluster. And then we support custom properties that you can, it's kind of passed through to the filter and
weight. This is a diagram of the new model. On the left hand side we can see a set of hosts within the cluster. Then we apply each filter, chaining each filter, one on top of the other, and then we construct a weight table,
which gets us the result of the selected host that we want to schedule the VM on. We had the same concept as NOVA and filters. The existing logic that we had previously, which
were basically validation, was migrated into internal filters and we can extend it in Python using the external scheduler that I will get into later. I want to show
you a really easy sample of how we can use filters. Basically this is a filter in Python. The name of the class will be the name that the server gets. The
properties validation are basically a set of properties that you can add within the server and then the filter will get it. This is the name of the filter and this is actually the signature that you need to override in order to extend a filter, to add a filter to the system.
This is how you get the custom properties within the filter. Basically, I didn't tell you what the filter is all about, but here you can see. We get the time and if the time is within the interval that we get
from the user, then we print the host, we return all the hosts that we got. If not, we just remove, exclude all the hosts from the filter. That's kind of a bank example.
Let's talk about the weights. It weights all the hosts that pass through the filters. We have the predefined weights. The first two are CPU load
filters and then in 3.4, we added a lot more weight models. Weight models is kind of easy now to add because we have the new architecture. Each filter can have factors. We can prioritize the filters.
Each weight can have factors. We can prioritize the weights using factors. We can also add external weights. Let's see another sample. In this sample, we use a connection to the server
using a Python SDK that we have. This SDK is backward compatible and stable. We connect to the SDK and then the logic of the weight is basically
we iterate over all the hosts and we append to a list of the host ID and the weight of that host. Here it's a little bit cut, but the weight of the host is the number of active VMs on that host.
So if we have a host, basically it would be ordered by a number of running VMs within that host. Let's continue, talk about the load balancing. That's the third model we have in the cluster
policy. Each cluster policy can have only one load balancing logic. Basically you can do whatever you want within the load balancing, connect to the SDK and you can shut down everything basically. But
what we support internally is that the load balancing logic will return a VM and a set of hosts. I will show it in a sample later on. And we will migrate that VM, a single VM within the server. It's basically more safe to migrate
a single VM within a period of time not to cause a migration rush and use all our resources for that. It's pretty not safe to do that.
We also have a pretty fine load balancing within Overt, the two legacy ones on CPU and now added in 3.4 even distribution one that we even VM distribution that we didn't have. The same balancing sample that I want to show
as I showed earlier is we get the same numbers that we want to shut down all the VMs but in this example we will actually do that and not exclude those that prevent users from running
hosts VMs like in the filter example. Here we will show how in that hour, in the wake up hour we will basically get all the hosts and if it's wake up hour we will activate all the hosts.
If we need to sleep then we will connect using our SDK and get all the VMs from that host. We will stop all the VMs and we will deactivate the host. Same bank example. This
is how we use it internally to migrate VMs. You get according to some logic you get the overloaded host. It's a code snippet then you select here it's random the first VM on that
host. Then we actually print it because we're using a CDIO to get the data from the model and we return it and we return a right listed host which is kind of a filter. The first filtering
we do before we activate the filter and weights and the normal process that we do. Basically as I said we have a cluster policy which is a container for all the filters and weights and a single load balancing logic and
we have two optimizations for a cluster policy. Speed and overbooking basically we run each time we schedule a VM we run it one by one because we want to prevent overbooking the same. We want to guarantee
the same resources for each VM. If we will try to schedule two VMs together we can fail because they both see the same resources. So basically the speed optimization
is to exclude the weight of the host. So later on the load balancing will do the following for us and balance out the cluster for us and the overbooking is to omit the, I don't know, just be able to
paralyze the scheduling process. Let's see if we can have, show those pictures within Overt. Okay this is
Overt. The VM is you know because of Wi-Fi and VPN the VM is unknown it's running somewhere. Here I can configure
a new cluster policy. I can give it a name like the shutdown one. I can
add the external filter that I've added to the system, the shutdown host filter to the enabled filters. The weight I will give optimal for power saving
that's tried to aggregate all the VMs within a single host as much as possible. Then I will select the balance that I created earlier. I can
give it wake up hour like 8 a.m. and shut down our 8. Press okay. Should be created. Okay let's go back
to the, no, shouldn't take some time.
What? Not connected to the VPN. No, no, something happened to the VPN. Doesn't like me. Yeah I have screen, I have
screen shots but you know maybe it will work second. Okay let's take a look at the screen. It works, believe me. Okay we were, we were
here. Never mind. I will show you
that one. Then you go to cluster. Forget about it. Okay I create the
cluster policy that I've showed you. Then I created to a cluster that I already defined so all the hosts within that cluster will act according to that cluster policy. Okay let's talk
about how it's implemented in the back. It's disabled by default. Whoever wants to extend to add filters should be able to know how to install the external scheduler. The external
scheduler is a separate process written in Python. We externalize it because we want to guarantee the engine safetiness. You know if a user writes a code it can be dangerous to the system. We want to allow other
languages as well. If you know the end overton is written in Java so and this model is written in Python and what we're going for we want to support SAS which is kind of a
scheduling as a service. It's a separate RPM you need to manually install it. How it works basically it's initialized. It runs, it reads
from local directory all the filters weights and balancing logic that you wrote. Then it's publishing an internal API. The engine, the server reads it and then it waits for calls from the
engine for filtering, weighing and balancing. This is how it looks like when it's loaded. The filter and load balancing here. Back to the users list now we can easily write a filter that
maximizes the number of running VMs per host. To sum it up we support
easy Python plugins for you to use for VM scheduling. You can manage a separate policy for each cluster for each group of hosts and every version that comes out for over it gets new models for scheduling. Questions?
So the question is that I have the possibility to read from what the server provides. Memory utilization, CPU utilization. Since you have a Python extensive API if I got it right you
may have the possibility to know if the storage behind or the storage framework behind is constrained so you have the possibility to say all right do a storage migration these virtual machines to the other storage which is SSD based or whatever. So you just need for example to have a good talk with all the storage big guys and
find more things about their IOPS and what we are doing on their storage. Because we have the memory, we have the CPU, maybe we have some network information about throughputs and stuff so what's left behind is the
IOPS and the quality of the storage. You can think of whatever you know You all heard a question? I think so. I think you in within you know when you extend you can do whatever you
want when you extend you know filter or when you basically can connect to every provider or use any SDK if you like so but we provide within the the engine is all that you ask you know memory and CPU load and if you
want to connect to other external provider that it's your own choice. Hi, so we had a few guys in the overt workshop in Bangalore and they
were asking us to connect the scheduler to a BMC system that is monitoring extra parameters for example they can monitor the CPU temperature and the fan speed and they can actually predict that if the fan speed is constant or zero and the CPU
temperature is high that host is going to crash and burn in a few minutes so what they asked from us is to give them a list of hosts and they can actually blacklist some of them because they are aware of more information than what over should have and there are so many other examples which are very similar for example
Cisco has very similar concepts they actually want to blacklist some of the hosts because the networking is going to go down there there are a lot of very similar scenarios in very big enterprise this is actually highly wanted. Absolutely, sounds promising. So now you have the power you can actually do it yourself.
Well my first simple question was all right I know about my CPU my memory maybe I know about my networking because we do the networking so the next good thing is if I had the storage IO information or information that had to do with the quality of my storage perhaps I could utilize
multi-tierings in a storage or storage we motion a virtual machine from one place to the other that could be nice with this engine. More questions?
Go drink a beer. Thank you. This is not it doesn't have a lot of
gain. Okay.
I think it's it's 20 past 20 past six
my talk.
So test test can you hear me is it okay? Perfect.
Okay so welcome everyone.