oVirt applying Nova scheduler concepts for data center virtualization
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32599 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2014132 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Scheduling (computing)State of matterNumberProjective planeBit
01:28
Video gameData storage deviceComputing platformConfiguration spaceNumberData managementHuman migrationElectronic mailing listBitScheduling (computing)Sampling (statistics)Multiplication signConnected spaceSoftwareComputer animation
02:57
AlgorithmGraph (mathematics)Distribution (mathematics)Multiplication signSampling (statistics)WeightSlide ruleFilter <Stochastik>Set (mathematics)BefehlsprozessorNumberLogicStructural loadNormal (geometry)Category of beingScheduling (computing)Single-precision floating-point formatPower (physics)Lecture/ConferenceComputer animation
05:05
Data centerWeightScheduling (computing)Process (computing)Human migrationBitDecision theoryDomain nameSocial classVirtualizationGraph coloringIterationLecture/ConferenceMeeting/Interview
06:00
Domain nameWeightScheduling (computing)Human migrationService (economics)Filter <Stochastik>Structural loadExtension (kinesiology)MereologyLastteilungComputer animation
07:21
Set (mathematics)Social classChainFilter <Stochastik>Endliche ModelltheorieResultantWeightTable (information)Category of beingStructural loadDiagramLecture/Conference
08:15
LogicKeyboard shortcutFilter <Stochastik>Validity (statistics)Scheduling (computing)Sampling (statistics)Social classField (computer science)BitServer (computing)Computer animationMeeting/InterviewLecture/Conference
08:56
Validity (statistics)Server (computing)Physical systemElectronic signatureWeightMultiplication signFilter <Stochastik>Category of beingField (computer science)Order (biology)Structural loadSet (mathematics)BefehlsprozessorMessage passingLetterpress printingJSONXML
10:24
WeightFilter <Stochastik>DivisorEndliche ModelltheorieSampling (statistics)Lecture/Conference
10:56
Roundness (object)WeightMathematical optimizationState of matterTrailSampling (statistics)CASE <Informatik>Stability theoryDistribution (mathematics)LastteilungConnected spaceLogicServer (computing)1 (number)Structural loadProcess (computing)Endliche ModelltheorieBefehlsprozessorGraph (mathematics)Filter <Stochastik>Chemical equationNumberFilm editingMultiplication signTheoremScheduling (computing)Field (computer science)FrequencySystem callElectronic mailing listHuman migrationCodeSet (mathematics)XML
16:04
Social classVariable (mathematics)LastteilungWeightWordScheduling (computing)Power (physics)Open setProcess (computing)Lecture/ConferenceMeeting/Interview
16:49
Power (physics)Social classFilter <Stochastik>WeightConfiguration spaceMathematical optimizationPhysical systemVirtuelles privates NetzwerkOpen setChemical equationComputer animation
18:34
Multiplication signStudent's t-testTouchscreen
19:50
Social classLecture/ConferenceMeeting/Interview
20:22
NumberUtility softwareEndliche ModelltheoriePlug-in (computing)CodeElectronic mailing listPhysical systemSystem callSemiconductor memoryExtension (kinesiology)Scheduling (computing)Filter <Stochastik>WeightLogicDefault (computer science)Server (computing)LastteilungDirectory serviceRevision controlService (economics)Group actionFormal languageLogic gateJava appletRight angleBound stateQuicksortExecution unitStaff (military)Pole (complex analysis)Roundness (object)Computer animation
23:20
Structural loadData storage devicePhysical lawInformationElectronic mailing listData managementStudent's t-testSemiconductor memoryMassBefehlsprozessorWordDialectScheduling (computing)Endliche ModelltheoriePhysical systemCASE <Informatik>Uniformer RaumVirtual machineAxiom of choiceHuman migrationHand fanInternet service providerSoftwarePower (physics)Parameter (computer programming)Open setMultitier architectureEnterprise architectureCrash (computing)Software frameworkLecture/Conference
26:44
BitFood energyJSONXMLUML
31:28
Computer animation
32:21
XMLComputer animation
32:55
Set (mathematics)Software testingPerfect groupComputer animationXML
35:13
Lecture/Conference
36:52
JSONXML
38:45
Multiplication signXML
Transcript: English(auto-generated)
01:01
Okay. Can you hear me? Okay. I want to start. Hi, my name is Gilad. I work for Red Hat. I'm a co-maintainer in the Ovid project and I'm working for the SLA and scheduling team. Today I'm going to talk about VM scheduling within Ovid, how it got evolved
01:23
and et cetera. Basically we'll talk about the problem of scheduling a little bit. Then we'll go ahead and talk about NOVA filter schedule concepts and then we'll deep dive into Ovid scheduling, giving a lot of samples. I think the best way to
01:43
understand stuff are through samples. Okay. Can you raise your hand if you are? It's a home crowd. So Ovid, for those of you who don't know what is Ovid,
02:03
Ovid is a management platform for VMs based on KVM hypervisors. It can handle thousands of VMs, has live snapshot, live storage migration, live VM migration, live, everything is live basically. And, sorry, we support also advanced network
02:27
configuration for those hosts and a lot of storage connections, Sun, NFS, Gluster, et cetera. Basically let's see what we've got. A long time ago we
02:41
got the questions from the user's list. I urge you to use that list if you have any questions about Ovid. It's quite active. How can I define a maximum number of running VMs per host? It's pretty trivial, but we didn't support that yet then. So we get back to it later. I just wanted to say
03:06
that you should use the user's list. Okay. What we had in Ovid, a long time ago, basically we had two distribution algorithms for running and migrating VMs. When we selected a VM, when we ran a VM, then we selected a
03:26
host based on CPU load policies, either even distribution CPU load or power saving, and then it's pretty limited. We have only two distribution
03:41
algorithms and we can't construct a user-defined one. Basically that is it. Then we take a look, we looked at the Nova scheduler, which brought us
04:05
filters and weights. Very simple and easy way to filter, to schedule VM on host. Basically the filters are very cut and clear logic that gets a
04:20
set of hosts on the left-hand side, then we run a filter on them, then we apply weights on top of them. Another cool slide from Nova scheduler. We collect a set of weights, then we sum them up, and then we
04:41
order the host that we got. Let's see a simple sample for Nova scheduler. This is a ROM filter written in Python. Basically it's a very simple method that gets a single host data and a set of properties, and then
05:05
run really simple code that either decide if this host is in the scheduling process or excluding it from the scheduling process
05:20
according to the requested ROM for that VM. When we looked at it and took it to the data center virtualization, which is over it, we saw that we have a little bit of problem with that concept, because
05:43
in Nova each filter and weight is applied on a single host, and we have a larger concept of clustering in over it, which is called migration domain. In the migration domain, each VM on a host can
06:02
be migrated to a separate host on that migration domain in that cluster, and we have another concept of load balancing for that cluster. And also a policy container, which a user can decide and create its own policy and apply that on that cluster.
06:22
So let's take a look what we had in over it. Basically we can create internal and external filter and weights. The internal part is used for better performance. We are within the server, so we get a quick access to the DB, and originally all the
06:47
filters and weights from the previous over back then when we didn't have filters and weights were migrated into that internal scheduler, and the external that basically all the
07:03
users can support and extend. We apply all the filters, we apply the filter and weights on all the hosts in the cluster for better performance, and we want to have a better grasp of how the cluster behaves. We have containers called
07:25
cluster policies. For each cluster we can define the set of filters, set of weights, and a single load balancing for each cluster. And then we support custom properties that you can, it's kind of passed through to the filter and
07:42
weight. This is a diagram of the new model. On the left hand side we can see a set of hosts within the cluster. Then we apply each filter, chaining each filter, one on top of the other, and then we construct a weight table,
08:03
which gets us the result of the selected host that we want to schedule the VM on. We had the same concept as NOVA and filters. The existing logic that we had previously, which
08:23
were basically validation, was migrated into internal filters and we can extend it in Python using the external scheduler that I will get into later. I want to show
08:40
you a really easy sample of how we can use filters. Basically this is a filter in Python. The name of the class will be the name that the server gets. The
09:02
properties validation are basically a set of properties that you can add within the server and then the filter will get it. This is the name of the filter and this is actually the signature that you need to override in order to extend a filter, to add a filter to the system.
09:24
This is how you get the custom properties within the filter. Basically, I didn't tell you what the filter is all about, but here you can see. We get the time and if the time is within the interval that we get
09:43
from the user, then we print the host, we return all the hosts that we got. If not, we just remove, exclude all the hosts from the filter. That's kind of a bank example.
10:05
Let's talk about the weights. It weights all the hosts that pass through the filters. We have the predefined weights. The first two are CPU load
10:21
filters and then in 3.4, we added a lot more weight models. Weight models is kind of easy now to add because we have the new architecture. Each filter can have factors. We can prioritize the filters.
10:44
Each weight can have factors. We can prioritize the weights using factors. We can also add external weights. Let's see another sample. In this sample, we use a connection to the server
11:05
using a Python SDK that we have. This SDK is backward compatible and stable. We connect to the SDK and then the logic of the weight is basically
11:21
we iterate over all the hosts and we append to a list of the host ID and the weight of that host. Here it's a little bit cut, but the weight of the host is the number of active VMs on that host.
11:42
So if we have a host, basically it would be ordered by a number of running VMs within that host. Let's continue, talk about the load balancing. That's the third model we have in the cluster
12:03
policy. Each cluster policy can have only one load balancing logic. Basically you can do whatever you want within the load balancing, connect to the SDK and you can shut down everything basically. But
12:20
what we support internally is that the load balancing logic will return a VM and a set of hosts. I will show it in a sample later on. And we will migrate that VM, a single VM within the server. It's basically more safe to migrate
12:44
a single VM within a period of time not to cause a migration rush and use all our resources for that. It's pretty not safe to do that.
13:00
We also have a pretty fine load balancing within Overt, the two legacy ones on CPU and now added in 3.4 even distribution one that we even VM distribution that we didn't have. The same balancing sample that I want to show
13:24
as I showed earlier is we get the same numbers that we want to shut down all the VMs but in this example we will actually do that and not exclude those that prevent users from running
13:41
hosts VMs like in the filter example. Here we will show how in that hour, in the wake up hour we will basically get all the hosts and if it's wake up hour we will activate all the hosts.
14:02
If we need to sleep then we will connect using our SDK and get all the VMs from that host. We will stop all the VMs and we will deactivate the host. Same bank example. This
14:21
is how we use it internally to migrate VMs. You get according to some logic you get the overloaded host. It's a code snippet then you select here it's random the first VM on that
14:40
host. Then we actually print it because we're using a CDIO to get the data from the model and we return it and we return a right listed host which is kind of a filter. The first filtering
15:02
we do before we activate the filter and weights and the normal process that we do. Basically as I said we have a cluster policy which is a container for all the filters and weights and a single load balancing logic and
15:25
we have two optimizations for a cluster policy. Speed and overbooking basically we run each time we schedule a VM we run it one by one because we want to prevent overbooking the same. We want to guarantee
15:43
the same resources for each VM. If we will try to schedule two VMs together we can fail because they both see the same resources. So basically the speed optimization
16:01
is to exclude the weight of the host. So later on the load balancing will do the following for us and balance out the cluster for us and the overbooking is to omit the, I don't know, just be able to
16:24
paralyze the scheduling process. Let's see if we can have, show those pictures within Overt. Okay this is
16:51
Overt. The VM is you know because of Wi-Fi and VPN the VM is unknown it's running somewhere. Here I can configure
17:10
a new cluster policy. I can give it a name like the shutdown one. I can
17:26
add the external filter that I've added to the system, the shutdown host filter to the enabled filters. The weight I will give optimal for power saving
17:43
that's tried to aggregate all the VMs within a single host as much as possible. Then I will select the balance that I created earlier. I can
18:03
give it wake up hour like 8 a.m. and shut down our 8. Press okay. Should be created. Okay let's go back
18:38
to the, no, shouldn't take some time.
18:43
What? Not connected to the VPN. No, no, something happened to the VPN. Doesn't like me. Yeah I have screen, I have
19:00
screen shots but you know maybe it will work second. Okay let's take a look at the screen. It works, believe me. Okay we were, we were
19:25
here. Never mind. I will show you
19:42
that one. Then you go to cluster. Forget about it. Okay I create the
20:04
cluster policy that I've showed you. Then I created to a cluster that I already defined so all the hosts within that cluster will act according to that cluster policy. Okay let's talk
20:23
about how it's implemented in the back. It's disabled by default. Whoever wants to extend to add filters should be able to know how to install the external scheduler. The external
20:41
scheduler is a separate process written in Python. We externalize it because we want to guarantee the engine safetiness. You know if a user writes a code it can be dangerous to the system. We want to allow other
21:02
languages as well. If you know the end overton is written in Java so and this model is written in Python and what we're going for we want to support SAS which is kind of a
21:22
scheduling as a service. It's a separate RPM you need to manually install it. How it works basically it's initialized. It runs, it reads
21:42
from local directory all the filters weights and balancing logic that you wrote. Then it's publishing an internal API. The engine, the server reads it and then it waits for calls from the
22:02
engine for filtering, weighing and balancing. This is how it looks like when it's loaded. The filter and load balancing here. Back to the users list now we can easily write a filter that
22:26
maximizes the number of running VMs per host. To sum it up we support
22:40
easy Python plugins for you to use for VM scheduling. You can manage a separate policy for each cluster for each group of hosts and every version that comes out for over it gets new models for scheduling. Questions?
23:08
So the question is that I have the possibility to read from what the server provides. Memory utilization, CPU utilization. Since you have a Python extensive API if I got it right you
23:23
may have the possibility to know if the storage behind or the storage framework behind is constrained so you have the possibility to say all right do a storage migration these virtual machines to the other storage which is SSD based or whatever. So you just need for example to have a good talk with all the storage big guys and
23:42
find more things about their IOPS and what we are doing on their storage. Because we have the memory, we have the CPU, maybe we have some network information about throughputs and stuff so what's left behind is the
24:00
IOPS and the quality of the storage. You can think of whatever you know You all heard a question? I think so. I think you in within you know when you extend you can do whatever you
24:22
want when you extend you know filter or when you basically can connect to every provider or use any SDK if you like so but we provide within the the engine is all that you ask you know memory and CPU load and if you
24:41
want to connect to other external provider that it's your own choice. Hi, so we had a few guys in the overt workshop in Bangalore and they
25:01
were asking us to connect the scheduler to a BMC system that is monitoring extra parameters for example they can monitor the CPU temperature and the fan speed and they can actually predict that if the fan speed is constant or zero and the CPU
25:21
temperature is high that host is going to crash and burn in a few minutes so what they asked from us is to give them a list of hosts and they can actually blacklist some of them because they are aware of more information than what over should have and there are so many other examples which are very similar for example
25:41
Cisco has very similar concepts they actually want to blacklist some of the hosts because the networking is going to go down there there are a lot of very similar scenarios in very big enterprise this is actually highly wanted. Absolutely, sounds promising. So now you have the power you can actually do it yourself.
26:01
Well my first simple question was all right I know about my CPU my memory maybe I know about my networking because we do the networking so the next good thing is if I had the storage IO information or information that had to do with the quality of my storage perhaps I could utilize
26:20
multi-tierings in a storage or storage we motion a virtual machine from one place to the other that could be nice with this engine. More questions?
27:08
Go drink a beer. Thank you. This is not it doesn't have a lot of
28:39
gain. Okay.
28:43
I think it's it's 20 past 20 past six
29:02
my talk.
33:45
So test test can you hear me is it okay? Perfect.
40:27
Okay so welcome everyone.