project by Oregon State Open Source Lab, or there are other possible interfaces, or using the CLI, just typing commands. If you want to see how the commands work and things

like that, please come tomorrow. Tomorrow I'll give an introduction to the most common commands and how to use them. Any www API or any other API that you might have, we'll talk to GANETI through REST, so through HTTP, to an RIPI daemon remote API just.

Those components both talk to GANETI master daemon through Luxi, which is just a JSON command over a local Unix socket. So everything there happens on the master node. If you want to use the GANETI CLI, you use it on the master node. Otherwise, you use something external

that talks to the remote API, or you can just send something to SSH to the master node and use the CLI. You really can't do something with their API. So Luxi is, as I said, the way in which we give commands to the master d, and in which we control the job queue, which is a vital part of the master d, which manages these jobs, which are the operations, as we said,

that GANETI uses. And then finally, the master talks to all the nodes through this RPC. The RPC is also JSON over HTTP, I think. I don't remember exactly, but basically, that layer is just send command to the various nodes,

and the nodes execute and manage the resources. So the resources are virtual disks that live on the various nodes, DRBD devices, so disks that are synced between two nodes, virtual machines, bridges, and NICs connected to one or another bridge. And everything that you need to actually virtualize a machine.

Internals. As you see, it's quite complicated. This is the same again, but the Python files and how they operate. I'm not going through it on this slide, but on the next slides, I'm going to tell you which files

are the various components. This is mostly a reference that you would use, for example, saying, OK, I need to touch the job queue. Which files is this? How do they interact? So let's try to see them. So jobs.

A job is a list of opcodes. An opcode is the basic GANETI operation. So start an instance is actually an opcode. But a job that starts an instance is a job with just one opcode inside that starts that instance. A job might be more complicated and say, OK, start these three instances. Or migrate all the instances on this node.

The opcodes are executed serially in sequence. So while jobs by themselves are executed in parallel. So if you want something that is executed after something else, and it's just a list of operations, you create one job. Now there is also a dependency system between jobs.

So you can say, look, these three jobs have to be executed after this one job has finished. So first do this, and then do these three in parallel. But right now, there's no way to say that through the CLI or RAPI. So it's just some internal stuff that is used for one opcode to schedule multiple other ones.

And make sure that they don't start before he is done. So where are jobs configured? Well, the job queue is the main part which executes jobs. So it reads them from Luxi, and then it stores them in the file system, copies them to all master candidates

so that if the master dies at that moment, we know that we were executing a job and which job we were executing. And then starts the execution through this other file, which is the master CPU. Which does just basically interacts with locking

and executes the opcode one after the other, and we'll see later how. The job store is just the place where the jobs are stored. Right now, it's just the file system plus the remote nodes. And the jobs are called from the CLI, which is basically CLI.py plus all the various clients,

scripts, like GNT instance, which is used, for example, to start instances or create instances, is on the client side just a Python file that creates the Luxi job for JQ to schedule, or from their API daemon. If you want to modify their API daemon, as you see, you modify it in their API directory.

Opcodes are defined in opcodes.py, which is the place where it says, well, this opcode takes these parameters and the types of the parameters. We code a quite typed Python, so that's a bit of a strangeness of Galetic.

Python is not typed, or not strongly typed, as you know, but we try to rather declare the types, for example, of an opcode and say, okay, this takes a number, rather than doing all random checks and then forgetting about them and things like that. And then each opcode is defined, again, in CMDlib. So if you have an opcode which is create instance,

then there is a logical unit in CMDlib that says create instance. And that's where the business logic is. While here is just the way in which they are serialized and then transmitted from Luxi to the master daemon. This is somehow shared between the clients that create one, serialize one,

puts them on the socket, and then it's unserialized here and parsed, checked, and scheduled as a logical unit. RPCs, so we have this main RPC part. Each RPC is just defined with the types it takes,

and then it creates automatically both the client side and the server side. CMDlib is the place that calls RPC. There's no other place. Maybe MCPU calls one for replicating some files, but mostly the logical units will call the RPC to implement what they're doing. So we'll tell the hypervisor, look, actually start this instance.

I checked about it. I created the block devices. I connected them. Everything is fine. Now go ahead and execute the xan or kvm command to actually do something with it. Backend.py is the place where they are implemented. So again, the business logic of the RPC is implemented that. So when we have an RPC called,

verify files that will return you the checksums of all these files on the nodes. The implementation is in backend, and then the call is defined in RPC. And then backend will use hypervisor and bdev and a few other things, but mostly these to actually do the work.

Bdev is the block devices. So anything storage related will be there. Hypervisor is just anything hypervisor related. And then it will just use things in utils like run CMD to just run a command for things that didn't have their own model to be abstracted.

These two are abstracted because we have multiple implementation of hypervisors and multiple implementation of different block devices. We'll see examples of them mostly tomorrow. And everything else is just, for example, just run a command to check some of these files. Well, we'll get it from utils. Utils, the constants are basically used

from everywhere in the code. And it's just like small common functionality. Configuration. The configuration resides in config.py and objects.py. Objects is, again, the definition of which components are in a cluster.

So the cluster object has, for example, an attribute called number of master candidates. And the cluster will try to make sure that at all times the number of master candidates is at least that number, not more, and not less unless there are less nodes available.

So this just defines how the various objects look like. And there's a bit of code to do things like serialization for things that are not straightforward to serialize, and a bit more code for upgrades. So when you upgrade from one granary version to the other, objects.py will say, well, look, this is an object that doesn't have this attribute,

so I know that the default value should be this. I just stick it in there. Config.py is instead the place where we access the config. And it actually does both locking so that cmdlib can, from multiple threads, call config.py without, hopefully, conflicts. And the other thing it does is that it takes care of,

for example, when you tell, okay, it takes care of accessing and retrieving these objects and passing them in a nice format to cmdlib. So the objects are stored somewhere on disk, and then they are copied to all nodes. We want to return them as objects to cmdlib.

We want sometimes to say, okay, just acquire the lock once and return all the instances rather than one at a time. So this is all the convenience code to actually access the config in a scalable manner. And finally, last thing, as you notice it appeared, it didn't exist,

it's ssconf, which is shared between noted and masterd. And it's a copy of just some parts of the config that is replicated to all nodes and not just master candidates. For example, the list of nodes is replicated to all nodes. Who the master daemon is is replicated to all nodes, and nodes can vote so that if an old master daemon

starts up and says, oops, I'm the master daemon, I should, then it gets stopped when all the nodes say, no, no, you're not the master daemon anymore, shut up. So this is basically information that all the cluster needs and not just the master daemon and the master candidates to actually restart the service, but anything that virtual machines need to access,

like keys that are shared and things like that. Customizing Ganeti. So we are at the second part of the talk in which we are seeing how to actually use all these files to actually do what you want with Ganeti.

Suppose you want to install it, fine, suppose you want to use it, that's all in the talk tomorrow, but once you start using it, you realize that it doesn't do everything you need because, well, nothing does everything you need, right? And then you need to modify it slightly and maybe do some minor modification. The objective of this talk is allow you to do this.

So how would you do, for example, a change on how the hypervisor behave or add an hypervisor parameter or all these kind of things that are, we consider common customizations. Why do we consider them common? Because that's the most patches we have received from outside

and that's the most common request we get on the mailing list, like, oh, but I just need to do this slightly different. How do I do that? Well, here is your answer. So for example, how do you alter the hypervisor behavior? Suppose you have KVM and you want to call it with some particular parameter that is not coded by us.

Well, that's very easy, actually. There's a hypervisor file, which is hvkvm, and there you'll see that the code is quite straightforward and just creates the KVM command line. So you can just edit the logic there and add your own flag and install it on your cluster, go ahead.

Or for example, for Xen, it's actually the config file that gets created because then you just call xm create, right? So you do the same. You just add your own option, statically coded, done. A bit more complicated, but still quite simple, is suppose I want, again, an hypervisor parameter,

but I don't want it to be the same on all instances. Or you want to contribute the code back to us because it's going to be useful for you not to maintain the patch forever and for everybody else to actually share this code. But then again, you need this flag, somebody else might need it, but not everybody. So in this example, for example,

there's this hvkvm spice use vda agent. So whoever's spice should use this vda agent or not, right? This is a chooseable parameter. Other things that might go here are, for example, things related to how the hypervisor actually works.

So you can pass a parameter which is not true, false, but is actually a number or a string that you want to add to the KVM common line or the exam config file. And to do that, you do the same. You edit the logic in your hypervisor's file, but then you also need to add the parameter and its types. So you choose a name for the parameter, hvoop,

then the hypervisor type, and then just a name that represents what your parameter is. You have to have a type. So it's either a string or it has to be a file path that then has to exist on all nodes or something or a number or a boolean variable, right?

And then in the hypervisor, you'll see many examples. This is an easy commit that does this, but basically it's just if bla is defined, then do this, add this flag, right? And as you see, you just modify a few things that are quite declarative. The only part of the code that you modify

that is actually business logic is this. Everything else, GANETI does for you. So if you add the parameter when you install a GANETI version, the config here and the objects will take care of seeing, oh, there's a new parameter. I don't have it for all the virtual machines.

You define both the parameter in constants and the default value, so it will just slap the default value on all the virtual machines. You don't need to take care of upgrading. Same for, here we were, right? Same for checking that the value, for example, is a number or a boolean variable.

GANETI will do all of this by just the virtue of the fact that you said, well, this parameter is actually a boolean or this parameter is actually a positive number. We have a small type system implemented for that, unfortunately, but that makes it easy for you to do something like that. Again, the only thing you need to edit is the actual logic of what the parameter controls

because we cannot know that just declaratively. A bit more complicated is altering cluster business logic. So suppose you don't want to change anything just in the parameter, but you want to change how the virtual machine gets started.

Why would you do that, basically, first of all, right? Suppose, for example, you're implementing a different type of migration or something that is more high level or something that interacts with more than one node, so some kind of functionality by which the nodes have to coordinate. This goes to the master daemon

and then the master daemon will call something else to do that. So you just change your logic in CMDlib, which is where, as we saw before, the logical unit were implemented. Here, so each opcode, as we said, is implemented in CMDlib. So we just change the logic there.

We are careful with regards to locking, so we know that opcodes can execute in parallel from different jobs, so we need to make sure that the logical unit we're changing locks the right type of object, like the node or the instance, in an exclusive manner, in a shared manner, to make sure that, for example, there's no exclusive job running at the same time,

for that instance. In general, you can just leave everything as it is, just check that the locking taken by the logical unit you're modifying is correct, and if you need more or you're removing something and you need less, just modify them. Then you need to add the RPCs.

Suppose that your new business logic requires some operation on the node side, like to start a particular daemon or to execute a particular command. In general, RPCs are not execute a command. They are a bit more specific because we don't want too much opening of what the master daemon could do. We want just a series of operations.

So you would need to modify backend.py to implement your RPC, RPC devs to define your RPC, and server node to call the right function in backend. This we should automate probably at some point, but right now those are the files you need to modify. And then, suppose you do something

which is hypervisor specific, like a new type of migration or something like that, then maybe the hypervisor interface changes. So you do it for KVM, but then XAN must do something. So you need to update all hypervisors, XAN, KVM, LXC, FAKE, CHROOT,

and for some of them maybe you'll just throw an exception, say operation not implemented for this particular operation, and then in CMD lib you'll need to check what hypervisor is it, and maybe bail out nicely before calling the RPC, but in general you need to keep the interface of the hypervisors consistent. So all the hypervisors have the same.

Adding an opcode? No, adding opcode options. So again, start an instance, maybe with a different parameter to do something specific at start time. Again, you just need to add the field in opcodes.py so that the clients can actually have a command line flag

and then pass it somewhere to a master daemon. Use the option in CMD lib, same thing as before, so maybe you'll also need to do everything that you did before. So basically this is an expansion, right? So you might still need to add RPCs, change hypervisor, and so on,

and then you'll need to add the command line flag to cli.py, and finally, the right utility in client. So if it's an instance operation that you're changing, you'll change client gnt instance. If it's a node operation, client in gnt node, node group, and so on. To see what all these clients do,

again, please come tomorrow. Finally, and I don't know if it's finally actually, there might be a couple more, adding a backend storage. So suppose that we have the RBD, actually. We do have the RBD, we do have LVM, we do have plain files, we do have storage area network.

You're not happy with any of them and you want to implement something new. Like recently they wanted RBD, which is the block device on which Ceph is implemented. You probably heard it, right? But maybe you want your own. You want sheepdog, which we don't have, or you want something completely different. How do you do that?

First of all, this is quite hard, by the way, so it's an advanced user operation, but it's been done, so I'm sure you can do it as well. First of all, you implement the block dev interface in bdev.py, so block dev is just a class that is implemented by all interfaces. You'll see an implementation for the RBD,

you'll see an implementation for LVM. You implement yours for RBD, for example. When you add the logic, for example, live migration might work differently. Cluster verify might verify different things. For the RBD, it verifies that the RBD children exist on the node and that the RBD devices are connected

and are not this class and things like that. In your case, they might do some verification specific to your type of storage. Then you want to add the new storage name to Constance. That's the easiest part. You would want to add any parameters,

like again, as an example for the RBD, a parameter is what the sync speed is. So to avoid flooding the cluster when the RBD re-syncs, you can limit the sync speed to a certain amount. Your device, for example, might support, let's say, replication on multiple nodes,

and you may want to say, well, replicate on three or on five nodes, so the level of redundancy and replication. This would be a parameter to your storage, and it goes into Constance declared as, I don't know, its name and its type, node number and then integer, right?

Then you want to modify objects disk. Just see this commit. It's basically quite easy, but it just says, well, when you receive this type of device, implement this class and things like that. That's the minor part. Everything else is the hard part, actually. Adding a new hypervisor.

Again, XAML, KVM, LXC, maybe you want VirtualBox, which we don't have right now because nobody ever did it. How would you do it? Well, just implement the hypervisor API. This is very easy, actually. Just a bunch of commands. You don't have to implement them all. On some of them, you can just return like unsupported operation. You need to implement at least a few of them,

like start instance, list all the instances and so on. So that's the way in which you can add the PULX to your hypervisor. And basically, the reference implementations are XAML, KVM, fake, and so on. You can just look at them, and this is the easiest part. The hard part is that your hypervisor

might need things done differently. To give you an example, when I was implementing KVM, in order to do live migration, I need first to start the KVM virtual machine on the destination node, and then tell the source node start transferring. With XAML, it was a lot easier. I just went to the source node and said,

XAML migrate destination node. So since we had XAML before KVM, the logical unit that implemented migration had just an RPC that was called at the moment of the migration starting on the node where the virtual machine

was currently running. What you want in this case is first to call something on the destination node, then to call something on the source node. So as you see, we fall back to the case before. We are changing the business logic in order to support different thing in our hypervisor. So we need, that's why it starts becoming harder

because you need to do everything else as you did before, plus more things, right? So this is the part, alter CMD lib as needed, add new RPCs and all these kind of things. Finally, change the hypervisor API. Again, before there was just initiate migration. Now there is accept instance and initiate migration.

So first, you call accept instance on the target node, which on KVM, on XAML it does nothing, but on KVM it does something, and then you initiate the migration to actually do the operation. In your case, maybe, I don't know, I haven't used virtualbox, for example, but if you were to implement it, you would maybe call different type of commands

or the concept would change in such a way that you would need to change the hypervisor API. Finally, from development to deployment. So this is just how you would develop for Granality, but when you need somehow to deploy this to your production clusters,

tomorrow in Chavan, we're going to see the latest features of Granality, we're going to see how to use it in practice, and what components are missing in Granality that we have somehow in Google and other people have implemented differently, or you could use different open source projects, for example, for monitoring and things like that, to actually have it reliable and in an enterprise.

Yes? So who runs Granality? Well, we do. The corporate computing infrastructure of Google runs mostly on Granality. We have a good number of clusters. GRNet, the Greek Research and Technology Network,

is running Granality for both their internal systems and for their public cloud that is incoming. The Oregon State Open Source Lab is running Granality, so if you have, for example, a sponsorship for an open source project and you ask them to host it, they'll probably tell you,

well, this is a bunch of machines that you have. We just installed Granality and the virtual machines on top so it's easier for us to manage your project and you see the virtual machines, but they are more reliable for you. Finally, Free Software Foundation of France, I found a track entry for bugs they had in Granality, but they never communicated with us, so I'm not sure if they're still using it.

Plus, other people, I mean, many small businesses do and ask us questions and then disappear as soon as we reply, so we don't ever know which kind of installation we have, how many nodes and how much it is used in reality, right? Finally, please check out the code.

You can just find it by searching on Granality on your favorite search engine, and then you can try to improve it. If you want to contribute back, we require a CLA. We promise you that we're keeping the code actually open even if the CLA says that we can re-license it on different licenses and so on. If we turn evil and we do something bad,

please fork it and go do the right thing, right? So, with this, I'm done, do you have any questions?

Anything at all?

You mentioned that Granity has support for ZAN-based disks. Is this really true, or is... No, I'm lying. No, actually, it is true from version 2.5,

which is at release candidate stage right now. This is a support that wasn't built by us. It was actually contributed by the GRNET people that are using it on ZANs. So yes, nowadays it does. You need to use the release candidate version. The only reason why that release candidate version is not stable is that internally, we're having some delays deploying it,

but it's actually quite stable for us. It's actually the fleet management tool that are a bit late implementing it. I will talk about the fleet management tool tomorrow, but you can use 2.5, and yes, it does support ZANs. I haven't been lying. Okay, thank you. You're welcome.

Any other question? If you have more questions like, I have no clue how to use this, what is it and so on, again, please come tomorrow.

Any other questions to Granity or development or things? Yes, please. Are there any plans to support managing of network like integration of open vSwitch? Sorry, I didn't get it. Are there any plans to implement management

of networking like open vSwitch? Open vSwitch? Yes. I've seen that open vSwitch has been merged in mainline. I'll be looking into it. I'm perfectly happy for someone to contribute it back. There are no strict plans right now, but if someone contributes the code, sure, or if I look at it and I find it particularly interesting,

sure, it's no plans I can speak of. Okay. Well, if there are no more questions,

thank you very much for attending, and see you tomorrow, hopefully. Thank you.