Raindrops: CentOS in the clouds
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | ||
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/40317 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
00:00
Projektive EbeneBestimmtheitsmaßFunktionalRechenschieberInterface <Schaltung>SystemverwaltungBitComputeranimationVorlesung/Konferenz
00:35
PunktSystemverwaltungSinusfunktionDienst <Informatik>PunktRechter WinkelTermComputeranimation
01:12
Notepad-ComputerDistributionenraumOperations ResearchZellularer AutomatBildgebendes VerfahrenDistributionenraumInstantiierungDatenverwaltungMechanismus-Design-TheorieUnternehmensarchitekturKernel <Informatik>Installation <Informatik>VersionsverwaltungComputersicherheitCASE <Informatik>ComputerarchitekturBitRichtungInterface <Schaltung>ProgrammfehlerProgrammierumgebungHypermediaVideokonferenzPerspektiveGraphfärbungVorlesung/KonferenzComputeranimation
03:01
Rechter WinkelUniformer RaumDistributionenraumWhiteboardMultiplikationsoperatorComputersicherheitBildgebendes VerfahrenSoftwareentwicklerVorlesung/Konferenz
04:15
Notepad-ComputerFlüssiger ZustandBenutzerbeteiligungZellularer AutomatComputeranimationVorlesung/Konferenz
04:38
WiderspruchsfreiheitSystemplattformSpezialrechnerSoftwareRPCBildgebendes VerfahrenQuaderPlastikkarteZahlenbereichQuick-SortStellenringProgrammierumgebungSystemplattformComputeranimation
05:27
WiderspruchsfreiheitProgrammierumgebungSystemplattformSystemplattformPerspektiveKlassische PhysikQuick-SortVirtuelle MaschineVirtualisierungVorlesung/KonferenzProgramm/QuellcodeComputeranimation
05:56
Orakel <Informatik>Fortsetzung <Mathematik>Nichtlineares ZuordnungsproblemKette <Mathematik>KrümmungsmaßApp <Programm>Objekt <Kategorie>NummernsystemInstantiierungElektronische PublikationRechter WinkelWeb logGeradeQuick-SortUmwandlungsenthalpieKernel <Informatik>Vorlesung/Konferenz
06:43
E-MailDienst <Informatik>KonfigurationsraumSoftwaretestSoftwareElektronischer FingerabdruckBildgebendes VerfahrenInternetworkingElektronische PublikationKonfigurationsraumPasswortVariableEin-AusgabeComputerarchitekturVersionsverwaltungProjektive EbeneCASE <Informatik>E-MailInterface <Schaltung>DifferenteVerknüpfungsgliedInformationOffice-PaketRootkitURLBenutzerbeteiligungPunktInteraktives FernsehenGeradeNormalvektorVollständigkeitTropfenProgramm/Quellcode
08:22
Interface <Schaltung>E-MailSchnittmengeElektronische PublikationSoftwaretestAdressraumBildgebendes VerfahrenLineare OptimierungComputerspielSkriptspracheBitEindringerkennungURLLastKonfigurationsraumZentrische StreckungGemeinsamer SpeicherSystemaufrufDokumentenserverInstallation <Informatik>VerknüpfungsgliedProgrammierumgebungGebäude <Mathematik>SchedulingTropfen
10:30
KonfigurationsraumGebäude <Mathematik>SoftwaretestZellularer AutomatSpezialrechnerDrahtloses lokales NetzKonfigurationsraumElektronische PublikationBildgebendes VerfahrenMinkowski-MetrikGebäude <Mathematik>E-MailProzess <Informatik>FinitismusInformationsspeicherungVirtuelle MaschineComputeranimation
11:03
Service providerRechter WinkelBildgebendes VerfahrenQuaderLokales MinimumE-MailInformationsspeicherungVirtuelle MaschineMinkowski-MetrikZentrische StreckungVorlesung/Konferenz
11:57
MIDI <Musikelektronik>KonfigurationsraumKonfigurationsraumBitMini-DiscWarteschlangeElektronische PublikationTypentheorieMomentenproblemInformationDateiformatUmwandlungsenthalpieGebäude <Mathematik>Schwach besetzte MatrixComputeranimation
12:36
Bildgebendes VerfahrenSkriptspracheDatenloggerDifferenteMultiplikationsoperatorMinkowski-MetrikDateiformatE-MailCASE <Informatik>Mini-DiscVirtuelle MaschineFunktion <Mathematik>URLTropfenRohdatenStapeldateiProzess <Informatik>Bit
13:41
ZeichenketteAutorisierungFiletransferprotokollE-MailBildgebendes VerfahrenInstantiierungSoftwareURLPasswortSynchronisierungGerichteter GraphUmwandlungsenthalpieAdressraumInterface <Schaltung>DefaultDynamic Host Configuration ProtocolRechter WinkelDatenverwaltungPhysikalisches SystemViereckNetzadresseKonfiguration <Informatik>SkriptspracheMechanismus-Design-TheorieInstallation <Informatik>DokumentenserverQuick-SortMomentenproblemVorlesung/Konferenz
16:33
SpezialrechnerKonfigurationsraumGebäude <Mathematik>SoftwaretestProzess <Informatik>SchnittmengeSkriptspracheMathematikKette <Mathematik>Elektronische PublikationBildgebendes VerfahrenPunktInternetworkingVirtuelle MaschineRechter WinkelVorlesung/KonferenzComputeranimationTafelbild
17:08
DifferenteLoopFiletransferprotokollBildgebendes VerfahrenMehrrechnersystemPhysikalisches SystemSystem FInterface <Schaltung>WärmeübergangSkriptspracheE-MailRohdatenFunktionalKernel <Informatik>Patch <Software>Rechter WinkelMomentenproblemStandardabweichungMessage-PassingZusammenhängender GraphURLWarteschlangeProzess <Informatik>MetadatenBetafunktionBootenSpieltheorieDokumentenserverVollständigkeitMultiplikationsoperatorPunktElektronische PublikationVorlesung/Konferenz
19:48
FunktionalZweiQuaderDokumentenserverp-BlockInstantiierungAdressraumMetadatenSchedulingGerichteter GraphVirtuelle MaschineWeb logGrundsätze ordnungsmäßiger DatenverarbeitungRechter WinkelSynchronisierungElektronisches BuchComputeranimationVorlesung/Konferenz
20:45
HardwareSkriptspracheSpezialrechnerProzess <Informatik>MomentenproblemInformationsspeicherungBenutzeroberflächeDokumentenserverVirtuelle MaschineBildgebendes VerfahrenQuaderDistributionenraumReelle ZahlBootenInstallation <Informatik>AusnahmebehandlungComputeranimation
21:11
BootenElektronische PublikationSkriptspracheBildgebendes VerfahrenDokumentenserverSchreib-Lese-KopfInstantiierungLoopRootkitPunktPasswortKlon <Mathematik>Vorlesung/Konferenz
22:12
SpezialrechnerElektronischer FingerabdruckProzess <Informatik>Bildgebendes VerfahrenZentrische StreckungMomentenproblemDatenkompressionSchedulingBefehlsprozessorVirtuelle MaschineDreiecksfreier GraphComputeranimationVorlesung/Konferenz
23:02
RechnernetzInformationsspeicherungPhysikalisches SystemInverser LimesBitrateBildgebendes VerfahrenMultiplikationsoperatorDemo <Programm>Programm/QuellcodeComputeranimationVorlesung/Konferenz
23:34
TropfenLoginElektronische PublikationTemplateSkriptspracheProzess <Informatik>KonfigurationsraumWeg <Topologie>DokumentenserverBitVersionsverwaltungPunktSchedulingBenutzeroberflächeE-MailMomentenproblemMini-DiscInterface <Schaltung>DateiformatInstantiierungDatenkompressionRechter WinkelResultanteSoftwareKonfiguration <Informatik>Front-End <Software>ZweiUmwandlungsenthalpiePasswortGüte der AnpassungGewicht <Ausgleichsrechnung>SoftwaretestPlastikkarteProgramm/Quellcode
27:50
DatenkompressionPhysikalisches SystemTouchscreenArithmetische FolgeThreadInstantiierungProgramm/QuellcodeJSONVorlesung/Konferenz
28:16
MultiplikationsoperatorProzess <Informatik>TropfenSoftwarePatch <Software>InstantiierungSchedulingKontrollstrukturKeller <Informatik>Minkowski-MetrikOffene MengeInterface <Schaltung>Programm/QuellcodeJSONVorlesung/Konferenz
Transkript: English(automatisch erzeugt)
00:00
Some of these slides have been recycled from talks, and some of you may have heard the talk last year, except the project has been evolving. So hopefully when you go through the whole thing, what you see here now, a lot of maybes and a lot of coming-soons have actually been replaced by functionality that exists right now. So just as a show of hands,
00:20
how many people here use the Cloud or a Cloud or the Cloud? Okay, so most people, excellent. So we don't need to go into the basic stuff. The raindrops interface is something that, okay, there's a little bit about me. I'm a CIS admin, I'm based out of London. I do centaur stuff. I like bullet points, and I like the rain.
00:46
And Belgian beers, yes, Belgian beers are good. Although I've lived in the UK now for about 12 years, I still prefer Belgian beers. I'm probably going to get assassinated on the way back or something for saying that. So the thing is, just looking at the problem definition, what we were trying to solve,
01:01
CentOS is everywhere in terms of the Cloud, right? It's a lot of people who are selling services in the Cloud use CentOS, a lot of people who are consuming the Cloud in various ways use CentOS. A lot of vendors, a lot of hosting companies, retail, and they sell CentOS images and instances. But the CentOS distribution itself is built for traditional use cases.
01:21
It wasn't built with the Cloud in mind. I mean, the way the installer works is meant to be sort of automatable in a conventional, traditional kind of an infrastructure environment. The way the management tools work is, again, it's kind of built for traditional enterprise users or traditional kind of setups. And that kind of caused a bit of a problem.
01:41
Because even a delivery for what we actually push out is we deliver ISOs on an HTTP interface that people can download, which are not usable directly. But what they deliver to you as a user is finite, is fixed. When you install CentOS, the kernel you get is predetermined. You know exactly what kernel you're going to get. The version of Bash you're going to get, the version of PHP or HTTP or Apache or whatever you're going to get is predetermined.
02:02
But the mechanism that we shift with and the mechanism that we should deliver with is tuned for traditional architectures as it were. And traditional media, we do ISOs or whatever. But how different are the Cloud guys? They're not really very different in that you still have to do a provision.
02:21
You still have to build an image. You still need to have something that you can run. You still need to manage that instance. You still need to patch manage within that instance for security issues, for bug fixes. So from that perspective, they're not very different from traditional use cases. But they're different enough to merit specific consideration. And to us, the biggest deal was
02:41
that we need to be able to talk to them about delivery mechanisms and how they consume what we ship and how they deliver to users what they further on ship to consume. So the issue was that we found a lot of people were coming back to us and saying, hey, I bought a VPS from hosting company A and they gave me this, this, this and it. Then I wanted to move to hosting company B
03:01
and I bought a CentOS 6 VPS and that had something completely different. What is that all about? And then I wanted to move to the AWS and that was a completely different thing. And then I wanted to set up a developer VM on my laptop and that looked like it was completely different. And basically what was happening was that every vendor takes what we push out and salts it to taste. And when I mean taste, that usually depends
03:21
on what distribution, what best practices, what issues, what history, what has been the lifetime experience of the guy who's actually creating the images. And many times it doesn't reflect company policy and many times it doesn't reflect what they're actually expecting to shift. It reflects what the guy building the image thought was what his users want.
03:40
And that's a dangerous situation to be in. So we started talking to vendors, we started talking to people who would come back and complain to us. And we found that there were issues like people were shipping 5.3 in 2011. People were shipping 6.0 in 2012. And we're like, do you know that those images have security issues? Do you know that you should not be doing this kind of thing?
04:00
And a lot of them were, we've put in the effort, we went through the pain of setting up a VM, it took us like three weeks to get it going. I don't have the resources to do it again. And that was pretty much uniform from all hosting companies from HPC places across the board. So we thought, okay, this is a problem that we can solve. So we spoke to some of the users that the vendors gave us access to.
04:20
So the vendor said, hey, I've got 100 users, here's 10 you can talk to, and see what the actual problems are. And we did that. We spoke to people who consume liquid web services, we spoke to people who host with pair one, we spoke to the big AWS users, we spoke to some people we knew who were running private clouds as well to see what we could actually do. And the problems like we were talking about, delayed access, people want,
04:42
vendors don't have resources to make that happen. So basically there was a massive gap between what the vendors were doing and what the users really wanted. So the problem we can really define down to, users want persistence for images, they want something that is reliable that looks like CentOS, irrespective of where they are going to, what vendors they're working with, what sort of environment they're on,
05:00
if they're on a local box or if they're on a remote box. But they still want to retain third-party support. So it's like if you want a VM from a hosting company, let's say pair one, you want to be able to get pair one to still support your image for you. You still want to have that phone number that you can call. Because if pair one does something like here, here's the official CentOS image, all the best, let me know if you have a problem
05:20
and just make sure your credit card is valid and we keep charging you. You don't want to be in a situation where you can't get support anymore, right? Which means we need a consistent platform from the vendors to CentOS and from CentOS wire those vendors across to the users as well. Does that kind of make sense? Yeah, okay. So we found that on-cloud and off-cloud are not,
05:43
they don't really look the same. You can still have VMs, but you could have a VM on your machine, you could have virtualized infrastructure or you could have a cloud. To me, those three things look very different. Virtualized infrastructure is classic sort of VMware kind of perspective. Cloud is classic AWS kind of a thing where HA may not be that big a deal. You HA your app rather than HA-ing your instance.
06:02
And all of them are kind of, they look just different enough that they merit sort of consideration specifically. And of course, kernel and app support, somebody's gonna run Oracle, somebody's gonna run DB2 or whatever, somebody's gonna run MySQL, somebody's gonna run SQLite. And then people like me still think that if you only have a thousand or two objects,
06:20
then flat files are fine. It's true, it's true. I mean, it's the amount of pain that people go through to set up MySQL with all sorts of schemas and everything, and then they run a blog which has five posts, right? Have you tried running flat files with five lines in it these days? Do you know what kind of performance you can get there?
06:41
So basically what we did was we got all of that stuff together and we came up with an instance called raindrops.centos.org. We're actually launching this as project raindrops.net, but it's also available from the raindrops.centos.org interface. What it is is it's a web email and a git interface. You create a kickstart file,
07:03
and this looks like a conventional kickstart file. You've got install, because that's what you want to do. Things that are different, and I'll point this out here. Your URL that you're installing from has to have repohost as the place it's going to pick up content from, because the builds happen privately. They're not available on the internet. So you have to have that, and you have to have whatever the centros mirror URL is.
07:22
It can't be, you can't use variables, so you can't use base art, or you can't use release version. It has to be fixed to six in the architecture that you want, or five in the architecture. Everything else is about what you'd have in a normal kickstart file. You've got root password, which should probably keep fairly secure. You have a network device.
07:41
Okay, so the other thing which is probably worth mentioning over here is that you have to have partitioning information, and you can set it up whichever way you want. And you have to have a kickstart which completes without user interaction. So if you remove, for example, the network config line, or you remove the root password file, it will get stuck. Anaconda cannot handle that. It is going to ask for input. So you have to have a kickstart file that's complete.
08:00
Is that kind of clear with everybody? Is there anything? Okay. Now what you also need to do is you need to create a config file that tells you what, that tells Raindrops what to do with that kickstart file, right? So in this case, we've got an image target, which is the architecture that we want to build on. Now, this sounds a bit silly, because if you look back, your URL has that in it.
08:21
But what happens is that we actually deploy host environment that we're going to build the image into, and you can control that. So you can say, I want, and this is me being completely fictitious here, you can say, hey, I want a MIPS build or run on an x86-64 rooted environment, which you can achieve with Raindrops. We don't have a MIPS build.
08:41
But if you did, it'd be the best MIPS build in the world. You have to have a UID, which is actually injected in if you don't put it in. So we recommend you don't use a UID here. And then things, life is a lot easier, because you won't have to remember three sets of UIDs, just get one. You can have a postscript, which is external to the kickstart.
09:01
If you want something to run, if you want kickstart to deploy an image, instantiate the image, and then run a script, you can inject that here, that'll get run in. Schedule is something which is coming soon. It's not there yet. But what you can say is that I want to schedule and give it, give me 30 days. And what'll happen is this particular config file will get reprocessed every 30 days.
09:20
So you get a new image every 30 days. This is not functional yet, because we want to see, we want to do a little bit of testing around how we can handle load and how we can handle scale. And then how we can actually get a scheduler, which kind of does the right thing. The updates, yes, again, is something like, if you're doing an install, we just need to tell what OS we're installing. Having an updates is equal to yes,
09:41
will mean it'll actually run a yum updates post install. It's just, you can have that in your kickstart if you want, or you can have it in here as well. This is handy when we do installations for certain hypervisors, which don't allow us to have multiple repositories in the kickstart. The kickstart file, this kickstart file has to match the name that you're going to use here.
10:01
So whatever kickstart you've called it, you can call it that. You have an image tag, if you want to add something in, you need to use what hypervisor target you want, and who you want to notify. So this could be an email address, it could also be a URL if you want to call back. So if you're automating your scripts and you're automating your kickstarts, you can have a URL over here, and when the image is done,
10:20
it'll actually do a call back and it'll give you some details. Okay, nobody can read that, but that's my email address. So if you were using the Git interface, this is what you would do. You would do git add my image, .ks, you would git add your config file, you'd git commit with build colon space, whatever you want, and you'd do a git push. And that's it, and that would trigger the build,
10:41
it would process the config file, get the kickstart into place, it'll trigger the image and do whatever you told it to do. This is literally all you need to do to get your image out. I'd asked for an email notification, so this is kind of like the notification you'll get, your job is complete, you get the download and the log. Now, we have finite storage,
11:00
because we've had to kind of get the hat around and pass it around to see who would actually donate the machines and infrastructure. I'm very grateful to Pier One for forking out six machines and the storage boxes have about two and a half terabytes of space, but this is going public. Two and a half terabytes, it does not last very long. So if you guys do start using it and you do ask for email notification, we highly recommend, or at least I highly recommend,
11:22
you get that image within a few hours, or maximum 24 hours. Because what's going to start happening is when we hit the 80% high water mark, we start deleting the oldest images. Right now, we don't know if that's going to scale or not, we're hoping it does, we think we should be able to retain images for two to three days. But if this becomes really popular, it goes viral
11:40
and hundreds of people start hating it, we may not be able to retain images for very long. So keep that in mind. How are we doing for time? Okay, we have 15 minutes in. That's literally all there is to it. You get your image, download it, and you can instantiate it on whatever you want. But not quite, because vendors being vendors still want to retain their ability of doing a little bit of salting, you know,
12:02
in whatever way to make it work for what you want to do. So we extend that a little bit. This is the base config file, right, what we were looking at earlier. And it's still the same. Now what you can do is you can add an extra stanza to your config file, and you can add a bit more information in it. So you can have a vendor, which at the moment is only CentOS, we're hoping we can add some other people in there as well.
12:21
You can ask for a specific kind of disk type. Now, at the moment it's sparse or raw, because of the way it's set up to do a zen build. But if you were doing a KVM build, you could have QCal up there, you could have whatever other format you want. Essentially there's a QEMU, IMG command that gets run.
12:40
So anything, any file format that QEMU can handle, the delivery agent within Raindrops will process it for you. And I think there's like, I don't know, correct me if I'm wrong, but there's about 10 or 15 different formats that you can use with QEMU. If you want a different format, if you can write a batch script which can take a standard in for an image and output whatever format you want, we can plumb that in. You can specify what disk size you want.
13:02
Ideally keep this to less than 50 gigs, please, because provisioning 50 gigs in, for example, a raw file takes up a lot of space and time. But there is no actual check on this. So if you go and say, give me two terabytes worth of a disk, the builder machines only have about 200 gigs. So it'll try and allocate that disk, it'll fail, and that'll come up in your log file
13:21
that hey, I couldn't, the disk process didn't complete. If you want the image compressed, you can ask for a few things. You can ask for different kind of formats. And this is the coolest bit. This is the thing that took me the most amount of time to write, is the delivery stanza. So you can either have this as pickup, in which case you will get an email with a URL saying, hey, this is the URL that you can download from.
13:43
Or you could say AWS, colon colon, your AWS authorization tag, the string, the author tag in here. And what Raindrops will do is it'll actually log into your account on AWS, push that image into an S3 backed, an instance backed image,
14:01
and send you an email with the AMI, the AKI, and the image ID that is used. So that's pretty cool. And then you can also have an FTP delivery. If you have an FTP server at your end, you can say give it to me at FTP colon with the whole URL. Hopefully add a user ID and password into it. It'll try and use LFTP to push it. It can also do rsync, but rsync is dangerous
14:21
because it requires a lot of network resources at our end, so we don't really want to use rsync. We're happier with FTP. So if you have an rsync target at your end, you say rsync colon whatever, it'll actually rsync the image to you. And you get the email notification once the delivery is complete. So if you've asked for AWS delivery, you'll get the email once the image is in AWS.
14:41
And then you can have extra stuff, like you want no IPv6, you want a specific MAC address to be used or whatever, you can add those in. By default, we only set up one network interface, but the system supports up to three. And this is something which is, I guess, more relevant to vendors, who may want to have every image coming up with a public interface, a private interface, and a management interface.
15:01
And this works, it handles up to three images at the moment. If you want any of those to be set up in a specific kind of way, you can specify a MAC address, or you can say DHCP and it won't set up an HWADDR line. How many people have used OpenNebula in the past? Right, a couple. So you guys know what I mean, right?
15:21
So OpenNebula does not need contextualization to set up IPs for VMs. It converts the MAC address into an IP, and the IP is the quad-notification of the MAC address. So over here, you can literally, what you're doing is you're specifying the IP address for the VM. Does that make sense, with the extra options? Is there any questions anybody has so far?
15:41
All right.
16:01
So the question is, how would you include software which isn't distributed by CentOS, whatever mechanism it may come from, right? The script that you can run here would be the script that you would influence those sort of things. And for example, if you wanted to include the Apple repository, and you wanted to install
16:22
something from Apple, what does Apple provide? Let's say you wanted to set up an RCD. So what you could do is you could have that in a script. And this would, what you put in over here would be my script. And then when you do your get add, you need to have that script added in as well. The only thing is you have to have changes in all three files, because the build up process
16:43
only checks out the last particular chain set. Does that make sense? Right. So the point at which the image gets created,
17:01
that happens privately. But the image is then pushed out to a machine which does have internet access, which is when the script gets run. So the reason why we have the script up there, rather than having it, rather than telling people to use it in the postscript for the kickstart, is the difference. At kickstart time, there is no public interface. There's no public access. But at this point, there is.
17:20
Because that's how even the callbacks and stuff like that would happen. Of course, it's probably for the sake of completeness, it's worth pointing out that you can only do that with a raw file. Because if you ask for a VMDK file, we can't really loop mount it to do these things. And because we don't have VMware running on the builders, we can't actually create an instance out of it to run that.
17:42
So there are some caveats. A lot of these things are not being tested for right at the moment. So for example, you can say, hey, give me a VBox image and deliver it to AWS. That's not going to work. Because firstly, there's no cloud in it, there's no metadata in it. Secondly, VBox would be doing HVM, not Zen.
18:00
So if you're looking for a CentOS 5 install where the Zen kernel is different from the stock kernel, it's not going to boot. And what we will do is we will very happily do everything. We will put it into S3 for you until you give you an email saying done. But obviously, it's not going to work. So we're kind of hoping that as we release this stuff out for beta today, tomorrow, a lot of you guys are going to go out there and break the system in as many ways as possible.
18:20
And then submit patches on how to fix it, hopefully. It is on Gittoris at the moment in components. It's also held, but the actual complete stuff is held on nazir.current.org. It's a public Git repository, I'll give you the URL for that. And the way the things are implemented is everything is run as, again, of Beanstalk workers.
18:42
And there's a concept of, we use standard in, standard out to cascade through drivers. So for example, I'll give you an example of how the delivery agent works. There's a message that comes through the delivery queue saying, hey, this job is ready for delivery. And then the worker will work out, how am I delivering this? I need to put it into an HTTP interface,
19:00
or do I need to deliver it to AWS, or do I need to send it by FTP? So let's assume that there's an FTP request. It'll actually fire up the drv-delivery-ftb.sh with the UID. And then that script will do whatever it needs to do to make that delivery happen. And everything is monitored for standard in, standard out. So if that script exits with a non-zero value,
19:22
we treat that as an error, transfer the log over for the user to see. Which essentially means that if you've got arcane, crazy stuff that you want to do, or let's say the HPE cloud guys want to have an interface that allows us to push images there, they can just write a delivery agent for it, which has all of that stuff in it. We do have some things which are available, like Python Boto is available anyway. We have the Jcloud stack that is available,
19:41
so you can consume that. So we've gone through this stuff. So this is stuff that is kind of functional now. We've got KVM, QVM stuff, LXC works, Zen works, VMware works, VBox works, with caveats. And I'll kind of come to that in a second. OpenStack HPE cloud is consumable because we have cloud in it available
20:02
within an extra repository, so you can use that. So that kind of works for Eucalyptus as well. CloudStack as well, I believe. You can do AWS, EBS works in a really weird kind of a way in that we inject stuff into an S3 instance, we request using your credentials a block device, and then we rsync the OS over to the block device,
20:22
remove the SSH keys, remove the metadata which is instance specific, and reboot the machine for you. So this has never failed for me, but a lot of people tell me there's a better way of doing it. But hey, you know, rsync is the gift of God, right? It works. And OpenNebula works because you don't need contextualization, the MAC addresses are converted in,
20:41
and that kind of works. This is how it's all set up. We have a scheduler with the Git repositories, which is not really there at the moment. We're hoping most people use the web interface. We have the repositories, the builder machines, storage nodes, all hosted at pair one, at no cost to us, so thanks for that, guys. This is kind of how it works. We do PXE images from distro to actually bootstrap a real install,
21:02
except for VMware and VBox where we use the distribution ISO and then boot it from that ISO to do the install. And a couple of things, we don't have EFI support, so if you need EFI, you're most welcome to engineer that. We don't have that support there, and cloud in it is possible. It's available over there. So big challenge is handling passwords from kickstart files.
21:21
Don't make your Git repositories public by putting them on GitHub, please. You're going to be cloning from here, but do not put them there because your kickstart files have to have a root password in there. So even if it's encrypted with MD5 or SHA, given enough resources, people can still go backwards from it, so do not do that. Images are delivered pre-boot run, so you need to handle first boot
21:41
if that's an issue for you. If you're asking for images, and if your infrastructure cannot handle first boot, then you need to kind of do something in the kickstart file so first boot does not kick in. Scripts are injected via loop mount. This is what we were talking about, that if you specify a post-run script, it'll be run after the VM is delivered to the head node. So you can do stuff like, you can wget, you can git clone.
22:02
You can do things like if you want your puppet manifest to be injected into the instance before it's delivered to you. That's a good point to do it from there. And the entire log is available. Very soon we're going to have stock images for CentOS available. So if you just want a basic CentOS image, you don't really have to go through the process. You can just use public images that we'll make available through the process.
22:21
And we'll update it every week, every month. It's not happening yet. We want to kind of see how scale works and we might need to get a dedicated builder for this. This should happen soon. Also, schedules are going to happen very soon, where you can specify schedules that you want an updated image delivered to you every month, every week, every five hours, whatever. And that just effectively uses cron at the moment.
22:42
It's not public yet, don't try to use it. Big challenges, will it scale? It takes between four minutes to 22 minutes per build for an image at the moment. If you've asked for compression, we do all compression for all images on a very small machine. Because we don't really want to waste CPU cycles doing compression, which means it could take
23:01
up to 30 minutes to compress the image before it's available for download. We don't know how good or bad that is. But everything else we think is going to scale quite well. It's secure, so we trust the images that are coming through. Third party ACL handling is tricky, but we're kind of hoping people do that on their own. We're doing AWS, we might do the Rackspace image as well, the Rackspace cloud stuff.
23:21
But we're hoping that vendors do their own thing for the other stuff. And big challenges, we're kind of running out of time. We've got like three minutes, so I just want to quickly, let me do one thing, let me quickly do a demo of what we have up there so you guys can take a look at it, and then we'll see if we can come back. Let me log out.
23:45
Can everybody see that up there? So you log on to the web interface. This is what you end up at, raindrops.net slash login. It is available right now. Please do not send hundreds of build instances to it yet.
24:02
This is what I hate doing is looking at stuff. So you can sign in with any of the options here. We're going to try and get a few more options going. We don't want to know who you are. We don't want to have to verify who you are. We kind of trust that GitHub and friends have done this already. So I'm going to try and log in with GitHub.
24:20
This should hopefully work. There we go. I'm actually logged into GitHub at the back so that you don't see me typing my password and acoustically decoded. This is what the interface looks like at the moment. You've got jobs that we've set up, none yet. We've got configuration files, none yet. Kickstarts, none yet. So let's start with the new Kickstart file.
24:40
We have a template set up already. So you can change a few things in there if you want. Whatever, it should kind of work. We need to give it a name. So let's say test one. So there's nothing specific in here in the Kickstart.
25:00
Because it's a template, it's actually rendered the URL bit there. Let's see what happens here. This web interface is really quick, really fast. And it's a network over here which is really slow. So when you actually go and see it away from here, you'll find that it is actually quite a responsive thing.
25:20
We keep version tracking for you with the aim that this is going to actually interface with the Git repository at some point. So you've got the Kickstart in. Let's try and get a configuration file going as well. What am I doing? Hitting the wrong button.
25:40
So you've got the Kickstart file, the config file. Let's just call this for the sake of interest. The same thing, let's set a notify by email. Most of this stuff should be pretty standard. I want it picked up, and I have compression turned on. I want a 10 gig size disk with a qcow2 format
26:03
and no script or anything like that. And this should then, it would help if I, did I go back? Sorry, what am I hitting? I'm hitting the wrong button here every second.
26:36
Which way am I going? Let me just see if I can do that.
26:47
Right, so you've got a config file. You should now be able to go and create a job.
27:01
And this should have a drop down with our existing stuff. And submit.
27:22
So what actually happens is you can open the job again, you can delete it, or you can see the results, which and you should see results as things are building. But what's also interesting here is that you can actually see the status of the backend workers, which are all sleeping at the moment.
27:41
Let's see if that picks up something. It's actually waiting for the next build schedule to come up. Hopefully that'll happen in the next second or so. Let me see here if I can influence that in any way. I'd actually pulled up the worker screen for the build that is going to run. So once the prep happens, this is effectively what's running at the back. You've got the delivery agent running.
28:02
So the one that I was talking about, the compression system. That's kind of like how it works. It should also be a builder instance here. So these are the builder threads that are waiting for the job to come through.
28:22
Are we out of time? We are out of time. Does anybody want to talk about anything in the minute or two that it takes to get the builder going? Because the build will only run once every five minutes for the schedule to come around. Sure.
28:41
Inside? Yes. So what we're actually hoping to do is actually deliver the entire Raindrop stack as a VM within Raindrops itself. So what you could say is, hey, give me a Raindrops instance for my Cloud Stack deployment. And then you have your own private setup, which is already there. And the software is all free.
29:01
I think we've been talking about OpenNebula. OpenNebula is going to try and ship a VM which runs Raindrops from within the OpenNebula market space. We're kind of hoping more people do that as well. Does that answer your question? Yeah. Is there anything else you want to talk about? We probably won't have time to see this build running.
29:20
We've got another two minutes to go for the next schedule. No? Well, thank you for coming along. Highly recommend everybody goes to the interface, check it, break it, and then submit patches to fix it, please. Submit patches is what you need to focus on. Everything else will happen automatically, I think. Thank you. Thanks, guys. Thanks.