Good afternoon everybody. Welcome to this talk, Relax and Recover automated testing. First of all, are there Relax and Recover users over here? Some of them?

Not yet. Not yet. Hopefully I will convince you after this talk. It's always good. This is the agenda that I will go over today. I will introduce myself. I will give you a short introduction into what is Relax and Recover.

Of course the main topic will be the automated testing, followed by a demonstration. So it's always nicer to see it than just to have two plain slides. The slides, anyhow, will be available on the FOSCUM site and will contain more or less the automated testing step by step.

So that at least you have some feeling of what the automated testing is. And of course you can do it yourself because it's open source. But I come back to that later. First of all, who am I? I'm already an old guy with a very long time in the UNIX and also in Linux from the 90s.

I think I was one of the first in Belgium that was using Linux. And I've been in RIR from 2006, but before RIR there was a make CIDROM recovery. Maybe some people have heard of that. That was the previous project that came before Relax and Recover.

That was also in the Linux disaster recovery project. And I have some other projects of course, but you can find it on my GitHub page and on my website. Okay, disaster recovery. So there are some basics. It's a very short basic introduction.

It is about you lost your hardware by fire, a crash, whatever. It can be also a virus attack for example, some guys. Anything that disturbs your business can be in the category of disaster.

Furthermore, you also have business continuity. That's going much farther, it's much broader. It's about your business that's out of business, your way of speaking. And then you have to build up your data center, your telecommunications, you have big plans around it. And disaster recovery is only a small part of it.

So to differentiate what is disaster recovery, for techies like us it's more like our computer or data center. Key components like a database server, mail server to restore that as soon as possible. But business continuity is about the business and business is always central. Because that's the one who pays us check at the end of the month.

So that is the last picture about business continuity. And that is where you can see very clearly that Relax and Recover really fits. It's in the recovery part of it. But you have a lot of steps in business continuity.

Sometimes I talk about business continuity itself, which is also a very interesting topic to talk about. And it is something that never stops, because your business is changing always. You introduce new tools, new business plans, new integrations of new buy-ins and buy-outs and whatever.

So it always changes, so you always have to rehearse, rethink and re-exercise. Also true for your disaster recovery, you have to make exercises. Don't blindly trust that the tool like Relax and Recover will work out of the box without testing. Very important tool to remember. Testing.

Okay, that is something I probably don't have to explain to you, what is disaster recovery. It's not the question, can it happen? It's just the question, when will it happen? And it does happen to everybody of us. It does happen to me, even I am an expert to it.

So always have the back-up plans ready. You can make your life easier by having a good inventory of your hardware, for example. Of your software, central database, whatever, that categorize everything and make plans of it, what is there.

Because when you lose it and you have no clue what is in your business, because your business can be worldwide, can be hundreds of different smaller sites, if you lose one of those sites and you have no clue what's in it, that's very difficult to rebuild, so you have to make plans of that. But Relax and Recover can help you with a tiny part, your computer part, in this case Linux only.

But it does cover different architectures, it does cover PowerPC, Intel, even Itanium is still possible. Alright, let's continue.

Still frequently people tell to me I have a very good back-up solution, and that's enough for me. It's only good for your data, it's not good for your overall business to be online again, so it has to be smaller very quickly.

This is maybe the best slide to explain to your management why back-ups are not enough. Back-ups are good for your data, but to do data recovery from scratch, then you have some good other tools needed for doing that. Relax and Recover is really fitting in that piece. It can shrink your recover time very drastically.

As you can see, if you do it from the first line, you have the OS that you have to reinstall, that takes time. You can tell me you are using the cloud, but even in the cloud you can lose your conference management tools,

because conference management does not cover everything because I'm daily using it. So there are still gaps in it that you don't cover with conference management, that you forget about it, that it's not ready yet, so therefore Relax and Recover can help you to have your OS and everything that's in it, your conference, your operating system can be covered with that,

and to restore that is much, much easier and much faster at the end, even with conference management. And we do practice it on the companies that I work for. I hope that I make myself clear that Relax and Recover is about disaster recovery of your Linux operating system.

Basically, the tool itself is completely written in the BAS language. Why did we choose BAS? Because every Linux system administrator is supposed to understand and use BAS, and in practice that is true, so therefore we chose BAS.

It's a very modular system, that means that you can write a very tiny script and put it somewhere in the tree, and it will be picked up automatically. So there's nothing that you have to do, especially you don't have to dig very deep in all the sources. If you know the concept of Relax and Recover, it can be used to add something yourself.

By the way, I have several other open source projects which are based on the framework, and the framework that we used was initially created by Shlomo Shapiro, which is my colleague from Berlin, and that is still used in other projects as well.

So it's not only dedicated for RIR, but it is so easy to plug in something in there. RIR is meant that you configure it on your system, or your systems if they are similar,

and you install it, configure it, test it, and forget about it. On a weekly basis you can have a new snapshot of your system disk and you're good to go. Furthermore, it integrates with many, many different backup solutions. So the basic RIR was created in the beginning, it was in fact using tar,

and make the tar of your operating system and put it somewhere on an NFS file system. And when you need it, you can boot from the ISO and you can restore it from that point in time. Furthermore, in time we also did integrations in many open source backup solutions,

but also commercial solutions, like Tivli from IBM, EMC Networker, Data Protector, and Barios is in there, Bakula is there, and many others that I forget about.

It makes it easy. The main point of reelection recovery is to make your task for disaster recovery as easy as possible, and design it in that way that it can be configured in a flexible way that fits your needs.

How does it work? Of course your operating system consists of disks, so the basic thing is that it has to save the layout of your disk environment, your logical volumes, your mirror disk, your hardware rates, software rates, whatever.

All your file systems, how it is labeled, is all written into a configuration file, and that is kept on your live system, but also on your recovery system. Also the boot loaders that you are using, the major boot loaders that are in use.

Furthermore, then it stores the files if you request it for it. If you use an external commercial backup solution, then it only stores your basic system layout. When you restore it, it recreates the disks and then restores from the backup. That is part of the integration.

Also, very important, create the boot loader, the boot system that you use to recover from it. All that can be done and is done online, so there is no offline thing needed. You can just install it on a production system, configure it, and it will work.

Never use it on a production system without prior testing in a development environment, similar to your production. Never trust the tool as such. Even I say it is a good tool, but never trust it. Always test it. The recovery system can be made on an ISO environment, it can be on a network via pixie booting,

or it can be on a USB disk. Those are the three main recovery possibilities that are in practice today.

Besides the recovery operating system, which is in fact from your live operating systems, it is the same kernel that we are using, so that is always nice. We copy a lot of other tools that you may require to do some rescue environment. You can use the rescue image to do file systems and repairs, for example,

not only to recreate everything from scratch. That is not always needed, so therefore we copy a lot of tools. Everything, your complete rescue operating system is in RAM, so you never touch your live disk unless you decide to do a recover.

Of course, your live disks will be overwritten. This is a graphical overview of the backup flow and output flow. If you look in the configuration files of Relaxion Recovery, you will see a backup definition and an output definition. The output definition is for the rescue image, where you want your rescue image to be.

Do you want it to be an ISO and a location where you want to store it? A backup is a fact about your operating system. If you do a backup, NetFS is the main default one.

That is for the tar or the rsync internally, but if you want to, for example, use Tivoli backup, then you define backup equals TSM, for example, or DP for data protectors. Then you are saying to RIR, your data of your operating system is in another backup solution,

so you don't use the internal one. An internal one is in fact tar or rsync. That is the main difference. In backup, you see the internal is tar and rsync, and the external ones can be any kind of open source or commercial backup solution. It also integrates, for example, with duplicity.

This is also a nice cloud backup solution. The uses of RIR, you have in fact three main parts. You have to make rescue. That is to make your rescue image only. Not backup, only the rescue image.

That is a fact. If you are using an integrated commercial or an open source backup solution, it only requires a make rescue because you don't need to make backup because make backup is meaningless at that point. Make backup is meaningful if you are using an internal backup solution,

like tar, rsync, or whatever. You can also run backup only. There are many others. But I would say make backup and the recovery are the two main options within RIR to do. One in make backup is to make a rescue image, including the backup. The recovery is a fact if you have a crash and you want to recover from the backup that was made prior.

As already said before, you can easily integrate it with, for example, config to HTML, which I find quite handy to have a configuration management overview in text or HTML included on the ISO or USB or on the network environment.

So you can always look what was the configuration of my system before I need to restore. Also very handy if you want to do cloning, for example, because RIR can be used for cloning systems, and it is used for cloning systems as a lot of corporations.

So you want to get started? These are the main pointers to the website, also to the RPM status option because we have a daily build of RIR with the OpenSUSE build services. This is a very helpful tool that OpenSUSE and SUSE provide to the developers.

So anything that we, once we do a check-in into our HID repository, then my server will detect it on a nightly basis and will automatically rebuild and push it to the OpenSUSE build service. And the next day you have it available or a few minutes later.

All the different architectures for Debian, Ubuntu, Fedora, SLES, RHEL, CentOS, etc. Furthermore, the official source is on GitHub, like I mentioned already. Also the issues are there.

If you have an issue with it, you can report it over there. And maybe you know it or you don't know it. RIR has been added to the base operating system of Red Hat. So you will find in the 7.3, 7.4 still the RIR version 1.72 with,

I think it's the ninth subversion that I created with bug fixes that were in our issues. But in the 7.4 that has been released in the beginning of August this month, it's rebased to the RIR version 2.0. And that is a major step from the RIR 1 version 2.0 because we did the renumbering.

And also a lot of new features and stability was included in RIR. So if you're still using a RIR 1 version, it's better to go to the RIR 2.0 because it's more stable, it's much more tested.

Also with the automated testing, for example. But okay, I just want to say it's in the official repositories of Red Hat and also SLES has the more or less official repositories in their thing. One of our main developers is also working for SLES, for Susan. So that helps, of course.

Right, I'm coming to the topic of today, in fact, as testing RIR. Because RIR is already more than 10 years old and in the beginning we were using physical hardware, a lot of virtual machines to test it. But it's very cumbersome because you have made some modifications,

we get new features from users who want to integrate their backup solution, for example, to give an example. And we have to blindly trust that it works. And of course, we're all developers and nobody is perfect. I'm also not perfect, of course. We make mistakes and sometimes mistakes are integrated

or checked in automatically without being tested properly. And it's, in fact, almost impossible to test everything because, like I said in the beginning, RIR is working mainly built for Intel, but now the IBM is pushing off,

one of the developers has come, is also now from IBM for the PPC and PPC LE46 environment. So it's growing fast in that community. Of course, I don't have any PPC system at home, so it's, for me, difficult to test on that kind of hardware.

So therefore, this environment was also built and IBM is also trying to integrate RIR automated testing to test their new features in it. So it's just too broad to test everything. We do our best to test the basic stuff. The internal backup solution is tested.

I can test the ISO, I can test the pixie booting. So that's convenient. And also, for example, for Barrios, if Barrios uses in here, that's the open source backup solution, that's the integration. They asked me to help them to do the integration testing with Barrios. And that's also done with the automated testing tools.

Like I said, it will be a challenge to test everything, but we try to do our best. RIR automated testing, in fact, is an open source project, to make it very clear, but it was initially written for a customer who paid for it.

We're doing more than 20 years of the source project for free and still have to live as an independent consultant, so it's nice to get paid sometimes. So doing the automated testing is finding a balance between continuous integration and automated testing, which is not easy.

If people here are in DevOps environment, they probably know it's always a battle, and even here it's a battle. But like I already mentioned, the open source build service is a very good first step, because that provides me with an automated way of building the RPMs,

and they are available, which is nice. So I don't have to do it myself every day for different platforms, which is good. It helps for developers, and we do get feedback from a lot of developers. It's on operating systems which I do not have or do not use. They come back with feature enhancements, with big fixes, and also with better forms.

And we do have developers which are game, helping us, and left, and we some have a few developers who are staying already on board for several years, which is nice. We have now five key developers in our team, which is good. Like I said, from Susie, we have a very good one.

From IBM, we have Slomu Šapir, myself, and we have also a guy from HPE from Slovenia, who is also a very excellent person, which is helping us. We have some previous very good ones, which are now left the team,

but still in background sometimes, but okay. That's open source. It's coming and going, and we are always lucky to have some good people on board, and if they lose their focus and do something else, that's fine. No hard feelings. We are glad with all the help that we can get. Support is done via the main channel is GitHub with the issues.

There's a lot of free support, as I said, but you can also have commercial support via my company. All right. The automated testing. We are coming closer to the demonstration. That's good. It was initially written for the customers who have a support contract

or a subscription contract. Currently, the first time that I talked about the automated testing was at FOSDEM beginning this year, and then I only had it for CentOS 7, and on the Lippert. I gave a demonstration with my Linux laptop with KVM Lippert.

But now we have Ubuntu, and to have it more convenient, also integrated into the Oracle VirtualBox, and I will give a demonstration with my Mac just to prove that it works on a non-native Linux system as well.

So I have a different operating system that I'm using on KVM, Linux with Oracle VirtualBox, but here it will be using my Mac system just to prove it. The next slide will show you the overview of the setup,

but the first of my scenes that I use are provisioned automatically with Ansible. In February this year at FOSDEM, it was still using a script, and I also gave a talk at Cbit, and then somebody in the audience talked to me and said, why don't you do it with Ansible or something else? I said, that's a very good idea

because I'm now also on DevOps, and it was a good exercise for myself to dig into Ansible and to do the complete provision of the virtual machines. That means, what do I do? The operating system is a box, so it's not needed. Updating it, installing Rea, of course. All the dependencies of Rea will be installed automatically.

For example, if I'm using Barrios as an integration test, I have to install Barrios. On the server side, I have to configure it. I have to become a Barrios server, and the client has to be configured. So these things are all automatically done, and it's done for Ubuntu as well because they're always between the different versions,

different dependencies. It's a bit awkward, but that's life, of course. So that's all covered by Ansible. This demonstrates I will not do the provision from scratch because that will take ten minutes. I will just demonstrate the automated testing itself. But if there's time or not, later on I can always show you

that provisioning is working with Ansible. In this setup, what it will do is automatically it will launch a client and a server virtual machine. It will automatically create a disaster recovery image from the client,

make a backup to the server. If I'm using Oracle Virtual Machines, VirtualBox in this case, that means booting needs to be done from the host system. If I'm using, for example, kvm-livevert, I can boot from my server VM, which is not possible from VirtualBox.

But that's the limitation of VirtualBox. It doesn't mean just a different way of starting up your recovery image. But that means that the client will, in this case, this demonstration will be using the internal network, NetFS solution, and I'm using TAR to make a backup of my client system to the server VM.

But the same can be done with Barrios. If time needs, I can always demonstrate it. In this case, I will boot via Pixy, but can also boot via ISO. When the backup is completely made,

I will stop the client VM and automatically start the recover VM. The recover VM will automatically do a restore from scratch. It will recreate your disk layout, restore everything from scratch. When that is done, it will reboot, and your client VM will be online again. So that's basically automated testing.

This is the graphical overview. On the top side, you will see the host system. I call it the Vagrant host, because Vagrant is used to drive everything. And on the left side, you see the client in the middle of the server,

and there the recover VM. So there are three VMs created. The client and the server are already provisioned. I did it on the front. So they are containing everything. Every time that I start to automate the testing system, I do not reprovision everything for the same operating system. If I want to use CentOS and it's provisioned,

and I want to test a new version of my Relax and Recover version, I do not reinstall the operating system from scratch. That's not needed. I just have to update the Relax and Recover version effect and restart the backup. So the reprovisioning is not required every time.

It's only required when I want to switch from CentOS to Ubuntu, because that's kind of logical. You can destroy your box and recreate it automatically. That's done automatically. So there are different ways. So you have an internal network. It's the 33 network, which I'm using to do the backup internally between the VMs.

And you also have an external network, an DHCP network, which I can talk to the internet if you want to download the latest Relax and Recover version from Open Source Build Services. It's going through the internet to download it, and you will see it in demonstration.

And to boot, to recover, with Pixy, it boots from the hypervisor. If I'm using LIFFER, it's going that way. So that's the difference between KVM, LIFFER, and the virtual box situation. This MEC system is using the virtual box,

so I'm booting from the hypervisor. But okay, you don't see the difference. And that's all covered by the script of the automated testing. It's also a script, automated testing. It's written in BAS. It's Open Source. You can find it. I will show you the point later. But to test the environment, I'm using Vagrant.

That's the main feature. Vagrant and virtual box of KVM. If you're using KVM, then you also need the LIFFER plugin. And hit is also required to download the latest version of the automated testing software,

and to clone the Vagrant files, because the Vagrant files are included in the project. I can show you. I will demonstrate it anyhow. To conclude, before I dig into the demonstration, it's Open Source. Everybody can use it, test it out,

and give feedback to me or to the project. What is missing, maybe prefixes, whatever. It's always welcome. New code is written only for customers, in fact. It's not doing it for free, unless it's a serious bug, of course.

You can clone it. This is the address. You just have to go into the directory and type miniscapes for the help. The real automated tests is the main script. There's only one script effect, and the rest is Ansible Playbooks. By default, you can use the Vagrant user to log in.

You can also use the root user to log in, and the password for both is Vagrant. That's also auto-provisioned by the Ansible scripts, Playbooks, Recipes, whatever. As said, there are different ways to log in. You can use the Vagrant subsystem to log in to your VMs,

or you can use Secure Shell, or even you can use VNC, or it's a virtual box. You can use the console of the virtual box, of course. By default, there are also three ports generated. That's for VNC that you can connect to the client, server, or the recover virtual machine.

Right. You want to try it yourself. That's in fact, minus P is the provisioner. If you're using KVM, then you have to give it. If you're using virtual box, the default is virtual box,

then you don't have to give any option at all. All right. There is also a testing part, but the testing part which is existing today is in fact a donation from Red Hat using BeakerLib, and the scripts are written in a BeakerLib language, in fact.

They're using that tool if at the Red Hat testing system was their main system, BeakerLib, to test internally the stability of Relax and Recover, and that is still used at Red Hat. As we are using more and more the automated testing system

for other operating systems, vendors like Debbie, Ubuntu, Sles, so I'm thinking very hard to replace these scripts by another system like a bus automated testing system, but that's for later.

I can demonstrate it. It's working because Red Hat is using it, and it's integrated in this tool with the minus T option. Well, it proves the stability effect of your OS, whatever you want to test. And now the most important stuff we give in demonstration.

Is it visible? I have to go up a bit maybe. All right. Here you can see the main script.

It's over here. The minus H is the help effect, and it shows you different options like the minus D, which is the distribution. The default is sent to seven, and that was the first customer who asked for it, but now there are two others like the Ubuntu 14 and Ubuntu 16.

You just have to mention it. If you want to know, okay, how do I know? Just do an LS, and you will see here the Ubuntu. These are the effective operating systems. This is CentOS. These are the test scripts. These are the internal real scripts.

These are the test stuff, and the templates is effective. It's always interesting to look at it. These are the configuration files which are used internally for testing here. You can see this is the ISO booting with NetFS NFS. This is a pixel booting with URL type.

This is the default, and that is for BIOS. BIOS death names is for Ubuntu, and this is with ISO for CentOS. That will change in one of the upcoming versions. I will do auto detection of BIOS death names.

Okay, it's going too deep maybe. If you're using SAP or this kind of hardware, then most of the time you're using BIOS death names. It's not Ethernet 0, but it's the real physical hardware names that you will see.

In this case, Ubuntu was using the physical hardware, and in the recovery it was the aliasing effect, so it was a bit awkward. I made a different configuration file for it, but okay. It doesn't matter. Right, for this test we will be using CentOS 7, so that's the default.

Using Pixy is also the default. First box is the default, so I don't have to give any parameter here. I can just announce it. Let's announce it. It tells me CentOS 7 will start up Vagrant,

the providers, and then the second client. It tells me there's already an update for the box itself. Okay, I have to do box in load for that. That's also useful to replace the box. If you have a new box, then you have to do a reprovisioning, of course.

Anyhow, it doesn't matter for the moment. It will start up the client, it will start up the server. Very soon you will see the server coming up. There is the server. Again, the same kind of text will appear. This is pure Vagrant. Vagrant is driving the start up of your boxes,

and it's using internally in the backend virtual box. I always find it nicer to demonstrate something, and then it's easier to understand the complete picture. Now, yep, the client and server are online.

Just five seconds delayed. If I was mistaken, control C. If you do it now, it will destroy the boxes. If the boxes are destroyed, then you have to do a reprovisioning. That takes more than ten minutes.

Hands off. Check if the networks are online. That thing test is okay. That means that the client and server can talk to each other. Of course, if you want to back up. Here it sees, yep, it's already in your rear. Uploaded. It goes very fast. Sorry for that. This is the configuration file that is being used.

That is coming from the templates. It's copied automatically. Every time that you start, it's automatically recopied to the client. It's already making a backup. You can see here, it's the version from the 7th of August that is being used. Relax and recover. It's making a snapshot of your bootloader file systems.

Now it's already making a backup to the client environment. It is in this case automatically, because it's designed to sit, that the backup is going to the TFTP area of your virtual hypervisor effect.

You have to do nothing for it. It is built into the tool that it knows which kind of hypervisor is being used. That takes only a few minutes. Is there something more interesting to know?

Maybe here, automatically add the kernel options. The net.ifnames equals zero. That is for using the alias names. To disable the biased dev names. This is convenient, because last year I gave a workshop at the open source backup conference,

and we had issues with that. I never had issues with Lippert, but most of the students were using VirtualBox and had problems with the alias names of your interfaces. If you add this to the kernel options, then it goes away.

You see backup, backup URL, it's done. Backup was successful, it says. The backup client area on the pixie boot area will be made readable. I'm halting the client virtual machine now, and starting the recovery machine.

I'm using another box, which is an empty box. Again, it's the same vagrant, it's driving the complete stuff, and now this is VirtualBox taking over. You will see it's pixie booting, so it's really live booting. No cheating.

Because it's VirtualBox, the first network will not work, but the second one will respond, and then it will load automatically your kernel and your initial RAM disk, your rescue image from Relax and Recover. It will boot it up.

You can see it will reconfigure the network, and it's already starting with reformatting your internal disk, so your disk layout, and automatically the restore of your data, or operating system will be restored automatically. That goes quite quickly.

Afterwards you will also see that it will rebuild the kernel. Not the kernel of course, but the initial RAM disk, because it's another size of disk, so it has to see. It's another box, that means internally it's another system.

So what I'm doing here is a kind of cloning. I'm cloning the client on another fresh hardware, and you can see it here. For every version of your kernel, it will rebuild your initial RAM disk of that operating system, the client in this case. It is rebuilt on another system. You will see it at least twice.

And it takes longer effect almost than the restore of your OS. Once that is done, you will notice that the system will reboot, and then we hope of course, if the Relax and Recover does not contain any bugs, that it will come up automatically.

So here is the rebooting in 30 seconds. That's also something done by Relax and Recover. So that's a timing that I can make shorter or longer, even zero. But okay. It's always nice if you see an error, that you can interrupt and check your restored environment.

Okay, 30 seconds takes some time. Okay. Stopping automatically the system, and it will restart. You will see automatically that again, it will pixiboot, that the pixiboot configuration was automatically changed, that it will boot from the first internal disk,

not from the ISO or network anymore. So that's all done automatically in the background. And here it is, booting up your client system. It takes a few seconds, and whoops, there is the prompt. You can just log in with vagrant as password,

and well, the client is there. I recovered the client VM on the recover VM. I had to do nothing except running the script. And I can do that every day. I can do it for different backup solutions, backup.

For the moment it's only barriers, but I can do the same thing for barriers on CentOS 7, Ubuntu 14, Ubuntu 16. And whatever is missing can be added, of course, but preferably my sponsor. That was my main talk. The slides that will be provided contain an effect and summary

of the complete recovery steps made by the script, and also the pointers to the main site, et cetera, et cetera. So are there any questions so far? Because this was my talk effect.

No questions? Yes? Is that related? You're also offering the regular recovery CD image technology. So what is the impact of maintaining this, you know? Because there are new kernel versions, dot, dot, dot, dot.

So how much effort goes into this? A lot of effort. You can see here. This is the issues. There are, in general, around 70, 80 issues open all the time. And we are working with, let's say, three people,

not full time, but in our spare times. And also, like, the person from Soozee is almost working 80% of his daytime. And the guy from IBM is working, let's say, 40, sometimes 50% of his daytime on cases. Because like I said in my talk, IBM

is hot these days for the SAP HANA environment on PPC environments. The customers are requesting it, that there is a decent relaxation recovery solution for it, or a disaster recovery solution, I must say. And therefore, they put a lot of effort to test it on their platform, of course.

But it's also, I mean, it's like a moving baseline, you know? I mean, it's internal, and when do you, how do you keep up with this? Is this somehow automatic, also? No, no. Take another manual step for it, as well. Two minutes?

OK, so the question was, how do we keep up with the rolling new features that are coming from operating systems, from new versions, et cetera, et cetera? Indeed, that's a very intensive labor. I'm using to do that for 10 years. Every new time, there's, for example, Ubuntu.

From going 14 to 16 major version, they were split up from ISO Linux into Pixel Linux and, what was it? USB Linux and stuff like that. All these packets dependencies were different namings, so you have to foresee that. And that is, well, normally, in most cases,

when there is a new version, we ask that the community, common users like yourself, are testing it, and a lot of feedback are coming from guys like you that make issues and say, OK, we have tested it in minutes, something, and it fails for that and that reason.

And then we ask, show me the log files, because it really has an excellent log basis, and we can find a common solution for it. That takes time. We do have customers who have a support contract. It goes much faster, of course, because we are paid to do that.

I'm working for customers who are renting me for maintaining their relax and recover basis, which is for 10,000 systems, making the engineering that goes better. But OK, it's going so fast, it's not easy to maintain everything, and sometimes it takes months to get something fixed.

But the main basis, let's say, for the main versions of Linux being less red-hat, Ubuntu, Debian, is working fine. But OK. Maybe another question, because I really will try this out, and I read enough that the whole circuit,

that backup can recover, and nobody wants backup. Everybody wants to recover. Everybody wants to be able to recover. You don't care about backup. You need to do it, but you want to be able to recover. And it's really important that you know that it works. And now, I mean, the POSIX subsystem in Windows

is getting hopefully a lot better. Do you have any intention, maybe, I mean, there is basically really no such thing in Windows. So is there any chance that this could someday also

backup and recover a Windows system? The question is, can this actually recover backup a Windows system? The question is, yes. Only on a laptop that contains a Linux system. It's called backup equals block clone.

That's written by one of our colleagues, Devlat from HPE, from Slovenia. He wrote it, and it's working fine. But OK, it's not, it will never work natively on Windows, I think. It's a Linux software,

and it's running on a broad version of Linux, different kinds of OSes, different kinds of hardware. We love to see the Arts 64 hardware being incorporated because we got already questions from Red Hat, and asked them, OK, can we have some sponsoring from you? And it goes quiet.

So it's always a struggle between doing it for free and doing, because if I work for RIR, I cannot be paid for other projects. But I love to work for RIR. Another question? You said RIR is for the collection of Bash scripts. So why does the hardware come into play at all?

Shouldn't it be a folder for any kind of processor? The question is, RIR is written in Bash. Why is the differentiation in the different hardware required? Well, the question is very easy. Different hardware is using different boot technology.

The PPC environment is using at least two different boot technologies, the YaBoot and Lylo. Within the Intel, you have the old legacy BIOS way of booting, that can be used in GRIP, GRIP2, Lylo, and GRIP8 these days,

or UFI, that's using a different kind of boot, GRIP2 most of the times. But even within UFI, there are different boot technologies. So the way of booting within that area is different. So all these things are already inside RIR, and that is a moving duck effect.

New technology is added. We are always coming after the facts, well, most of the times. It's only a matter of time that it gets integrated, and that's why in RIR you will see different tracks for different hardware technologies. Okay, if there are no more questions,

I thank you for your time and for your interesting questions. Have a nice day. Thank you.