Good morning, I'm Guillaume Gardet. I work for Arm as a partner engineer dedicated to SUSE and OpenSUSE. I will cover what happened

since about a year on OpenSUSE side for Arm. I will start with what is OpenSUSE on Arm. Then I will give another view of OpenSUSE on Arm workflow of the development and

we will see what happens on OBS side, on OpenQA side, and specifically on Tumbleweed, Cubic, Leap, a little word about OpenSUSE wiki and finally the to-do list and we will have some time to

for questions at the end of the talk. So what is OpenSUSE on Arm? The short answer is obviously OpenSUSE running on Arm architectures. What does it mean?

We support 32-bit Arm architectures. Arm v6 for Tumbleweed only. We also support Arm v7 on Tumbleweed and Leap and also 64-bit architectures on both Tumbleweed and Leap. Arm systems

are very wide. We cover very small embedded systems such as Raspberry Pi, but we also have some very powerful server system such as the Thunder X2 from formerly Cavium, now Marvel. And

there is some differences between those systems. On Embedded World we are more with custom bootloaders which is a bit painful sometimes. And on the server side we use UEFI so we have a single image and

you can run it on your server. Before talking about OpenSUSE on Arm workflow, a little word about an OpenSUSE workflow on x86.

For Tumbleweed when we want to update some packages you submit it to factory, it is reviewed, it is tested and once accepted to factory it is pushed to OpenQA for further tests and

if it's okay the update is released to users on download server as Tumbleweed. For the Arm side we have OpenSUSE factory Arm project

which is simply a link to OpenSUSE factory x86. The same for Leap. We have a project link from OpenSUSE Leap 15.1 Arm to the x86 project which is OpenSUSE Leap 15.1.

So we reuse all the sources. It's really the same. No patches on top. It's updating in real time. So when you have an update on x86 factory you get the update on Arm side and

there is just a very small overlay which is available to handle the snapshot version, the content of ISO and FTP trees on the Arm side. Here you have a screenshot of the OpenSUSE Leap project. So you can see that we have only four packages for the overlay and

all the rest are inherited packages from the x86 project. So if you want to get an update on Arm side you have to push it on the

x86 side because we share the sources. So the workflow is the following. You submit it to factory as previously. It's reviewed. It's a little bit tested and then when it reaches factory, it is inherited in factory Arm and

then it is pushed to OpenQA Arm and if it's always okay, it is released as Tumbleweed Arm. If the OpenQA is not okay, the update is blocked so that users are always fine.

Now let's talk about the OBS. In OBS we get some servers to get more power to build packages for Arm. So thanks to our sponsor Marvel, formerly Cavium, which gives

some DirectX 2 machine. Those build powers allowed to remove snapshotting between x86 factory project and Arm project.

So now the sources are updated in real time. Previously we needed to block the updates because it was very too fast and then Tumbleweed didn't have any time to rebuild and push to OpenQA. So now it's okay.

It allowed to enable more Arm builds in DevOps projects so that maintainers can see build failures on Arm earlier and they can fix it when they care. Here is a little word about how to enable Arm builds in your project.

So it can be your Arm project or DevOps project. It's the same process. You just go to the repositories tab and then click on add repositories. Then on the second page you just select the Arm distribution you want to build for. For example

OpenSUSE Leap 15.1 Arm or OpenSUSE Factor Arm. Previously it was OpenSUSE Leap 15.0 ports because Arm and PowerPC was part of the same OBS project.

So maintainers, please enable it and catch build failures for Arm early. Arm containers are now published when Arm is released not when X86 is released. It seems obvious that it was not the case previously.

Some Arm OBS workers have been updated to better fit requirements of packages. In fact, we decrease the number of build workers but increase the CPU,

RAM available. On OpenQA side, how many people know OpenQA here? Yeah, do you know how it works?

Yeah, not so much. So a little word about how it works. You have one server chart for all architectures where you have a web interface and an API. These servers, all the files, I mean ISO, hard disk images,

repositories to test. It has also the test suite information and then it controls the test run and stores the test results. Connected to this server, you have a number of machines which can be X86, PowerPC, Arm, whatever.

On those machines, you are running some virtual machine. Here, it's QEMU and you run tests inside it. For each test, you run some actions and check if the result is expected or not.

If you want a very complex schematic, you can read this. Just go to open.qa slash docs or ask OpenQA guys here.

Just a very small example. This is the media check test. It just puts the ISO, starts to check installation media and makes sure no errors are found. So you can see screenshots and the screenshots are used to check if

the test is okay or not. On Arm side, we just test Armv8, so the 64-bit flavor. Currently, we do not test for Armv6 or Armv7. Last year, we used a Seattle machine

which was 6 CPU with 32 gig of RAM and we only run two rockers on it. We added a few tests for Tumbleweed and Leap, but with two rockers, you are very limited if you want

not so long test in OpenQA. Then, at the end of the year, we get a new machine, a D05, which was very more powerful with 64 CPU, 128 gig of RAM, a big SATA disk and

initially, we enabled 10 rockers on it and now we are at 16 rockers. So, we added even more tests for Tumbleweed and Leap and we now have a very good coverage of OpenQA on Arm.

We test, for example, upgrades, but if it's tools, virtualization, RAID, multi-pass and etc. Of course, we needed to update some tests for Arm because sometimes only

x86 is supported in Arm in the tests, so you need to update. We got non-Arm specific updates, which include a new test ideal, of course. OpenQA has now a good developer mode, which allows you to stop the virtual machine, take a screenshot, save it and

so on. There is a good script to use when you make a pull request on the GitHub to update tests or add new tests and

with this script, you can run the tests on OpenQA.openssa.org and show that it's okay. We added support for huge pages with KEMU backend and also generic options to be passed to KEMU. A few numbers

from one or two weeks ago, on Tumbleweed, we have 72 tests now plus 69 tests for the kernel and on Leap, we have

59 tests. On Tumbleweed, there is a test for cubic micro-res. You don't have it on Leap, of course, because cubic and micro-res is a Tumbleweed flavor. On the DVD, you have three more tests on Tumbleweed side

because LXD pattern is only available on Tumbleweed DVD, so you can test it only on Tumbleweed. And there is just two more tests added recently, OpenQA bootstrap and OpenScape. On the NetEyeso,

we have two more tests on Tumbleweed side too because we create an hard disk image with a release Tumbleweed and test the upgrade from this hard disk image to the tested image. And we also test, we have two to test suite for

Justinus RS. A few screenshots from two days ago, so you can see that Tumbleweed ARM is not so bad. Leap 15 has been released and it was in pretty good shape.

The six failures are not all real failures. You are false positive and only one real bug. And on Justinus RS images, it's not so good.

But it's not very big things. In fact, when you run a very long test run, a single test failure make it all red. So now let's talk about

Tumbleweed. Tumbleweed is now officially supported and is no more best effort port for ARM. But for ARM, I mean for AR64, it's not the case for ARMv6 or ARMv7.

Lots of package have been fixed at build time on OBS, but also on runtime on OpenQA. It includes kernel, Firefox, Chromium and more. We also added new packages or enable build for

64-bit. It includes LDC compiler, free Pascal compiler and more. And we also added some features to existing packages, for example in Mesa where we enable some ARM specific options. On ARMv7, we enabled

two more boards on EFI grab to support. It's Sabolite and Chromebook Snow. And we get some boards out of country

projects and are now in the Upstream Factory ARM project, such as you do Neo. And thanks to the always up-to-date kernel in Tumbleweed, we get all the improvements we get in kernel upstream. ARM

AR64 supports also Cubic since January this year. It is tested in OpenQA

with Tumbleweed. If you want to get more information on it, I added the link on the slide. Or just go to cubic.opensuzio.org and you will find it.

On leap 15.1, just a little word about how it is built. In fact, leap 15.1 inherits some packages from sleep 15 SP1. It's for core packages such as GCC, kernel, Camu.

And for the rest, it inherits from leap 15.0. And if package maintainers want to, they can push updates from Tumbleweed. So this is the case, for example, for Carita or Firefox. And we, of course, fixed some packages built and runtime failure.

As for Tumbleweed, we switched two boards from ARM v7 family. It's again Sabolite and Chromebook Snow. And now, the ready-to-use images such as justin.os OS or Docker

are testing in OpenQA. It was not the case before. And we release always up-to-date images from leap 15.1 project.

Yep. A little word about OpenSuzio Wiki. The main page for ARM is portal-arm.

On this wiki, you can find some updates such as on portal-arm, of course, but also on hardware compatibility pages. We added some, for example, the T05, the OverDrive 1000.

And there is an interesting page with tests on systems which is OpenSuzio supported ARM boards, where you can find if USB or video output is supported and working with Tumbleweed or Leap.

So if you have a board, please add it. And if you want to get information, go to this page. So a lot have been done, but still a long to-do list.

We should continue to improve the wiki with new information and up-to-date information. We can improve on OBS side. What we need to do is to speed up ARMv7 images

because this is a cure on bottleneck to get more Tumbleweed snapshots. I think we have about 100 ARMv7 images to build. Each type.

And we should enable ARMv build in more developed projects so that people, so that maintainers can catch build failures and fix it early. Otherwise, you have to wait that updates reach Tumbleweed and someone notices its failure and the fix is much longer this way.

We can improve on OpenQA side. Maybe add more tests for AR64. Why not testing ARMv7 images?

OpenQA allow tests on the real hardware, so why not test the Raspberry Pi 3, for example? We can increase the number of OpenQA workers to speed up tests, maybe. And we should monitor continuously build failures and test failure and fix it as soon as possible.

And people, please report the bug you have because I often meet people who told me, hey, I have this problem on this board, okay, did you feel the bug, a bug or something?

Oh, no. So, please do it. We need to help some, we need some help to test and get feedback on systems people use. Could be a simple board on the embedded side

or big server and told us what is okay, what is broken and we may fix it. We can add new features such as secure boot, for example. And we need to improve the graphic stack on ARM.

So, it includes PCI express card such as NVIDIA, AMD, but also integrated system and chip into the system and chip such as NVIDIA, Adreno or Mali. And stay tuned on Tumbleweed because kernel 5.2 and Mesa 19.1 are coming rather soon.

And it adds Mali GPU upstream support. So, it's currently just an initial support, so don't expect a full OpenCL or Vulkan or whatever.

But most part of OpenCL ES 2.0 is supported. So, let's see how it goes. And please join us on IRC or on the mailing list.

Have fun. Do you have any questions? You have a microphone behind, please.

Hi. My question is, you said on ARMv7 we don't have OpenQA at the moment. So, is this a problem of human resources to enable and look after them

or is it a problem of hardware or, yeah? It's just a problem of nobody takes care to do it. So, we have two options. Either add an ARMv7 board such as an ARM Dell board, for example,

and run rockers on it. Otherwise, you can run rockers on 64-bit machine with KEMU. It just specify that you want to use 32-bit instruction.

But I think we will do it shortly. Hi. My question about the testing, you mentioned about Raspberry Pi. So, does it work?

So, how does it work for OpenQA with actual hardware? Which part of the hardware? With actual hardware, not virtual machine. You mean testing the real hardware inside OpenQA? You mentioned that in the last slide. It is already done in SUSE OpenQA, so it works.

You just need to add it to OpenSUSE OpenQA. So, it's just a matter of hardware. All the software is ready to support it. So, that means that it doesn't test any specific Raspberry Pi hardware features?

Or what kind of things are tested on Raspberry Pi? I'm not sure if you should ask SUSE people to know what they test, but I think you just run the same test as in KEMU. You run it on the Raspberry Pi. OK, I'll ask later, thank you.

Any more questions? OK, thanks.