Hi, I'm Takashi. So I've been working for SUSE fairly for a long time. And still, I'm working for the kernel stuff. And this year, again, my talk is about the kernel.

So this outline of my talk today, at first, I'll start something clarifying meters and followed by what's new in the deep 15.1 kernel. And then going to the OpenSUSE kernel development process.

And asked something to show how things get fixed and how the things get tested. So the first topic is about clarifying meters. That means just kind of FAQ about our kernel.

And here's my talk about OpenSUSE leap and SUSE Linux Enterprise. And one of the common questions is, well, leap and SUSE kernels, they're the same, identical?

Well, it's yes and no. Yes, both packets are built from the very same source tree. However, SUSE kernel and deep kernel, they have major differences in the binary form.

And first of all, the kernel configurations, they're completely different between two. The SUSE Linux Enterprise kernel have reduced kernel configuration, while OpenSUSE deep contains almost whole enablement of all features.

And SUSE kernel has a split for two packages. One is supported, and the other is unsupported modules. And while deep has only one kind of package, contains everything. And both packages are built in different environment with services. IBS internal bit service versus OBS, OpenSUSE bit service.

And then SLE kernel supports live patching by K-graph, while OpenSUSE leap doesn't provide that yet. So they're also the same. They are built from the very same source tree.

The result on binary packages are completely different. And maybe that is the most commonly asked or seen thing. The OpenSUSE leap 15051, 1412 kernel, that is very, very, very old.

Yeah, yes, it's old. And actually, looking at our history, OpenSUSE leap takes relatively old kind of basis. So OpenSUSE leap 42.2 took a folder to four kind of basis.

And OpenSUSE leap 15.0 and 1 are based on folder 12 kernel. That's correct. However, we took very huge amount of patches on top of that. And now, well, OpenSUSE leap 15.0,

we already took 22,000 patches on top. Now, guess what? For 51,000, 46,000. So almost 50,000 patches on top. And that's why it is no longer folder 12 kernel.

It is a kind of question of the shape of T-cells. That's a famous sort of experiment. So this is no longer original folder 12, but quite containing so much different components on top.

And is it deep? Is the kernel stable? In that case, the answer is yes. So actually, this is the reason, very much reason, why we take that old kernel code base. And that's another point about the stability

is that we guarantee, as a kind of guarantee, to provide a consistent kernel ABI. That means if you build kernel module package once, then this package can be built for all kernels after upgrades, updates of the same version.

And we do proactive backporting fixes, and also that are from the several trusted sources. And we do CI and QA testing regularly on the kernel. So let's continue to the next thing,

that what's new in deep 15.1. So, as I said, there are so many changes in the kernel in deep 15.1. And this is a table showing which directories, top directories in the kernel, contain the changes.

And as you can see, that's majority of changes have been done to the drivers. This astonishing over 86% of the source code changes. So in the end, we almost had 5 million lines of changes,

and about 86%. And this is not surprising, because in general, device drivers that tend to have a bunch of changes in the code, while the important changes

like the memory management core or file system, their amount of changes are short, so small, however they are important. And for deep and the three kernels, we are kind of conservative. So we don't touch too much about the core parts

intensively, while we are backporting many things to support the new features of the new machines or new systems. That's one of the reasons that we get these statistics. So now 51. So let's start from the server side.

We had, as you can guess, many, many storage and file system block layer and network updates. Most of SCSI, so recent SCSI drivers have been updated, and if you need a bunch of DMA. And another interesting thing is NVME over five weeks

that is required for net app stuff, and we catch up upstream. File system got also a bunch of updates, especially Battle Express. This is our default file system, and also XFS, XA4. And at this time, we got a bunch of big cache updates

and MD rate and safe and CIFS. And block layer, now we are still, I think we still didn't switch default to multi-queue, but we got the updates to the most recent code for the block memory queue,

and that includes a BFQ IO scheduler. And network, yeah, of course the network core has been updated, and ethernet, yeah. Broadcom, Kabin, Chelsea, Cisco, oh well, you name it, most of the vendors have been updated.

The desktop usage, first of all, that is video driver DRM stack updates that we raise the whole code up to 4.19 or later stage. And actually, this is a very much, many, many changes,

and this is about 20% of the whole 50,000 lines of changes. And also Wi-Fi drivers, we almost, we update almost the whole Wi-Fi drivers and Wi-Fi stack up to 4.19 plus. And storage, MMC, SD, and there have been updated.

And sound drivers, that's my area, so I update, upgraded whole HD audio, USB audio to 5.0 or even 5.1. And then bus platforms, et cetera,

we got a Thunderbolt update, PCI hot block, and FGPA, small bit still. And TPM, 1.2.0 things, and RDT, memory bandwidth allocation stuff, also hardware, crypto, and x86 is WMI,

this is for laptops and desktops. And virtual machines, KVM, Hyper-Vs, and they have been updated. And security app armor, and another interesting thing is AMD SEV, that is a secure encrypted virtual machine, I forgot the acronym.

And tools, as a perf and BPF, of course these stuff have been updated. And architectures, and for x86, we add support for the recent Intel and AMD chipsets,

like, I forgot the name, Whiskey Lake, Amber Lake, or Vee Beach, also Ice Lake, and also AMD Ryzen Zen 2, Zen 2 is just a little bit. And ARM 64, well, there are so many changes, and I cannot list up.

So if you have a question, here is Matthias Brugger, then he can answer, yeah. Or something's broken that it's because his updates. And ARM 32-bit, for that, we didn't have much updates,

so much back ports, but only casual fixes that are spotted from the trusted stable tree fixes. And ARM 32-bit is provided only for LIP, this is not for SUSE Linux Enterprise. And on the other hand, PowerPC 64 and S3 19,

they are mostly for SUSE Linux Enterprise, and they got also updates. Also, we provide, I think, we provide that these architectures package also for ports on OpenSUSE LIP.

So continue on SUSE Linux kernel development, how that happens. So, not surprisingly, we manage that in Git. Yes, of course, everything is on the Git. And what is different in OpenSUSE kernel source management

from other distribution is that we keep all code changes in individual patches, instead of applying the patch on the Git kernel tree. Instead, Git kernel repository contains patches,

patch, patch, patch, patch, patch files. And this is applied dynamically at the building package, so just like a code. And this series.conf contains a list of patches, so which patch is applied first, and next, so on. And the repository contains patch files.

When you look at the kernel source packages, you can find that over 97% of patches are from upstream. So I mean, this means upstream, the industry. And that is the result. We really try hard to push upstream first rule.

So we basically accept only patches that are upstream or that will be upstream. And the recent changes in the development process is that when we apply sorted patches. And what is sorted patches?

That means basically we apply the patches in the same order like upstream tree. So suppose that upstream industry had changes A, B, C. And then we applied patches A, B, and C, that order.

That's all. That is sorted patches. That means we, for example, when we hatch the patch B and C first applied, and we found out that the patch A is missing, then we have to apply patch A.

And in that case, we don't apply patch B, C, A. Instead, we apply patch A, B, and C. So for that, always reordering the patches to adapt the upstream, the topological order. So that is the key of the sorted patches.

And why we do that? Because by keeping that sorted order, that each patch becomes closer to the original form. And this has a big merit that makes the backport easier,

backport cleaner. And also it makes easier to catch any backport mistakes. You can just compare the backported patches, patch and the original commit. So how to expand patches?

Yeah, we have 50,000 patches, as I said. And this may take really long time. If you run code script for these 50,000 patches, I measure that, and that takes six hours

on this machine. Six hours patch up time. It's not good for the daily job. Morning, I plan patches, then sleeping, the day's over. It's a good job. So that can be actually faster.

So we had already a script, so-called a sequence patch, and that applies the patches, and just like code, but a little bit, yeah. Optimized way. And this takes nine minutes, 25 minutes for 50,000 patches.

It's faster, yeah, but still take time. But there is a trick. The script got fast mode, and what does mode do? The problem of the previous approach was

that script invokes patch program at each time. That means 50,000 times patch was invoked, so patch program executed. That took so many time. So instead, this mode, we gather all 50,000 patches in a single patch file and feeds to patch program.

Then that works. Then that's 80 seconds. So that's the approach. However, the drawback of this approach is that you cannot roll back to the patch that fails to apply.

Is there any better way? Yeah, there is. The reason we developed a program called Rapid Coach, and this is a program written by Rust, and does apply the patches in parallel, and also supports a rollback at the patch failure.

If you use that program on my machine, that's with eight-core machine, it takes only three seconds for 5,000, applying 5,000 patches instead of six hours. That's really awesome.

Then SUSE kernel D3, this is publicly available, and you can see that kernel.suse.com at any time. This Git tree contains several different branches, and each branch, mostly, yeah, each branch represents for each, so to say, the product.

Like, it's the 15, what is it, 15sp1, opens at 15-0, opens at 51, and also, TumbleWidge is taken from the stable branch that tracks to the upstream stable kernel.

Also, there is a master and a head, this tracking the Linux tree, that's currently 5.2 RC1, and also, there is vanilla Linux next branches that automatically fetch the Git tree from the upstream, and these are all just for testing.

And Git workflow. We do, yeah, kind of GitHub-like, or normal Git workflow, just taking each Git branch maintainer, takes a pull request from each developer,

and merge after integration test and review. That's the way. There is a kbuild bot running that's testing the builds and also doing the sanity checks, like a patch can be applied cleanly or something wrong, contained, and so on.

And if everything's okay, then kbuild bot says that, yes, this branch can be merged, and the branch will be able to reuse and merge that stuff. And one thing to be noted, that is the branch, some branches are shared by other branches.

For example, it's 15, it's shared by many other branches, and that branch is automatically merged to other branches. This is like that. So it's a branch, it's merged to say 12sp4, and it's 15sp1, and it's 12sp5,

and it's 15, it's dragged to 15.0, and so on. That's for users and developers. One good thing to know is kernel of the day. This is really, I would recommend to remember.

And actually, this is a kernel package built from the very latest kernel Git branch. And in OBS, it's updated daily, so every day it's updated and fetch the very latest Git repository and rebuild the package.

And that OBS kernel colon sl15, or this kernel colon branch, that contains the kernel package, kernel of the day. So why this is good? So you can install kernel of the day package from other branches too.

That means if you have a brand new laptop that's OpenSUSE 51 still doesn't support, then you can install OpenSUSE tumble weight kernel, and bury this one from the kernel stable tree. Or if you have a regression after upgrading to the OpenSUSE VP51,

then you can just install the old OpenSUSE VP50 kernel on top of your 51 system, and see whether the problem gets fixed by that. And if yes, then this is a kernel regression of 51. You can report that.

Then we see what changes are done and so on. Once it to be noted that it's better to increase the number of multi-installable limit in zip.conf file beforehand. And as a result, I think only you can install

two or three kernels on the system. But usually I increase that number to five or six. So bug fixes. So as I said, the deep or in general OpenSUSE,

we apply the bug fixes on by our hands. And usually we take the bug fixes from upstream commit and how we can find that bug fix. Nowadays, the kernel developer is supposed to mark fixes tag if the commit is supposed

to be a regression fix. Then there is a script program called git fixes. And this program can scan the upstream changes and reports which commit may fix the bug fix.

The bug that's found in our kernel. And this is one way to find out the fixes from the upstream. Another way is just looking at the stable kernel trees. Currently there are 4.14 and 4.19 long-time support kernel.

And there is script, also git fix can take a look at that and see which commits are missing. Possibly fix that problem. And if we have a problem, then of course you can report OpenSUSE backscatter.

Or we can take a look at upstream backtrackers too. And now, this is something new. Now we have a lightweight CI tests for the kernel. And actually this is running hourly on the git

on my desktop, currently not the cloud. And it fetches the git commit. And if something changed, then running the tests. And it tests only KVM and boots to the desktop system.

And doing also suspend regime testing. And there are different images built from the different file systems and also every legacy boots and different QM graphics backends. And so that helps sometimes to catch their regression as early as possible.

And another new thing is that we deploy OpenQA tests for the kernel of the day. So thanks to the QA team, they take certain branches,

currently SF15, SF15, SF12, SP4 and something else. And they test kernel of the day, so every day basically. And that's OpenQA, so it's currently limited only on virtual machine. And test scenarios are seen also limited

to catch currently they test only LTP. So that's basically all my topics. Then the resource, the kernel.studio.com and OBS repository is there. So if you want to find something, then you can take a look at that.

Okay, so that's all. Any questions or a bashing to the kernel package or something else? Yeah. Hi, it's not directly a question,

but more like a comment. So I think as a OpenSUSE community, we should think about how to, how to manage, how to handle requests for OpenSUSE leap kernel development.

Because I think we use an OpenSUSE leap, we use this leap kernel, which I think makes a lot of sense because of the stability. But we sometimes had the problem that afterwards someone came and said, hey, this driver or this peripheral is not working, this is a bug.

And we told him, no, we can't add this driver now because it's already closed. So I think we would need to formalize in some way the possibility for the community to create feature requests against the leap kernel to include their needs.

Yeah, thank you, yeah. Very much I agree that we used to have open page in the past, but it was discontinued, I think. So the current way to request for the OpenSUSE leap is either back to there,

just open the back to there, entry to report that, or ask on the OpenSUSE kernel mailing list or factory. Maybe the OpenSUSE kernel is better. But yes, it would be better to have some more formal way because we want to track the feature request itself.

Yeah, but I think it's, yeah, little bit above my hands.

Good, okay, thank you. Thank you for your attention.