So, welcome back from lunch, everyone. I hope you found something to eat, and I'm not hungry any more. I'm here to talk about how you can build images containing OpenSUSE with OpenSUSE. I'm working for SUSE as release engineer, specialising in building OS images, and

by images, I mean containers, images, stuff you find on Docker Hub, OpenSUSE for instance, for various versions, and for leap, juice, live CDs, also for tumbleweed.

This is how such an image looks in OBS. You can see some files there and on the right side, you see some builds which luckily succeeded. That's how the behind the scenes stuff works. As you can already see, we're using the open build service, of course. It's not only used for packages but also supports images, so you probably used OBS for downloading

RPMs, but it can also handle images for building images and pulling in dependencies of images, and all of that stuff. It's a central tool to develop OpenSUSE.

So every distro you can find for OpenSUSE is built on the open build service. It's the central tool for everything. And OBS uses, of course, a tool for building images. OBS doesn't do anything itself, it just calls other tools, and, in this case, it's Kiwi. Yes, that's the official Kiwi logo, and I did not find it in a higher resolution,

but if you did, yes, just send it to me, but I think it's not really worth it, showing it bigger. It's literally a Kiwi with some OpenSUSE logos copy-pasted onto it. It has integration into OBS, that's the OBS part, which you'll learn more about later.

It can not only build images with OpenSUSE, it can also build images Fedoro, Ubuntu, all that stuff. The components of an image are various. In the end, we, of course, want to have some container images or bootable disk images, which using Kiwi are actually not that different.

Kiwi, of course, produces those images from a configuration file, which is called image.Kiwi. You can also call it a different file name, but you have to make sure that it ends at .Kiwi, otherwise OBS does not know how to actually build your package.

This Kiwi file contains metadata about the image, stuff like what the file name should be called, whether it's bootable, which architecture should have which packages, and also the version number. OBS reads this Kiwi file and knows then which packages it has to download from other projects, which are defined in the OBS project config, and then starts a build using those RPM packages

it collected and calls Kiwi. You can also supply a shell script, which is called in the root of the image, which is just a shell script called using bash normally, and you can really do whatever

you want in there. You can call systemctl, you can remove files, write files, also remove all files you installed, but that's probably a bad idea, and you can also add custom files using archives. In this case, it's just a single archive. Normally you should avoid to add custom files in image builds, as we have RPMs for that,

works much better. Kiwi image destruction is a fairly complex XML file, and you can probably not read any of that text, but don't worry. The top part of the Kiwi file contains obviously an XML header because it's an XML

file, and then some mysterious magic comments which tell OBS stuff it doesn't know otherwise, like Kiwi profiles or Docker containers which repositories to use. Below there, the image method info is defined, which means the name of the image file name,

the author of the image, and also profiles, which I will get into more detail later if I have enough time. The next part is somewhat more image specific. You have, for containers, you can define labels there, for other image types you define

which file system to use, for instance, whether to use grub in legacy mode, or UEFI mode whether to enable secure boot, or which to use. It's also defining the version of the image which is part of the file name. And some miscellaneous stuff like should you want to install documentation of packages

or not, but I will get into more detail later on that as well. The last part is probably the most important part of the image, because it defines the list of packages actually installed in the image. At the top, you also have to define which repositories to use, but as we are building

on OBS, we don't have any URLs to define, we just tell it to use the repositories OBS has configured. And then we have the list of packages in various groups, like Bootstrap which you normally shouldn't care about unless you do some weird stuff, and the list of packages in the image.

The good thing is you don't have to write such an XML file from scratch, and you also don't have to take care of the OBS project configuration for that. You can just go on, build open source org, click on new image after you're logged in, and then you have a selection like this, where you can select I want an image based on juice for leap, I want to build an image using kiwi as a Docker container, I want

to build a container using Docker file. For containers and other kinds of images, size is very important. For instance, for Docker images, it's usually around 40 megabytes, sometimes just below

50, but we really try to not exceed 50 megabytes, because the base image is, of course, the base image, and if you make it bigger, everyone using any base image or derived image also has that much space on their hard drive lost. And if you use multiple derived images using different versions of the base containers,

that is obviously even more severe if you have some megabytes wasted space in there. Also for ISO images like the live CDs, those have a limit based on which flavour they are. The rescue CD has to obviously fit on the CD, so it cannot exceed 700 megabytes, and the other live media have a hard limit which is artificial of one gigabyte.

We just cannot exceed that. The most important part of the image building process is the package selection. First, you have to be sure that you actually know the use case of the image. If you don't know what the image is used for, you can't actually do a reliable and useful package selection.

If you aren't sure whether a package should be included or not, you should redefine your use case and look at it. Maybe you should build two images for two different use cases if they are too different. Is it working? It's also important to not break hard dependencies.

It definitely causes breakage if you say I want to install package A but package A depends on package B but I don't want B. That doesn't work. If there is actually a use case where you can install a package but you ignore some of the hard dependencies, that's a sign that the dependency is actually wrong

and should be replaced by a soft dependency like recommends. What makes this easier is if you use patterns which you probably have heard of a bit in the previous talk about DNF. Then it looks like this instead of a 200-line list of packages.

If you just say, yeah, I want this pattern in this profile and the other pattern in the other profile instead of actually listing 200 packages per KV file, that's just unmaintainable. And it also makes it easier to have a synchronised package selection between the DVD installation, which has no such defined package list, and the TV image

files you have. Then you can just edit a pattern spec file and say, yes, I want this package A in all micro images, for instance, and then it's automatically rebuilding the image with those packages. Yeah, I just mentioned soft dependencies. Those are actually something to look out for.

Soft dependencies are expressed in RPM files using recommends and supplements. Technically, all the such as and enhances count, but for KV files, they don't actually matter as they are not pulling in any packages. Normally, they just give hints to the solver if one's branding is recommended,

for instance, or required, and also suggested it uses the open flavor instead of the upstream flavor, for instance. The issue is that a huge amount of packages are recommended by some other packages somewhere. Too much to fit on a CD or DVD, which is quite of a mess, so we

actually have tools which try to balance those recommended packages. And if you pull in, for instance, the X11 pattern, you also get the X11 optional pattern recommended, which then also has some recommends on it. For instance, if you install just X11 and you want to have a basic SWM desktop,

you end up with Yast. You don't really want that, especially not if you are ZastCon trained like a CD and you just want to install a minimal desktop. They are definitely important, though, because some stuff just doesn't work right if recommends are ignored. We have some bug reports about that.

Users install using the only requires option sometimes because they say, yeah, it saves two gigabytes of space. But yes, you also might actually lose a bootloader. That's not a good idea. We can't actually make the bootloader hard dependency for the minimal base pattern in this case, as you can definitely have images which do not need grub. You can use different bootloaders technically or just not have a bootloader

at all if you have something else doing it. Now, the supplements example is the Breeze 4 style package, which is only installed if you have Breeze 5 style installed, which is required by a pattern. And libQt 4. If the supplements get ignored and you install a package using Qt 4,

you don't have the fitting style, and it just looks weird. Yeah, the question is what you can actually do with those sub-dependencies and those issues. One option is to just ignore them. Just mention every package you actually need manually and disable honoring of sub-dependencies, which sometimes break because patterns

change, and other other commands change, and, yes, you would need to take care of that manually, and I heavily recommend using OpenQA for that stuff. The other option is, of course, to enable sub-dependencies and take care of blacklisting those packages and explicitly telling GV to not include those.

That can be also a mess, and it's pretty much hard to maintain, as when the build fails due to a size constraint violation, you have to find out which package to ignore. As recommended packages can then also recommend other packages, you have to find a place

in the tree where you can basically just say, no, I don't want this package anymore, and then ten other packages are also not installed anymore. This is the approach which the live CDs use, but none of the other images I'm aware of. So this should probably be ignored. The better approach is to have patterns which pull in all the packages which are normally

only pulled using sub-dependencies. Another option is RPM exclude docs which can also be a mess, as some packages don't actually take that into account, that documentation files can be excluded. And some files in RPM files are automatically marked as documentation, everything in user-shared

documentation, user-shared doc. And so, if a user says that he doesn't want to install documentation on his system, those files are just not there, and RPM doesn't even complain if you say yes, verify that this package is installed. Yes, the file isn't there, and RPM doesn't help you.

Sometimes configuration files are marked as doc by mistake, and that's quite a mess. To enable exclude docs, you also have to call Kivy with a specific option, RPM exclude docs in the XML file, but you also have to take care of setting the option in the install

system. Otherwise, every package should then install enabled image, will then again pull in documentation. A different problem is that also licence files were marked as documentation previously, and so many image files saying that documentation should not be installed, also didn't have

any licence files in them, which is, for most licences, actually a violation, so those images can't really be distributed. This is mostly fixed in Tumbleread and also mostly fixed in Leap, but for older versions like Leap 42 or Leap 12, this wasn't really completely done, so you have to really check

that every package you install has a licence file on the image. Container images are special in the way that they don't normally have something to do with block devices. You don't have a file system, and, kernel, you don't have to care about the booting.

Which also means you don't have to add a system like system D. Metadata is for containers much more important because containers are just global. If you say I want to pull open the Leap, you get Leap, and then you have a table with Leap inside but you don't actually know which Leap.

So we need to use labels to actually tag those images with yes, this contains Leap 15.1 and, yes, this got built in 2019. This contains this version of the package, and other labels which might be useful. The live containers are built on top of other images which is a difference compared to other

images like KUKA images, VM images, so you actually have dependencies for those images. If you want to build a container based on open source Leap, like let's just say a Selium container, you say I want to base Selium on an open source Leap container, then

download the latest version of the open source Leap container, and KV just adds packages on top of the image. Container labels are quite complex. We need labels for image version at build count, and also much more images, so we

actually have 16 labels to define. We have to define each label twice as the current system of labelling and Docker containers means that, if you derive an image, it overwrites all labels defined in the base image. That means we have to, for every base image, and for every derived image, add another

set of labels duplicating the existing information so that you don't overwrite all copies of this information. Of course, this is a mess if you have to define all those 16 labels by hand, but we have OBS helping us.

We can run custom scripts to build using OBS services. It looks like this in an underscore service file. You can say, yes, I want the KV meta info script to one before building, and the KV label helper to one. That way, the labels only look like this which is much less messy than the previous selection as you can see there.

It's actually doable, and most of this content is actually auto-generated as you can see by the macros with the percentage marks, like percent build time. About official images, what makes an image official? Of course, it has to build an OBS so that everyone can download the source, and uses

the official OpenSUSE signing keys that everyone can verify that the binary is authentic. It takes care of actually ... everyone can download the image sources to this corresponding image which OBS actually makes it easier using this tag which is also defined on the list here, so you always know if you have a Docker container or other image which source

on OBS it is built from, and you can check it out using OSC directly if you want to. An official image also has to go through the whole OpenSUSE review process which not only means independent review from a community member from the OpenSUSE review team but

also legal checks that you don't include any files on a proprietary licence, a licence you cannot distribute binaries of, and it also can only use packages from the OpenSUSE project on the OBS which also went through the review process, and so you can be pretty

confident that an image officially built by OpenSUSE is actually only containing OpenSUSE. If you want to submit your own image to OpenSUSE, it's actually as easy as just submitting to an OpenSUSE project. If you build it in your home, you can do whatever changes you want, and then just submit

to OpenSUSE update after doing the usual stuff like picking a development project and maybe finding some other co-maintainers that can take care of if you're not available. For new images, there has to be an extra step to actually make the image downloadable

which is normally calling someone from the community using from the release team using IRC or writing a mail. That should be a relatively painless process as the image is at that point normally building already. Kivy profiles are a really handy feature to build multiple images from the same Kivy file,

so if you want to have one container for LXC and one for Docker, you don't actually need to do it twice. You can say I have this Kivy file and please build this for Docker and please also build it for LXC. For that, you can of course define a profile in the Kivy header which you also probably

can't read. Then you can say I want for the LXC profile, I want a different package selection. You tell the Kivy using the package profiles equals LXC tag, and then a different package selection as well. It is especially handy if you combine this with the OBS multiple feature which contains

one of those magic strings above which you can see on the right, the OBS build profile, the command above there which is passed by OBS before building, and the multiple file which defines which profiles to build. In this case, just a single package builds two images, one with the Docker package selection

and one with the LXC package selection. One complex example is micro S which combines flavour and platform, flavour being something like container host or just bare micro S which is micro S without any container engine on it, and a platform can be anything, and you definitely do not want to create a Kivy

image for every one of those combinations, which is too much work, and profiles can be used for that. What got implemented just for this use case is having profiles which depend on other profiles.

In this case, it is defined that the container host depends on the container host profile and the KVM profile. The container host profile puts in packages you need for the container one time, and the KVM XAMPP profile contains packages you need for KVM and XAMPP, like the spiced guest agent, for instance.

Another useful package is live which you definitely want to use if you build your own custom image and not based on an existing container. It's a package which sets up the official OpenSUSE repositories in your image build by just using the scale CD control XML which is used for the installations.

It's passed during build time, and it then adds the fitting repositories for your distribution. If you build against leap, it pulls in the repository, for instance, and if you build against Tumbleweed, it pulls in the Tumbleweed. This also works for different architectures, as, for instance, the Tumbleweed have a different

URL than the Tumbleweed X86 repositories. You can easily use that by just installing the live package in your KV file and calling a simple config.sh and it's taken care of. The package can be removed again but you can keep it if you want.

The release process for OpenSUSE images is fairly complicated but you also don't need to know that if you just tell the release team that your image is now ready for publishing and download OpenSUSE. Images are all built inside the OpenSUSE factory main project in the images repo, and then

the binaries are released when the new snapshot is built into a separate project. From there, OpenQA notices that the new image is built and pulls the image once a few tests, depending on the image, and, if it's green, the project is released into the publishing project.

For containers, they take a different path and go to the OpenSUSE container registry. The Docker Hub images are a different work flow. They are actually pulled after 15 minutes from a different bot running on a different system, and copied from the OpenSUSE container registry to Docker Hub.

And here are some resources you can definitely use if you want to build your own images. There's a wiki page about building containers specifically, and there's also a link to the image template page on RBS. And what you definitely do not want to miss if you work with Kivy is the official auto-generated documentation for the XML format which documents literally everything you can do with

Kivy and always the latest version is at this link. It tells you exactly which XML element is expected where and also what this XML element means. It's the ultimate source of truth as it's generated from the files which Kivy uses internally to pass those XML files. So it's always correct.

Yes, and that's it. Any questions? So now, okay.

Does it support multi-architecture? So creating container images for not just x86 but ARM and power, et cetera?

Yes, OBS does that natively. If there are multiple architectures in a container project enabled, OBS actually measures them together and pushes them all at once, and you can even mix them from different projects. For instance, the OpenSUSE images for ARM are built in a different project as the OpenSUSE images for x86.

Just merge into the same project and OBS does the magic so they are both available as OpenSUSE slash leap, for instance. This is kind of more of a statement, but if you look at the soft dependencies with patterns in Tumbleweed again, you will find some things fixed like the minimal

base pattern doesn't recommend grub anymore, and we'd like the patterns to be more useful for containers, so if anyone can suggest more changes, we're happy to consider them.

So you said that the dependent containers are pulling, let's say, the container they should be used as a base. Does that also mean it creates layers instead of basically rebuilding the container?

Yes, that's done by a scopio automatically. So basically, if I use them all together, my registry only downloads the base container once and then merges the layers? Yes, OBS keeps those layers separately. It's not visible on OBS directly, but they are actually the layers themselves at different

tables on top of each other using the usual hash. So with this control XML mechanism that you've shown, is this limited to OBS-provided repositories,

or can this be any URL of any third-party repository as well?

It can be any URL which OBS can map to a local project, which means you can just pull repositories from any HTTP URL. It has to be a URL reachable from OBS inside OBS packages. If you write, for instance, it has to map to a home project, but you can't just pull

packages from if you want to. That doesn't work as OBS builds don't have internet access. OBS has to resolve those dependencies all internally, and you can't access packages which are not inside OBS. For official OpenSUSE images, you have to make sure that the image only pulls in OBS

repositories, which means that it honors the project configuration. This is necessary so that, if in staging, you have an updated image, it builds against packages in the staging and doesn't pull in anything from a home project, for instance.

No questions? Okay, so thanks for listening.