We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

openSUSE Maintenance

00:00

Formal Metadata

Title
openSUSE Maintenance
Subtitle
Leaping to new heights
Title of Series
Number of Parts
63
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
While openSUSE has a rolling release, which is constantly integrating new versions and bugfixes, we also have releases where the main codebase is frozen and we release bugfix and security updates via online updates. This process has been established for a while and is coordinated by the openSUSE Maintenance Team. Also, last year a new release came with new and changed update origins, openSUSE Leap 42.1. This release inherits some parts of its updates from SUSE Linux Enterprise 12, which might be confusing at times. This talk will offer a refresher on the maintenance processes and also some details you as packager needs to observe, also especially in regards to Leap.
13
38
Thumbnail
23:08
44
Thumbnail
25:53
Data managementMereologyProjective planeSoftware maintenanceProcess (computing)Information securitySoftware maintenanceUMLXMLComputer animation
Data managementCoordinate systemMereologyProjective planeSoftware maintenanceSoftware maintenanceLecture/Conference
CodeData structureMathematicsLibrary (computing)Product (business)Incidence algebraLevel (video gaming)Software testingBand matrixComputer configurationConstraint (mathematics)Arithmetic meanBinary codeBitLine (geometry)MultilaterationPower (physics)Physical systemProjective planeSoftware maintenanceRevision controlBranch (computer science)Process (computing)Software bugInstance (computer science)Information securityPoint (geometry)Control flowKernel (computing)Java appletStability theorySoftware frameworkSource codeMultiplication signRepository (publishing)Software developerLibrary (computing)BuildingBand matrixLimit (category theory)Power (physics)Software maintenanceRevision controlFactory (trading post)Software bugInformation securityKernel (computing)Focus (optics)Java appletKeyboard shortcutFreewareXMLComputer animation
InformationMathematicsComputer architectureRobotDescriptive statisticsIncidence algebraCategory of beingUser interfaceSoftware testingComputer configurationForm (programming)Binary codeBitLine (geometry)Content (media)Coordinate systemMultiplicationPhysical systemProjective planeNumberSoftware maintenanceConfiguration spaceRevision controlText editorRegular graphComputer virusBranch (computer science)Factory (trading post)Process (computing)Software bugInstance (computer science)Information securitySoftware repositoryInheritance (object-oriented programming)Set (mathematics)Open setKernel (computing)Latent heatMetadataElectronic mailing listTraffic reportingSource codeComputer fileBootingMotion captureOpen sourcePatch (Unix)Proxy serverMultiplication signRule of inference2 (number)Repository (publishing)Software developerInformationMathematicsDescriptive statisticsBuildingIncidence algebraTask (computing)Group actionContent (media)Projective planeSoftware maintenanceLinearizationDependent and independent variablesComputer virusError messageSoftware repositoryElectronic mailing listComputer filePatch (Unix)Rule of inferenceUMLXMLProgram flowchart
DatabaseInformationMathematicsTheory of relativityRobotSoftwareStatisticsIncidence algebraCategory of beingSoftware testingComputer configurationBinary codeBitLine (geometry)Projective planeCountingNumberEmailLink (knot theory)Software maintenanceRevision controlRegular graphComputer virusCASE <Informatik>Factory (trading post)Process (computing)Information securityNetzwerkverwaltungSet (mathematics)Kernel (computing)Electronic mailing listScripting languageChemical equationSource codeComputer fileBootingMixed realityPatch (Unix)Meta elementMultiplication signFreewareWeb 2.0Mechanism designRepository (publishing)Software developerDatabaseInformationMathematicsStatisticsIncidence algebraINTEGRALSoftware testingBinary codePhysical systemExecution unitTime zoneEmailLink (knot theory)Software maintenanceFactory (trading post)Process (computing)Open setJava appletSource codeQuiltPatch (Unix)Standard deviationFirefox <Programm>Demo (music)Mechanism designComputer animation
InformationEmailConvex hullDemo (music)UMLXMLComputer animationLecture/Conference
Transcript: English(auto-generated)
Welcome to a basic or to an introduction to the OpenSUSE maintenance process. Some look at the technology and some of the philosophies and also the people behind it.
My name is Markus Meissner, I'm project manager security at SUSE and I'm also part of the OpenSUSE maintenance team. The OpenSUSE maintenance team is basically currently just staffed by SUSE employees but could also take some volunteers.
So we have Benjamin Brunner, Jens Mammen, who is our intern, Benny is part of the team I'm in, and also Andreas Stiger, the other project manager of security, we four basically take care of the OpenSUSE maintenance as coordinators.
What are the goals of the maintenance? So basically maintenance happens after the release and our goals are to keep the users free of security issues, free of bugs and also if possible enhance the experience in a carefree way. What does it mean carefree? Carefree means that you can install the updates and don't need to worry that something
breaks. So it's not as hot off the press like Tumbleweed where like for instance the GC6 update yesterday broke some parts, we want to avoid that, we want to have it like that the users
that are not so familiar with all the low level technology or with rescuing your system really can install the updates and don't have problems operating them. But they still stay secure and also have less bugs. So what do we do updates for?
As said we do bugs, security fixes, if something is missing, occasionally the release process misses stuff or some people say yeah it would be totally cool if we would have this package for this product that we can also add it. Some improvements, but for improvements usually we rely on the future products and also some
features so we are not strictly against for instance doing version updates or feature updates, it's just that we still need to fulfill the carefree aspect that nothing should break during the update.
So nothing should be a risky update. What we cannot do updates for, we cannot do updates for the installation, of course we cannot change the installation medium, we don't want to do new kernel versions so we do minor version updates if the kernel is still in a stable process but major versions
are always a bit too risky. So the Everclean project tried or did a 3.12 update and that went surprisingly well so that might be revisited at some point in time but so far we were processing or we were working quite good with the same kernel version over the lifetime of a product.
Major library versions are out of the question because the major version that we would need to recompile and we are not recompiling the packages, we are just releasing the updates for the packages that we fix bugs in. New build tools or change dependencies is difficult and we also cannot really remove
packages, we can of course do a hack, let the package be obsoleted by another package but removing packages is also usually not really possible. What are our limits, the constraints that we face, there is a manpower issue so we
don't have that many developers over all of OpenSUSE, there are a lot of people focusing on Tumbleweed, on the new stuff, less on the old, I don't know, it is of course a topic, I am not even thinking about it with my FET DSL line but some people might
have less than 16 megabits at home so pushing out multi-gigabyte updates every week is definitely not an option because that would, with our mirror structure we would cope with that but people might have slower bandwidth or paid bandwidth that would cause
some troubles. We are also not doing much QA so not doing much manual QA, we have a testing setup so we have a test repository that people can look at and since some weeks, some months we have OpenQA, I will talk about it a bit later.
So we cannot do much QA, also our build power is not really limited but we also have to take care about the build power and especially Hulu, Dominik and so on, we really want to focus on Tumbleweed and factory, bringing new stuff to the people.
So we do version updates for the bigger things like Firefox, Opera, Chromium, we also did version updates for LibreOffice, LibreOffice is a bit bigger, a bit more problematic but we did version updates for that, minor version updates for the usual suspect that
you might imagine, PHP 5 bind, the kernel, for Java and KDE, for Java we ship what the OpenJDK Foundation gives to us, so basically the minor version updates, our KDE team always supplies us the minor version updates for the KDE frameworks, no longer KDE 4 but now
KDE 5, the framework minor version updates and we also ship a lot of bug fix updates. So how does the technical project setup look like, for 13.2 as an example, 13.2 GA, that
is the product development code stream, it's frozen, as we get to it, then we have the update repository, the update repository is actually not building, it gets copies of binaries and sources and out of that project is the update repository, the actually the Yum repository generated and all the staging, all the preparation happens in so-called
maintenance incidents, named openSUSE colon maintenance colon running number, for every update we have one such incident and you will probably have seen them already and
the user submissions, they usually look like home, username, branches, OBS maintained and package name, so it's a two-stage process, the user submits to us, we coordinate it, we stage it in this maintenance incident and in the end we release it to the update project. How does the packager prepare the update, so he branches the package he
wants to fix, he or she, either with the maintenance branch command, M branch or with the option minus M, which basically does the same thing, from openSUSE 13.2 or 13.2 colon update and it will result in a package named hello dot suffix openSUSE 13.2 update, basic
package, you can edit it, you can apply your changes and in the end when you are done, if you have multiple packages, you can use or see maintenance request to submit,
alternatively you can also submit the normal package with a submit request, that will convert it into a maintenance request in the end. After we accept the incident, it will be looking like this, so if you submitted both hellos for 42.1 and 32, we have both
sources in the same incident and now it becomes also clear why there is this funny suffix to it, because we can have the versions of hello for 32 and 42.1 in the same project, but we somehow need to differentiate between the different sources. Also, the
project is set up via its configuration that the 32 package builds only against the 42 updates and the 42.1 against the 42.1 updates. There is a magic file, a semi-magic file called patch info, so it looks, it is a bit of XML, you can write it already
with the oc patch info command or we will write it if it is not present, basically says who the packager is, what the category of the update is, which is either recommended
or security, and there is also the optional category. The rating, interesting mostly for security updates, which is critical, important, moderate or low. The summary line, the summary line is the one that is printed in the online update in the overview and also in the super info somewhere. And the long description, so the summary is a one
line thing and the description can be a multi paragraph textual information. And specifically we list in a special issue list the bugs that are fixed and also the CVEs that are fixed. These are represented in also special metadata in the YUM repository and are also
shown by the, for instance, just online update or by the super patch information. If you submit something to us, please follow some of our policies, our rules, try small
self-contained fixes because if you do larger changes there is always more risk of breakage than if you just do a small fix, just fixing the bug. The changes entry should be linear to the previous update, so just add changes entries on top. The bug number should be mentioned. Here I still use BNC for bugzilla.com, but more commonly
for open source is BOO bugzilla.opensource.org or you can use bugzilla.source.com BSC. So all three prefixes for bugs are working. And also please also mention similar as the
submit. And if you touch a spec file that is not usually maintained by you, try to follow the rules that these guys follow in the patch file. For instance, the GNOME team have a strict patch tagging style where you have a command, a comment where the patch
is from, who did the patch, and so follow the style that the other guys already did in the same spec file. The patch info editor that is used by us that can, the patch info edit the patch info that I showed before, that is in the web interface but usually
used by us, probably not by you. So once you have the package submitted, the maintenance instance and we will review it and accept it or we might also decline it if something is broken. There is also a review bot. So if you are submitting a package where you
are not a maintainer, the maintainer will get a review request. So if you are seeing suddenly a review request for maintenance submission, that means that that specific developer that submitted the fix is not listed as a maintainer. And you can at this
time review the submit request, either decline or approve the review or contact the submitter and work with him if something is not really working out. So this is to avoid bypassing the regular package maintainers with such fixes. So once we accept the incident, there will be
this maintenance incident, number is currently around 5,000 something. The maintenance coordination team is responsible for it, so the packager has no write access there anymore. We will take care that it builds. If it doesn't build, we will get back to the packager or we
will try to fix it ourselves or for secondary architectures, just disable the architecture. If it's too troublesome, we write the patch info, we check that all the rules are applied and also check for dependencies. In the end, once it has all built, in the
download.obesuso.org, the update test repo will get automatically the content of this update. Okay, that's still the overview that I showed in the text form before, but it's not so interesting.
So once we are finished with setting up the incident, we issue a so-called release request that will lock this incident, so the incident will not change anymore, so that the exact binaries and the exact sources as we had at that log request will be pushed into the update channel. Here's the update test repository that you can add to your systems
if you want, if you want to have testing or if you just only want to test the specific incident, also the repositories of every incident are published on download.obesuso.org, so if you want to test it. The release request then has three review steps,
or two review steps actually, so the source review, like with the factory submissions, every source change must be reviewed by a second set of eyes, so basically the review team will
change and approve or reject it. Optionally we have an open source QA team that tested so far the kernels and the update stack, where the goal is basically that the system should boot after the update, so if you have updates that are critical to booting, like the kernel,
like the update stack, then we will request specific manual QA, but usually we will not request manual QA, and for LEAP 42.1 you will also see the open QA maintenance review board, so that will, so once this review is added, it will trigger an open QA job,
once the QA job has finished, it will approve this review, and after all the reviews are done, and usually after a wait time of like seven days to capture reports from people using the test repositories, we release the updates, so if you are wondering why your update is going out slow,
we usually wait for one week, if it's more urgent or a critical security fix, we will do that quicker, so once it's released, we land in this human repository for 13.2,
in the update repository you will see the source link of package hello going to hello dot this number from the incident, and also the patch info dot number of the incident, which contains the binaries and the update info XML, which will, which has all the meta information for the update stack for super or just online update to show.
We can remove the update easily by just removing this patch info, so if you know that something broke in your update or if you notice something, tell us, we can then decide if we want to remove it, it's just as simple as removing this virtual package. What changed with LEAP, so
what changed in last year, so LEAP is a synergy or a mix between SLES and OpenSUSE, there are a lot of talks about it, I will not go into detail, but there's one magic file that is used, there is a file called zero zero meta, a package called zero zero meta, and there's
just one file inside, a lookup dot yml, which is one line based format, and it contains like this, for instance, network manager, colon, and the project it was taken from, it was for 42.1, it was, network manager was taken from factory,
or glibc was taken for 42.1 from the SLES 12, SP1, GA, or at this time the update repository, and third option is, for instance, docker, docker was specifically made for 42.1, it was not taken from factory, but there was a specific docker version.
We can change this adjustment, but we usually do not want to do massive changes, if you can migrate to a SLES package that would ease the maintenance effort by the OpenSUSE team,
that would be good, yeah, but it's possible, but usually we try not to. So how do updates get into leap for the 42.1? The SLES packages, once SLES releases the update, we have a script that runs every hour that auto imports an update from the external OBS-facing
SUSE C12 or C12 SP1 update project, creates an incident, sets up the incident, copies over the patch info from SLES, and starts building, so we only have to review at that time that
it builds and that the patch info is correctly bordered, otherwise it is fully automatically copied over. That also makes it a bit difficult for OpenSUSE contributors to make changes to SLES-specific packages, so in those cases where you want to touch a SLES-specific
package, it's best to open a bug and assign it to the maintainer, because if you accept update for such a SLES package, it would be overwritten by the next SLES update of the specific package. So that's one process step that is a bit difficult that we still need to
work a bit on. The factory and fork-level packages are handled like OpenSUSE 13.2 and all older versions, you basically submit the maintenance request and we release it as a commercial update. I have some statistics run begin of this week. So far we shipped 307
SLES-imported updates and 386 non-SLES-imported updates, so there is quite a balance. If you compare the packages on OpenSUSE leap, there is more of a one-to-five relationship, so
1,000 from SLES and the rest, I'm not sure, 1,000, I don't know, but there's a different balance, different relation than what we released as updates. Of course, we have 33
Clamav-V database updates that, because we, in SLES we do weekly Clamav-V database updates for built virus scanning, so these could be removed and it would still leave 270 versus 386. And we see XAML, PHP5, GRUB2, the OpenSUSE maintainers did not need to take
care about it, or NTP or systemd, OpenSSL, so quite a number of packages that are high maintenance have been taken care of by the SLES maintainers. For the non-SLES updates,
the highest counts are the web browsers, Chromium we do not ship on SLES, so we cannot take it from there, and Firefox we have a specific, we are not following, so SLES is following the ESR release lines, but OpenSUSE is following the regular release lines, so there we cannot easily align it, and I don't think the OpenSUSE users would appreciate using the ESR release,
because it's a bit more outdated. I talked about OpenQA, we use OpenQA, so Ludwig has set this up, and it's a bit better testing still, so we're still watching it, it's not, it's flaky
occasionally. So what it does, it does install the updates in a very specific manner, it sees problems during installation, it does various desktop testings, it is taken from the factory OpenQA, so there's a list used from the factory OpenQA testing, and it is hooked in
via request review bot mechanisms. We have actually three categories where we run it, so we run OpenQA on the released updates, OpenQA has also run on all the updates that are currently in testing, all of them at once, and also we run it for specific incidents. The specific incident
is hooked up to the review bot, and if you look at OpenQA.OpenSUSE.org, this is one of the incidents that I looked at on Monday, and it looks a bit broken,
so it doesn't boot on the prepared images, I'm not sure why, but usually there would be more green, so there's green down there, there's a bit yellow down here, but once everything is working, it would auto approve the update.
So I think our OpenQA developers are working on it, and so that's still a work in progress, but it's also very useful because we tested the kernel with it, usually the kernel waits for a free QA slot of our maintenance QA team, but the kernel QA is
the last 42.1 update kernel with the latest security fix, I just looked at the OpenQA past, and so it didn't need manual waiting for someone manually, so OpenQA already helped there a lot.
So basically that's it from my side, we have a portal page, portal maintenance, that shows how to submit, how to work with maintenance, and you can always mail us at this email address, or if you have a question in Baxila to us,
said you need info to maintenance at OpenSUSE.org, then the maintenance team will help you. Yeah, so that's it for a quick maintenance introduction. Are there any questions? Okay, then thank you.