Collaboration in the openSUSE Build Service
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 70 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39533 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 200915 / 70
3
4
6
16
17
18
19
20
21
22
23
24
25
26
27
29
30
31
32
33
35
40
41
44
46
47
48
50
51
54
55
57
58
59
60
62
65
67
68
69
70
00:00
BuildingService (economics)CollaborationismMereologyPresentation of a groupService (economics)BuildingContent (media)Projective planeLine (geometry)CASE <Informatik>Level (video gaming)Computer fileCollaborationismWeb browserChaos (cosmogony)Repository (publishing)Patch (Unix)Right angleSoftware maintenanceOpen sourceSoftware developerMultiplication signPhysical systemBitResultantBeat (acoustics)HoaxBootingMathematicsProcess (computing)Client (computing)Computer fontWeb 2.0CodeHookingConfiguration spaceOpen setOscillationLecture/Conference
06:34
Arithmetic meanProjective planeBranch (computer science)Open setOscillationComputer fileBuildingPatch (Unix)PasswordService (economics)WordRevision controlData miningOffice suiteBitMathematicsPhysical systemForm (programming)Source codeXML
07:50
Pulse (signal processing)Projective planeRevision controlNetwork topologyConfiguration spaceProcess (computing)Pascal's triangleEmailWeb 2.0XML
09:01
Process (computing)Branch (computer science)
09:32
Design by contractProjective planeProcess (computing)Software maintenanceFactory (trading post)CASE <Informatik>Rule of inferencePatch (Unix)Moment (mathematics)Endliche ModelltheorieBitLine (geometry)CollaborationismDisk read-and-write headStreaming mediaTuring testPoint (geometry)BootingBranch (computer science)Monster groupInstance (computer science)Multiplication signOffice suiteRepository (publishing)Universe (mathematics)Product (business)3 (number)Software developerOpen setDistanceDistribution (mathematics)Ferry CorstenForestRevision controlService (economics)BuildingDrop (liquid)Beta functionSet (mathematics)Lecture/Conference
16:30
State of matterFactory (trading post)Moment (mathematics)ChainProjective planeMathematicsPatch (Unix)Physical systemInstance (computer science)BootingQuicksortMenu (computing)DivisorRevision controlBranch (computer science)WordWeightScheduling (computing)Multiplication signComputer configurationOcean currentService (economics)Software developerBitDifferent (Kate Ryan album)Software maintenanceReal numberSlide ruleVideo gameRight angleScaling (geometry)Open sourceCountingBuildingCollaborationismDesign by contractLink (knot theory)TheoryKernel (computing)Mechanism designLecture/Conference
23:28
Factory (trading post)BootingProjective planeMultiplication signSemiconductor memoryClosed setDesign by contractCASE <Informatik>Patch (Unix)International Date LineMoment (mathematics)Open sourceSoftware maintenanceRight angleDifferent (Kate Ryan album)Repository (publishing)State of matterSet (mathematics)Web pageStandard deviationRule of inferenceRevision controlInstance (computer science)Branch (computer science)1 (number)HypermediaParameter (computer programming)Physical systemSparse matrixMereologyPlanningDatabaseStiff equationInformation securityIntegrated development environmentForm (programming)System callLink (knot theory)Software developerStapeldateiView (database)Formal grammarArithmetic meanProcess (computing)Kernel (computing)RootMultimediaGoodness of fitService (economics)Compilation albumStability theoryDecision theoryElectronic mailing listEmailDirection (geometry)Lecture/Conference
Transcript: English(auto-generated)
00:00
So, I see that everybody was here already for the overview talk, which is quite nice because then I can skip the first part of my presentation, which will be about how
00:20
to collaborate in the OpenSoup's build service, and the general content will be What do I mean with collaborate? How does the build service allow me to do that? And in the end show you some examples how people are doing it right now.
00:44
So, first the what part. As Adrian nicely explained, the build service is just, if you never heard of it, it's something and I'm pretty sure for somebody that never heard of it, it's total chaos.
01:00
You get projects and packages and builds and repositories and there are maintainers and command line clients, blah blah blah. So, I just want to make clear what the build service really is. The build service is a maintainer creates a project which contains a package or more packages.
01:27
The package contains sources and spec files which then gets built and the result is a repository containing binary packages. So, that's the part where the end users are, that's the part where the developers are.
01:46
So, with that cleared up, let's have a look at how the build service supports me in collaboration with others. So, there are two hooks you have to collaborate in the build service, two easy hooks.
02:04
The one is based on the project level which, as we can recall, is just some place that contains one or more packages. The other one is at the package level, that means I can do contribution purely on the package level.
02:27
So, if I have more than one package in my repository, I can collaborate with Adrian on one but not the other. So, first is how do I collaborate on a project level. It's fairly easy.
02:47
You can either, in the web client, you can either click add user and I can, for my project, add Adrian as another maintainer and be done with it.
03:03
So, Adrian has all the same rights that I have. You can add packages, delete packages, submit other code or submit patches, everything. So, I can do it in the web client with a simple click or I can do it in a command line client like I've shown here,
03:23
which is just not a good font, so it wandered down, but it's just adding one line to some XML file. So, I have my project, I'm the maintainer of that project and I added just another user as maintainer, in this case, Derex.
03:46
So, this is collaboration on the project level. So, now, me, Henel and Derex can do everything in that project. Like I told you before, I can have another easy hook to collaborate, which is the package.
04:07
And, surprisingly enough, it is totally the same. We have just another XML file and I can just add another line and collaborate on the package. The same way on the web client, I can just add user as maintainer.
04:28
So, in this case, I have rights to do something to that package like adding another patch, updating it, whatever, changing configure lines, whatever.
04:44
And I just granted Derex the same rights. So, this is the second easy hook to do collaboration. And then, there is the usual practice we have in the open source world that people are just saying,
05:04
oh, no problem, just submit me a patch. And this is the usual case because nobody wants others to mess with his own system. So, if I don't want people to add packages to my project, remove packages from my project,
05:27
if I don't want to have other people like me to have the ability to change one single package, I can always say, okay, come on, submit me a patch. So, there is tooling for that in the open SUSE build service. I mean, it's always easy to just do that.
05:48
I mean, come up to someone and say, dude, I've updated your package to 1.1, I made you a patch. And that's easy, but we have tooling around that process.
06:00
So, the same as this, you can do with an osc command, which means, obviously, if I updated Pascal's package, then I had a look at the package.
06:21
I grabbed it from somewhere, I gone over it, did my changes, now I have them compiled in the patch, and I say to Pascal, oh, come on, I submitted to you. So, the same is just here as tooling in the open SUSE build service,
06:42
means I just get Pascal's package from somewhere, which is osc branch. So, I branch it from Pascal's project. This will create a copy of that package in a project of mine,
07:03
which is under my home in the open SUSE build service, which I then can check out. So, I have the actual files on my system, can do my changes, and can do the telling Pascal that I have updated his foo to 1.1,
07:27
and telling him that I mailed him a patch, I can do that with a command too, I can just do osc submit request create, and then create a patch from my project,
07:41
which I updated to his version, which is not updated, and Pascal gets a mail, and we'll see that there is a submit request for his package foo. So, from Pascal's side, this looks like, okay, there is a submit request,
08:02
if there's more than one, he can list them on his project, and he can look at what I did with the tool, which is submit web show. So, either Pascal now says, oh, I like that, then he commits it to his tree,
08:25
or he can say no, it's nice that you've done that, and I can take like 90% of it, but there is stuff that I won't take, so he can either accept these, or not accept these,
08:43
and if he does not accept these, I get a mail again, and he can tell me in that mail, okay, you updated to version 1.1, but you changed this one configure switch, which I don't like, I want to have it otherwise, and stuff like that. So, the whole process of two people talking to each other,
09:05
is like automated in the tool, which makes it easy to scale it up. So, I mean, if me and Pascal talk, it's probably better if we do it on some IRC channel, than to me branching his package, and blah, blah, blah, blah.
09:22
But, if you have more than two users involved, it really gets easy to get mixed up in that. So, tooling around this process is really nice. Okay, so, what I want to show you is now some examples
10:08
of how people do stuff and collaborate in the build service today. As you can imagine, this tooling I just showed you,
10:22
you can use it in all kinds of ways. For instance, we have a new repository in the OpenSUSE project, which is called Contrip, which is kind of a universal third-party package repository,
10:41
where people that are not directly contributing to the OpenSUSE factory have some place to put packages in. So, with the tooling we have for OpenSUSE factory Contrip, which is the development branch, we do it like this.
11:03
For new packages, if you want to submit your package that didn't exist before in factory Contrip, you have to do a submit request. Which means you have to do this, oh, I have this new package, I sent you a patch that adds it to your repository.
11:21
Then, there are people more than, I mean, there's a whole team behind that, that just reviews the submit requests that come in for new packages. Once you are granted access to that package, to the repository, you are the package maintainer, you are not the repository maintainer.
11:44
That means the only thing you can change is your own package. You cannot add another one, you cannot remove other packages, you cannot do anything to the repository, you can only do stuff to your own package.
12:02
And this means, process-wise, if you update a package, you just work on it and check in and stuff like that. If you don't want to maintain it anymore in that repository, if you say, okay, maybe I maintain KDE 2 in it and now KDE 4 is there,
12:26
so I want to get rid of it because everybody has moved on to KDE 4 anyway, I can just drop it, for instance. So I can do whatever I want to my package, but not to the repository.
12:43
Then there's kind of an extra repository for every released version of OpenSUSE. So for instance, with 11.2, we will have 11.2 contrib, where the tooling and the process is completely different.
13:03
So new packages are simply not allowed. The project maintainers means that can do stuff to the project, like deleting packages and stuff like that, or having it built for some other distribution or whatever, are still the contrib team.
13:21
In this case, for the individual packages, the maintainers are also the contrib team, which means if you had your package in factory contrib, you could do everything. If it wandered to 11.2 contrib, you can't do anything.
13:41
For everything you do to that package, you have to submit a patch again. So you can see in that example that we are trying to follow the general factory rule we have at the moment. This is unreadable, so I can explain a little bit.
14:04
This is actually the usual OpenSUSE development model. On the upper line you have the factory stream, which is just like in your SVN, your head branch or whatever, always the newest stuff.
14:22
Then somewhere, at some point in time, we start the development of 11.2, then do something, release alphas, betas and then goldmasters, and then somewhere in time the development for 11.2 stops.
14:41
We branch away 11.2 and have the same here. So this is the general OpenSUSE development model, which is kind of reflected here. You can do everything in factory, you can't do shit in release projects.
15:01
So this is how we today in OpenSUSE contrib use the tooling around the OpenSUSE build service to just have a repository that follows the general OpenSUSE development model.
15:23
So how do we do it today for factory, which is the OpenSUSE development distribution? At the moment we have for each package in the distribution,
15:42
we have someone directly responsible as maintainer, which means it's kind of easy to set tooling up around that, because if you want to submit something to the Vim package of OpenSUSE factory,
16:03
you have to go to the Vim maintainer and tell him, I have changed this and that and that and can send you a patch. So the idea at the moment is that you just do a submit request to the OpenSUSE factory Vim package
16:23
and the Vim maintainer takes care of all the OpenSUSE factory Vim submit requests. This is how we are doing it now, which kind of scales badly, because for every package you need someone directly responsible,
16:44
which means if we have like today somewhat 3000 packages, you have like either you add more people that care for factory, if you add more packages,
17:01
or the package count for individual developers goes up. And what we see at the moment in the OpenSUSE build services that factory cannot really keep up anymore, because you need to find other methods than having someone directly responsible for a package.
17:22
And this is kind of the state we are at now and where we are trying to come up with solutions. And one of the things we thought of in the moment is to have some kind of review team,
17:44
like the OpenSUSE contract team that is not directly responsible for one package, but where people review just simple patches and stuff like that. And that's the way we currently think we can do it.
18:05
But there we are in discussions and that is because for factory there are a lot of restrictions. For instance, factory gets used to build a lot of other packages.
18:27
So if you submit something to factory, it gets immediately used for building other stuff and a lot of stuff, like the whole build service. I mean every project in the build service,
18:42
at least every OpenSUSE project has an OpenSUSE factory branch where they build against factory. So that's one restriction we have that there has to be some review, otherwise you can't do it.
19:01
Then there has to be some scheduling, because if all of you have one after the other great ideas for the glibc package and you submit and submit and submit and submit, nothing will ever get built
19:21
because if glibc changes, like the whole system changes and gets rebuilt. So if you check something into glibc and then you and then you and then you, nothing gets ever built. That's another thing we need to do with factory scheduling.
19:42
And as you can see there are a lot of restrictions but I think the tooling around the OpenSUSE build service is actually equipped to handle that. So that's the current state of collaboration in the OpenSUSE build service.
20:01
If you need me to show you stuff, we can go down to the booth and I can show you how to set up your project to handle the different roles of maintainers and stuff like that. Or I can show you how a submit request really looks like
20:23
and not on a slide but in real life and stuff like that. And if you have other questions you can ask them too. So are there any questions now? You would like me to answer?
20:45
No? No? Yeah?
21:00
You talk about branching and I have two questions related to that. When you branch a package, if the original package is changed, do you get a notification of that? No. Not really.
21:20
You could get a notification but actually it's not needed because the branching works with the source link mechanism. That means that you and your branch project has a reference to the original one still and when you change something the build service is storing your change. So when the other guy, what you link to, is changing also
21:43
the system automatically tries to merge both changes and it's still working. Of course there can be the situation that stuff is conflicting and there we still have a problem that the merging is not nice enough. This is something that we need to work on at the moment.
22:01
But in theory that's also a good thing when there are, for example, five submit requests to one package and now you accept the first one and then you see, okay, the other four three of them are still applying but one is not anymore
22:21
so you can decline it, please adapt it again and the other three you can see, okay, from then I take that one and then you can see if the other both are still working. So as long as the changes are not conflicting it's really easy to see if they have an influence to each other
22:40
and to merge them together. Just the manual merge conflict solving tool is not there where it should be. That's something what we need to improve. My question was more on the, like, if I branch your package
23:00
and I want to maintain that branch not that you want to merge my patch on yours like, I have my patch and every time you do a new version I will get my branch rebuilt. That's true. That works with the sourcing mechanism. If you never make a submit request, okay, then the other guy will never...
23:25
So that's the same with the sourcing mechanism. I mean, that's the example with the kernel. You have the kernel and you want the multimedia patch but the kernel developers don't want it because it reduces the speed of the Oracle database. Then you can perfectly easily set up your branch project
23:41
with the source link and this is what happens in the end with the oc-branch command at your patch and as long as your patch applies you will all the time get the latest package plus your patch compiled. So if the other guy is changing something, some source your package gets compiled also again including your patch.
24:00
So to make that quick, this oc-branch command does nothing else than creating another project underneath your home project and linking the sources. Can you close a branch? Sorry? Can you close a branch? Like, you open a branch and you maintain that branch
24:23
and when it's merged to the original... You just delete the project. You delete the project. And then you talk about the contrib team. Who is the contrib team? Is it the SUSE employees, you mean? No.
24:40
I mean, there are SUSE employees. I, for instance, I'm in the contrib team but there are others as well that are not employed by SUSE or anything. So... But what's the difference, I mean, how someone gets in the contrib team? You said you can... We initially started just doing a call and who wants to?
25:05
Because it means work. I mean, you have to review patches from all over the place and you have to review full packages and in the end if we have a released contrib like for 11.2
25:21
you will have to review security patches and stuff like that. So it means work. So initially we just asked who wants to do it and those guys who said yes are now the contrib team. So there was no formal decision making process or anything.
25:41
We just asked and we're done with it. You're welcome. Yes. If you yourself want to be maintainer of contrib you can just come to the mailing list and say OK, I want to help out. But in the end we will probably clean up this team every once in a while
26:04
and throw out people that are not contributing anymore. And why do you have to have... I mean, when people in the factory contrib has to be not the same on the contrib team. Sorry? I mean, you have people that would be in the factory contrib and people on the contrib.
26:23
Why it's not just the same people? No, we have... You have two repositories and you have one contrib team that is caring for in factory contrib only for new package submits.
26:40
So if a package that wasn't there before gets submitted to contrib this team has to take care of reviewing their package if it's OK. You are the package maintainer of the factory contrib. For example, I do a package and I ask for this package to be in the factory contrib
27:03
and I'm the package maintainer. Then when this gets to contrib you mean when this gets to the release state? I mean, I could still be the package maintainer. Why it has to be a different person? I can explain too.
27:24
You are not the package maintainer anymore because otherwise you could change whatever you want in the release state of the repository. That means for the end users that use your packages they have a moving target all the time. So we want to have quality packages in open-user contrib
27:45
so we follow the same rules as we have with factory which means, for instance, no new packages in released repositories. No version updates or only sparse version updates for security problems.
28:03
Or we say, OK, we are not changing the way the whole thing is built too often so we don't get rebuilt too often and stuff like that. So that's why package maintainers, individual package maintainers
28:23
cannot maintain their own stuff in the release state of the contrib repository anymore. That's the stuff the contrib team will care for. Which does not mean you are not responsible anymore or you are not able to help anymore.
28:41
It's just you cannot check in every time you want to. Maintaining it in factory contrib is fun. Maintaining it in 11.2 contrib is work. Yeah, but if you want to have a package that works with factory and you don't want to maintain that package
29:01
you just have your home projects. But if you want a package to be in the contrib I think you should be the one to maintain that package. I think it might be that sometimes the...
29:20
There are probably some cases where the original maintainer is also in the contrib team and he is then of course also maintaining his own package. In the moment you join the contrib team you are no longer allowed to maintain your own package.
29:41
That's not the problem. But if you are maintaining it for the contrib team you have a different set of standards to obey to. No version updates. No, oh that's a funny feature, I enabled it now. No, you can do that in factory but don't do that in the stable contrib team. It's basically the same like we work at SUSE when we package.
30:02
Sometimes I would like to do a version update for a released version but it's very hard, you need very good arguments to be able to do that. Adrian also wants to... What you need to keep in mind is that we want to have really lots of people using contrib and we want to have as much trust into this repository as possible.
30:23
What's important here is with the build service always the stuff gets built automatically and people who have added this repository get directly affected so it's way more risky than with just an SVN where people can review it or usually any standard user
30:43
but a package which gets installed gets installed as root and so there is some risk that it damages the system and especially for a stable system like an OpenOS 11.2 there must be some review in our opinion to keep the trust.
31:06
And as you maybe can hear from the comments from Stefan and Adrian it's kind of my evil hidden plan to form factory contrib after factory so in the end we can just merge them back together
31:24
and have contributions to factory directly. But first we need to resolve this review step somehow because factory contrib will have a lot of users
31:41
but factory has really a lot of users and you don't want to change stuff in there that breaks people's system or that opens root holes or whatever. So there has to be a review and that's my personal evil plan
32:01
how I designed factory contrib to adhere to the same standards as factory because in the end we will just merge them back together but that's stuff for the future and this is what we have now. So any more questions? Okay.
32:27
Then I'm finished early. Good. Thanks.