Rust packaging: Cross-distro collaboration done right
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 40 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/54423 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
openSUSE Conference 201914 / 40
7
11
12
13
15
16
17
18
19
20
23
25
26
27
34
38
40
00:00
E-learningCollaborationismCollaborationismComputer animation
00:24
StatisticsMacro (computer science)Identical particlesSimilarity (geometry)FeedbackSource codeDistribution (mathematics)Repository (publishing)File systemOpen setFlagProof theorySuite (music)Computer fileMereologyDistribution (mathematics)Core dumpRevision controlDescriptive statisticsDefault (computer science)Open sourceEmailResultantProjective planeAuthorizationNetwork topologyLibrary (computing)Macro (computer science)Different (Kate Ryan album)Ocean currentElectric generatorMultiplication signDistribution (mathematics)Musical ensemblePatch (Unix)Associative propertyFreewareInterior (topology)Software developerGUI widgetWeb pageComputer animation
06:48
Workstation <Musikinstrument>File systemSource codeDistribution (mathematics)Repository (publishing)Macro (computer science)Similarity (geometry)FeedbackCodeGoodness of fitCartesian coordinate systemCompilerMathematicsMereologyNP-hardSource codeTemplate (C++)Repository (publishing)Computer fileSimilarity (geometry)Distribution (mathematics)Distribution (mathematics)Different (Kate Ryan album)EmailElectronic mailing listProof theorySoftware testingCollaborationismOpen setShared memoryPhysical systemMacro (computer science)FeedbackComputer architectureCondition numberGUI widgetFiber bundleInstallable File SystemSoftware repositoryLibrary (computing)Open sourceComputer animation
13:11
Open setNetwork topologyFirefox <Programm>Template (C++)Web pageComputer fileDistribution (mathematics)Open setEmailTemplate (C++)Different (Kate Ryan album)Revision controlComputer fileSoftware repositoryMathematicsGroup actionMereologyConfiguration spaceComputer animation
14:18
EmailComputer fileNetwork topologyFirefox <Programm>Telephone number mappingTape driveBranch (computer science)Default (computer science)CollaborationismDefault (computer science)Distribution (mathematics)Distribution (mathematics)Different (Kate Ryan album)Computer fileConfiguration spaceSheaf (mathematics)Open setElectric generatorComputer animation
15:20
Network topologyFirefox <Programm>Online helpMultiplication signPatch (Unix)Gastropod shellDistribution (mathematics)ArmComputer architecture1 (number)Computer animation
17:08
Network topologyFirefox <Programm>Open sourceRevision controlPattern languageFreewareMereologyRepository (publishing)CollaborationismStreaming mediaFormal languageDistribution (mathematics)Distribution (mathematics)Computer fileBitPhysical systemProcedural programmingPoint (geometry)EmailOnline helpMacro (computer science)Core dumpTerm (mathematics)BuildingElectronic mailing listAutomationComputer animation
21:49
Videoconferencing
Transcript: English(auto-generated)
00:06
Hello, my name is Igor. So today we're going to talk about Not very popular thing like cross-distro collaboration because everyone does it his own thing and nobody he doesn't care about
00:22
other people So I'm federal contributor for six years though. My open source account is probably older than that. I Couldn't find a way how to check that I'm Member of federal packaging committee the people who are defining the packaging guidelines for Federer. I'm author of rust RPM the tool which is
00:44
generating the spec files for the rust packages I'm contributor for open-source here basically mostly for rust stuff For a few months and I'm also a contributor to many upstream projects randomly pushing pitches everywhere
01:01
So for today we have Look at Current status between distribution differences in RPM packaging Then basically is some small Why why we actually care about that? Why do we want to have same packaging between different distributions?
01:24
then I will tell you how it happened with the rust and Then basically just small recap what to do and how to do so current status is that for C and C++ the Packaging is more or less same like you have same macro make build make install
01:44
Which is more or less works for C make it's actually quite different for open SUSE and Federer because With open SUSE it builds out of the tree by default and for Federer it does not and there are many things which are different there The package names are also different in many cases in federal
02:03
We just name packages like glib2 while in open SUSE It's split to the sub packages like lib glib2 0 by so name, but actually I found out that it's not everywhere It's just half and half probably For Python go and Ruby
02:22
It's kind of similar the Package package names for Ruby. It's Ruby gem and the gem name for Python. It's Python 2 Python 3 package name For Golang, it's golang github whatsoever But actually the macros inside and the how spec file looks is really different
02:42
Like really different open SUSE does a lot of magic to automate the parts federal does not So it's it's just horrible because you cannot simply take open SUSE spec and build it for federal You just need to rewrite it entirely So and rust is actually identical you can take federal spec file for the rust package
03:03
Build it on open SUSE and install it on my J and it will work without any problems So Why we actually care about all this? Well, first thing is we don't want people to duplicate work because People can spend time on writing some automation tools improving generators for rust or PM for gem to RPM whatsoever
03:25
They can work on infrastructure instead of doing the monkey work by writing spec files between distributions. That doesn't make any sense and Obviously users are confused because when they switch to a different distribution They see oh it used to be Leap G leap and now there is no such package. What do I do now? How do I install so basically for upstream developers? It's
03:47
They need to have ten different ways of describing which dependencies do you need to build your package build a tool? So how did it went with the rust first thing we did is I created proof of concept which works for federal
04:01
So it's generating spec file and it builds for federal. It was not that easy, but we did it Yeah, then differently. We just went and started using and didn't tell anybody like everyone else does but not really So then we actually Went to my GM and open suits and asks people. Okay, here is we have some proof of concept. What do you think about it?
04:27
like tell tell me your preference what you don't like what we need to improve and stuff like that and Apparently we got some Debian guys coming to us back and also telling oh this looks cool
04:41
We definitely cannot use it because we have their packages, but we can make some agreements on the where we install files how we package things and basically stuff like that and Also, we collaborated on the some upstream pitches for example for cargo So we're working together. We got some flag
05:02
Minus minus target which is specifying for which distribution you want to generate spec files It is when I say you can take spec file from feather and build it on open SUSE It will work but open SUSE has its own guidelines for example the license header in the spec file Which we don't have in federal because all the spec files are MIT license by default unless you specify otherwise
05:24
Well in open source, it's from what I know. It's requirement to have the license header on the spec files So What were the challenges they were obviously none? Probably no The main challenge was that we needed to change RPM. So for the RPM we needed to change
05:49
We needed to have new dependency description so we needed to have Way to specify that I need create full with a version more equals 1 0 0 but less than 1 to 0
06:03
for example and that's actually quite complicated in the RPM ecosystem because RPM is the core part of the whole distribution and you cannot just implement something and then hey guys start using it because Yeah, it will take take a while. So I think we had first proof of concept done in beginning of
06:22
2018 But we could not start using it everywhere in open SUSE until probably 2019 almost So what are the results of the of the work? Basically we established IRC channel Federal rust where we have people from open SUSE. We have people from Debian. We have people from Federer from magia
06:47
Probably some other distributions, which I don't know about We got as I said, we got compatibility between Distributions that you can take spec file from there build it there and install it somewhere else. It will work
07:01
So we got only one repository which contains all the macro all the tooling and everything so people just use it Basically file system path user share cargo whatever names are there is the same between distributions Yeah, and spec files are the same as well and if we compare RPM and deb distributions we have
07:26
Basically agreed on the file path and even the package names are somewhat similar because they don't use devil But they use dev so for example in Federer and open SUSE we have rust certain derived devil While in Debian they have lib rust derived dev so looks very similar. So people are not that confused
07:49
So basically some short instruction how people should work on this is basically Create proof of concept which works for you Then basically no before creating that
08:03
People should actually look what's available there probably already start somebody started working on something what you want to create so if it's there just join and Work together if not just create proof of concept and then ask people for feedback find out who is interested and
08:21
Make sure that basically other distributions are aware of what you are doing and they can give their comments Obviously you need some place to discuss everything not just in five different mailing lists you send to open SUSE devil RPM ecosystem Federer devil and everywhere just get some IRC channel or one mailing list and discuss everything there and
08:46
Yeah, don't forget that once you've got some people you cannot just go and break everything because you just wanted to have something Nicer in your distribution, but then everyone else will say oh this doesn't work for us So we are just going to fork it and be done with it
09:05
and Basically what I did I became a more contributor to the other distributions to basically try it out and see basically whenever I do any changes in the rust RPM I'm testing it on the open source here to see basically whether
09:20
somehow Basically, how does it affect the distribution and do changes which help both distributions? Yeah, this was actually much faster than I expected So I suppose we can go to the questions or suggestions or anything like that
09:46
Obviously the collaboration is really hard between the sources not easy We know so what was the hardest part for you to get three destroyers together even four so Mostly it was talking finding right people. That was quite hard. It's good that we had a Neil Gompa who actually
10:04
Told me. Hey, this guy is doing something similar in Open SUSE these guys doing it in Maggia and What helped a lot was that federal was I think the first distribution of which packaged compiler rust compiler in the RPM
10:21
So actually other people when trying to package it, it's horrible to package compiler So they actually started looking at federal and they actually go to further us big before we started doing packaging for the rust applications Other hard part was Basically RPM RPM changes, it's it's really hard
10:42
All right, if I get it correctly, you're mostly talking about rust applications Is there anything that you can also share across distributions? Regarding building the compiler and cargo and all the tools So actually the I'm not sure about open sushi
11:02
But differently between federal and Maggia actually the spec file is the absolutely same The it has I think three four conditionals in there like if Maggia then because they have architecture I 586 and That is not supported by upstream. I mean, it's not supported by upstream and it's it doesn't exist in federal
11:22
So there are few ifs but apart from that the spec file, I think is like 99% same Yeah, I mean, yeah, it turns out for example on on open sushi. We would like to have the same spec file For the current or running distribution the tumbleweed and for some older distribution so that you can get a rust compiler on
11:44
Say sleep well, and we don't have to maintain two different packages And that is a as you know, that is that is a challenge because rust compiles rust is compiled by rust Which is in turn compiled by LLVM So you would either have to bundle in the LLVM or you would have to depend on I don't know
12:07
Which LLVM when you start to get the differences between different like, yeah, I think Yeah, federal and Maggia do not bundle LLVM We just use your library But basically if you try to compile it for older distribution
12:20
We just does not have new enough LLVM you probably will need to bundle it and it's becoming just a mess so Yeah, I think in people we do bundle LLVM because LLVM coming from rallies is to old Okay fair enough
12:42
so is there some repository that keeps all those tools and and template spec templates at some place or is just in some packages so it is rust or pm repo I Think you can open it
13:08
Yes, it looks horrible Yeah, whatever. So we have rust or pm repo which has actually some templates between the distributions and You can see that. Oh, no, it's not this folder. It's rust or pm templates
13:31
Yeah, so basically main spec is the same in just between name version release whatsoever Some templating magic. So the only difference part between distributions is
13:43
Yeah license header some basically it's for open SUSE Yeah The group because in federal we don't use some group thingy yeah, and the change log So basically we have one repo which contains all the templates and it's just generator
14:04
Which is generating spec file and we have same you can supply some configuration For example, you can I probably can show it
14:41
Yeah, so basically if you have such configuration file you can use rust or pm for different distributions and it will this actually Will you can override for open SUSE for each distribution? You can have different requires if they're really different but apart from that you have just default section Which is compatible between all of them. So basically when I'm packaging
15:02
Things like this for open SUSE just copy this configuration file there and generate spec file for open SUSE Using same tool on federal so it's great to see the Collaboration across the different distros. What about collaboration with upstream rust?
15:26
Well, we do have some pitches for the upstream which we obviously send them but for now We had three major things. I think to fix one is that Previously when you publish some crate it it still saves workspace path and stuff like this in the cargo tunnel
15:47
Which is creating a pain for the packagers. So this was fixed by the rust upstream Quite quite fast. We got a few more things which needs to be fixed and seems that upstream does not care much about that So it seems
16:02
We just need to go and write pitches for it and they will just accept it They seems do not have some time to to work on it And what architectures are you testing against? So basically for Federer we have six architectures, which is x86-64, i686
16:24
PPC-64 LE, S390X ARMv7HL and AR64 We used to have PPC-64 the big Indian one, but we don't have it anymore We never had S390 in the Federer
16:44
as the primary architecture. Is there any interest of smaller distributions to join this or to take over the work? Which exactly distributions those small ones because actually I did not see those small distributions coming and
17:06
basically asking for any help They probably just exist somewhere. I don't know what they're doing. I never heard about them. Okay And one more question, so if this Rust stuff is done, what's the next language you're doing according to this
17:24
procedure so the next thing is I want to Get Federer closer to the open source and the other way around. So for example what I Really enjoy about open source is the build system. It's amazing, but it has broken version control system at least for me
17:42
It's painful to work with So basically what one thing which I want to get Federer to do is I want people to stop writing Release in there because obvious is generating that automatically So and in Federer you have to always rebuild yourself increment release and do all the manual work
18:00
Which people hate so basically I want to get Federer closer to open source in this way But on the other hand, I want open source to start using for example Git as the version control system. So So we can get a bit closer and
18:20
Probably at some point. I want to get Ruby and Python stuff to be better With help of the RPM folks. I'm looking at you Florian So from what you have seen how to improve cross-distort collaboration
18:42
Obviously it worked for you Rust guys, but mostly it's not where it should be So I think we need Actually RPM RPM upstream wanted to get this was closer So they created the RPM ecosystem mailing list, which is supposed to be read by different distributions Which is not very true at this point
19:03
so I would probably suggest to subscribe to RPM ecosystem at RPM dot org and basically, like whenever we want to do something with the packaging like an entirely new ecosystem, which didn't exist before or Basically somebody wants to rework old stuff basically bring it there discuss it with me between the people and
19:25
Get from there not just do it So we also created an repository on GitHub for those stuff So the idea is in the long term to move more of those trips out of our PM core because we really can't Take care of it properly
19:41
Like things like Python and go and all those languages to have the tooling for those packages move to separate repositories that that are maintained by the community is shared between the Distributions and basically RPM only lending its name and place as a neutral
20:01
Playground for for people to meet and do that, but we have we created a repository But we didn't put too much work into getting people in at something We need to do Far Probably rather sooner than later because there's still a lot of stuff that can be improved and it's currently
20:22
Distributed over all those small parts in some distribution some files and probably not even being uniform within the distributions and So there's a lot still to be done to make Make this easier for everyone Yeah, and probably stop doing all this magic
20:42
Inside the RPM macros and get some pitches to RPM to improve it not just try to work around it But try to get some features in the RPM nowadays. It's much easier than it used to be many years ago So like for example the sub packages for Python to Python free
21:01
Florian is working somehow on eating the RPM upstream So let's just help RPM upstream to get some patterns for the packages Whatever we need to automate packaging. Let's get it to the RPM stream instead of trying to work around it in distributions more questions
21:32
So I think if no more questions such stickers are available somewhere around feel free to grab them
21:41
Supposedly sit. Thanks You