Linux distro should be an upstream contributor too
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 44 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/46122 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
All Systems Go! 201919 / 44
5
10
17
20
21
30
32
36
44
00:00
DisintegrationSystem programmingPhysical systemSoftware engineeringINTEGRALHamming distancePhysical systemProjective planeDistanceComputer animationJSON
00:45
Physical systemCommon Language InfrastructureService (economics)Software maintenanceWordInteractive televisionCompilation albumBitException handlingSoftwareComputer animation
01:55
Source codeSystem programmingPatch (Unix)Computer fileSoftware developerPhysical systemSoftware maintenancePartial derivativeFeedbackMaxima and minimaCommon Language InfrastructureIndependence (probability theory)Client (computing)Component-based software engineeringPatch (Unix)ImplementationInteractive televisionRevision controlFile Transfer ProtocolDemonClient (computing)Common Language InfrastructureProcess (computing)Software developerCASE <Informatik>Service (economics)AuthenticationRepository (publishing)Online helpTask (computing)Partial derivativeLogic gateSource codeComputer fileSoftware maintenanceSoftwareSource codeComputer animation
04:36
System programmingPhysical systemCommon Language InfrastructureRepository (publishing)Client (computing)Service (economics)Repository (publishing)Computer fileMathematicsType theoryLogic gateDiscrete groupProjective planeSource codeComputer animationProgram flowchart
06:07
Common Language InfrastructureoutputCache (computing)Ring (mathematics)Default (computer science)Service (economics)Web pagePhysical systemKey (cryptography)Repository (publishing)Computer fileToken ringProjective planeOrder (biology)Configuration spaceWeb 2.0Source codeKerberos <Kryptologie>Logic gateFloppy diskComputer animation
08:01
SummierbarkeitSource codeSystem programmingOpen setPhysical systemRootCorrelation and dependenceRevision controlRepository (publishing)Exception handlingSource codeComputer fileMathematicsLoginComputer animationSource code
09:09
Source codeEvent horizonRootProcess (computing)System programmingPhysical systemInflection pointRepository (publishing)Software repositoryContent (media)Patch (Unix)InformationDifferent (Kate Ryan album)Branch (computer science)Software developerSource codeInformationFile Transfer ProtocolProjective planeRepository (publishing)Computer fileCondition numberUtility softwareMathematicsCASE <Informatik>Content (media)Software maintenanceGastropod shellMultiplication signConfiguration spaceScripting languagePosition operatorDemonPartial derivativeLogic gateComputer animation
12:45
System programmingDirectory serviceProjective planeCASE <Informatik>Computer fileComputer configuration
13:59
Maxima and minimaSystem programmingInformationComputer fileProjective planeInformationSoftwareComputer animation
14:40
System programmingPoint cloudService (economics)Point cloudRadical (chemistry)Continuous integrationRepository (publishing)Computer animation
15:20
System programmingRepository (publishing)Physical systemBuildingSineBranch (computer science)Service (economics)Projective planeBuildingComputer animation
15:46
System programmingSoftware testingInstallation artService (economics)MehrplatzsystemTask (computing)Projective planeOrder (biology)Parallel portInstance (computer science)Sinc functionQueue (abstract data type)Computer animation
17:13
QuadrilateralRepository (publishing)Menu (computing)Physical systemExplosionSystem programmingRepository (publishing)Software developerOrder (biology)MathematicsBuildingWeb 2.0Computer fileLogic gateHookingComputer animation
19:14
System programmingHydraulic jumpEnterprise architectureSynchronizationNavigationComputer-generated imageryComputer fileRepository (publishing)MathematicsPattern languageComputer animation
19:45
System programmingPhysical systemRepository (publishing)ExplosionService (economics)BuildingEvent horizonCheat <Computerspiel>Service (economics)AuthenticationPhysical systemSoftware developerComputer animation
20:26
Service (economics)Event horizonBuildingSystem programmingElectronic mailing listLine (geometry)Revision controlSoftware maintenanceMathematicsPlanningComputer animation
21:19
System programmingError messageGroup actionBlogSoftware testingPersonal identification numberEmailoutputRepository (publishing)FeedbackFreewarePlanningPhysical systemINTEGRALService (economics)Uniform resource locatorPoint (geometry)Ocean currentEmailImplementationBitConfiguration spaceSoftware developerComputer fileSoftware bugoutputInformationSoftware testingComputer animation
23:46
System programmingSource codePlanningRevision controlProjective planePatch (Unix)Scripting languagePhysical systemKerberos <Kryptologie>Service (economics)Standard deviationRepository (publishing)Group actionComputer programmingRight angleMultiplication signHookingComputer filePosition operatorSoftware maintenanceFile Transfer ProtocolCASE <Informatik>DatabaseINTEGRALMathematicsConfiguration spaceDifferent (Kate Ryan album)Content (media)Computer clusterBuildingSystem call2 (number)Commitment schemeTraffic reportingCircleControl flowWeightLogic gateSoftwareCore dumpWeb pageLecture/Conference
32:59
System programmingWebsiteLattice (order)Computer animation
Transcript: English(auto-generated)
00:05
Hello, my name is Martin Sahnotka and I work as a software engineer for Red Hat and I would like to share with you my experience with Packit and it's a tool for easier integration of your upstream projects into Fedora.
00:26
So, first of all, let me stress that there is a hamming distance of 1 from Packit to Packit, that's the solar sponsor of all systems go and it's something completely different.
00:46
So, today I will try to explain what problems is Packit trying to solve and how you can get started using the CLI tool and also the service.
01:01
So, first of all, let me briefly introduce you into packaging. So, I think you all know that packages are very popular way to consume software and I personally almost never install things from upstream except for some,
01:28
I don't know, nightly compilers or something like this. So, it would be really nice if the interaction between downstream and upstream was a little bit better because downstream maintainers often don't interact with upstream at all.
01:48
So, I have this nice example of a software that I used to work with. It's very popular implementation of FTP daemon, it's called vsftpd and this is our package.
02:04
As you can see, we have like almost 60 patches on top of the upstream version because the interaction is, well, there is no interaction.
02:21
Yeah, that's it. So, for example, when a packager wants to do an update, there are a few tasks that the packager needs to do. For example, download the source tarball, upload it to our repository, modify
02:42
the spec file, refresh all those patches, build RPM, create, update, etc. And it's mostly boring, especially for upstream developers because I guess if upstream developers tried to support all Linux distrust, they wouldn't do anything else.
03:03
So, that's why we are trying to help with packaging. So, packet has these goals. It's trying to partially automate the task of packaging software and I
03:21
must stress that it's only partial help because it's not possible to completely automate the process of packaging. Then it's also trying to create some easier interaction between downstream maintainers and upstream developers. And finally, in those cases, like I was showing with VS FTPD, it provides like easier way to manage this Git for maintainers with source Git.
04:01
I will explain later. So, you can use Packet either through CLI or as a service. And again, please note that Packet CLI is not a client for the service. Right now, it's actually more capable than the service because the service has some problems with authentication against the Fedora infrastructure.
04:31
So, currently it cannot do all that the command line client can do. So, what can the command line client do for you?
04:42
Well, as I said, it's about bringing upstream and downstream together. So, on the top, you have the upstream repository. Currently, I think they only support GitHub, maybe GitLab.
05:01
But they are definitely working on it. And if you want to, for example, create an update for Fedora repository, you can generate it automatically from GitHub release. So, whenever there is a new release, you just type Packet proposed update.
05:25
And it will download the sources, upload them, modify the spec file and propose this change as a pull request to our federalist Git. You can also use it to trigger builds in our community build service.
05:45
So, when you are working on some new features in your upstream project, you can quickly see if the RPM still builds. And you can also build the RPM in our build service and then propose update.
06:08
So, this is how it looks like. When you run the Packet command line tool in CLI, and it looks very simple, right?
06:23
Well, unfortunately, in order to execute such simple command, you need all of this configuration. So, if you want to know the status of your project in Fedora, you need to create the configuration file in config.packet.yaml.
06:51
And you need to generate the GitHub token. You need to generate the user token. If you don't know, Pejor is the web UI we use for this Git.
07:03
And unfortunately, you also need Kerberos ticket for Fedora. And you also need SSH key in your Fedora account. So, that sounds like a lot of work. Unfortunately, this is how Packet works like now.
07:24
Yeah. But when you do all of this, you can, as I said, you can create source RPM, propose the update, build it and create the update in Bode.
07:40
So, let me show you some examples. For example, so, this is the, this is the DisKit repository that is completely maintained by Packet. Oh, sorry.
08:02
This is our repository that is completely maintained by Packet. And as you can see in pull request, all of the updates were automatically generated.
08:23
I didn't have to do anything except for, except for accepting these PRs and, well, this is not really interesting. What is interesting is here, it was able to automatically update the spec file, create a changelog entry and upload the new sources.
08:52
So, yeah. When you, then when you merge this PR, you can easily build the new RPM and propose it to Fedora.
09:04
Okay. Let's go back. Yes. So, again, this needs some configuration.
09:22
So, they have this little YAML file in the upstream repository. But even though it's pretty long, I don't think it contains any magic. Basically, just specifying where you can find the spec file, how the package is named in Fedora was the upstream name.
09:50
Because sometimes there is a difference. The downstream package has a different name than the upstream project simply because the name was already occupied.
10:01
And the rest is just about what branches you want to build, etc. It's nothing complicated. Yeah. So, this is what I have already mentioned. I don't think that all upstream developers have Fedora accounts.
10:25
And that's unfortunately nothing you can do. There is nothing I can do about it right now. You just need one. Or you can wait until package developers remove this condition.
10:46
Okay. And what to do if you are a maintainer and you don't have access to the upstream repository. This is a common case for maintainers in Red Hat.
11:01
In that case, you can create something they call source Git. It's a fork. So, on GitHub, you create a fork of the upstream repository. You rebase commits that contain the spec file and maybe some downstream changes on top of this repository.
11:22
And then you automatically generate the content of this Git repository. Even though it sounds almost the same, it's much easier. Actually, when I was maintaining the FTP daemon, I did exactly the same just for myself.
11:43
But it was a little fragile because it was a shell script. So, this time it should be more robust. And I hope it will work. And finally, what Packet can do for you. As I said, it's about partial automation. And people often ask me if Packet can generate the spec file for them.
12:06
And the answer is no. Because in general case, it's impossible to generate the spec file. For example, in the project I maintain, the spec file simply contains information that cannot be found anywhere else.
12:23
So, it's impossible to generate it if there is nothing you can use as a source. But you can still use all those utilities for generating Ruby packages, Rust packages, Python packages, et cetera, et cetera.
12:46
Okay. But this still looks like a lot of work. Yes. Microphone, please.
13:06
There. Theoretically, if they had a makefile that had a duster option, couldn't you just make dirt to a temp directory and then figure out what it installed when you did a makeinstall?
13:24
Yes. In case the project has makefile, it should be possible to determine what it installs. But, for example, we don't have makefile. We just don't. No, I'm saying if I'm doing it to a GitHub account, it probably has a makefile.
13:44
If I do a makeinstall duster, you could generate a spec file based on the file paths that were installed. Well, that's the problem. It doesn't work like this for every project. Like, you can have as you have the gem to RPM, Rust to RPM, you could probably have something like make to RPM that would use makefiles.
14:08
But, for example, the project I work on, we have just setuppy. But setuppy does not contain all the information. Because it's not supposed to be used as a system-wide installation.
14:25
So the spec file contains some data that cannot be found anywhere else. So it's pretty complicated to do this for any software project.
14:42
Yeah. So, but, yeah. Still, it's a lot of typing in terminal. So, can we do better than this? And, of course, with cloud. Because that's the current answer to anything.
15:01
So, you can use Packet as a service. It's available on GitHub in the marketplace under continuous integration. And you can install it and then enable it for your repository. It currently has only a few features.
15:21
The first one is built on push. So, whenever you create a new pull request in your project, and you push to this branch, it will trigger a build in our community build service.
15:40
So you can immediately see if the project still at least builds. It looks like this. It's very simple. So you push and in few minutes or hours, you should get RPM. Now, why am I saying few hours maybe?
16:02
Well, the community build service is sometimes a little busy. This is the task queue. But if you are from some, for example, if you are from US, you are lucky.
16:21
Because the peak is usually during the working hours in Europe. So, if you work during some other hours, you should be fine. And, yeah, the unfortunate thing about the community build service working with Packet
16:41
is that in order to somehow manage the build instances, each user in the community build service can run only seven builds in parallel. And since Packet uses only single user for all their projects,
17:01
it can build only seven projects in parallel. So, if Packet becomes popular, this will become a problem. And these are the features that it should have or will have.
17:21
The first one is the blue one. The blue one, it should automatically create the updates, as I was showing earlier. So, whenever it receives a webhook from GitHub, it should create the PR automatically.
17:42
But there is a problem. When you create the new release, you can just wait. There is nothing you can observe. There is nothing you can do in order to somehow check that Packet is working.
18:04
There is simply nothing. All you can do is to contact upstream developers and ask them, hey, is there any lock in your cluster? So, that's a pity. I tried to use this feature. Unfortunately, it didn't work for me.
18:23
So, I had to generate the updates by hand. Now, the other feature that should work is the other way around. When you change the spec file, for example, in this Git, this happens a lot when we have, for example, mastery builds.
18:45
Before federal release, there is some new entry in the changelog. So, you need to propagate this change to upstream. And Packet, again, should automatically create a pull request for the upstream repository containing these changes.
19:07
Again, I can show you how it looks like. So, you can see this is the upstream repository was built.
19:22
And when there was a change in Python packaging rebuilt for Python 3.8, Packet automatically generated this pull request on GitHub. So, I could just merge it and the downstream and upstream spec files were synchronized.
19:48
Okay. So, yes. And finally, this is not yet implemented. It should also create the new RPM automatically and propose it as an update.
20:05
From what I've heard from the Packet developers, this is currently blocked on the authentication system we use in our federal build service and update service.
20:23
And I hope they are working on it. Okay. This is what I was talking about. And of course, there are still some unresolved questions. So, for example, what to do with RPM changelog? This is something, again, everybody has a different opinion.
20:45
So, there is simple no general way to generate the changelog so that everybody is happy. I'm personally happy with one line saying that there is update to the newest upstream version.
21:07
But some maintainers want a list of new features or something like this. So, again, something that is very difficult to do in general.
21:21
Some future plans. So, as I have already mentioned during the talk, there are still some problems with current implementation. So, we should or they should mostly stabilize and fix bugs.
21:42
I hope they will improve the user experience. Because as I said, when you do something and it doesn't work, you have no idea what went wrong. For example, there were failing copper builds for a few weeks
22:00
and we had no idea what's wrong. And we couldn't do anything about it. So, I hope this will get better. And finally, this is something I'm really looking forward. They promised to spin Fedora VM for each copper build
22:20
so that when you have the RPM, you can actually run some integration tests. Because I think, at least for me, that's the point of building the RPM. So, I can test that it actually still works with the rest of the system. So, that would be really, really nice.
22:41
Finally, the packet developers, they would like to hear your input. So, first of all, is it interesting for you? Then, if yes, are you willing to include the packet YAML configuration file
23:02
in your upstream repository? And are you interested in the information if it is still possible to build it for Fedora?
23:21
And if it works with the rest of the system? You can use their GitHub issues. I use them a lot. Here you can see the URL github.com slash packet service or you can use IRC channel or this email.
23:44
Yeah, I was a little bit faster than I expected. So, thank you. That was it.
24:01
If the goal is to automatically also build upstream stuff, I think then solution actually would fit for 70% or 80% of upstream software, which is better than to do everything manually. And the other thing, I was wondering, so how would that solve your FTP program issue where you had 50 patches?
24:24
That's the approach with the source Git. You can create a GitHub repository with the sources and you mark the commit that is the upstream release
24:42
and then on top of it you have all those patches but this time as commits and then you generate the content of this Git repository from this Git repository, standard Git repository. It's actually easier than, no, in this case not
25:08
because upstream releases new versions as starballs and there is no upstream repository. For projects with Git repository, oh yeah, I should probably...
25:26
Yeah, so, okay, in this case they don't have a Git repository but if you have a project with 50 patches... There is a GitHub or whatever... Yes, for example, if there is a project that has a GitHub repository
25:40
and don't want to include the packet.yaml file, you would fork the project and then you would again rebase the commits with the packet.yaml file, spec file and downstream patches. So whenever there is a new release, you just pull in the changes from the upstream repository
26:05
and again rebase the patches. I know it sounds almost the same but when I was maintaining these packages, it was actually much easier than maintaining all those files by hand.
26:24
Hi, yeah, so I wanted to ask because you mentioned adding a packet.yaml file which is necessary but there is also the spec file and I wanted to ask whether there are restrictions to the spec file. Must it be like just one source? Can it have other sources, patches? Because that's all going to the upstream repository
26:42
that's not really pulled from the Fedora side in Pagger, right? Yes, actually you don't even have to have the spec file in the upstream repository. They have a concept of like actions or hooks.
27:03
So when they trigger some action, they can run some little script that is written in the yaml file so you can use curl for example to download the spec file from somewhere else. So basically you don't have to even include it in the upstream repository
27:23
but the packet.yaml file is mandatory and if they don't want to include even the yaml file, then you have to fork it and use the source grid approach.
27:40
Okay, two questions. First of all, about automatic versioning. You said that pocket automatically increases the version but version of different packages maybe worry. I mean a schema of versioning. So how do you deal if, not if, then your packet fails the versioning?
28:06
Yes, so this is again complicated question and again you can use custom command. For example, we did use Python script to generate the new version
28:27
and packet would just call this script to generate the version for it because in general case you have no idea how to automatically generate the new version.
28:42
So whenever upstream uses some unknown or specific versioning, they can always include their own script to generate the version. Okay, thanks. And the second question. Any plans for non-Fidora specs?
29:03
Non-Fidora specs. Very good question. I was wondering about this as well but I don't think they have any plans right now. No. Because they are pretty busy just implementing features around Fedora.
29:25
So I think they don't have any plans right now, unfortunately. But yeah, I was thinking about this as well because for example I was maintaining Wireshark and they have a spec file in their upstream repository
29:42
and it's possible to use it with both SUSE and Fedora therefore I don't think they would be interested in changelog entries with Fedora rebuilt or something like this. I mean in the end there are already a lot of CI systems
30:07
like Travis or Circle CI or appware or something like this so some projects are willing to add even more configurations files for CI stuff but I think you might be able to look at these files actually
30:23
and somehow if the automatic stuff works and normally you have built stuff in there use this to do this integration stuff and second, if you build stuff you need of course to make sure that there are no false positives so maybe there's some kind of database where you log
30:45
if there was a successful build and then only afterwards if it breaks in the future then you report issues and not before but to avoid false positives. Yeah, I'm pretty sure there is a lot of efforts like this
31:02
as I said when I was a package maintainer I tried to implement something like this myself and I tried to use whatever was available but as I said none of this works for every project so this is a brand new project to solve this again.
31:29
Any other questions? There is one over there. I missed some of the talk, maybe this has been answered before
31:44
in Packet as a service are there any plans to allow Koji builds instead of just copper builds? Yes, there is but there is a problem that if you want to trigger a Koji build and build the update
32:01
you need to authenticate yourself using Kerberos and as far as I know there is a problem with running Kerberos inside of OpenShift cluster right now so they are trying to debug it what's wrong or how to do this
32:23
but currently there is no plan to or no specific date I could promise you when this will be available. And that's just for Koji builds.
32:50
So if there are no more questions, thank you for listening.