Package management and creation in Gentoo Linux
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 90 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40311 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201330 / 90
2
5
8
10
12
13
14
15
17
19
21
24
25
28
29
31
32
34
36
39
40
43
44
46
50
51
52
54
55
57
58
62
65
66
67
78
79
87
88
00:00
Gentoo LinuxData managementSoftware developerBitMultiplication signFile formatData storage deviceData managementGroup action2 (number)WeightTerm (mathematics)QuicksortWebsiteData managementDistribution (mathematics)Modul <Datentyp>Lecture/ConferenceComputer animation
01:50
DivisorSoftware developerSource codeVariable (mathematics)BuildingGoodness of fitCodeInteractive televisionTotal S.A.Right angleCohesion (computer science)Lecture/ConferenceComputer animation
03:20
Software maintenanceArchitectureSoftwareCommon Language InfrastructureCodeCASE <Informatik>Data managementBlack box2 (number)Software developerGraphical user interfacePoint (geometry)File formatBitTerm (mathematics)UsabilityMathematicsData managementMultiplication signInterface (computing)Common Language InfrastructureLevel (video gaming)Scripting languageRadical (chemistry)Computer animation
05:09
Common Language InfrastructureMultiplication signMereologySoftware maintenanceSoftwareFile formatPoint (geometry)Computer architectureScripting languageAdditionPhysical systemInstallation artLecture/Conference
06:03
Configuration spaceBinary fileImplementationRevision controlSource codeFlagMassPhysical systemCompilation albumParallel computingStructural loadParallel port1 (number)Point (geometry)Software developerBinary codeSource codeGroup actionView (database)File formatRevision controlQuicksortInheritance (object-oriented programming)SpacetimePhysical systemMultiplication signCoprocessorComputer animation
08:15
Embedded systemGeneric programmingSet (mathematics)Repository (publishing)SoftwareVariable (mathematics)Function (mathematics)Process (computing)Software developerHeegaard splittingFunctional (mathematics)Repository (publishing)Installation artPhysical systemGroup actionSeries (mathematics)Configuration spaceScripting languageComputer filePatch (Unix)File formatBitSource codeLecture/Conference
09:51
Function (mathematics)Variable (mathematics)SoftwareInheritance (object-oriented programming)Level (video gaming)File formatFiber bundleComputer fileObject-oriented programmingConfiguration spaceRight angleDefault (computer science)Physical systemQuicksortComputer architectureRun time (program lifecycle phase)Revision controlMetadataType theoryLibrary (computing)Complex (psychology)Functional (mathematics)BitDifferent (Kate Ryan album)CuboidTerm (mathematics)Point (geometry)Source codeDistribution (mathematics)Multiplication signEmailCompilation albumBuildingLecture/Conference
12:02
Type theoryRevision controlDifferent (Kate Ryan album)Computer fileData managementFile formatFile systemBitQuicksortDirectory serviceLecture/Conference
12:42
Configuration spaceComputer configurationDefault (computer science)User profileAdditionInstallation artConfiguration spaceFlagSurjective functionProfil (magazine)Functional (mathematics)Computer configurationQuicksortMultiplicationDifferent (Kate Ryan album)Sinc functionLevel (video gaming)Interface (computing)Multiplication signUser profilePoint (geometry)Type theoryData managementSoftware developerLibrary (computing)Source codeAverageSpacetimeComputer animation
14:29
Default (computer science)Core dumpData managementDifferent (Kate Ryan album)1 (number)Computer fileBitFile formatLecture/Conference
15:03
Default (computer science)Regular graphCodeComputer animationLecture/Conference
15:42
Hill differential equationMIDISlide ruleNetwork topologyComputer animationSource code
16:17
EmailComa BerenicesEwe languageHill differential equationBitRight angleData managementLibrary (computing)QuicksortRevision controlPhysical systemNumberSet (mathematics)InformationMultiplication1 (number)Computer architectureFunctional (mathematics)32-bitUtility softwareVariable (mathematics)Positional notationSubsetRun time (program lifecycle phase)CASE <Informatik>EmailRegular graphSocial classMultiplication signSoftwareMetadataDefault (computer science)Computer fileCategory of beingEquals signSource codeComputer animationJSON
18:50
Information systemsDefault (computer science)Functional (mathematics)QuicksortBuildingVideo gameDefault (computer science)Source codeMultiplication signDistribution (mathematics)Order (biology)File archiverReal numberFile systemSoftware developerDirectory service1 (number)FlagConfiguration spaceMereologyPhysical systemRight angleCASE <Informatik>Installation artComputer fileParameter (computer programming)BitPresentation of a groupSource code
22:03
Default (computer science)1 (number)FlagSeries (mathematics)Physical systemMultiplicationGraphical user interfaceRevision controlQuicksortPoint (geometry)Default (computer science)CASE <Informatik>NumberProfil (magazine)Parallel portStatisticsSoftware developerInstallation artFunction (mathematics)Computer architectureLecture/Conference
24:06
MIDIInformation securityMereologyCore dumpJava appletVariable (mathematics)Software developerRight angleSource codeSeries (mathematics)NumberUtility softwareJava appletFunctional (mathematics)Multiplication signRevision controlPatch (Unix)Default (computer science)EmailUsabilityIndependence (probability theory)Different (Kate Ryan album)QuicksortEnterprise architectureComputer architectureDirectory serviceSoftwareRun time (program lifecycle phase)Computer animationSource codeJSON
26:02
Java appletArrow of timeFlagData managementSoftware developerArrow of timeSource codeMultiplication signLevel (video gaming)Hash functionPhysical systemNumberUniform resource locatorComputer fileForcing (mathematics)Library (computing)Revision controlBuildingDifferent (Kate Ryan album)File formatProcess (computing)Functional (mathematics)Parallel portCASE <Informatik>CollisionForm (programming)String (computer science)Cycle (graph theory)Right angleSource codeJSON
30:46
Arrow of timeFlagLibrary (computing)Level (video gaming)CASE <Informatik>BuildingLecture/Conference
31:29
Web pageMetadataNumberMotif (narrative)Library (computing)CASE <Informatik>Functional (mathematics)Revision controlSocial classLogicRight angleSource codeJSON
32:45
Phase transitionDefault (computer science)Configuration spaceMereologyMultiplication signConfiguration spaceRight angleFunctional (mathematics)Flow separationBlock (periodic table)Doubling the cubeFile systemNP-hardFlag1 (number)Term (mathematics)Parameter (computer programming)CodeBuildingLecture/Conference
33:54
Function (mathematics)Default (computer science)Configuration spaceCompilerMereologyPhase transitionDefault (computer science)Software developerParameter (computer programming)Configuration spaceComputer fileFunctional (mathematics)BitPatch (Unix)
34:58
Default (computer science)Phase transitionPoint (geometry)Reverse engineeringRight angleSource codeDefault (computer science)MereologyParameter (computer programming)Configuration spaceFlagPatch (Unix)Computer fileCompilation albumFunctional (mathematics)InformationQuicksortFile formatMessage passingInstallation artLecture/ConferenceSource codeJSON
36:07
HypermediaRootDirectory serviceSubsetMultiplication signRight angleFile systemComputer animation
36:48
Default (computer science)FlagInformationRootRight angleSoftware developerFunctional (mathematics)Default (computer science)Directory serviceInstallation artBuildingLecture/Conference
38:02
Default (computer science)FlagDefault (computer science)Installation artCASE <Informatik>Right angleCalculationPhase transitionLecture/Conference
39:18
Default (computer science)InformationFlagHausdorff spaceMIDIHaar measureComputer fontUniform resource nameEmailComputer-generated imageryWeb pageBeat (acoustics)Revision controlObject-oriented programmingDefault (computer science)Functional (mathematics)Right angleSuite (music)Social classSoftware testingDirectory serviceProper mapSource codeJSON
40:32
User profileGroup actionOperator (mathematics)Revision controlHierarchyMathematicsFlagAutomatic differentiationLibrary (computing)Odds ratioExclusive or
41:48
User profileBlock (periodic table)Gentoo LinuxData managementSoftware developerSoftware developerMultiplication signConfiguration spaceSource codePhysical systemRight angleFunctional (mathematics)QuicksortRepository (publishing)Default (computer science)Library (computing)BuildingSocial classDifferent (Kate Ryan album)Revision controlComputer fileCodeLecture/ConferenceComputer animation
43:47
Physical systemBitSet (mathematics)Right angleProcess (computing)Equals signComplex (psychology)PlastikkarteMaxima and minimaPoint (geometry)Sign (mathematics)Lecture/Conference
44:34
Lecture/Conference
Transcript: English(auto-generated)
00:00
I'm Donny, I've done Gentoo for a pretty long time now, and I'll tell you a little bit more about that in a minute. But basically this talk is about what kind of value we can provide to the other distros in terms of what kind of things you could learn and take away and apply to your own. So what we're going to do is talk through some of the philosophies,
00:23
think about how you can take a philosophy from a distribution into how you turn that into package management and that sort of thing, and then run through some of the individual features we've got and the way we've iterated on those features in our package format over the past 10, 15 years to continue thinking about how can we keep improving the packaging format
00:45
and not just how we handle things on top of that format like the last talk, which was mostly iterating on top of RPM rather than iterating RPM itself, although RPM does very slowly move forward.
01:00
So I apologize for the terrible quality of that picture. I stole it off somebody else's site. This is basically what I used to do in Gentoo. I was somewhat overweight, and I also wore a lot of hats. I did things including maintaining most of our X packages, which was about... Well, first it was like five, and then the modularization happened, and then it was 400.
01:23
And then there was a lot of other science packages and miscellaneous stuff. I am, I think, still the desktop group manager on the council, which is the group of seven people that decides all the technical stuff about Gentoo and so on. But really these days, first of all, I lost some weight,
01:41
but second of all, I pretty much just talk because I had a few kids, I got married, I graduated from college, and suddenly there's not so much free time as there used to be. So the philosophy of Gentoo really derives from two things. One is the guy who started it named Daniel Robbins,
02:01
previously worked on a distro called Stampede, and that's purposely pixelated because that's how old the distro is, and how long since it went anywhere. Stampede was a very old school distro back in the late 90s, early 2000s. And then Daniel Robbins went to FreeBSD because he got pissed off at the developer community,
02:23
at the guys just being total dicks about everything. So he left, decided, turns out he thinks the developer community is a really important factor, not just the technical code itself, but actually the quality of the interactions within the community. So he left, went to FreeBSD, and he said, oh, FreeBSD has a really nice way to handle the packages, right?
02:43
Because they've got, they build everything from source, they've got these makefiles, but on the other hand, all the makefiles, everything is controlled by an individual variable, right? Everything is a unique snowflake. So the same feature in one package versus another package might be controlled by a variable with a totally separate name,
03:03
and it makes it really hard to deal with it. And so he said, you know, I wish I had this FreeBSD thing, but in a more cohesive way and on Linux with a good developer community. And that's why he started Gentoo. This is a picture of a Gentoo penguin. It's supposedly the fastest penguin,
03:21
which is terrible for Gentoo's reputation because really it's not about speed at all. It's about flexibility. But we've got all these ricers who think, oh, I use Gentoo because I get faster code. No, that's not at all the case.
03:40
Well, you might get 5% if you're lucky, but that's after quite a bit of work. I don't know what's going on with all these weird bullet points. So in terms of philosophies, it's really... Do I have more bullets or not? Oh, yeah, I do. That was weird. So basically we were thinking,
04:01
how can we have something that's user-friendly despite being something we spend a lot of time on a command line? And so we spend a lot of time working on our package manager portage, which the main command for that is emerge, and working on the CLI actually being something that's very colorful and easy to understand because we don't really support any GUIs for package management.
04:23
Everything is done through the command line on a terminal. And so the title is Gentoo Package Management Goals up there. So thinking about, if we've got these goals, what can we do to make that a reality? And really the goals were, one, it's great ease of use despite a very low-level interface
04:41
because we're trying to understand what's inside the black box of Linux distros. The second goal is really, how can you minimize the developer effort for people who are writing the packages? And that pervades through a lot of the changes we've made to the formats over the years, and even more so a lot of the changes we've refused to make over the years because they make the format too complex.
05:03
And one of the benefits we've had is that it's very easy to start writing packages, and this has helped us get a lot more people involved because packages look like bash scripts. And everybody who runs Gentoo, you have to run a lot of bash stuff to just install the thing, a lot of it, and you probably screw up four or five times and then you figure out it's important to read the docs.
05:25
And so we've got people who, by the time they even manage to install Gentoo, they're already at a point where they could build a very basic package because it's just a bash script. And wrapped around with some fancy features to build software more conveniently.
05:40
And so simplicity is a very, very important thing when you're thinking about what should your packaging format look like. And that's the easy maintenance part. And in addition to that, it's thinking about what does a package format have to look like if you want to support tons of architectures and tons of operating systems? What kind of things do you have to add to deal with that?
06:06
These transitions are wacky, aren't they? And then the other major features, these aren't as interesting, but really the ones that are cool are thinking about how do you support the possibility of installing, say,
06:20
five versions of GCC on the same system? Or dealing with GTK1 and GTK2 and GTK3 all installed at the same time? And then thinking, you know, Gentoo is a source-based distro so we have to handle source-based packages but we also want to enable people to do what they want to do. So how do you support binary packages
06:40
for the people who want to do binary packages? And all of these kinds of thoughts go into how do you create a package format that makes sense? I don't even understand what is happening with these transitions. It's so weird. I'm just going to skip that stuff. That's not very interesting.
07:01
It's always about flexibility from Gentoo's point of view and thinking how can you enable, whether it's users or developers, to do whatever they want to do? How do you enable them to use all their processors? We added a bunch of parallelization stuff. How do you make it less of a pain in the ass to upgrade things when you're sitting around, if you're compiling for like 12 hours or something crazy like that,
07:23
you don't want to find out that after three hours when you went to sleep after too many beers that it failed. So we said, okay, let's resume that thing. Or dealing with... It turns out it's amazing how many packages out there install binaries with these same names. It's really a pain in the ass for us to deal with this
07:42
when you've got some weird obscure thing that installs a binary with a certain name and then you've got something that's super common that installs the same thing. It just makes you want to kill somebody. But then also things like making it super easy for developers building on Gentoo to do what they want to do by saying, okay, it's really easy to build packages,
08:03
keep them integrated with the system, and yet have them very easy to work with and code on by adding debugging support, installing the sources without stripping the binaries and that sort of thing. Is there anything that's worth saying there?
08:22
Not really. I have no idea what's going on with those transitions. I apologize about that. And then the other things that are really interesting to think about are how do you deal with multiple different repositories and thinking about do you want to add dependencies
08:43
across those repositories so you can actually depend on a certain repository being installed. But that really creates some issues when you're thinking about, okay, but somebody might derive from that repository and then do something incompatible and change the name. How can you deal with these kinds of things inside of a package format?
09:08
So the ebuild file itself is the package format, and basically it's a big bash script that's split into a series of functions that map logically to what a build and installation process looks like.
09:21
So you've got some main functions that are like, okay, let's unpack the source, let's screw around with it a little bit, patch it up, let's configure it, let's compile it, let's install it, and then let's do whatever weird screwing around with the system you have to do to actually make the thing work. And how many of you are actually developers for Distro?
09:40
Or packagers? Awesome. So all of you know that build and install processes when you're actually writing a package are nowhere near as simple as you wish they were. There's always some stupid fiddly thing you have to do on any package that's anything beyond the trivial. And our goal with the package format
10:01
is to make it really easy to do that kind of thing. So you do, okay, here's the default, and then let me tack some things on to that to do whatever screwy, fiddly things you have to do, whether it's doing a sed on a configuration file or fiddling with a couple of files on the live system, renaming some things so they don't overlap, and so on.
10:21
And then the last thing that's really interesting about the package format is this idea of we have some level of object-oriented style inheritance. So we have this thing called e-classes, where they're essentially like a library of functions, and the individual e-builds can actually inherit from that library, use all the defaults from that library
10:43
in terms of the unpacking, the compilation, the installation, and then vary from those a little bit as they have to. And these other things are just some basic metadata about the package. What architectures does it work on? The slot is the thing that enables you to install multiple versions
11:01
of the same thing at the same time, so GCC 3.5, 3.6, 4.0, 4.7, and so on. And then we've got a few different types of dependencies, and we're actually thinking about adding more because we haven't made it complex enough yet. So it turns out when you're making a source-based distribution, you have to think about a lot more types of dependencies that have to be on every user's system.
11:22
It's not just about the runtime dependencies. It's also about the build time dependencies. It's about the dependencies that have to be installed after the package is installed because they depend on it, but it won't run without them. And you've got some other styles of dependencies like, oh, I need the headers to be available while I'm building it,
11:41
but I'm building it for an embedded system on this box, and so things get really weird and complex. Yeah? I'm sorry? Oh, yeah, we have conflicts between packages. Yep. Yeah. Thank you. Yeah, so we deal with conflicts,
12:02
and I'm going to mention that in a little bit because one of the things we did as we were iterating on the package format was think about what are the different types of conflicts and how can we enable the package manager at Portage to deal with those types of conflicts more effectively. So there's one possibility, which is the two packages, one can overwrite the other before the other is uninstalled,
12:22
but then the other type is a mutually exclusive thing where they can't be touching the live file system at the same time at all. So the first one has to be uninstalled completely before the second one can be put into place, which may happen if you're replacing a symlink with a file, for example, or a directory with a file.
12:47
So then we've got this thing, since we're source-based, we have to think, okay, how do we enable people to build only the futures they want to use? And that's what we created with use flags. They sort of map onto configure options.
13:01
Let's say you've got a package that supports a GTK interface or a Qt interface, and you have a use flag for GTK, you'll have a use flag for Qt. It might add some extra functionality if you actually build for GNOME, specifically beyond GTK. So there might be another use flag for that that will pull in another 15 libraries.
13:21
And we have lots of different profiles so that people can basically pick a reasonable starting point and then iterate from there. Both on the user and the developer level, it's thinking, how can you set up people as close to where they need to be as possible and then just let them define a couple of things to vary from that? So the value of actually providing a lot of different types of user personas
13:42
or user profiles is actually really high. The basic story behind how we've iterated on our ebuild API or the eAPI over the years, every year to year and a half for the past six years, we've actually come out with a new eAPI,
14:02
some of them adding more features than others. You can see eAPI 1 was the first time we changed things. Before that, it was eAPI 0. And so 0 was basically like, what is this thing we have now? It's been around for like 10 years and it's just a gigantic mess
14:21
and it was a huge effort to just document the thing. And in fact, it actually has turned out to be pretty valuable to have multiple different package managers implementing the same spec. So we've got Portage, which is our default package manager, but then there's also a couple of other ones. There's Pollutus and PackageCore that implement the same spec
14:42
and all of them deal with ebuild files. This has been pretty nice and actually forcing us to really think through what does a package format look like, how do you describe it in such a way that it's useful to lots of different people. And so we came out with 1. A year later, we came out with 2.
15:00
A little bit over a year later, we came out with 3. Another year later, 4. Another about a year and a half later, 5. But the funny thing that you don't appreciate looking at this, it looks like this is a nice regular march. But in fact, it turns out that 3 happened because we were so sick of the stuff we were working on for 3, we decided to push most of it to 4 and just get something out the door.
15:21
So in reality, it's like big features, big features, almost nothing, huge pile of features, a bunch of features. And we're just going to run through some of those so you can get a feel for what those feature sets look like. Yeah.
15:40
So actually, why don't we just pop out and look at some actual code instead of just sitting here looking at slides all day. What did I say? That was MPC? Yeah, OK. DevLib is MPC.
16:02
So this is what it looks like when you're in a package tree. Yeah, it looks beautiful over here, guys. Yeah, I'll slide it over there. Just maximize the thing.
16:21
And maybe make it a bit bigger. Can you read this in the back yet? Cool. So this is basically what an e-build looks like.
16:42
And it's just a simple text file. I use nano for e-builds because I'm terribly pathetic. So at the top, you've got some basic header info, copyrights, and so on. Then here, one of the first things we do is define the API so that the package manager understands how do you parse the rest of this thing.
17:04
We inherit some e-classes to give us some default ways to handle various things and a few sets of utility functions. So e-utils is sort of a basic subset of functions. We've got a set of functions to deal with libtool. We've got a set of functions to try and handle multi-lib systems
17:20
like 32-bit and 64-bit on the same system simultaneously. And then you've got your basic metadata. What is this package? Where do you find it? How do you download the thing? What's the license? Because we actually do support things like only installing packages with a certain set of licenses. So you can have, for example, a completely free or Libre software system.
17:41
You've got a slot, and the slot is what you change if you want to install multiple versions at the same time. You've got what architectures you work on. This IUSE thing defines the set of possible features you can vary on this package. So in this case, there's only one. You can build the library statically or not. And that means you'll always get it dynamically,
18:01
but then you can also tack on the static ones too. And then you've got the dependencies. And these are just very basic standard notations for indicating the numbers and things. So you've got a greater than or equal to. Here's a category. Here's a package name and then the version number. And the same thing in the second place.
18:21
And we've actually got a fairly flexible syntax for dealing with that. So you can have greater than or equal or less than or equal or just greater or less. And you can also have a star, an asterisk, if you don't care about the version or you don't care about some set of the version. And then it's just regular old bash, right?
18:40
So the runtime dependencies, you just use the depend variable. And then you can go down and look at what the functions look like when you're doing this thing. And you've got an unpack function, which just calls out to unpack the tarball,
19:00
go into the source directory, and run some libtools stuff. Then you compile, so you have a configure function that has some convenience features, like a bunch of default arguments. And then you've got, here's handling the use flags. So use enable, and then when this use flag is specified,
19:20
here's the configure flag you pass. So you pass enable static. And then if it doesn't work, you fail, die. Yeah, right. So I'm purposely showing these in order of how we came up with them. So this is the old school EAPI 0.
19:41
We actually decided in the case of following our philosophy, trying to make it simpler for developers to write these things, that in fact it would be nice if you didn't ever have to write this die thing. And that was how it worked by default. Because normally if something wrong happens when you're building or installing a package, you don't want to install the thing.
20:00
It's going to be broken. So you want it to die by default and instead specify something if you don't want it to die. So one of the things we ended up adding in a later EAPI was this idea of a sort of non-fatal prefix where it doesn't die. And normally it just happens that it always dies or something goes wrong with nearly every one of these commands.
20:23
So you unpack, you compile, you install, which is just basically a make install with a destination directory, and die if it fails. And then there's the little bit of fiddling that I'm talking about. Every time you're writing a package, there's always some fiddling that has to happen.
20:40
So in this case it's getting rid of libtool archives, installing a couple of docs. And then something I should say is every one of these functions that's prefixed with source, those ones happen when you're doing a source build, only when you're doing a source build. But every time you're installing a package that's merged onto the live file system
21:00
from the source directory where it's compiled, then you've got package functions that run, package prefixes. So you've got a pre-installation function, a post-installation function that you can run before and after all the files from the build are merged onto the live file system, which enables you to get things ready, make sure nothing is in the way.
21:21
And then it seems like there's always packages out there with terrible build systems, and so you have to tweak things in the files to get installed, because they don't realize they were going to get moved after they were built. It's surprising how many people write just awful build systems that expect you to download the thing, unpack it, build it, and then run it in place
21:40
and never install it onto a real system as part of a distribution. Okay, so that's what a very basic EAPI 0 eBuild looks like. Now let's see if I can manage to get back into the presentation.
22:00
Okay, so then we came out with the EAPA 1 about five, six years ago, and we added a few really interesting features. Nothing huge, and I'm only showing the ones that are kind of interesting versus the ones that are minor tweaks. So we added this idea that you could actually have a use flag default in a single package
22:20
that wasn't the same as the default for the whole system, because sometimes with a package it makes more sense to enable a feature than not. And so we said, okay, in this case, you could turn off GTK support, but it would be kind of stupid because then you wouldn't have a GUI at all. So if they haven't manually specified
22:42
switching away from their global default in their profile, then it will, for example, add GTK instead of not adding GTK, even if the global default is to not add GTK. But if they say, I don't want GTK, it will still override that. So there's a series of overrides.
23:01
We also had this idea of being able to actually properly depend on things in a certain slot. This was very useful, for example, with GTK 1 and GTK 2, where it became increasingly difficult to specify all of the things that matched a slot using a series of version numbers. And so it ended up working out pretty well to add a way to depend on a certain slot.
23:23
And a lot of distros, instead of doing the slot thing, they decided, oh, we're just going to put the version number into the package name. So there was a package that was GTK 1-1.2.10, for example. And we thought that wasn't really an elegant way to handle that, so we added a new concept to deal with the parallel installations instead.
23:43
And it actually turned out to be really useful on development systems, because you could have multiple versions of GCC. You could have multiple versions of GCC that cross-compile two different architectures on the same system. And it actually had a lot more benefits than anybody ever expected, I think. So we'll just take a quick look at one example of an e-build that does this.
24:05
devvcs-stat-cvs 0.4.1. Let's see.
24:24
It's all the same stuff from last time, right? You've got your basic header. This time, the difference is eapi1, right? You scan down, most of the stuff looks about the same, but you start looking into dependencies, and now we've got this colon and a version number after, indicating that we're depending on a certain slot.
24:42
So it's only the jcommon1 slot, and presumably jcommon also has a 2 series, right? This is actually really common with Java packages, is that they'll have simultaneous independent series, all of which are maintained ongoing, because Java is a really popular architecture
25:01
for doing enterprise software development, and those things have to be supported for a decade. This is just an example of what the slot dependencies look like. Other than that, all of this should look a little bit familiar, because you saw it a few minutes ago.
25:20
You've still got your basic build dependencies, runtime dependencies, unpack function. We're doing some utility functions that will unpack the source code, apply some patches, and some convenience variables to say, okay, the patches are in that files directory for that package, and then this one expands to actually the package name
25:42
and the package version. I'm trying to remember. Does that one actually have any use defaults in it?
26:14
So that was a couple of minor features. But then we came out with EAPI 2,
26:21
and we added a couple of really cool things to it. One of them was this idea of arrows for where to download it, and this is just another... Yeah, okay, sure. So the notion of slot is primarily intended for packages
26:40
that were intended to be installed in parallel at the same time. So GTK1 and GTK2 are a really nice example of this. There's GCC, for example, so that you could have GCC 3.6, 4.0, 4.5 all at the same time on the same system. And the slot is just a way you tell the package manager
27:03
that these are all the same package, and we should enable them to be installed at the same time instead of forcing the lower version to be upgraded and removed. So that means that the package manager is the one who decides what would be the slot number?
27:26
So the slot number is specified in the e-build itself. So the packager writing the e-build specifies the slot number. And normally the slot number would be zero. But in a case where you're actually dealing with multiple slots,
27:42
it normally corresponds to something like the major version. Or in the case of Python, you might want 2.5, 2.6, 2.7, 3.1, 3.2 all installed at the same time, and the slots would map exactly to those major.miners. So it's just a string. It's not numeric necessarily or anything.
28:02
It's just a way of saying, install all of these things at the same time rather than removing the lower versions when you upgrade to a higher version. So in EAPI 2, we said, all right, let's deal with some more of just the weird build stuff that people have to deal with. And so we added these things called source URL arrows.
28:23
The source URL is the download location for files. And so normally it looks something like this first piece before the arrow. But it turns out there's a lot of developers who don't really understand how to make packaging friendly, who don't put a version on their tarballs, for example, or who name their tarballs in such a way
28:42
that the suffix doesn't map to what the file format actually is. Like, this .tar might actually be a .tar.gz. And so how do you deal with that in a way that doesn't force people to download them manually so that everything else works nicely after the fact and so you're not having hash collisions because everything is hashed?
29:02
And so we say, OK, foo.tar.gz has got an MD5 and a SHA-1 and a SHA-256 with X. And so you download the thing, and if it's not versioned, then you can't do that, right, because all the different versions are going to have the same file name and there's going to be five different hashes for the same file name,
29:23
which doesn't work out too well. And so we have this thing that says, OK, download this file with this name but actually rename it to this other name so that your hashes work out, so you're not overwriting files with each other, and so on. And then we also added this idea of use dependencies, which is not just depending on a certain package,
29:43
because it used to be before this. We would depend on a package, and then in the functions within the ebuild, check and see how that package was built. And so you would depend on it, and then you would start building your package, and you would say, let me go look at that library, see if it was built the right way. If it wasn't, I'd fail and I'd die.
30:02
But that sucked, because what would happen is you would die three hours into a 12-hour upgrade cycle, and then people would come back the next day and they'd be like, what happened here? Why did I die? I wanted that thing to keep going. I don't care whether it doesn't have the right feature. You should have just skipped the package and kept going.
30:20
And so we said, yeah, you're right. Let's add a way to depend on that properly instead of hacking around later on after we've actually started the build. So let's depend on the library built with the right feature in the first place. And so it's about simplifying the developer's job but also making things easier for the user
30:41
and pissing them off less, basically. And so we've got a lot of different styles of this, and they all basically map to build with this flag, build without this flag. If I'm building with this feature enabled, make sure my libraries underlying me are built with this feature enabled, or vice versa.
31:00
If I'm building without this feature, maybe my libraries also have to be built without this feature because they may be incompatible features. And let's pop up an example of this one and check it out quick. R5.1.
31:29
In this case, you can see again EAPI 2. Everything else looks very similar. You've got your basic metadata. You've got your inheriting your E classes for convenience functions and so on.
31:43
But right here, you see they name their tarball arbsource.tgz. No version numbers or anything. Cool. Thanks. Which is a serious pain. And so we just say, okay, let's have an arrow, and let's name it arb-5.1 because that's the name of this package which this thing expands to.
32:08
Let's see if we actually have any... Oh, cool. In this case, we've also got use dependencies as well. Depends on a bunch of stuff. You've got some logic in there. If you want OpenGL, you have to get this, this, and one of these things.
32:24
One of them is the Mesa library built with motif support, and the other one is this libGLW thing. If you avoid that, then you get this and another thing. Does that make sense?
32:42
Cool. That's two. I guess we're going to have to move a little faster because I have ten minutes left and three more to get through. Here's some more stuff in two. Here's the blocker syntax that somebody was asking about earlier.
33:01
We've got blockers that can be installed on the file system at the same time, or ones where the first package has to be completely gone before the second one can go over. That's like, okay, let's block this really hard. Let's double block this thing. We also added more granular builds.
33:21
This turned out to be really nice in terms of letting the packages be shorter because we started out having just a big compile function, do all of the build things. Then we said, actually, it would be nice if configure and make were actually separate functions so that if the make part is normal, we can just let that happen and not have to write that code.
33:42
But if we have to tweak around with configure, add some more arguments and so on, so every time you have use flags, you always have to deal with those for configure. We have the separate configure function where we just basically call configure with the right arguments for the features we want, but then make just happens and it all works. The same thing, we split out the unpacking from the pre-pairing.
34:03
Pre-pairing is basically, for the most part, patching. Don't worry about the unpacking. That just happens by itself. But we're going to have to apply some patches, maybe tweak some autotools files around, do an autorecomp, and so on.
34:21
The other thing that really made developers' lives easier was adding defaults for every one of these functions. All of these steps are called phases. These are phase functions with unpack, configure, compile, and so on. We made it really easy to do the default behavior and then tweak on top by having a function within each phase that's just called default.
34:43
If you're starting the configure function, you can just say, okay, default, that's it, and then it runs configure and everything. Then you can play around a little bit afterwards instead of having to manually specify the configure part. We'll pop up a quick example of this.
35:01
Cairo, whatever version that is, 1.8.10. Hopefully we've got some default functions in here.
35:22
Here's an example of prepare. We don't specify unpack at all. It just happens because it knows what kind of format the file is. It downloads that file and it says, oh, that file's got the right suffix. I'm just going to unpack it and put you in the right place. Then some patching happens.
35:40
Let's see what we've got. The same thing with configure. We've got source configure. You don't see source compile anywhere. It goes straight to the install. The compilation just happens. It uses the default information. The configure part we have to tweak around, skewer around based on what the host is, append some C flags, pass all our configure arguments
36:01
based on the use flags, that sort of thing. Now, 3, I'm not going to bother showing you this one because we're running out of time here, basically. What we did with 3 is we were going to add a ton of stuff. We worked on it for a year and a half. Then we were like, this sucks. We're never going to get this out the door.
36:21
We'll just do something now to help people who really need something and then keep working on these other features. The main thing we did with 3 was prefix support. This is a Mac here. It's actually running OS X right now because I haven't installed Gentoo on it yet. It's got Gentoo installed on it in a prefix. In my home directory on this thing,
36:41
I've got a Gentoo subset on its own little piece of the file system. It doesn't install on the root. It just sits there in my home directory. That's what prefix support is. You have to do a lot of stuff to deal with that based on packages that assume they're going to be living in the root and look for other packages from the root on down
37:01
instead of from the top of the Gentoo installation on down. We had to do some tweaking around for that. Now, 4 is a big one. We added a ton of stuff, again, trying to make developers' lives easier. Thinking about, okay, some of these things are just ludicrous where it was just like a historical accident that we didn't have it.
37:22
We've got this install function, but the default behavior for the install function was doNothing. You would build the thing. It would be sitting there in the build directory, and then if you didn't specify an install function, it would just sit there in the build directory, get deleted, and nothing would happen, which is ludicrous.
37:42
Finally, we said, okay, we're going to make install doSomething to make sense. It's going to install stuff by default instead of doing nothing by default and just leaving things there. We added a bunch more capabilities to basically fail fast and fail early so that users, again, aren't getting screwed
38:01
two or three hours into this four- or five-hour-long thing so that you can actually run some code from inside the packages before the full installation starts happening while it's still at the dependency calculation and asking whether you want to proceed phase. Some more things for use dependencies.
38:20
Here's one of the questions we had earlier on. We added this non-fatal prefix to say, okay, actually, the normal behavior we want is to die by default if anything goes wrong. We shouldn't have to specify that. Instead, we're going to make that the normal behavior, and if we don't want to die for some reason, for something that truly doesn't matter,
38:41
then you have to say non-fatal before that. Then we added a new variable for basically dealing with installations that have to be interactive for some reason or another. There always seem to be a few of them out there for some weird packages, and we're always adding things to deal with all the weirdness things
39:01
that obscure packages do. In some cases, they have to be interactive because you have to accept some license before you can proceed, for example, and you have to actually visually see that license. You can't just accept it implicitly. Let's pop up Git Quick and take a look at that.
39:23
Whatever version that was. Git 1.8.1. You've got EAPI 4 up top.
39:43
All this other stuff is pretty normal. You've got some bashiness. You've got inheriting some functionality from e-classes, your default metadata, your dependencies.
40:04
I'm pretty sure there's a non-fatal here. Here's an example. If we're running the test suite and it doesn't successfully clean out beforehand, let's not die and just keep going, because nobody wants to fail out
40:20
because the test suite directory didn't clean up first. Everybody's going to want to install and have something that works, even if that doesn't work properly. And then a bunch more stuff. One really cool thing was we've got package dependencies that can go in these hierarchies
40:41
and do ORs and mutually exclusives and so on, but we couldn't do the same thing for features, for use flags. We added the same capability to specify feature dependencies in the same way we could specify package dependencies, because sometimes features do depend on one another. I've only got one minute left, so I'm not going to show you that,
41:02
but I hope you get the idea, because it looks exactly like package dependencies. Sometimes you have a stack of features, none of which work without the next one down. If you've got GNOME support, you might require GTK support first, is a great example of this. EPI 5, which is the last one we came out with,
41:20
adds capabilities to do rebuilding better when library versions change, which is the main idea here. You've got these ideas of sub-slots and slot dependencies that are smarter, so that, for example, if the slot version stays the same, then I keep working, but if the slot version changes,
41:41
then it will forcibly rebuild all the packages that correctly specify that they require that slot. That's really all I'm going to say about that, so we can just wrap up. EAPS 6 is something we're now thinking about, thinking about how can we keep making developers' lives easier and users' lives easier.
42:02
I'd like to thank you for your time. Do we have time for questions? Anybody got questions? It seems like a lot of this stuff that you have built in
42:22
is set up for autotools, but what about other stuff like scons? That's a great question. I was talking about e-classes, which are basically our libraries of functions, and we've got e-classes for dealing with different build systems. Autotools is by far the most common, so our default functionality is set to deal with that.
42:41
But we've got a scons e-class, for example, or a CMake e-class, where you inherit that e-class and that replaces all the default functions. Now, source configure and source compile are set up to deal with scons or CMake instead of being the default autotools behavior. We try really hard to make it easy to deal with that.
43:01
It's the same thing for version control systems. We've got ways to very easily build from live version control systems using e-classes. If it's CVS subversion, now we've got Git, we've got BZR, Mercurial, and so on, so that you just inherit something, say, okay, the repository is over there, and it knows how to get the code,
43:21
how to download it, how to build it, and everything. Yeah, yeah, then DOS upstream. Well, GitHub doesn't DOS, though, right? Given how complex it's got now to write an e-build file, is there anything that, looking back,
43:41
any features you wouldn't add if you were doing it all again? The features that I kind of regret, we needed them to have a complete system, but are the things with really complex sets of syntax like all the different ways to use dependencies, for example, make it a little bit complex to remember, like you've got an exclamation point in front
44:02
and a question mark in back, and there's an equals sign there somewhere where most people actually have to look at a reference card when they want to figure out how to get it right, and that's exactly what you don't want. But that said, we try to keep things as simple as possible, but no simpler, right? Sometimes there's just not a way to do it simply,
44:22
so we try and just do the minimum we have to do to get the job done, but not adding extra syntax for its own sake. Yeah, so it's easy to write e-builds if the upstream behaves,
44:43
so if you get upstream to change, then it's easy to write e-builds, so it's just kind of hand-in-hand with the distro and the upstream. Yeah, yeah. Upstream cooperation definitely makes doing packaging a lot easier, no question about it.