Is distribution-level package management obsolete?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 199 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/32553 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 2014178 / 199
2
3
10
11
12
13
14
17
18
19
23
24
25
27
29
45
46
47
48
49
50
51
52
55
56
57
58
65
67
68
70
72
74
75
76
77
78
79
81
82
84
86
87
89
90
93
94
100
101
102
103
104
106
107
109
110
111
112
113
114
115
119
122
123
124
126
127
128
129
137
139
141
143
145
147
150
154
155
158
161
163
164
165
166
168
171
174
175
176
179
182
183
185
188
190
191
192
194
195
196
00:00
Repository (publishing)ResultantTemplate (C++)VideoconferencingInformation securityInclusion mapOffice suiteServer (computing)ChainSoftware developerStructural loadMaxima and minimaStress (mechanics)Real numberProcess (computing)Computer configurationPhysical systemDirectory serviceDirection (geometry)Link (knot theory)Data managementCartesian coordinate systemEndliche ModelltheorieVapor barrierStaff (military)Inheritance (object-oriented programming)Address spaceCASE <Informatik>Goodness of fitCore dumpEmailWeb browserScripting languageTerm (mathematics)Different (Kate Ryan album)Point (geometry)Computer fileVotingFigurate numberTheory of everythingNP-hardConsistencyPressureSource codeDomain-specific languageObservational studyVideo gameNumberTraffic reportingMereologyQuicksortContext awarenessAreaComputer architectureInsertion lossRandomizationSineHierarchyPoint cloudSoftwareGame controllerForm (programming)FingerprintDistanceFormal languagePerfect groupWordScaling (geometry)State of matterArithmetic meanConfiguration spaceMatching (graph theory)Object (grammar)Service (economics)Normal (geometry)CuboidVolumenvisualisierungException handlingInstallation artGraph (mathematics)Expert systemVirtualizationFood energyVolume (thermodynamics)Moving averageSoftware bugIncidence algebraSphereSocial classProjective planeElectronic mailing listMultiplication signSlide ruleKey (cryptography)Bit ratePlotterFreewarePrice indexNoise (electronics)Sampling (statistics)Single-precision floating-point formatOpen setMedical imagingLevel (video gaming)MultiplicationDemosceneCopula (linguistics)GoogolMessage passingTranslation (relic)BitModule (mathematics)Software maintenanceAnalogyGreatest elementSystem callSoftware testingMetropolitan area networkArchaeological field surveyVirtual machineState observerLoginOnline helpRevision controlBuffer overflowPower (physics)Stack (abstract data type)Hacker (term)Data miningDigital electronicsPhase transition1 (number)Right angleInformation managementWindowInformation Technology Infrastructure LibraryRun time (program lifecycle phase)Distribution (mathematics)Operating systemRuby on RailsFile systemWeb 2.0Vertex (graph theory)TwitterDistribution (mathematics)FreezingGeneric programmingGradientBuildingMathematicsStandard deviationLibrary (computing)Nichtlineares GleichungssystemContinuous integrationPixelSpacetimeCategory of beingTouch typingSolid geometryMultitier architectureINTEGRALWebsiteInstance (computer science)SubsetData storage deviceFiber bundleVulnerability (computing)Dependent and independent variablesDebuggerCurvatureStatisticsView (database)Group actionRadical (chemistry)Integrated development environmentUniformer RaumSpectrum (functional analysis)Disk read-and-write headFluxBus (computing)Covering spaceAlgorithmDemonLatent heatCloningSoftware engineeringContent (media)Expected valueCodeOcean currentSinc functionShared memoryOpen sourceVarianceUsabilityComputer-assisted translationAnalytic continuationGraphical user interfaceSet (mathematics)EncryptionPerturbation theoryMobile appFront and back endsMeta elementLecture/Conference
Transcript: English(auto-generated)
00:06
Alright. Are we good from the AV crew? Perfect. Thanks. I apologize for my voice. It was a great night last night at the beer event. And I got here about five minutes ago after scrambling to catch the last bus
00:23
that would get me here in time. So I won't have any references to earlier talks from this morning because I was sleeping. And probably too drunk to get a talk anyways, so that was good. It's great to be back. I love coming to Fosse every year. And I've been doing Gentoo stuff for a long time now.
00:42
I started, I became a Gentoo developer, I think it was in June of 2003. So I've been doing this just for a long time. I've been thinking about it for a long time. And now, in the past couple of years, I was fortunate enough to become employed by a company that lets me do research and think about this stuff full time. So now I work at RedMonk.
01:02
My job is called industry analyst, which nobody really knows what that means. But what it means to me is that I can spend all of my time doing research about the state of the art in software development and how that compares to what people are actually doing and where they actually are. And now I'm able to sort of bring that back to my initial love and my initial open source love of Gentoo
01:21
and say, look, where is Gentoo? But not just that, more importantly, where is everybody today in the distro world? And where do we need to go? And why are so many developers running Macs instead of Linux? And part of that is just because the experience sucks. So we're going to
01:42
just sort of walk through that today. And Joe is going to see some of these slides for the third time, although I'm saying something completely different. Because I've given three talks in the past four days. But it will not be the same content. I promise you that. So some of the questions, and I'm trying to avoid bullet points,
02:01
but sometimes I just want to start with a few of them. The key questions for me and one thing about where are distros today? Where do they need to go? How are actual software developers in the real world doing things? And how could distros do a better job of catering to them? How are users doing things? How could distros better cater to them? Because there's all of these things happening out there
02:21
that we, as distros, and I am curious, how many of you are volunteers for a Linux distribution? Just raise your hand. Excellent. Perfect. How many of you think your distribution has solved all of the difficult problems? Again, good. So you're here for a reason.
02:43
You want to think about this stuff. And feel free to catch me after and we can have beers and talk about it even more. But the things I want to go after today is where are developers? Where are users today? Where do they need to be? What do we need to do to build the right kinds of things for them?
03:01
And less importantly, done at the bottom, is what do we want? It's great to build a distro that's friendly for new maintainers to get into the distro but it's more important to build one that people actually want to run. If you don't have people who want to run the thing, you're never going to get anybody who wants to become a maintainer
03:20
because it's this whole funnel thinking about things like, nobody becomes a container magically without being a user first. So we need to build software that users and that developers want to run. And that's not happening in the way it has to today. I'm going to start with a little bit of data, and I apologize for the black and white, but I have no excuse. I did a little bit of data
03:44
mining across a few different data sources and you may not know it but when you talk in places like Hacker News and do Google searches I'm searching for what you did. I'm looking behind the scenes and trying to understand what are people doing and why and some of this data is more reliable than others, so if you don't like one particular data source
04:04
that's fine. What makes it interesting is the trends across multiple data sources and that's also what makes it more reliable to me. I can look at this stuff and say, ok, Google Trends, fine, if you don't like that, that's ok. But you look at the trends across Google Trends, across Hacker News, across Stack Overflow for example, across GitHub, I've even done some mining of IRC logs
04:24
that I've had on my machine because I've been doing this stuff for long enough that there's some data in there and you can do some fun things. And then across, in this example one of the others is Linux Journal, where they actually ran a survey of what distro are you running. And what you see is not really surprising. There's a ton of Ubuntu users.
04:44
Just loads of them. But what is interesting is is Arch, right? And somebody pointed out on Twitter, I think it was last night or this morning, I don't think there are any Arch talks here. How many of you are Arch users? So what's going on with that, right?
05:02
And it's one of those things where you've got this set of traditional distros the set of new distros, and even the new ones I don't think have solved the problems that need to be solved. And that's what we're going to talk about today. But if you look at this stuff, you see some interesting trends emerging in terms of, if you think about this and try and build some intuition about
05:22
what are these people like who are using these different data sources at least to me I think, Google Trends, that's just normal people. At least insofar as you can be normal and be running Linux at the same time. Now, Linux Journal, alright, these people are interested in technology, they're a little bit more technically savvy. Hacker News, they're actual developers
05:43
they're not just users and enthusiasts. And so that was sort of my intuition with this. And what's interesting about looking at this data is that intuition holds true. You get trends that sort of make sense when you think about where are the developers, where are the enthusiasts, and where are the more normal people running Linux. And I don't think any of that last group is here today.
06:05
So we're just going to look at data from them and talk about them without seeing the rarely observed normal Linux user. If you look at this stuff you see some trends that make sense. You see things like, wow, there's so many developers running Arch. But in terms of
06:23
enthusiasts and normal users, there's a little bit less. Whereas you go look at Mint and I love to make fun of Mint just for, well, we won't get into that today. But in terms of normal people running Mint, there's loads of them. People love Mint. It's super easy it's super, it just tells you, like, here's what you need to do and we're going to do it for you
06:43
and you don't need to think about that and you don't need flexibility. I think all of these searches included Linux, say it was Mint Linux, Arch Linux, Debian Linux, what have you, and not GNU Linux unfortunately, so if RMS is here, I'm sorry for that.
07:05
Yeah, so there's ways to do searches such that you can keep out some of the stuff you don't want to happen, although it's never perfect. Like, for example, searching for Arch gives you all of the architecture stuff related to Linux as well. You have to deal with it and do some controls to see, you know, if you actually look through the results, you realize
07:24
that after you look through 100 of them, less than one of them talks about architecture and the other 99.9 talk about Arch. But it's interesting to look at these things and say, oh, this is where things are in the closest sense we can get to real data, because almost nobody
07:43
actually installs PopCon or something like it. We don't know how many Linux users there are. We don't even know how many, what the breakdown looks like across different distros. This is about as close as we can get to that sort of thing. Where you look at multiple data sources, you build this list of, ok, it's pretty consistent across a lot of them. The only one where there's
08:03
a great deal of confusion is Mint. I don't know, is this number more right or is that number more right? But in almost every other case, it's fairly consistent and consistent with at least my intuition of so many people use Ubuntu. And unfortunately for me, not so many use Gentoo anymore.
08:23
But 2004 was a great year for us. And you can look at this a little bit differently and say, ok, if Hacker News is the zero point, what do the other data sources look like in comparison? And again, you see things work out
08:41
sort of in the way you would expect. Some things are much more biased toward developers, some are more biased toward normal users, and some of them what makes it really weird is some of them, like Arch is a great example, apparently enthusiasts don't like Arch, but everybody else
09:03
does. So people who just enjoy playing around with stuff, I don't know. I'm not sure how to explain that. But most of this stuff does actually make sense when you think about it. But I want to get away from the data and start talking about ideas. And if you look at just trends over time without any access to show you volume, just look at
09:23
raw trends over time. You can get some very interesting ideas of which way things are going. And I'm not saying by any means that there's more people running this or that. But which way are things going? What do people care about enough to talk about it? And I put in red
09:42
everything that's going up or remaining stable, and in gray is everything else. And in fact, it's interesting to me, in part because there's two new distros, and I'm going to call them new even though they're not that new. Two of the newer distros are going up. And then CentOS is actually pretty stable, which is interesting to me.
10:02
I'm sorry? The scale? Oh, the time scale. This is since 2002? 2003? So it's about the last ten years or so. Which is why Mint basically doesn't exist until more recently.
10:22
In Arch you can see some of the noise that I was talking about, or people were just talking about architectures in Linux back then, and then it picks up. And the same thing with OpenSUSE, because it was SUSE. Until they decided to call it open and make it more open, I guess.
10:43
But it is of value to look at this thing and think about, wow, people don't seem to care quite so much about things like, well, Slackware, Gentoo, Fedora, Debian as they used to. This doesn't mean they aren't running it, but they aren't talking about it. It's an assumption, it just underlies their lives at this point and they don't care. And so that's one of the points of this talk, is how do you keep the distro relevant?
11:04
How do you keep people caring about it? And one of those things is you have to integrate with the way they want to work. And distros built this idea of here's how package management should work, we're just going to keep it that way forever instead of moving forward with the times. And that's created loads of issues.
11:20
And one of these trends that's happened over the past eight years or so is cloud. How many of you have ever run anything in a cloud? And did your distro do a good job of that? Or were you running a Mac at the time, maybe? Just integrating with this new style
11:40
of development is something that distros have not done a great job of doing. They don't, for the most part, do things like natively cluster. Like share configuration data. There's this guy named Randy Bias, started a company called Cloud Scaling. He has this fantastic analogy of cattle
12:00
versus pets, where in a cloud world, all of the servers are cattle. And if one gets sick, you just take it out back and you shoot it. But in the world we live in, the servers are pets. They're carefully taken care of. They're certainly not kicked, Joe. Somebody made a joke
12:22
about kicking a cat in the elevator with him the other day, and I think he just about flipped. But distros are still in that world of naming everything carefully and carefully installing it. For the most part, all of these new things that have come along, and I tried the reason I'm here late is because it took me 15 minutes to get past the configuration management
12:44
room. And that's something that people care deeply about. And that distros, for the most part, don't handle well. There's some modules that are written to interface with the distros and run RPM or whatever behind the scenes, but it's not native.
13:01
And at the distro side, we're like, oh, we don't really care about that. I mean, sure, people can run that if they want to, but for example, why don't we install all the distros using Puppet or using Chef? Why don't we just set that on the machine? And that's the way you do the package management. Because that's the way people want to run the stuff. And so they slap on their own package management on top after the fact, and you've got this weird
13:24
mishmash of things happening. Or, okay, they've got CPAN, they've got PyPy, they're installing gems, and none of that stuff talks to each other. And so you've got 20 copies of the same thing in different versions, and things break, and it's just not a great experience.
13:40
And that's the sort of configuration management thing. That's the point, right? Is that developers today, maybe not you, but you're not the only developers I do research on, are looking for an experience that uses configuration management. And it's not today either. I mean, even Gentoo's own infrastructure runs, well, still CF Engine.
14:03
Slowly moving forward to something a little more modern. CF Engine's been around for more than 20 years now. It was 1993 when CF Engine 1.0 came out. And we're still not thinking about any distros here. Do you think you do a good job of integrating with config management? Okay. But that's the way people
14:26
want to manage their systems. Because we're working at high scale, we're working with transient things, we don't want to sit around installing things. And you know, we came up, I think Red Hot initially came up with the idea of Kickstart quite a while ago. And we kind of did that, and then we stopped.
14:42
And we need to go farther. We need to start integrating these things into the way not just the way we develop the distro, but the way users install the distro. I mean, why don't you give people an option to upload an encrypted repository for configuration management? And we could store it for them. It would be easy. We already manage loads of weird servers doing all kinds of random things for a distro
15:02
anyways. Why not make it a better experience for the users? And when they destroy their install, as they inevitably do, and I inevitably do, we've got the data form, and it's ready to replicate. And one great example of this, of how poorly
15:20
we integrate with configuration management, is Chef, which told the Debian folks, you guys are packaging us in such an out-of-date and terrible way that we don't even want to be in your repo anymore. Just please take us out, because we're doing a better job of being up-to-date and having all the packages that matter to us. We're the domain experts.
15:40
Get us out of here. And they did, which was great. They listened. But that's a real problem. And it's a problem for a lot of reasons, because, you know, as distros, we create all of these policies for packaging and guidelines and code for a reason. And so everything works together, and it's all happy.
16:02
And now we're in this weird sort of middle ground where if you've got ISVs, software vendors writing their own packaging, it doesn't understand how to talk to the rest of the distro. And it's in between, in this point where it depends on things in the distro, other things depend on it. But it doesn't integrate right. And so you end up with all kinds
16:24
of bugs that, well, some of my colleagues would love to talk to you about for hours. And so we're in this weird place where, as distros, for the most part, we want to be stable. But software like Chef, especially
16:43
software that's sort of cloud-like or devops-like, you know, there's a lot of innovation happening very quickly. How do you keep up with the pace of that? Because users want to run the newest software. They want the latest features. But at the same time, how do we put out something that's stable? And this is something I don't think anybody's really solved it yet. Some people are taking an effort
17:02
like, if you talk to the Red Hat folks, they'll tell you about software collections where they're trying to put newer software for developers on top of rel. But it's not in a place where that's a normal behavior yet, and it's not in a place that users and the developers running the distros are looking for.
17:22
In all honesty, I'm not sure of the right answer for this question, but it's a question you have to think about and decide what is the right answer for you so you understand what you're optimizing for and what you're getting up. I am curious, how many distros actually have continuous integration set up
17:41
for all your packages? Which one? Okay. So for the core packages. But almost none of them. But that's the way that, if you're a professional software engineer
18:01
working at a modern company, which most of them are not, you have CI set up, and every commit is tested and built. But we're not doing that with distros. And we're not doing things like, how many of you have heard of continuous delivery or continuous deployment? Excellent. Are you doing it? Wow. Awesome.
18:24
Great job, guys. That's a good 25% of you who are doing continuous delivery. But most of us aren't doing it with a distro. And this is one point where I am actually very proud of Genti, because we've been shipping software to users within an hour for the past 10 years.
18:42
But we're not actually testing it. The users are the testers. And that's why they're called betas, right? But we need to start delivering software continuously and tested so that they can get the modern stuff
19:02
but it also works. And we're not. And so we've got all of these different package managers all over the place. Every language has its own. Some languages have like three or four. JavaScript is a great example. I mean, there's NPM, there's
19:21
Bower, there's Jam. How many different package managers do you actually need? And I guess we haven't gotten to a large enough number to answer that question yet. But none of them talk to each other. We've put some work into Genti on trying to integrate this stuff and say hey, let's download stuff from PlaPy and then generate a
19:40
Genti e-build with it. And that's going part of the way toward the right answer. But I think the answer that we need is that package managers need to natively understand this stuff. Having a translator isn't good enough. Because people are installing this stuff from all over the place. And especially more modern developers using things like Node and Ruby on Rails
20:03
as much as we don't want to talk about that here because we're all Libre and we love Copyleft. That's where people are and that's what we need to build for. And we're doing a terrible job of this stuff. And so people end up with five different copies of some of the lower level things in Node, for example, installed by
20:24
five different package managers. And it's just not an ideal situation because you never know what's using what with all kinds of interesting issues. And so what do we do about that? And I think that integration is going to be the answer.
20:40
You've got normal developers today are using all kinds of crazy data stores. For the most part, we're still storing package management data in flat files. Why aren't we keeping it in a more queryable way, in a more reliable way, than just sticking stuff in a directory somewhere in a bunch of text files? It gets corrupted all the time and it creates
21:04
all kinds of problems. And I know some distros actually do, but a lot of them don't. And if you store things in a way that's more friendly and more transparent to the developers running the stuff, you know what happens? It lowers the barrier for them to get involved. Because the distro runs in a way that they are used to working, and so they're more likely to become a distro
21:24
developer, a distro contributor, because it's friendly to the way they want to develop. And now all of these things are packaged, but we don't rely on any of them. And there is a point, too, and I'm going to agree with the criticism you want to say,
21:42
that we shouldn't have to rely on this stuff. Why would you ever want to get a Cassandra cluster up and running just to install a package? Sure, you're right. But at the same time, you have to optimize for something and I don't think that something should be text files at the expense of data consistency. I'm not sure that's the right trade-off.
22:03
You know what else is happening out there in the real world where we're not, for the most part, except for the 25% of you doing continuous delivery, people are doing all kinds of containerization. Virtualization isn't even interesting anymore. Now it's got to be containerization. It's got to be LXC and
22:21
fun front-ends on top of LXC, or on top of VirtualBox in the case of Vagrant. Everything is happening in containers, but none of us are doing anything with containers. I mean, none of the distros, to my knowledge, understand how to work with containers or install anything in containers
22:42
with, I think, one exception, Core OS, which is built around Docker. And that's one of the case studies I'm going to bring up over and over, if I have time and I don't just run out. I'm halfway through. Cool.
23:02
I mean, we need to start taking the software that developers want to use and optimizing for that, instead of pretending distros are a thing that we figured out where they should be in 1998 and just staying there. And that's what we have done, and it's really a shame. Bentu has done a great job on the UI side for normal users, but
23:22
for developers and for technical people, even they've fallen short, I think. So why, for example, aren't we able to install every single application and application stack into its own container? And just use copy-on-write across containers to manage all the dependencies?
23:43
Why is it so tough to start a distro in the first place and just get it installed? Why aren't we shipping a Vagrant box? Do you know, any of you who work on distros, do you ship a Vagrant image, a Vagrant box?
24:01
Which one? Ubuntu. Excellent. So we've got the vast majority covered. Unfortunately, for the rest of us, we're not Ubuntu. I bet nobody does anything with Docker. I know they said we're going to support it
24:26
because of OpenShift and PaaSes are awesome and no-ops and all that good stuff. Can you tell me more? What does Fedora do with Docker? And I'll tell everybody else.
24:45
In the config management room? Oh, so nobody will be able to get to hear it, unfortunately. But perhaps it's on video. So we've got all these things happening out there in the real world. And this is another good example. Well, two examples I want to talk about. One of them is Chef has this thing called omnibus packaging
25:04
where what they do is just bundle everything up and ship it. Pretend the distro doesn't exist, basically, which means we have failed. If they have to go that far, we have utterly failed in our purpose. And our purpose should not just be, Gentoo is great or Fedora is great, Ubuntu is great, but Linux is great.
25:24
And people in the real world have to run stuff across multiple distros, but we're not talking to each other enough. We come here once a year and do it, and then we go home and mostly don't. I've been on this list at free-design.org called the distributions list for a number of years. I don't actually remember when the last post was. So hopefully some of you will go home and subscribe
25:44
to it and start collaborating a little bit more. If that was the only thing that happened as a result of this talk, I'd be very happy. You've got Chef doing things, and Chef keeps coming up over and over. They first pulled their packages and then they said, screw package managers entirely, we're just going to do this all ourselves.
26:01
But there's more and more companies going that way. Chef isn't the first example, it's not the last example. And they say omnibus because it's got everything in it. They just ship the entire application stack all the way down to and including Ruby itself. So they're not even using the system Ruby installation. We failed. Another example is
26:22
FPM. Depending on who you talk to it stands for various levels of profanity. With the second two being package managers. And that's one of those things where people in the real world who have to build packages
26:41
and even want to integrate are having such a hard time with it that they had to write their own solution to build packages because it's such a painful experience. So what FPM does is you unpack a tarball build and install it, and then run FPM and it generates a Deborah and RPM for you. Because people have such a hard time writing this stuff, we've made it too difficult.
27:02
We haven't thought enough about the barriers to entry to creating a package because we've been too focused on making the experience perfect for the experts. And what else is happening now? Well, we've got a lot of people who don't even bother making releases.
27:22
They don't even make a tag in Git. They just say go clone it from GitHub and run it. And this is a thing that I see people shaking their head. I don't like it. You don't like it. But it's happening and so we have to deal with it. People are doing this stuff, again, because the experience is so painful.
27:42
What are we optimizing for? Well, it's not what developers want to do today. We're not optimizing for ease of entry into writing packages, into creating packages. We're not optimizing for people who need to run stuff across multiple distros. We're not optimizing for people who need to do configuration management or want to do configuration management
28:01
and who are sick of editing text files and configuration files in Etsy. But we need to because the distro is becoming irrelevant. There's a lot of people who don't even care. And in fact it's...am I on video? I guess I am.
28:20
It's actually kind of embarrassing now because Windows is doing a better job with some of this stuff. Windows package management has finally reached the Windows world. I just learned about this myself like six months ago. These are Windows package managers. They all work. They do stuff. There are repositories. And that wasn't the case a couple of years ago. It was like go install Sigwin and then
28:44
you can do some kind of package management. And you know the thing that Windows really has going for them is they really understand people who like to click GUIs. They really deeply understand people who like to do points and click and getting a good user experience. And once they're bringing that to the developers who are using this stuff
29:03
there's a good chance that they'll get this problem solved before we do it in the Linux world. And so what kinds of directions do we need to go? And I put up a few examples here. If you actually know the story behind them, you'll know that two of them are at some level based on Gen2 or it's package manager. I'm kind of proud of that that they're
29:23
in the directions that I think everybody else needs to go. The third one, I think there's actually talk on MixOS at some point. I would encourage all of you to go check it out because it's a really really interesting ideas in terms of where we need to go as distros. And so we need things like, hey, everything is ephemeral.
29:42
You don't need to care about package managers or in the case of Chrome the browser, we've solved that for you so you don't need to think about it and we'll just keep all of the data. You can tell us everything. But CoreOS is a very interesting idea. It's super minimal system
30:00
it's designed for the cloud, it's built using Portage, which is Gen2's package manager, but nobody who runs it knows that. Because inside of CoreOS if you care about packages, you just use Docker. And again, it's one of those things where this is what developers want to actually do. They want to use things like Docker, they don't want to care about package managers.
30:23
If you don't make a package manager a good experience for them, they're going to ignore it and go just start installing random stuff and using Chef omnibus and picking a language level package manager perhaps. I know this isn't something you have to do. It's not mandatory for distros to matter. It's not mandatory for them to succeed.
30:44
If they die, it's okay. Nobody else is going to care. But if you and I want to remain relevant, we have to keep up with these things that are happening. One of the reasons I put this text down here, I'm not sure if you can read it, but there's this key phrase here that says Use Docker as a package manager to build and push your app.
31:04
Docker is not a package manager. It doesn't know anything about dependencies. What you do is you write this Dockerfile script and in that script you run something like yum install xyz and that's the concept of package management. It's not the way we think about it, it's the way people actually want to use it.
31:23
They don't care about package management. They don't really care about even about things like eventual consistency, like configuration management does for you. They just want to get stuff up and running. Go check out all of these if you haven't yet, because these are the kinds of ideas that people want
31:42
today. This CoreOS thing is just going nuts. People love it, because it works in the way people want to work today. It does things like and they use Go, of course, and Go is, I don't know if you've heard it yet, but it's the trendiest language ever since Node.
32:05
They use Go, they have this sort of clustering daemon using the Raft algorithm that maintains consistency of configuration data across Etsy on every single installation. It's based on Docker, and these are the technologies people are interested in using, people who aren't distro people.
32:23
We need to be there for them, otherwise they're going to pick something else. Yeah, and bundling is great. One of the things that's really not done yet, NixOS does, which is really cool, is atomic upgrades. Install stuff somewhere, flip the bit, suddenly everything's updated
32:42
at once instead of breaking halfway through the process, because you have libraries that change and you haven't upgraded everything that depends on the libraries yet, and we've done that forever and we still haven't fixed it. But they did. It's not mandatory for us to succeed, because they will. So what kinds of things do we actually need? We need to understand
33:04
bundling, we need to be able to have some kind of insight into what's happening inside of those bundles so that we can integrate with them and understand what's there and understand if there are security vulnerabilities in the 27th version of Zlib because that's what's happening out there. People are bundling in things like Zlib and like OpenSSL
33:24
and just name your library and it's being bundled so many times you have no idea. That's not going to go away. So what we need to do as a response is to understand those bundles and integrate with those bundles, and integrate with the language-level package managers and integrate with things like Docker and things like Vagrant
33:42
and some distros are doing some of that, but almost nobody's doing all of it. You know, store our data in ways that people expect it to work if you want to get more developers. It's not necessary that your community stay healthy or that it grow. It's fine if it shrinks, nobody else is going to care. I lived it.
34:02
Integrate with configuration management. Why not? It's a better way to deal with stuff. We're still not doing it. We're still in this world of pets where that server over there is named pretty and you carefully maintain it and groom it and configure it and spend...
34:22
I used to run Fluxbox a long time ago and how many hours do you think I spent configuring my menus? You have no idea. And then it turns out I decided I had other things I have to do besides screw around with my desktop environment, so I just installed GNOME and forgot about it. And now I run two applications.
34:43
A terminal and a web browser. And I don't care about what else is happening over there. And so there's this difference in world views between the developers running us and us. And they're code centric and we're system centric. We need to think about what's in the middle. It's applications. And that's where we need to
35:02
meet. Just thinking about the applications that people want to use and run and live there instead of living at a systems level. And then maybe we'll all get along. So I'm about five or ten minutes short, which is great because I hope I've generated some discussion. Thanks for your time. And you can hit me up on
35:22
IRC or Twitter or one of my twenty different email addresses or wherever you like. We can take questions too. Yeah, questions now as well, of course. Thank you. Raise your hand if you have a question I'll bring you the mic.
35:47
Anybody? We need it for the video I think.
36:01
So how much time have you spent looking at tools like Juju and OpenStack Heat? Because they're designed to solve pretty much exactly this problem. Well, yeah, but Juju is, and I don't know, does it run anywhere but Ubuntu right now? Okay, then it's not going to solve this problem because people in the real world have to run and deal with multiple distros. That's true. So maybe it can solve that
36:23
problem. And one of the things that drives me nuts about software in general is NIH syndrome. And I'm sure it would not be a huge amount of effort to add multi-distro support to Juju. Probably the biggest issue would be getting Mark Shuttleworth to say it's okay. So take a look at
36:42
OpenStack Heat. It's not distro specific.
37:02
One of the difficulties of packaging software that comes from multiple sources is where to place the files. And what are your thoughts on the file system hierarchy standard and what distros should be doing about abiding to it or rethinking it?
37:22
I think FHS is great. And I like it and it's another one of those things where at some point roughly ten years ago people decided it was good enough and then it was mostly left to fall by the wayside. It's kind of getting picked up again. It's useful to have these things like consistency. It's actually
37:42
surprisingly hard to parse and figure out where you're supposed to put something even after reading it. And in fact that's part of our recruitment quiz for new developers is figuring out which subdirectory of var are you actually supposed to put certain things in. And it's not as clear as you would like it to be. But it's
38:02
definitely something that's useful at Gentoo. We obey it in so far as we feel like it makes sense or is up to date with current standards. That's the problem with these things that got written ten years ago and have had very few changes since then. Like I said, I think it's been picked up again. I hope it goes somewhere because it's great.
38:22
But you know what's going to happen with it is nobody who is not a distro developer will ever read it. And that's the problem. We need to build tools that encourage the right behavior rather than writing long documents that tell you the right behavior. Some of this that I'm hearing is really developers are the problem.
38:41
But no, developers are never the problem. Users are never the problem. It's always your fault. It's always my fault. We're doing it wrong. I hate to use the devil's advocate phrase, but a lot of the things that developers don't want to do is because they want to treat your operating system as their own little playground and assume that
39:00
nothing else but their application exists. Which is one of the things that is problematic. Yeah, but I mean the fact of the matter is these are the users. They're going to do whatever they feel like doing. It doesn't matter what we want them to do. They're just going to go on and keep running Mongo and losing data or whatever. Well, one of the problems that we have
39:23
is that there are two completely different, in many cases, categories of users. And that's blurring a little with some of the DevOps stuff. But you've got the users who are the developers who want to build with all this latest technology. They want to do whatever is the native package management for that system. And they want to get it out there quickly. But then you also have the other side
39:42
of the equation, which is the deployers. And they are almost entirely distinct in the real world right now. You should, okay, before we keep going with this, I'm sorry to interrupt. I have a slide deck on SlideShare about this exact topic I gave this talk three days ago in London. So go check it out. I will. But my
40:02
point is that in the real world, you have these cool technologies that you want to be able to use, but if you want to put them into a business where it's going to be the difference between making a million dollars and losing a million dollars in stock trades, for example, a fly-by-night
40:22
bundled package with known vulnerabilities is not an acceptable deployment scenario. And so that's a place where the distros are much better at it, but we can't convince the developers that they're not producing something that can be deployed cleanly. Right. Yeah. I agree. Goodbye.
40:43
Thank you. How, as a developer, do I care about my security updates to find packaging in, say, Docker? So I ship my code in this little
41:02
bubble, and it's got a bunch of dependencies within that bubble, but then, as you say, there's a security issue with OpenSSL. I don't want my end users to be caught out by that. They're data exposed just because I didn't care about package management. So how, as a developer, would I deal with that? Right. And that's kind of an interesting quandary, is that people
41:22
care, but what's been happening is that a lot more of the effort has been thrust upon the users rather than the creators of the software. I mean, there's a lot of upstreams nowadays that aren't even bothering to issue security releases when there are security issues. They'll commit them
41:42
and push out a release, but there won't be any working with search or what have you to do this stuff. We have time for one more.
42:02
How can meta distributions help to fill those containers? I think it's through some of the things that I brought up, which is this meta distributions thing, that's been Gentoo's thing for a long time, so I'm not sure if that question was specifically asked for
42:23
that reason. Thank you for that great marketing opportunity. People are looking for a certain level of flexibility and they want to do different things with the containers versus outside of the containers. Most of the distros are not good at being installed inside of a container right now. One of the things
42:44
that Gentoo and a number of OS X package managers do is let you install things into a prefix where you don't need the full system. You just need a certain level of it. So you can have package management even on a system that doesn't care. I think Debian, one of the Debian BSD things might do the same thing, I'm not sure. But just being able
43:03
to have a user who has package management in their home directory. That's an important direction. Not because that's an interesting use case in itself, but because it deals with other use cases like the Docker thing. So you can get some level of package management within every single container.
43:26
This could help for security updates too. So you should all run Gentoo. Thanks for coming everybody, I really appreciate it.