Okay. Does this work? Okay, so thank you for coming. Okay, that's right, okay, that's going to be too painful, but...

Okay, never mind. So, thank you for coming. Just a quick show of hands. Who in the room is a Debian contributor, Debian maintainer, or Debian developer? Okay, and who is a Debian user? Okay. So, I'm the current Debian project leader. This talk is not...

It works? Okay. Okay, so I'm the Debian project leader. This talk is not specifically about Debian. I hope that even if you are not involved in Debian in any way, it will be interesting to you.

Another title I briefly considered when submitting the talk back in December, but I rejected, was this. But the truth about this talk is really that it's about hard problems I would like someone to solve. So, I'm just sharing my, trying to share my to-do list.

So, it started when I saw this quote from Matthew Miller. The problems that these tools are trying to solve a decade ago largely are seen as not only solved, but kind of boring. And so, it was featured in LWN a few months ago, and I started thinking, and I was thinking, okay, which problems should we try to solve today as distributions?

So, let's first take a step back and look at what distributions are in the pre-software world. So, there are many upstream projects, different kind of upstream projects as well. Some of them developing one big piece of software, other developing

closely related smaller pieces of software, some of them related developing collections of software, and we have users. And in the middle, we have distributions like Debian. And what distributions do is take software from upstream projects and push it to users in the form of, transform it, and push it to users in the form of packages.

And in the other direction, distributions get feedback and bugs from users and then to forward it to upstream projects. But sometimes the picture is more complicated because we have some distributions with derivatives, like Debian and Tails and Ubuntu.

And some distributions even have their own derivatives, which has their own derivative or users. So, what's the role of distribution in the pre-software world? So, what we do well is we provide the UniFade interface

for users to upstream projects using package managers and upstream and neural networks to hide all the subtle differences between between each upstream project. And in some case, we end up replacing upstream projects completely.

We also integrate upstream projects, resolving all incompatibilities between them, and we kind of end up cleaning free software from problems that upstreams often ignore. Well, distributions are a kind of big washing machine where we put upstream projects and they end up cleaner without, with many issues such as

ABI compatibility, legal issues removed. There are things we don't do that well, and I will talk about that in my talk, provide an intermediate support layer, act as mediators between upstreams, derivatives, and users, and meet all our users' needs.

So, can we do better about all of that? First, about meeting all our users' needs. So, quick poll, who in the room during the last year had to install something from sources using an official packages,

gems or CPAN or whatever? Okay. There's a lot of software that is not packaged, and we are losing the race. So, these graphs from Modulecounts.com shows a number of packages in various

language specific repositories. So, Debbie and Jesse, which is going to be released soon, has 21,000 source packages. So that's that line. So clearly, there's a lot of software that is not packaged in distributions. There's also often not the version you need in the release you are using, or in backports.

So, in Debian, we have backports. We have currently 974 packages in with the backports, 21,000 in Jesse. So, if you end up, well, if you end up being able to use a specific package from backports, you are actually quite lucky. So,

how can we aim, 400% coverage of our users needs? And I think that two parts to these questions. First one is get more efficient packaging and providing additional levels of support.

So, first, one slide about distribution contributors. With distribution contributors tend to be more DevOps and pure developers. Most of them, especially in volunteer projects, started by scratching a niche, which is easing the installation of software they need. And actually, that's a need that is very frequent among sysadmins,

but not so much for developers. So, we end up, we have a community that is excellent at dealing with obscure, dirty, unique stuff in various languages like build system, etc. And forcing various things into working together. But we end up also with a community, which is not necessarily not so great at designing and writing complex frameworks.

People who like to do that kind of work tend to go to other projects. And also, we tend to, we have a tendency to act using, like sysadmins, using

adding layers of glue to get things to work together. And to avoid deep refactoring, because that's not usually what sysadmin does. It means just to take things together. And that's also why it's harder to recruit compared to other projects, because that mix of sysadminning skills and development skills that we need in distributions is actually something that you cannot find,

or don't find in typical university curriculum. So, I teach at university, both in sysadmin oriented courses and development oriented courses, but those are generally very different sets of students. And finding someone with both skills is really difficult.

Especially someone young, because most of, usually they, one side is learned through education, the other ones through experience. And finding someone who learned both early, it's quite, it's quite hard. So, let's look at the Debian packaging stack. At the bottom of it, we have

dpkg-dev and shell commands, like install. On top of that, we introduced an abstraction layer called developer, with all those dh underscore something commands. On top of that, so that with developer

alone resulted in packages that were, that shared a lot of similar code between, between, yeah. And so, we decided to add assertion layers. The first one was CDBS, which was introduced in 2003. And there was another one introduced in 2008, dh.

On top of that, there are tools like dh-mcperl, python, st-dev, gen2-dev, npm2-dev, or cabal-debian. Cabal-debian is the one for Haskell. Most of them are now based on top of dh. I'm not sure if there's still some of them based on top of CDBS. And

on top of that, we have tools like git build-package, svn build-package, git, that I actually used to, what I'm more, it's not really about packaging code, but about maintenance of the packaging code. So, there are several problems with that stack. The first one is that we are not really moving away from duplicated tool.

When CDBS was introduced in 2008, it was often seen as a CDBS killer. People did not like CDBS for various reasons. So, they hoped that dh would just kill CDBS. And that did not happen at all, actually. So, in that graph,

the red line here is the number of packages in Debian using developer. The green line here is the number of packages using CDBS. And the blue line here is the number of packages using dh.

So, what dh managed to do is actually, it made maintainers move away from developer, but not from CDBS, but still about the same number of packages using CDBS than in 2010.

Yeah, sure, okay. Yeah, but I think that if you ask Debian developers, most of them would agree with you and think that it should be duplicated. Okay, who in the room is a Debian developer and does not think that CDBS should probably be duplicated?

Well, we are also not hiding lower-level tools.

So, those are slides from Joey's talk about developer at DEPCONF9, where he introduced developer. So, he made the point that the surface, the visible surface area of developer was this, 138 items. CDBS added

153 items to that, which was one of the reasons why people thought that CDBS was not such a great idea. And he made the point that dh was only adding 12 items. But the thing is, it's 12 plus 138. We did not simplify the interface that people have to learn to do packaging.

We just added a layer on top of it and required people to know both developer and dh. So, when you look at this graph and wonder what people need to master, to need to know, to master to do Debian packaging,

the answer is all of it, actually. The third problem is that we are not good at maintaining our packaging code. The packaging code is influenced, so by packaging code, I mean everything in the Debian directory, so both

description of packages and the Debian rules files that describe all packages are built. And this code is influenced by changes in the version code, like the new version that does things differently, so you need to change your packaging code.

Changes in the Debian policy and changing in the packaging teams policy and practices, because each team tends to have its own way to do packaging and sometimes there are changes in a specific teams that need to be pushed to all packages. So, packaging is kind of manual merging of those three different branches and

we do that all the time and we have tools for some of it. For example, for a trim code, we have you update, but we don't really have tools for changes, we don't have tools for changing the Debian policy. There are some teams that have ad hoc tools to push one specific change to all packages in the team.

It's quite easy to write a script that just replace something by something else in all of the teams packages. So, we end up with a lot of duplication is in our about 3,000 Perl modules or 700 Python libs.

We also have a lot of outdated packaging code, because this manual merging is not really easy to do. So, what should we do? So, is Henry Kozeny in the room? No, okay. So, oops. So, we should design a higher-level packaging framework.

Probably. And actually, there was one discussed at DebConf called DebDry by Henry Kozeny. That's why I asked if he's in the room. And when DebDry is run, what it does, that's from the DebDry documentation. So, it moves the content of your Debian directory aside.

It chooses and run an automatic degonization tool, for example, dhmake-per. It applies your manual changes on top of the autogenerated Debian directory to produce final source package. And it stores the original Debian directory in Debian DebDry, so the process can be reversed. So, in a sense, your packaging code becomes the output of

dhmake-per or dhmake-ruby or whatever tool is specific to your kind of package. Plus, the diff of Debian for all the manual changes that you made on top of that. So, I think that's clearly a step in the right direction to use the output of a standard tool, higher-level standard tool,

as a basis for the packaging work. But still, the maintainer is still editing files in Debian after dhmake-per or dhmake-ruby is run. So, it's still low-level. So, diffing and patching might fail in some cases. For example,

if you have new packages, replacing old packaging with a different name, and it's yet another tool. It would make the higher entry barriers. I mean, on that graph, it would be something there, on top of those tools. So, what we should really look at is, look at how to

design a tool that relies on our known working tools and formats, like developer. But I'd say, in 99% of cases, if you think that

programming languages, if you want to learn C, you don't need to learn assembly first. That's actually kind of what we are doing. If you want to learn packaging, Perl packages, you don't need to learn, you shouldn't need to learn developer first.

So, it should be something like a compiler or generator for the Debian directory. And if we do that, it means that we get to spend less time on growing pure packaging stuff, more time on hard, interesting problems. We would lower the entry barrier for newcomers.

We would automate the packaging work, because we'd be able to move more common tasks in the packaging framework. And finally, it would not prevent NMUs, because it would still be possible for bug fixes. So, the NMUs is a Debian process for a Debian name for non-maintainer uploads. That's when someone,

not the maintainer of the package, wants to make a change to a package. And because you could still work from the generated files using the well-known tools to patch some bug in the package.

So, if we look specifically at automating the packaging work, what could it bring? So, we could have automated backports, to stable for most of testing and stable and experimental. We could have automated packages for repositories like CPAN, PyPy, RubyGems, Maven Central, or NPM.

Of course, if we do that, we don't provide the same level of support as for the packages that are really part of Debian. We would not be able to provide security support for them, no manual testing, but we are getting quite good at

automated testing. This is something that would still be an improvement compared to the current state of things, where you have to install stuff manually using third-party tools. So, if we manage to do something like that, it means that distribution packages, so DEB or RPM files,

become, again, a universal way to manage software, which is no longer the case today. we end up being able to package most of the free software world, including every version of it, because it would be quite cheap to just instantiate that for every version available on various repositories.

Which is something that our users need, because they end up installing a lot of stuff from source. So, this was actually discussed on the Debian mailing list over the last two weeks.

So, there's a few additional pointers in that thread. And one, of course, difficult problem to deal with about providing automated packages is the legal side of this, but if you think about it, CPAN or RubyGems are doing it already, so probably we can find a way to

do it as well. So, also, I also would like to talk about other challenges, which are a bit less technical. So, the first one is

application and services versus packages. So, what users want, typically, is working applications and services that can install their machines. And what we provide is working packages. And actually, there's quite a difference between those two things. For simple application services, it's not really a problem. We have a clear mapping between packages and applications.

But for complex services, it actually requires interactions between various packages. It's not so obvious. For example, you cannot get install a mail server. Okay, you can get install ExSim, for example, but not a mail server with

spam filtering plus your IMAP server, etc. You cannot install your cloud infrastructure. Quite works in the behind, with OpenStack, but it's not quite there yet. Our web application using complex stack. We are kind of behind on web application packaging.

And that's a very important problem, to be able to enable users to keep controlling their computing. We have users using the cloud, typically, because it's too hard. We have users using Gmail, because it's too hard to install your own mail server. And it's actually a big, a big failure

probably of most of the free software community for not being able to make it easy, easier to install mail servers or web applications.

So there was a great talk from Stefano Zaccheroli at DEPCONF about that specific problem. You might want to to watch it. So, one thing that makes it hard to package complex application services is that

the technical issue is that packages that configure other packages is hard. How to do that is hinted in the Debian policy. Really, working with that, if you think about web applications and

databases, for example, or web servers, it's quite hard to get it right. So we should look at all the things happening in configuration management and containers, which are really popular topics currently. And I think that, well, we are probably all familiar with

Richard Salmon's analogy between free software and cooking recipes. And there's a kind of similar analogy about packages and cooking recipes. Packages are, in the case of complex services, are kind of ingredients not really the cooking recipe itself.

So we could ask ourselves whether we should package cooking recipes, which would mean, for example, packaging puppet or chef recipes that would automate the configuration of set of packages. We could do that. Maybe we should ship fully prepared meals,

the containers with everything installed and configured that just work. But if we do that, that's not really a preferred form for making modification. If you get something, you don't know exactly how it was configured. And also, if we do that, it's going to be hard to

to come to combine different, it's going to be hard to combine different services or different applications on the same machine, which is also something that is important. Or you can eat buffets. So maybe we should think hard about it and

help users install complex application services by thinking, by inventing something that would be in the middle between packages, containers, the tasks. Tasks are sets of packages, so you can get installed fully working desktop, for example,

or blends. And I think that there's maybe something there that distributions would be in the right position to look at, to fix that issue of people moving away from distributions to move, to run the mailing structure, for example.

So, yeah, a lot of computing today is moving to new architectures, smart phones and tablets. Currently, you can run Debian in Shrutes or in some cases using bind mounts. Cloud infrastructure as well.

The current status that you can run, you can, there are semi-official images for several public clouds. And it's actually a similar situation. It's about users giving up their freedom, control and trust in action of comfort. It's easier. You give up your freedom, control and trust when you choose to use a non-free

smart phone platform or public cloud infrastructure. So, the question we should ask here is, can we help users regain those freedom, control and trust without losing the comfort?

And the question where I have no answer is, what could we bring with a Debian-powered smart phone or tablet? What's additional values that distribution could provide and that non-free smart phone platforms would never be able to provide?

Also, there's a question of semi-official images in clouds. How much should we work with public clouds to provide official images of, well, Debian vetted images in those environments?

And how can we enforce the quality of those images? How do we make sure that, say, the quality of official, so for Amazon, the situation is quite good. So, I'm using Amazon as an example. How can we make sure that the Debian images on Amazon are actually

of a suitable quality? And maybe we should work on a certification kit for cloud providers to help them improve on that. I'd like to talk also about increasing trust in distributions. So,

as you know, well, security issues have been, have had a lot of focus during the last month, or probably you know about it. And our users are actually putting a lot of trust in us. Users are just blindly running our binaries and maintain our scripts.

And are we really that trustworthy? That's not so, yes, we are. So, we are quite good at providing a trustable package manager and an archive. I think that many people have looked at that.

What's, yeah, that's, okay. About development process, quite. We only have 25% of packages. Not maintained using a VCS.

Yeah, so one could argue that's probably not the 25 most important packages that are in that case. I did not compute it compared to popcorn or something. We also only have 5% modifying upstream sources without using a patch system. So,

just to explain that, so in Debian we have So, in Debian, when you, when the maintainer needs to modify the upstream source, the recommended practice is to use a patch system, which means that each change to the upstream source is separated in a different patch,

with a description. And those 5% of packages are just actually providing a single big patch with all the changes, which is not quite good for auditing what was changed. About trustable packages.

So, one of the Debian's dirtiest secrets, not so many people know actually, outside Debian. If that's one way to make it go away, you know, talk about it, is that the packages built by developers are used in the archive, which means that if you are using a package

I maintain and you are using the same architecture as the one I'm building packages for, for example, AMD64. The package you are installing on your machine is actually the one I built on my laptop and uploaded. It's not getting rebuilt in the Debian infrastructure. We did not know about that.

Okay, so there's some work for that, because it's no longer completely true. Source only uploads, which means uploading only the source and gets rebuilt in the Debian infrastructure are not possible, but not naturally yet.

Also, it's not working for architecture all packages yet. Architecture all packages are packages like per libraries that don't need to be rebuilt on every Debian architecture. The same package works everywhere. So there's some work to do here, because it's actually, this is actually quite scary.

But we have a solution, actually, or a partial solution to that. There's a great project called Reproducible Builds. And one of the goals of Reproducible Builds is to provide auditability via a bit-for-bit comparison. So the idea is to take the package that is in the archive,

rebuild it, and see if you get the exact same package. If you do, it means that even if the package was uploaded by the maintainer, it doesn't really matter, because if you had rebuilt it in the infrastructure, it would be exactly the same package.

It's also, well, I think that already is also a good step in the right direction, because it means that if the maintainer thinks about uploading a package with a backdoor to the Debian infrastructure, it's like it will be noticed at some point, thanks to that work.

Yeah. Well, it's a way for people to make sure that Debian is doing the right thing.

Providing that means that you don't need to trust Debian anymore. You can just make sure that if you were to, if you want to check what Debian did, you can audit the work Debian did there. Or for a derivative distribution, because everybody trusts Debian, but you might not trust derivative distribution as much as Debian.

No? Okay, okay. So to do that, actually, if you think about it, it's quite really super hard. Because there are lots of things that get randomly changed in packages during builds, like timestamps,

etc. So there are several solutions to that. There's a file used to record versions of all dependencies used to build the package, but also various tricks to deal with timestamps and randomness. And I'm not going to say too much about that, because there's a talk today at 4 p.m.

by Olga Levzan, who is sitting here. So just go. Yeah, with Manar. And finally, a trustable runtime environment. That's something

for which we don't do much, or almost nothing at this time. Distributions are in a really good place to help run each application in a separate secure container, so that you can isolate applications from each other and make sure that one service that would get

hacked does not result in the whole system getting hacked. That's especially interesting in the context of web applications. That's also something that, well, there are lots of, there's some work going on in that area in the context of web application distributions should probably look at

following there. The last point is about increasing the bandwidth to upstreams and derivatives. That's very important to, if we go back to the picture in the beginning with distributions, upstream projects, etc.

Because that's the way, that's our way to improve free software as a whole and not just our distribution. So the current status is the following. In Debian at least, we have some structured contact points for derivatives. We have the Debian Derivatives front desk

that actively contacts Debian Derivatives to inform them about everything available in Debian to make their lives easier. We have some services to monitor new upstream versions. So, maintainers of packages in Debian can provide a Debian watch file that describes how to watch for new upstream versions,

so the, it is a website of the upstream project. And we have services in the Debian infrastructure that fetch those files and then go, well, go web browsing the upstream web pages and notify the maintainer when a new version appears.

There's some manual forwarding of bugs happening. There are some services to track bugs in other bug trackers, so Launchpad has one, has this capability, so you can link one bug to other bug trackers. In Debian, there's a tool called BTS link.

There are some attempts, so BTS link, I thought it was dead actually, and I've seen commits. So apparently it's still working. I don't know if anybody is maintaining or using it currently. No?

Okay, so you are using it, it works. Yeah, okay. Yeah. So maybe, so it's probably a lack of advertising because, yeah. There are some attempts at facilitating the exchange of patches between projects. So for example, something Ubuntu, what that Ubuntu does, and that is quite nice,

is that the last changelog entry for packages that diverge from Debian summarizes all the changes that were made to the Debian package. So in a few lines, you know, what were the changes in the Ubuntu side. So Debian has a patch tagging guidelines document that explains how to tag each patch with standard headers.

I'm not sure actually how much this is followed. I would say we could use a automated, more automated tests to make sure that this is actually followed.

We used to have a service that exposed all patches in Debian packages. Unfortunately, so it was called patchtracker.debian.org. It's currently done. There's no maintainer. If you're interested in taking this over, probably get in touch with

Isam or the DSA team. Quite, but if you want to point your upstream author to a single page where it can monitor to make sure that he knows about everything happening

on the Debian side, it's not so nice to point to sources Debian net. I think there's a point in having something there, but I agree that maybe one way to recreate it would be to create, recreate it on top of sources Debian net.

And we have some dashboards which point us to other distributions. For example, if you go to tracker.debian.net, there's a new Ubuntu box where you can check the status of your package in Ubuntu. So that's not working very smoothly. There are many upstreams that actually complain that they never hear from the Debian maintainer.

So if you talk to your upstream, well first, it's good to talk to your upstream and make sure that he or she knows what you are doing. And there are also many bugs and patches not forwarded.

It depends on the distribution, but some distributions are pretty bad at forwarding bug reported against the packages. So that's something where a lot of progress could be made. For example, I think that that's really typically a problem that could be addressed in a cross distribution way.

It doesn't need to be a Debian specific solution. Most of it doesn't need to be a Debian specific solution. It would be actually quite nice if we had a way to provide a cross distribution dashboard that upstreams could use to monitor the status of packages in all distributions that

package their software. One problem here is that the naming of packages is not necessarily the same over different distributions, but that's not such a hard problem. It can be worked around or it could work as a starter for all packages, which has the same names.

So to conclude, I talked about several challenges scaling, automating our packaging practices and tools, bringing complex services and applications to users, improve our support of new computing environments, increasing trust and improving collaboration.

So to answer the comment that actually started that talk, I don't think that's boring. There's a lot of interesting work to do in those areas and other areas, but it would be better to release Jesse first.

Thank you. Questions?

If I'm supposed to repeat, you could as well come forward.

Oh yeah, sure. Well, yeah. Yeah, if we add such a dashboard, it could be cross distro and makes it easier to talk to the maintainer of the same package in another distribution. That's something we don't do much.

Yeah, I'm not sure I heard the beginning of your point, but you said that the distribution maintainer should be on the upstream developers mailing list.

Yeah, the thing is when it works, this works well when it's about maintaining one big piece of software. When you are looking at, for example, at the Perl team with 3,000 Perl libraries,

it's not so easy to actually keep track in a manual way of the development of all all libraries packages in Debian. That's why I think that we should try to look at more automated things to look at that. I think fresh meat is dead. I'm not sure. Yeah.

It has been a while, but fresh meat was kind of that. Well, we're addressing part of the problem, but yeah, yeah. Yeah, I agree that

in an ideal world, all patches and distributions should be forwarded to upstream bug trackers. The thing is,

in reality, we don't do it that well. The same is true for derivatives. Every patch to an Ubuntu package that is relevant to Debian should be in Debian bug, but that doesn't happen in reality. That's why I think that we need to look at other solutions

also to kind of replace that kind of things. And if you, from the upstream maintainer point of view, if you have something like that, okay, they're just one single place to check from time to time in addition to bug trackers. Okay, that's painful. They're still better than the current state, which is no bugs get forwarded or a few bugs don't get forwarded.

Yeah. Yeah, so, but that's

isn't that what I talked about, about automated packaging for CPAN? What? Sorry, okay, so I didn't get you.

Maybe there's not something that it's not a problem in Debian, but that something close to Debian could actually provide DEBS because depending on the, so I'm not really familiar with Perl libraries, but depending on the language, it might be much easier to use DEBS rather than installing from source. And if you have a large infrastructure,

you need some kind of packaging to deploy your software to every machine. That's something that well, if you need to automate the building from source using Perpet, for example,

tends to be a bit dirty and error prone. So,

I'm not sure it's that complicated, actually. Our automate, our packaging tools are already working quite well, depending on the language. If you take Ruby gems, for example, it will just, the generated Debian package will already work for probably 80% of cases. So, maybe the last 20% are really hard, but I

don't think we are that far.

Sure, but that's also the point that if you have, if you use, if you start a project by using a Debian-based stack, using Debian packages, and

if you, in the end, you have like 5% of your dependencies missing in Debian packages, relying on being able to stay with a Debian plus AutoBuild packages stack is much better than having to move to a mix of Debian and custom-built stuff. And also, that's,

if we provide something like that, it means that we kind of close the opportunity for tools like RVM in the Ruby world that are just rebuilding the Ruby interpreter from source, because if we get people used to rebuilding from source, there's a risk that

just goes down the stack, and in the end, the tendency to completely replace distributions or only see distributions as a foundation on which you build your custom compiled from source stack. Yeah, well,

I think there are two sides. First,

distributions are in a kind of difficult position because they try to work to meet many different needs. For example, well, booting with a lot of logging is something that many users actually like and need for their daily work. It's true that for some other users, booting without logging is also

could be better, but there are ways to do that. I mean, I'm not sure what's the default in Debian currently, but

well, I must admit, I don't know what's, I don't, yeah, but and about, well, upgrades failing, etc. That's, well, that's a hard problem. I think we are working on it and making improvements on it. I don't think that's something that is

easy to solve and, well, it's mostly incremental fixes to that, but

well, because if we just try to copy what is already done, probably we'll fail. While distributions, well, I think there are things that distributions could offer in that area regarding trust, for example, that are much harder to, for others to offer. So we should probably

be focusing on what added value distributions could bring based on our experience doing other things. Then look at yeah, then just try to copy what is already being done.

But then it means also that a lot of people are okay with compromising in the other direction. If there are some solutions to run phones that are free software based or more or less free software based, more or less is the important part of it.

And a lot of people are actually looking forward to those solutions because they would like to go in that direction and yeah, and are okay with losing some other functionality in exchange of more trust. So other comments or questions?

Thank you.