both source and port. I'm responsible for Packaging and the last three years we have done a couple of small changes in the port 3. And I'm going to talk about that and to explain

what we have done, why, what is coming next, and what changes you can expect in the next years. So first, I'll give you a couple of numbers about the port 3. So the numbers should be

two days old, which should be quite accurate. We have more than 24,000 ports. A few of them are broken, not that much. We have a couple of deprecated ports, so if you want to save them before I rip them out, have a look at them and try to

either find a new upstream or look why it's deprecated and what it's going to remove. The reality is the package we managed to build, because on all those ports, some are just for one architecture or one other. So the reality is, last Wednesday,

the latest build was around 23,000 packages built for 8 and 9. We have a bit less packages for 10 and 11 because mostly it's Clang refusing to build because it's using bad codes and there

is a lot of errors you can fix. Most of the time it's quite easy to fix. So if you want to improve that, you can just go to portsmon and have a look at the packages that doesn't build and try to get one, fix it. The repository is not only used on FreeBSD, it's also used on

Dragonfly since one year, I think, about. And they managed to get 20,000 packages, almost 21,000, so they are pretty close to the same numbers as we have on FreeBSD. And I think there is a couple of issues that have been fixed recently, so the next build

should be even better. We have since December created a new branch in the repository for which we build packages. It's called quarterly. It's a branch where we keep stable ports

for three months and maintain them for three months. So we just commit into that branch, build fixes, security fixes, and runtime fixes. We maintain them for three months. Right now, we just build them for 10. So if you're using package.ng and you can change the repository

instead of saying I want the latest packages, you replace latest by quarterly and you get packages that just get the security fixes. In three months, it will be a new snapshot from the port 3 and we will rebuild packages on that and we will keep them for three months.

It's an experiment. We want to provide stable packages because a lot of people are expecting them, but it's also a lot of work for committers to be able to maintain that list. So we are experimenting

one branch by one branch for now, so one branch three months. If it works, if we manage to do that without too much workload, then we will probably push the support for one year and we maintain four different branches during one year. So the port 3 is great because it allows us to

build a new package. The port 3 has a lot of problems. I'd like to say had a lot of problems. We have solved a lot of them in the last three years.

So it has bad reputation compared to some other package system. Why? Because we have a lot of deprecated code. We are very good at adding new things all the time, but we never remove the old things and we never make sure that all the port 3 is up to date with the new things.

We have changed that. Make is not the best tool to do what we do with the port 3, but we have it, so we will keep it, but we have to know how it works exactly,

how it does things in the background so that the port 3 is efficient in terms of performance, in terms of features using that tool. That's something that was left behind for a long time in the port 3 and there was a lot of inefficient code and a lot of inefficient behavior in the

ports because a lot of people don't really know how makes work. It's not just some variables and do things. It's way more behind. For example, we have way too many targets defined. I don't remember the exact number, but in the basic port you have something like 300

targets defined and all of this cost because make has to go through all of them and with bmake it's better because bmake has, if I remember well, a hash table to how to get them, so you can get it fast, but with the old make we had it was a linked list, so each time

you have to go through the whole list again and again and again. We have too many files to parse and we parse those files too many times. Each time you do include bsd.port on something, it will reparse all the bsd.port.mk. This is totally inefficient.

As the port 3 has grown, there was little care done in the infrastructure to to be able to have a clean separation and between what you predefined, all the variables,

all the things and then what you will do. So bsd.port.mk has become a huge mess of things and it's really hard to say oh I need to add this target, I will add it in this place, not in this place because I don't know it should be done before pre-mk or post-mk. It's really

hard because it can blow up because one variable is not yet defined. So we have to separate to make sure that all targets are defined only when all variables are defined. We had a lot of old badly maintained ports. Basically when someone is not interested anymore in the port,

they just say okay I dropped my maintainership on it, I put ports at or I just don't reply the mail so I still have my my maintainership on it but I don't do anything. And we keep those ports in the port 3. They are often broken, outdated, not using the new features,

not up to date with the new way we do things in the port 3 and they are taking a lot of time in the clusters, they are taking a lot of time everywhere. So we don't need to have the port 3 being something like web archive. We just need things that works and that people care about.

We had some arbitrary limitations. For example someone one day decided that FreeBSD could not load libraries where its name is dot so dot one dot three dot four.

Someone decided that we can only have one number after dot so. This is totally wrong, we can prop we can properly handle whatever name is the name of the library. But in the ports 3 we decided in that case to limit everything to that name,

meaning that we have a lot of hacks everywhere so that the port when it builds the library change the name to just have one number after that so. We have a lot of complex code. Because we have added new features or trying to fix things on the flow without a

large vision of what we will what do you really want that thing how we want that thing to behave then we are building more and more complex code. So we need to and we have to go through

all that complex code try to figure out what is the real problem it tries to fix and write something cleaner and simpler but we don't have anything to enforce policy. For example if we had a new macro that does a lot of fancy good things we had nothing to say

okay I had to change the whole port 3 so that we don't have three four five different version of something doing the same thing. We have a lot of inconsistencies in the name the options in in in the way we are trying to fix things in the port for example

for the ports to put the pc file in the right place each time people are reinventing the same in a different way and sometimes in a bad way. So one of the first things we have done

has been to trim all the dead code because you had a lot of dead code in the in the port 3 to in order to to make the bsd.port.mk more readable and more performant. So for example we have removed a code that was handling make version there was

a change between 3bsd4 and 3bsd5 in make and we had ports to handle that we have a code to handle that that's totally not necessary anymore we have removed that we do not handle any category page for a long time anymore but we still have a huge we still had a huge code to handle that

we have removed that we have a lot of obsolete macros that are not in use anymore in the port 3 and but we still we still have code to handle them we have removed that we have fetched in the base system for very very long now so we removed the ability to fall back to ftp

and all the code that goes with that we have changed the option framework i will speak a bit more about it but we have changed the option framework so we have removed all the complex code to handle the old option framework and we had a lot of old unnecessary anymore uses

like uses to handle x3 x3 is gone for long we have xorg for very long so we have removed that so we have also tried to figure out how we can improve our make usage it can

each time we can we can save one second on the building is very important for us because one second times the number of ports can give you one more day to build the whole pot stream so one of the things that has recently been committed into the into the

framework is we do not anymore define target all the time we define the target and then we create the target and then we really define them if they are needed by the port

we don't fall back anymore on do nada which will run the bin true for nothing we just say okay this pot doesn't need to do anything in post patch we just don't define post patch that allow us to remove around 80 targets on the basic port and save a lot of time on

the cluster we have reduced the number of shell invocation in make each line on the target will invoke a shell so it's better to have inline shell but the problem of inline shell it's pretty ugly to read one of the way we have done to one of the thing we have done to reduce

the number of shell invocation is to basically remove all this inline code and all the complex target move it into a script and having one line calling that script so we invoke the shell only one that save a lot of time on the clusters we are trying to reduce we are trying

to reduce the size of the infrastructure the size of the files that will be passed by make one of the thing is we have introduced what we call the uses uses basically is a very very small make file which is loaded if and only if a pot needed so basically it says it's

try to do one things one thing so it's easy to read you know exactly what it's concerned and how to you can easily understand it you can easily maintain it and when you create a pot

just say i use this feature so only in that case i will pass this file that reduces the size of bs.com that reduce the number of things we have to pass and that's better for performance we have also um reorganized a bit uh the code it's not finished yet but we have

and um i have pushed all the target all the target or most of the target definition in the end of bsd.com so that when we are actually doing something with the target we are sure that all the variables all the testing whatever you can do on variables list

whatever is done before and it's very easier now to say okay this part is for the target i know what i'm going to modify there and i know the impact on the rest one of the other

thing is we often do modification on files like changing the shebang changing stripping binaries converting from those to unix and we used to do that file by file

if you have a port you're converting from those to unix and you're doing that um on i don't know 100 source files it's highly inefficient so most of the new code try to always use xargs and that improves a lot of time in the clusters

we are now trying yes well when we first try to build the on the new machines

the pottery we were around 24 hours for a full build now we are 18 hours for a full bit so we saved around six hours but it's hard to define what exactly helped in what place because

we also improved the code of the the builder itself since but basically we save around six so we have now we're we are now doing some cleanup in the pottery to avoid keeping all the old stuff we don't care anymore because it's broken or whatever in three years we have

removed around uh it's a bit more now we have removed about um 5 000 ports from the pottery and the number of pots we have is still growing we have finally managed to remove gnome one

yeah we still had that gnome one we removed caduce kde3 we still have xmm s and unfortunately we still have at least one user of xmm s so i'm not sure it will be removed too

we have also reduced port duplication a lot of ports are duplicated each time they are changing their abi or have a lot of change in the api so that we can still have the ports that have not been switched yet to the new version of the said library be able to build and be usable

the problem is one day all the ports using the old libraries are using the new one and no one care about the new one so we kept having for icu for example we had five different versions in the pottery only one was used icu is a complicated beast it takes a lot of time to

build we don't need so we now try to take care about the versions we have in the pottery and to remove old version we also try to avoid having always a numbered version we can have something

like if you have a look at virtualbox you don't have virtualbox version a version b you have virtualbox and virtualbox legacy so that each time we go to a new version we do it in the we push the old version into legacy and that avoid us forgetting about removing the old version we don't need when it's not supported anymore reduce code complexity well make is

quite can quickly be quite complex and quite unreadable trying to factorize code in make and reuse code in make is also very complicated make is not done to

really be able to factorize but we still managed to do it we had complex code to handle the different build system like gmake make ninja and whatever other things exist they were all

basically running the same command with just a few arguments different so what we did is okay it's the same command few arguments different why do we hard code each time why do we duplicate it code and why do we hard code each time the name of the builder just create a new macro which is the name of the builder and if you have arguments you do not

support just over overwrites the other variables doing that we removed a lot of duplicated code the other good thing about that is now if we had to change we had to add a new environment variable for the build system we just have to do it in one place and all the all the

targets will benefit from that in one place before that we used to add it for gmake and then we forgot about ant and we forgot about ruby whatever and thing like this we also with the

uses we also try to share the tips that usually you'd you'll had from a commuter that knows that oh i know that this is often failing you should do that and duplicate duplicate that code in or create a very complicated thing in a bsd.something.mk for example we had

gnome hack which was in bsd.gnome.mk you could have only with used gnome so people do working on something totally different than gnome most of the time didn't figure out that we already have something to fix their problem but it was in gnome so what we did is we extract that into

a small uses we try to name it with a name explaining what it does we document it in the handbook and now we have a lot of things to help you to help you doing things for example we have desk hack that's something i stole from the open bsd port which is something helping with

staging stageifying ports that are not dsd aware i will speak again about stage in a few minutes we have pass fix with which is what was called before gnome hack which basically goes through the make files for autotool port and try to figure out common

paths that are wrong and that doesn't match is what FreeBSD wants we have a uid fix which allows our make system to install thing as a user we have shebang

fix to be able to fix all the shebang so that we point to the right place instead of each time having each maintainer adding a new set something we have fixed it depends so now we do not require having that so dot one number but we can have a full normal line

we have almost fixed lip tool because for the same reason we had a problem with lip depends we had a problem with lip tool because someone told the little guy that we had this limitation

we don't have in fact so they wrote lip tool that way the problem is we don't have this limitation and if you go through glib ports for example you will discover that with our version of lip tool every single minar version will bump the library number which is very not good

that's really something you don't want because you will end up having to rebuild all the ports depending on glib for nothing what we end up doing is instead of going through why lip tool does that why we have this problem we say oh let's write a new hack and we we wrote

lt hack lt hack i don't remember uh both yeah so we wrote we wrote a couple of hacks plus all those things that was the manually port by port by other people where we could just we could just fix lip tool so what we ended up doing is we created a new uses which is called

lip tool what it does is it goes through all the configuration files all the new configure files and apply a couple of said lines to fix lip tool and we don't have this problem anymore and we are working on upstreaming the change so that our main lip tool will be properly working on

3bsd by default and we have now that we have the stage directory we can automate a lot of tasks we couldn't before that that helps to enforce policies or help maintainers to check for example if binaries are stripped if there is a script with the wrong shebang if we are

referencing bad pass if all of this kind of qa and it also helps to simplify things until now

we were installing everything on the system and then we try to go through each man page to compress it so we had to define all the man page right now we can just go through the stage directory and say oh there is new man page i can take all of them compress them do what i want to do with them and we will soon do the same thing for the info page we have improved the qa

with the stage directory we have the new stage qa i was just saying about before it's not activated by default because it costs a lot of time but if your maintainer just add developer equal yes into your make.conf and then it will be activated by default it takes a lot of things

you i really recommend that you add developer equal yes into your make.conf we have also been able to handle the stage configurator the stage support very very quickly i think if i

remember well in package source it took something like three years it's not finished i thought i've seen a male saying it was finished recently okay okay so what we have done is in four months we have converted 62 percent of the pottery and my plan is to say in four months

everything should be converted if something is not converted there is there should be only one reason a bug in the framework and then we have to fix that bug if not that means no one cares about this pot and this pot will be removed from the pot

string and we have the new bsd.sanity.mk to help people to keep in touch with the new things to be aware about the new way of doing things the new way of defining

a feature or a functionality in the port we now report them what is deprecated what is new we can report them that they should use this uses because their port have a desktop file so you probably want to register the desktop the desktop file on import

installation we can also warn the users that without nls is deprecated and unsupported we now have with a new option a very convenient way to define always the same way and consistent way global options like i don't want dogs i don't want it's not oh this time it's no pot

dogs i know this time is without x11 oh this time it's i don't want something so we can warn user when we remove things and we can run maintainer when things are have changed or things are not supported anymore and we have added two new shiny targets which will help a lot of people make make p-list that will just create the p-list for you you just have to

review add eventually some p-list sub but that does it and make checker fonts you do an update you do make stage you have everything in your stage you do make checker font and it just gives you what files you have forgotten in your p-list and last thing we are now able to package

without the root credential so you can just go somewhere say make package and you have a package some 100 built as a user we have improved consistencies through the new option

framework so what it does is basically allow the normal option we had before but it also had single option you say you need one and only one option one or zero among n options we have the radio which is one and only one in that list group option is just for convenience and

saying okay this is all concerning the same thing i can choose how many i want in them but i want them to be packed together so that for the user it's cleaner and we have a multi option which say i need at least one but probably more in that options we also added a feature

few people knows about only the apache port is using it for now but if you're a maintainer and you had the package dash help in your ports tree you have the dialogue for the option you you it have one and you have a new dialogue opening with a description a help you can provide

a long help you can provide on on the port so if you're a maintainer you think your options are complex to understand just like this we are deprecating deprecating knobs all the knobs are inconsistent used or not so we are removing it we will go through option for everything

we promote uses instead of adding all the time using into bsd.py.mk just write a new use we can add it test it the good thing is if you modify your use you don't break all the pot 3 but only a couple of pots so it's easier to maintain easier to get in because even if it's

half broken half working we can add it and then improve on the flow and we have introduced default version that's for users right now if you want to specify the version of php you want when you're building you have one thing to add to make.conf but if you want to specify

the version of ruby you want it's another thing in another string now you have just one macro which is default version and you say luo equal 5.2 and you have 5.2 by default you say python equal 3.3 and you have python 3 by default and every single thing with multi version will

go through that way slowly so we are each time we are improving those ports we are adding the support into default version so for users there is only one thing to care about we have added optional first i just show a quick example on how it helps it helps in two things

first it's shorter so it's easier to read second it it avoid having to include twice bsd.port.mk which improve the speed of make because it doesn't have to sparse it twice and last i find it more readable and more understandable we have way more options

than configure we have configure enable configure with we have configure off configure on we can do that for cmake for all the environment variables for for a lot of things have a look at the it's in the handbook have a look at the handbook you

have a lot of a lot of helpers most of the time now you just need the single included in the end nothing more how things were done before we had staging so basically you when you start

you just fetch so it tries to fetch then you you the next target is usually extract and all that goes with you extract you configure you build you think you will install but you mess your system there is different way of messing your system with the old way first there is the files

that are not properly tracked that's a problem but you can live with it it's not good it should be fixed but you can live with it but there is also make install fails in the middle it has installed a couple of files somewhere in your file system it has broken a lot of things other

things and then you end up with a system a non-working system if it's getx that fails you end up with half of your ports not working because you don't have leave getx installed but you have some things from getx still in your first system how things goes now with

the new way of of doing things with staging we still fetch we still extract we still configure with stage and then we can decide to package from the stage no need to install or we can install the install phase what it does is it makes sure that the stage is

clean that all the thing i want to install are in the stage and does installation only when the stage is clean so if getx fails it won't be installed on your system and you still have the old version of getx and your system is still working this is with package ng with

package install it's a bit different because package install so what next we are dropping support for package install package install is now the bottleneck for improvement in

the post tree and for improvement in in in the way we handle the packages we have we are doing way too dirty things right now in the package even in package ng just because we have to support the way package install did things so we will drop it and we will drop it for all release of free bsd at the same time we were aiming at doing that for may this year

but it's we haven't announced it so we don't have time we don't let leave time enough for people to to migrate so we will probably push that to september this year the reason why we aim

that may is because every single version of free bsd release but 8.3 has a bootstrap and is able to install package automatically but free bsd 8.3 doesn't have and free bsd 8.3 is out of line in may

we will provide sub packages instead of having tons of cute ports tons of php ports you will have just one php port you do make and it will create all the different packages and it will provide a way for the user to say i just want to build this package it will say

okay this package needs this package this package and only build what you need you won't build everything but you have one single port to maintain and you can create multiple ports out of that we add provide requires people are often asking why we don't have flavors yet because openbsd has flavors provide requires is a way to do the same thing but it's in my

opinion way better provide require with instead of saying i depend on poll version i say i require poll the binary pearl so there is no reason why i need this particular version of pearl or this

particular version of pearl right now on our package system when you install a per script it will depend on a version a precise version of pearl so you just say now require poll and the poll say i do provide poll and you can have it properly you can also say i'm image magic

so i provide image magic and i provide both image magic and image magic uh without x11 and image magic x11 without x11 will just provide image magic without x11 so if you need

image magic with x11 you say i require image magic with x11 doing that you can do exactly what the the flavors are doing but in a simpler way we will provide auto p-list maintaining the p-list is painful for most most of people

the thing is the p-list is very important for package install because package installs it as a script and it follows the sequence to do things with the enough line of package install we don't have the limitation we had before on the p-list so we will provide a way to say pack everything that is in the state directory you don't need to

pack to specify because i know that and we provide a way to say oh wait i need to apply this keyword i used to do in the p-list for this file so you will just say i have a special file all the rest you pack it normally it's a bit like what dibian does in in the bridge system

and we are going to cross compilation and to have a working cross compilation right now i'm able to build on an md64 machine i don't remember the number it was quite close to 10,000 to cross build 10,000 ports for

arm and i'm going further and i want to be able to cross build at least 90 of the ports tree but cross compilation is quite complicated it's simpler to do one by one but if you want to

automate that and to make sure that the framework is properly getting the right dependencies and not doing too many things it's very complicated and right now all what i've done is not able to go through the pottery like this because it will require too many manual

things from the users basically when you're cross compiling you need a cross a target arch you need to say i'm building for maybe 64 whatever you need a destination directory where your libraries and binaries for the target are installed but you also need to have the dependencies

smart dependencies because if my port is depending on python and not linking to python i just need at the moment i build the port the native python i don't

if i'm using libxml2 i will need to be linked to the the version the target version so i'll have to have libxml2 install both on the native and the target or sometimes it's just i need i

only need the library on the target that's quite complicated to do we have two way to handle cross compilation we have cross compilation and we have emulation we want cross compilation for everything so basically we'll have a new target which is x build safe and say this port

is safe for cross compilation just do it but there is some very complicated ports to to cross build and for those ports if we can't manage to have cross compilation working we will just mark them as unsafe in that case it will switch instead of cross compiling we

will just use the qmu user emulation for this particular port we have a lot of work that has been done by stacy son on qmu user emulation so the bsd user emulation at least for free bsd is now working for mips and we have a modified version of pudria to be able to handle this and

we have a new kernel module that is not committed yet but will be soon which is able to say if you are going to run this for example a lip64 binary then prepend qmu the right line before that

so we can use without having to emulate the whole box we can use emulation to be able to handle those ports and we can go even further we can say okay every binaries but the compiler will be run through the emulation so we can still have the cross based compiler running at the native

the native speed and have the rest emulated so that the the overhead of the the emulation is not that high that's all do you have any questions which one it is in it is in that's

a summary of what i did for the last three years and what is coming so basically most of

the cleanup has been done there is still a lot to do but yeah all done except what was well except that part and right now for the staging we only have 62 percent of the ports so

we have still around 9 000 pots to convert yeah the handbook is up to date we try to to make it up to date it was lagging a bit behind for for two years last two years because

i'm not very good at documenting but right now we are but right now we have people that are going through my commit and say yeah this needs to be committed this needs to be documented

overhead for package install we do that because package install has no way to do another way package is able to use the directory as a package so what it does is basically the same as creating the package but instead of putting everything into a tar file you put everything into your system there is no problem just go through the problem

you want and send a pr and if your pr is still in gnats after two or three weeks go through irc and yells to someone say hey but yeah we we have a lot of pr and

we lack people to go through gnats and have a look at prs and try to take them and commit them but basically just go on the mailing list and say why my PR is still not there is it not good is it or just try to find a committer somewhere in the channel or a mail directly you know that

i don't know you have something using java then you send a mail to one of the people you know that is using java for for example we saw whoever is interested in java because he works on that and you say what do you think about that the thing is also for we have now a lot of good

infrastructure to help people which are non-committers to work on on our port 3 we have redpots redpots is basically a repository where you push your it's it's a svn repository you can put your work in progress ports and then you can say okay try to build it on every single

version of 3bsd support it and it will send you a message saying okay you have this problem here this problem here so you can prepare everything properly into into redpots you don't have to create your own to set up your own tinderbox you don't have to do too many things yourself on your box you can use the resources we have and we propose to the users

yeah no redpots.org it should be one day redpots.3bsd.org

i have no idea i'm not a maintainer of redpots i think there is already something for that they're not i don't remember because i don't use much redpots i have my own build system but

i think there is an option it's not trivial and i'm not sure it's documented but probably send a mail to to dk or or matt and ask them yes it's on my laptop

and my head because now the reason is because it's rather complex right now so

it needs a modified version of pudria it needs a lot of manual patches into the port 3 it it's really not user-friendly yet it's and it needs a lot of cleanup because i've tried a lot of direction i have i have been looking at what package source is doing i've been looking at

how gbn is doing how others are doing so i try different direction until i have something that is able to build more than 5000 packages with 5000 packaging including complex one now i think i have the a good direction i need to clean up to make that user-friendly

and as user-friendly as we can and then yes it will be available right now you can cross build um pretty straightforward i can send uh or put a wiki for that for mips using emulation because pudria is already knowing about about all these qmu things and you just have to

grab stasis and patches for the the kernels there is a new kernel module and there is a new module and it does all the magic by itself so mips works but for others not yet and

qmu user emulation for arm is totally broken and someone needs to have a look at it and basically it's not that hard it's mapping syscalls so you have a huge you have a huge use case and say oh i have this syscall it should be that in arm i will call this one which is local one and whatever so that's that's uh a port for the prime for specific to subversion it's not

a global port problem so no nothing has been done i wasn't aware of this problem since i

only used the others version and uh you should still be able to install both and there is not that much thing that depend on subversion except uh git svn is that the one you want

yeah yeah it says you need to talk to the maintainer or subversion so that he fix that the patch is yeah yeah it's subversion should definitely go through the default version so you

can you can set and the other thing is subversion support renaming the binaries so version svn become becomes svn i don't know one seven for example and then you can i've both installed without any conflict so nobody is working on the as far as i know nobody

is working on that but i think i think center pr center pr read that and both maintainer of of subversion should have a look at it and probably fix thank you so thanks all for coming

this is the end of the bsc dev room at fosdom when you go out please take out any trash you find laying around even if it's not your own thank you