Okay. Welcome, everyone. This talk is meant to be on reproducible builds. There was one session last year. I hope most of you might have seen it because this time we have a little

less time to cover it. And what's that about? It's we have all the software coming from developers that commit to get repos secured by hash sums. Then they package it up through tar and sign it hopefully and then the package goes into OBS and then some magic

happens and it goes out to the mirrors. But the tricky part is the magic that happens. Do we know if it doesn't insert vectors? And to find out we can use reproducible builds. Because when we really have that, everyone can build the same software at his home machine

where he can control how it's built and if it produces identical results, bit by bit, ideally then it's the same and can't contain any vectors that are inserted only on the build service. And there's also some second consideration where we apply some filters

because we know we have the build host name in the RPMs and currently it's still needed because we want to build know where an RPM was built to trace down errors that are from specific build hosts. So we have this build compare and it knows to fit all certain differences.

So why is it good? So first you need less trust in the build hosts and the OBS. Also, if we produce identical binaries, you know, okay, it's the same so we don't have

to republish it and we don't have to build the depending packages like we have a new Python and then we don't have to build all the Python modules on top and these things. So we get less load on the build servers. For the update channels we have delta RPMs

and they become smaller when we have smaller diffs because normally if you don't have any change you have zero diff but if you have a change you have only a small diff usually. And sometimes we even find bugs that are completely unrelated like these two here that screwed up some packages and they didn't throw an error. They just produced packages but packages

contained incorrect data and each time a bit different and that's why I noticed them and we got them fixed. And typically prevents packages from being reproducible as timestamps.

That's easy to fix because we have a variable called source date epoch and we replace the current time with that value and then we always have this one value and currently that comes out of the latest change log entry date from the RPM change log and that's nice.

That's an image I brought upstream to RPM. Sometimes packages put rebuild counters in there but it's only very few and if you really try to build an identical package you set the rebuild counter to the identical value anyway so it doesn't matter so much.

So .05 order affects some dozen packages I found, maybe even 90 and I started to submit some fixes there. You see some more on the next slide and a few packages even did compile

time CPU detection and that's very broken because if you build it on a very modern machine it won't run on older machines because those older machines don't support CPU instructions that are compiled in them. So that's a real bug and it needs to be fixed anyway.

And during the last year we discovered some new sources of randomness or non-determinism inserted. One is profile-based optimizations which is a feature of GCC where you build

your software with some special flags and you run it a bit and it generates some counters of which functions were used and what it did and then you build it again using this information from the stuff in between. And it turned out that some packages like gzip and bash did stuff in between that produced different counters between different builds

and these different counters produced different optimizations in GCC output. That's a bit tricky because either you completely drop the optimizations but then your software runs

slower or you make really sure that the stuff you do to run your software always happens in exactly the same way and it can be a bit tricky to get right. For example with gzip I found that even the file name you give it matters because there are some two lower functions. The two lower functions depend on if you have uppercase letters

or lowercase letters in your file name and the temp file is sometimes one and sometimes the other. Okay, yeah, RPM. RPM is tricky because it has sizes for directories and you don't see it in RPMQL. It's always zero but in the actual binary they have sizes

and sometimes they vary because your file system grows when you add files and it doesn't shrink when you remove files again. And even ghost files which are not actually part of your package are tracked with a size and a checksum even. So there were some variations

and I guess I will propose some patches to get rid of these when we want to do reproducible builds. The other part, the lower part here is about the link order. So many packages have in the make file a dollar wildcard, something C and then they produce a list

of o files from that and then they link these o files into their result binary and if they just do it as on the left side, it will be a random order and the linker optimizes in different ways and sorts of functions in different ways. So if you want to really

have reproducible builds, you need to do a sort and in Python a sorted list and even the boost only, no, boost owns a build system called gem, had the same problem so I did one patch to one usage there. Oh yeah, there is another build system called

bum and it has exactly the same problem. And the fun part is, I saw the Python glob, there is a libc function called glob and that does sort by default but the Python glob is not using the libc glob so that's why it doesn't have it in it. They apparently

make and boost, they don't use the libc glob for some reason. Not sure why. So a lot of work I did, plenty submit requests, filed for openSUSE, that's for example when we had stuff in the spec file that needed fixing, like calling gzip without

the minus n option, so then I just added minus n to leave out the current time and then it became reproducible. Very easy fixes and a lot of upstream fixes actually went in already. Many dozens. And sometimes I just file bugs because I can't

figure out how to fix it and then I let others have a look at the code and decide how to get it right. Okay, so if you want to test out if a package builds reproducibly

or why it doesn't, then you can use mygit repo which is nicely named to remember reproducible openSUSE. There is also this presentation's source code in markdown and they're using this nice script called ODP down to run that into openOffice and apart

from that there's also a wiki page on the openSUSE wiki giving some pointers to the upstream reproducible builds.org website and some more things. But the stuff I work

on is a git repo mostly and there's a readme telling you how to use it. I also did some patches to oc and build to make it easier to do this, mostly to pass flex from oc to build to kvm and telling kvm that it's already 2018 and I want to build this package in

2018 and then I find java packages to say documentation copyright 2018 which is already interesting but yeah, they do and others do as well. And last year the state was that I was able to build one package twice and it produced

the identical results and this year I built 9,000 something of our 11,000 factor packages and they produce identical results. So that's quite good.

Yet some tweaks that are not in factory yet, like the hostname, the build-host.producible, that's only in the project where I built currently but at some future point in time you might have it in factory if you want to and the other one that's not in factory

yet is clamp-mtime that modifies timestamps in rpm and makes sure that they are not the current time but changelog time. So older timestamps will remain as they are but anything

that's newer than changelog time that gets set exactly to changelog time and then you get reproducible bits for that one. And the plan is to work some more on that. Maybe you want to work on some build-compare issues as well. And finally when we are all done

we might be able to have a whole factory with all packages building bit-adding to kill rpms and it would be quite nice to see that. And until we are there I keep publishing on my web server called rb.zq1de some build results

and the latest one is always the symlink here called factory which is actually a symlink to this version. And in here are the diffs I get from building these few thousand packages that are not yet reproducible and in there you can see what exactly divergent. Maybe

fix it yourself but oh yeah Haskell is still an issue. But also others so it's not limited to some Java sometimes tricky. KDE has a sorting problem where I don't know

how to put it here but overall it's progressing, it's getting better and you're finding more fixes over time. So questions so far?

If any file has data in future then you can get very wrong results for some build dates.

OK, so if our table contained a file that had a date in future? If the source contains a date in future then very wrong things can happen. So it's good to check source and there are some sources of unreproducibility that cannot be

observed in OBS but they can be observed when you are trying to rebuild on your local machine just by RP and build. There are for example language and local settings.

Other problem of some spec files could be that if you have more packages installed than build requires defines then you will get a different result than build with just

packages that are there. So when fixing it's good to think about building also outside the OBS. Currently I'm doing local builds using oscbuild but that's using the same build tool behind that. That's a clean environment where you have no local, no time zone, only the

packages that are build required except if you overwrite it with minus minus no in it. So it's a pretty clean build environment and yes we are not hitting all of the issues that you could hit but at least we are hitting the ones that are most interesting

to OBS right now. So we are fixing them first and when we are done with them we might not filter the others but RPM build is not that much used anymore. The last guy said OBS is so much nicer. For example when I am creating spec files then I'm explicitly writing disable some

feature that we don't have in build requires. So even if somebody builds it locally it will fail. Yeah, that works. Okay, more questions?

Thanks. So are you doing these reproducibles currently only for x86-64 and if not are you seeing differences there between the various platforms that we have in OBS?

So currently it's only this one architecture and I think the vast majority of non-grid reproducibility is independent like inserting timestamps is independent, ordering input files is independent, linker optimizations maybe but it might even happen on other architecture so I think it's over 90% and not depending on architecture. So if

you fix it for one it will work for others. More questions? There's one.

Do we have this linked more visibly somewhere than just on some obscure website? Which link exactly? Well for the diffs for example for the packages. Okay, where exactly would you want to wish it to have this link? For example RPM lint gives us results in OBS and this is not visible anywhere.

I'm not sure if it's anywhere centrally located because for example in Debian when there's a QA page for a package you get a list of is the package reproducible and then you get a diff right in there instead of finding it in a mysterious location.

I can try to add these links in some more prominent ways but we can discuss later. Well I can imagine OBS test case that it shows a warning that this package has never

equal RPMs in output. In OBS or in RPM lint? Probably in OBS because OBS knows the previous result and it could tell that it

had never the same RPM as output. Yeah, if you don't look at changelogs and these things. OBS does sometimes rebuild and it compares the builds but...

So we could collect these statistics and use these as extra input. That's true. Just another question. Do we already have repair-usable builds of debug info and debug source?

Not exactly sure. I think in my tests I currently build without debug info but I think it's even possible with debug info because if your binaries are identical your debug info will be identical as well. We need to check but I think it's working.

I think time is over so thanks for joining. Have a nice day.