Reproducible builds ecosystem

good morning on I'm a solid lives and I will talk about with this will this ecosystem today only yeah you talk so slides some of the fully there but combining this now on models things and then in some of the most confident in Boston deviance and QA on the some things I used to learn this at control organizer that we need to and it yes and not doing those anymore still very much interested to talk about that if you have questions ask me any time on the tall would have a lot of all of these names so we started having this Jenkins instance then made reproducible that an alternative and will boost AUC they're all on the same system it's all the same time and sometimes it's only are on the slides a case on sunday you can enroll in this proposal builds in the end of 2014 arms in May to fall 15 remind me ask respondents funded by the mean of the nation to work on this on but there's many more people in the team working on that I mostly working on this testing framework on I'm not really working on other things cause this testing framework as you will see is quite big so I'm quite busy was the of and 1 other thing which I would like to say some people say using 2 to the difficult using 2 was really super easy up installed to overall the launch and you've told brawler configured was staunchest of community and the net in case you didn't know you so anyway as I said I'm just 1 of these people versus the the theory because this team and this is just me and there's many more people working on and on and equally as Jenkins about these are all the contribute us through the drink stuff and these other contributors who contributed court to the Jenkins study in that thing which did not non related comments so I'm also they're concerned that some stuff on all the other things unstressed of stuff on F-Droid and I will explain what people are working on so this is it started as theory in what has become across this to project Saul as selected know that about you who you contributes to free software game worse you talk about reproducible this last OK it's quite quite many and has contributed to reproducible this creatinine what's appointment good and so briefly explain the motivation for those who haven't seen

drops but yes this what's this talk from the from the last but 1 Congress on which explains very red the motivation was what is the problem we are trying to solve

on so 1 example is was 1 would route exploit and as as ht and there was only the difference in the binary was a single bit they are a was on the core was greater than and had to be greater equal then and the difference in the binary 1 bit you 500 you by binary so 1 that can have a huge impact and they also the lifetime was a kernel module which modify the source in memory but not on this so if you look at the cold the cold it's fine to compile the code we get a vector or complied and so it was a just a proof of concept but it's forces and they also nice to show how much financial sense of doubt to break development machines known if you can had fixed on have exploit 1 machine and it gets distributed and millions of calls you can get whatever keep financial data can get private data and its own to circumvent censorship competent countries have hundreds of millions of but it was really lots of money to create 1 machine could you develop a machine or it could be that big demands of a softer project like W and offered all audible and Otago research a 50 million dollar invites you release in the water and you probably also need it you don't really know what's running when you few machine all the time on the net and you also usually if your computers and on other people have physical access to it and can use these 3 as direct memory from excess so if you if your computers alone you compute or endangered and everybody does that the and another 1 it's last example is there was no from this northern documents the CIA talked about an embedding of modifying its decays in this case of forceful IOS but there's lots of other things on Monday to discuss how they can put it on the back door into the case when developers use this decay to build applications that as a vector in on the applications and what this was theoretically in this northern documents last year there was this speaks called coastline ability which is I respect all which was found in the wired and we said it was the most popular application was made tens of millions of users having the spectral and the source code was fine I was just this decay which was spectral on solution to that is that we rebuild the sources everybody should be able to read sources and get the exact same binary bit by bit identity and we call this reproducible boots and I will build a mall at

so from the the so this been variant package 5 times takes 20 seconds or something and the resides use and we produce a word is that different checks from all the time but then you see on the left the checksums sums and they're all different yeah well from and now I switch engine but reproducibility is wrong the the same the and they should be all identical lot of that would be the case the the instructions for the few

the and we think this should become the norm and we also think before change the main want to change the meaning of free software that is only free software when it's produces this not reproducible it's software India so all the on we have developed some what we started with steady in read by now have some common resources so we now have this webpage reproducible minus this but all where we explain the concept we have documentation how to address issues on we then and the who now gave a talk at the CCC camp last year from way explain many issues in detail the come what what they are how to fix them on the talk is available also answers you see video that if you have read the documentation on all web page or watch the talk that explain how to and the fix the issues we have identified the most insane common problems and the most common problems timestamps timestamps are and get invented by everything you moths this so much so that the binaries but it's often documentation and as we want the whole package to be identically the times and the documentation is equally annoying and basically all the documentation to and that the bit timestamp and these things needs to be fixed and then in another source for variations time so and so on time zones and locates so if you for example and if you unpack the time archive from 19 80 or whenever you want packets and packets of a you look at times only be applied and it's so the fights from ninety eighty get modified by your over time so so if you if you with the different time so that the original that you get different results and this these tuitions time sons-in-law clients can be addressed by normalizing the bit environment some projects to that and I mean we don't live and so on so we want to fix these in 80 and invariant fixes in the upstream sources because we don't want to it fixed and the images of the TU was adjusted log normalizing the bid environment to try to fix the upstream no OK it's also important because they for example assaulting is affected by the OK so French solve differently than John and that gets a wineries and then there's the other issues on where the we have identified quite men but there's also some which we have identified we don't know what this are but to give you some numbers that's all of 20 thousand packages it's like 300 packages in this category so it's really small arrest they're probably hot but the majority of things up timestamps time songs and movies not and this is all explained in the web page and we must talk i will mostly not good for example the soul this there and that daytime since usually not really useful for the user of I thought it just says when is something was the but if this if you can read it at any time and again the same reside the big day becomes meaningless what is important for the user the last modification of the source code and for developers it's also interesting to know the environment which it was created and just having a big data is an approximation of the environment but it's not really describing the environment so it's not useful so we came up with the source state people which is a variable and this is defined as the last modification of the source and this can be used as of the date but as a date because deterministic the source will not get modified afterward it can also be used for random seeds and other stuff in there in we said it from mean change the entry can also that from the last beat commit or the last modification of the fire on has adopted by and that is the previous the Artstein of schooling so using it we also put it into other our we have this written aspect which is like 2 pages document very small very technical describing the stolen must hold this looks like and we use this arms and heads these was already on to it respects state people so when they embed used in the embeddable data going on that the source data people date on and thus become deterministic there's still some negative well with patches spending we together into GCC thanks to the army there's some more of what we need to get in but it's did the adoption is quite good on the so and then on test you should try to get the slides it the mortgage

we

use yeah and so this is the so

on test refusal this 7 or protests deviance stable testing announced experimental unstable and we also know stopped test of Open Government unit is the previously but we looks at all of them soon afterward and weeks he's on the so this 230 Jenkins upwards of Jenkins said on and it's 40 to a screw some Tyson some banish prices using only used to generate the web pages where the best that's just from beta far recently so it's by now it's again 10 best that's because all of the under this result is just 1 best script it's really really simple Court it's also not much on the and I cannot maintain all of these this process so I happily take pictures of you don't need a test set up to pensions when you work on whatever asteroid the robot just send the the patches I would have been you know it's them with whatever breaks I don't care cause it's quite an isolated I explain this in greater detail so what are we test the basically we built a software twice the and then compare and for the 2nd good we introduce variation so we change the host name we change the time so so the time differences more than 24 hours that made us find many of the shows already you change the user name the economists change the you must changed and receive you type is only change some on the moment it on a new 64 think all but will be happening in the months of something we don't test variable fire system at the moment low Andrew a our world s disorder fuse 5 system where the 3 year returns the on off fires in a random way so that will on find things which were and because now systems which return that in runways multitudinous the ways this will find things when you build on different 5 systems on and on in 64 so we have to a host running in the future so now they run in February 2007 to arm and thus the also find these variations only and for the non and test we have a little bit of bit less variations of the moment because sometimes not as is or at best after implemented differently in the of time but most variations are applied to all everything and it's sometimes the big environment just normalize the settings sometimes not and we want to find all these cases and to find the differences long status right to for school on the divorce called takes two objects and compares them and of course because into them so if you give a to whatever 2 rpm which will look into the European find the pdf and the people in the copy and then look into the PDF find Pewsey in there and will analyze this and will display the the difference is like this side by side so you can see on the left is that on the right is the content their own what follows that to binary comparison also and has been there it's available you and the from give me a pipe Iet's package for the arginine notes sticks homebrew it's been tested on BST in all we want to maintain a somewhat distributions and if you want to try it you can just go to try different school but all and give it to objects and to objects can be 2 directories through CD-ROMs whatever you want to objects to for school but compare to it's really really cool and that the score was only used for the body on because for testing whether something is reproducible we just want to compare the Czech sentence and that's it we don't want analyze the contents was analyzing the content potentially error-prone and can be URIs so for checking whether something is reducible interval score was not used not sure whether you really want to use different scope on at totally untrusted input comparing 2 this is definitely find definitions and the the whole set up is running on on several machines in the cloud funded by profit rates so we have more than 100 courses and 300 gigabyte run on at the moment in the ends of mostly testing they're beginning to see this come what is that 64 the 2nd some of the most models the and so this 4 times more resources for in and for the rest I am happy to switch this over cause you're really fast in building 64 so unhappy if the Fedora test used more or the video obstacles or other things we have huge resources and happy to switch resources the way that we can do other things faster and onset of is more funny it's 40 small and also the the stressed very high orange whatever hosted by vagrant together they only draw 100 watts of Paul which is really nice for 50 machines was as 40 machines was assessed these on but it also takes 4 weeks to read 1 deviance into there and we are beginning to so it takes a few weeks while on new 64 rebate boasts testing and unstable and so the only been 1 of them we would 4 days was 4 weeks on on but it's still we can build an undefined issues with just 1 side effect of us constantly rebuilding W and we also find many many bucks very so as soon as they uploaded because other people don't read but the whole lot of all the times of the binomial to file of molecule a books the and

so is so since yesterday we reached 20 thousand reproducible packages and unstable in all current test framework can which is a few from the Lorentz graph shows the unreproducible packages and the red ones of failing to build from source and the black ones are either not for in the 64 dependencies among available or others and the green ones are the ones on then we have no lots and issues on so we have 100 79 categorized issues like what I said time so once in PPI dropped on location this and round and we have me felt almost 3 thousand 700 notes in 3 thousand 300 packages but other and reproducible packages and only 100 39 don't have an old so these are the ones that we could really make anything out of it not all packages was have old are completely are a complete so there might be just a note 1 issue in the packages wishes on but these are the ones which probably on what I don't want to or just have been uploaded history on we maintain these notes and get a sense of the amplifier which then once you commit to it and Jenkins of picks it up is the nodes update the web pages since RISC notification to our channel arm and at the moment it's only Debbie and what we want to have the source across the street president of know what is the issue isn't something it's likely also in the other distribution and this is getting the nodes is strongly take the sum of our would protect sometime and then you can roll for any package that is negative and what reproducible adopted in . net less whatever restless notes that I see is a stressful messes and and look at the package State of Texas package names and because they're being is way too big was nobody cares about 23 thousand packages on we have package sets on 29 on them and they concede that requires only 68 % reproducible so that 10 which unreproducible we have patches from I think for a of to all of the 10 on the other packets sets have better status of Pegasus we have this this essential we have all packages which have a security issue all packages installed under the machines the KDD package set the norm package so that we have all the of our practices all come with packages so we can look at it you'll area of things and say OK this is the state is there if you have more packets sets you think I interest and here is new we have take expected set also include the active on and this is the progress in the Darien backtracking the green box on the fixed box and the orange 1 other blunt fixed once and as a rule we usually fireboxes patches and things were not 95 % of all box and these have pictures we 5 more than a a thousand bucks was patches and there's another graph was them on packages failing to bridge from source because we also know fired I think 607 hundred bucks failing to bridge from source when a packet is updated doesn't bid against all libraries were just there and we see these spot so we also find them the and so on talk about some of this for them in we agreed to always be it in a fixed the past clusters C compiler and other compilers hostile come with them a bit and that the bill passed in the binary when the objects that would so for now being read just a bit anesthetic parcels SMEs but it and then the package name of the source was for GCC we've now find a way to get rid of this so for Jesus you could be an arbitrary passes but for other compilers we cannot do this yet so we still do it do 1 thing the bit explaining the 2nd strip non determinism is another 2 will be rolled which removes non-determinate mystics stuff which can be safely dropped 1 example is time stamps and p ath it can emit in images I forgot what other things that are we just applied the at that help us all packages which could be in the the use that money to mean as in probably that's a tool which is also useful for other distributions to use the it was called already explained as the rest of exploiting children and then sends last year every every single week now has written a report about all or a false on which this patches we rolled packages which kept fixed development of the testing framework on its some Planet Debbie and what you can alter dozens of blocks just subscribe to the RSS feed so we had a summit in reasoning seminar was 40 people from 16 projects and that's where we also renamed this test set up we want would like to have another summit in April home-made lowered step was planned probably would let from if you're interested in collaborating please talk to me the if you want to attend the summit come and we also have to all you will some of quot students in 2015 and they're totally new contribute a civilian as far as I know and they made really great contributions that's really depends it looks very complex but you can also start a lot of money but not without much knowledge you just can't take 1 small package with is broken analyze it and that's where you start and end the war on so all I said is already has been involved flies if you 1 or to be 100 % sure that you can read build the packets 1 way is to create the exact same that environment if it's a bit different that might create the same results so maybe in works 1 way to be sure it's just create the exact same environment on yeah and so for that and we have sort of this I know that code is the line for that I spoke with the 4 developer yesterday and to find a way to do that it's designed for that but it's not well documented hold wouldn't wouldn't G which in the words and out some open wrt recall it's the environment but I'm not sure where they are saving the results that we enhance the snapshots witches 20 terrabytes I thing which has already been packages ever uploaded since the year 2000 was something so you can treat get any Rosen there are managed to all other projects of resolving this found is the soul the half the speed and profiles and every and which are the few better package they used to be a changes file and Hungarian binary packages also written for file it has the checksums sums of resources of the generated binaries and all the dependencies so used to wouldn't and this can be later used to create the exact same environment you have a patch for s the we need patches for people there and other things but we have 1 working Rosen and this is celebrated final so it's simple RCA 22 former confessed me I eye-contact of source name the beta passes the artist's exons and debate environment this always lies at the beat environment consonants check some school and install binaries for the dependencies and for that in to get what this is 85 per cent on our testing framework it's not and then use it now so we have to block as at the moment and once these as of fixed variance would start to become reproducible for real modern testing framework so this the

package has some patches than which piano started to look at them and sensor and what optimistic that was will happen so and that the Darien archive students at the moment accepts up Rulz was that amplifies the throws them away that needs to be fetched that the speed and 5 circuits and then we have parts of them mean reproducible on and we in the end we would also like we want the policy to be changed that Pegasus must be reproducible on forest over here we hope this will happen after stretch so 1st threats reproducibility is still this list and nice-to-have but for Damien tend to on this must things which would be 2017 material and 2016 we want feature changed of sources so shall be reproducible on their it is not as that's a proof of concept we need these 3 is to to the brackets and that there's some other answers we have like we have at the moment we have modified aid packages so it's the package the accessory to me the and 5 other packages was a smaller things on these are the main focus so I hope that deviance stretched will be possible partially reproducible in a meaningful way hopefully beta sensor will be reproducible I don't know what you can exactly on and I hope they get was really select and to 3 months in the next over the next but 1 up of detectors we get finally reproducible deviance part but even then we among their because how to read words come to that how we handle these fit in 4 fighters and user to was needs some design still and accordance most in and now to the nonlinear worked on all the there some minor so we do a test corpus the moment of 250 images for what images to 1 not reproducible was the sea by us pay notes and not looking at the other payload but I think there are visible as well the mid unclear what the next steps on on course called doesn't really binaries so we can say it is reproducible but yeah and it's mostly summonses links this work here I'm open W. T. O. MWI bit stream in which the i-th Texas from all the diversity and Tommyrot packages I think and most of them are reproducible on with the starting pitchers sent upstream last week biotic summer again membrane you want on whether 1 can be a free trade they did environment Fulton w In as it was in the past needs to be test you to be tested on fault wt we could also opt instead of rebuilding twice also you could just be once and compare this with the release binaries but this has not been done it should be noted the it's the that's where the more things Thomas clustering was walking on this serpent his he has a branch now well around 50 to all of these 54 classifiers are reproducible there's need to set the variable to make that mistake that contributors of others this and covered for yes and you need to set K tag timestamp to so state the pulse of FTAA getting re reproducible and use the and I have no idea what we create a bit environment because I'm I don't know everything about many things on the next step is was what to where the previously the base system is not yet reproducible on what almost as 5 patches the port system was already reproducible for 65 % between years ago there was a wiki page saying this that has that once and I'm not sure whether they have all the variations we have in our testing framework so there's a talk later today about the status of previously come and on his your eyes you can go to these webpages now and take the status that show this we have we're testing is then we have a lecture obviously I have no idea what the lecture via status on there's talk today but also says on the web page collectively the would the free software when it's released I'm curious about the talk this the dual of army we have now as bidding feudal while I using mock so far not was supporters of just mock DealBook William um patch up here so that the RPM 5 what includes the b at time of the abuse holes and also has a signature embedded and it's the circuit something we haven't addressed at the time we have addressed in this repository but it's not enough to give reproducible packages there but we also enabling should 23 at the moment and not by the my 24 1 or or or or of there we need to spend will work on it on I'm sure it was almost as reported as attribute is a list of the inputs of the same sources just getting up and ended to be and somebody needs to come and when also this view was very busy and I don't know what the so I'm not the right person to that I a set of this test I'm happy that I understand how complete Fedora pictures of and then there are notes on your reading options to was and he wrote patches for that obstacles package main moment and so on which creates packages that we have not integrated the patches analyticity framework yet which is stuff of February on when the school and S Troy at our last week as such as this consists of Steiner to date F-Droid on this Jenkins of the same thing but the work has just begun and we have techniques which is reproducible or recreating the bit environment done by design have little idea how it worked as a left the talk yesterday to see this kind of thing on what's interesting going which says this is the patient would mixtures which found would a package and compare this was the reason binaries only which is something we should also have for that we and I think that looks very nice but I cannot more it's also works by defining the bid environment normalizing variables there on another mainly my inbox which tells me how to set up these tests which it should read and there's wall on the sold it calling and to what I actually the boat to project who were the 1st who have these reproducible binaries at all it's going was afraid because they are pending leader money now and lots of money bitcoins 4 billion I think is the loss of the colon and the developers so afraid that if somebody has a trolley and binary that they would be blamed so they reproduce so so they can prove the binaries there these really come from the source and same was to what to also want to show mix OSS design this 1 what Europe so this the next release that won't it's only when or something is planning to produce the goods and this probably also some commercial software and other software which this kind of you know some more projects working on that piece turning and so

guess which commercial software is reproducible windmills medical devices and you want to ons nuclear power plants safe driving because no gambling machines have in their lives in Germany and France scanning machines have to be reproducible binaries by all means of and it's also in the embedded in parts of the embedded in the stream of and more binaries they also be visible binaries but and then there's also other projects on hold and so that could be tested using the cause the figure was stuff analysis just use was up here and so I just put a different opinion revolves repository in there so the work 1 2 is waiting for progress in that region open BST I have no idea then to also uses binaries that would also be interesting to read those binaries and from the rest of us so on 1st said there's the things needed to be done 1 problem in the Darien release process as that we've getting unstable now and this becomes stretch later some parts of the Indians threats would be the was wasn't which are not part of stretch and 1 solution could be revealing everything months before what we still have passive not being baby with soft out of sight stretch and it's unclear how we will solve this thing other projects will have the same problem when those rolling rolling distributions especially on then we have this info fires so there would be at 100 thousand flights per variants suitable 50 Solomon was wrong because there's book is right this 25 resource practices and 10 I detectors and part of them are takes all or something so would not be 200 30 thousand 100 less 100 thousand and these are almost as many fights as and the Norris now so we can output is being revised and the Morris and you probably have a seen walls of another neural network we need to think about on test that's and you and we also want to catch signatures because when people read Bishop also assigned the same in 4 5 and say yes I reproduce this is the use of the and I signed so we get lots of more 2nd shows and revoking signatures and all very with the and then rebooting and sharing signed sex sex and so on there has been some ideas floating around how to do it in scale we don't know 1 1 way a systematic reboots the other is that developers just crawl sources and probably for different projects that would be different solutions and put in that in that we have a bit of trust with the GP GP so that power work but I'm not sure whether it's scale really and because every bit as run by large organizations whether a series you was sound lots of bunching as they would have the US Army bit at the Russian on and on the of mostly on the on the on the check some problems with on a no alright good tools to do all we will study and and every and we open source and that is the whatever so we need to figure out loud this example actually way better than these armies and then user to lose their 81 to install this unrecognizable software on do you want to get this package was has not been the biggest before what do you want to do build that a new machine all and how many signatures other needed to have a softer called rebound of of 1 is amount of new 1 3 in which we build those arm then which reversed you trust and how to put this in the virtuous and for all the test set and we wanted to we want walk on hardware for sure but we also want to test other architectures we want other on test more projects of possible if you're interested to have your distribution tested I'm happy to have you been working but we also want to read more people looking at the results were at the moment I only the parts of the 1 parts of Argentina looks because there's only 1 person in each looking at the results so I don't be all of the 1 of Argentina's but just a subset arm and also we really want you don't want to do it twice but rather being ones and compare was released binds if the in and and this is just a testing framework only it's important to work on this but most of the work for it and so this needs to be done on the source of so fixing and that indicates the 3 thousand packages on is also and so to some without having come a long way made quite some progress but this the not we don't know we are not seeing the end of the tunnel on the we we don't know where we're going to get but I hope next year will be here and it's really really fun and thanks to everybody who already country wouldn't and it also has said that's the big problems and as many many smaller problems and if you just submit 1 patch away and it's 20 people and all that will be done at the end of and before your time I just didn't want as a software developer of his problem using the dates you saw state people on that's really important if you want to have more than just this something twice wonderful school and compare that to the documentation is there for them you can store this which chain which I was using the reproducibility of fruit so you get to pick expects you can either ask on the deviance and IRC channel all we also have their own reproducible minus this I answered channel but we and and and a really happy to head anybody to get anything reproducible on you could join the is a listing the 11 a group of people very nice to work was I learn something new almost every day I think something that I don't wanna know and some other fascinating stuff you you find things we found about this software figure to be in the future because the test that in the future and when he does this but people will be made in the future time will pass and that will have and lastly me things on and as many things that we like can you packets is identified shows find solutions so bright patches right of mentation different things or I could start another team might as I said we think of all the words that should be reproducible so if Europe project was not listed here please get started and best to what talks today the selective use in the food levels in the BSD develop room and it's that's basically these are the resources it's area we have no don't so the highest external gets the highest notifications of changes and we have to maintain this means that in many of course but we also generalizing as long as the lost and we also have a Twitter account now for those of you who are into that and that's it thank for listening thank contributing if

you build up here and now and that's the the time in the army and container and also the band holes and what's the signature

in the states will prevent had the opportunity to define the identity of each time you reverted related changes so we need to change of the annotation the so the tools to perform the same constant or through all takes that ends in Section 6 the the history of everyone's the immediately of thank you thank you

so what is the impact of the instance packages that are not listed to this dependencies what about if you have a system status is salts that you foods the builds the kernel explicitly listed as dependencies is that's going on that's not common any more costly but we already did very anomaly and on the environment will obviously dependencies and store these just moment box packages will find them sometimes but these kind of parts are invariant usually 1 lead thanks the thank you thank you the the jaw and my migration is something could use of blacklisted because they require too much resources like liberal office and what is the vision how to hold that what was your thing of all profits because the the the on on on your lips eyes the all liberals is slow packages object listed from jinking fast because users are Users explanation that they require too much the sources while is the property of that that in the support I think I know of factors that are modeled in music what anyway with we've the we looked at the moment of the big picture so leave offices of the important package for sure but it says 1 of 20 thousand away so we look at a later by analyzing manually or look so the target distributors due to test the tool that I want to test all necklace of packet of usually we have that was what gets where the tests you in ran exponentially long lead the war cause you so and we will do is going cost of that time so that has surrendered relied on all days of something so we just as a package cause there's so many but fixed there's something which break the tests you disablement likeness of that the when the kitchen and the ideology a and happy to talk about anything enjoy force and thanks for listening thank you that