The Tumbleweed Factory

at the other than anyone me says OK thank you part of so my name is Difranco Norton I'm the open released met miniature are working for losers in this thing too long to was so quite a while that I am developing on these processes and victory and opens the insulin and but I would like to know who you are because as was really the biggest barrier why have been various taught to have you know so even develops so from time to time open that's good because I actually prepared talk mostly for those who who not so I'd to have less slides than in the previous to then terrible but I have more time so if you have questions show them and me I'm actually about and time for that so what is open tumbleweeds the question was already I asked tumbleweed is a project that started father of like 4 years ago by a great KEH who is was then working polluted about was formed have and and and the three-time project from from Greg the poet Russian updates of popular software on top of the latest opens the readings this Russian by available in the open field service for everyone but not in 1 repository so we have 1 for the 1 for the known 1 to the wrong 1 for the ground floor there will be a tight control whatever but tumbleweed the as started by Greg was the 1st time that everything was in 1 report the variables quite popular Allen was introduced sometimes people confuse the rolling distribution because it was said that it's family but into but it's actually not rolling distribution as Wikipedian finds that it is the rolling updates on top of the ones they will call or why the role in the solution is that every software in that distribution changes on a rolling and so we reuse that name as tumbleweed this now obsolete divided what we call so far affect the what is the factory open the factory has always feel the definition of a rolling distribution as every software changes all the time there's no freezes an ongoing development but the key Pierce says that the development branch of and software is not a donor's affected not actually fulfill that role you know but we attach the development branch of mark from factory lady but it has factories really the main the biggest the most defining project in the open loop of service most software asked that is only open the bid service Ling's directly or indirectly to this factory so everything comes and goes on most stuff in there is around open victory all the little factory the various both releases and fluent enterprise releases but we are not when I'm going there and stopped at some point we're at 13 1 and then we have that that into but we called this ongoing development branch and then release that talk at some point so it's really it attribution of its all it's only in world so its development it is it has always been a development version is meant to be for developers developed by developers but it's we removed what we really hope that so that's really given

an IP the so from battery that that's

why if the the talks all about and what what we did last year so but how factory is developed in general this as I said something that I expect most of the audience not to know this is actually from the Wikipedia from opened Libicki this is not something for the talk so you have the bank lot factory open the factory in the wilderness and then you have several development groups within the field service that developed their area of interest they have the blue in year and the green 1 team member the current guys that the that the whole world they have their own skin with their own policies their own connection to upstream and they submit things to the me and basically my role is here Blum I'm the gatekeeper of all these random through the guys and try to make the good health of of you laughing when it's and and it out there just challenge so this development the this development process got quite right so everyone was doing what what he was doing on his own rules everyone in try to be gentle of there's no 1 tried to for something on and there's and wait it worked in working for quite some years quite we released openly and really normal no animals words during that but some you always felt right there from something is missing and so because like most of these images the connecting defectors somehow but so the problem is the development process that we that we live was set up here so don't don't mind that this is a book like this so that the development potable fed up when we were really young project there just then we can really thought about how to fed up this process that like 6 7 of these development groups in mind when now we have a thing 108 of these groups so we have and this our contributors per week so we have my the very active on interface that 1 several things that are conflicting and that not every of these from users have at all times behind that behind the word the way we went into the sum troubling and follow of that obvious from the start but their submissions to factory I want to be tested in the context of the development group we have the common guys they have become a pond upstream and the baby they even have testing somewhere upstream for this summit factory and suddenly you notice that a lot of mobility build send is not supported yet that red just recently that takes livened of no longer build because it relies on the sick fast handling of the common so we the problem of this integration is so complex that it's the theory of that you can really expect upstream to to to test it in the context of proposal that that we have a lot of these untested of submissions and again when we set up this process we were having a hundred changes the week and by now we are happy we have less than 700 train this week and all these are coming to of that to work so so they have a lot of changes that come up you have to integrate them you find a problem you slow down to say OK I I can't accept any more changes I have to be back what's going on and I have to fix that why you do that it might take you 2 weeks because that that might be a complex

issue after these 2 weeks you have almost 2000 changes pending that you have to integrate them they're finally thinks that really critical issue and individual that was held submissions and makes it again and then you figure words that worked better than before and free up to that from scratch and you're all they have to struggle with always broken called base so but in the end the whole process of renewals and integration and depending on the package being responsible and being the being responsive to to the feedback if you say there's something broken that random develop tends to claim that is that it is not his fault just like any developer and but with 2 thousand changes coming it wants claiming 1 single developer it's not possible usually times it's really you're the because you know I just in the year 2000 changes I think it's yours so the packages will most likely not be responsible for having caused the mess hall tis upstream even but is instead if you start at the end of the problem after the integration is done usually look at at a mess and you have no idea what what went wrong I I could and list and I guess every diffusion maker can do that then list so many examples of things that look so different so just to give you 1 example the alive cities did not build any number and it was nothing obvious changed so I tried to get the life maintainer looking at this problem is that I did not change anything that's can't be mindful of so I look into the problem and the turned on that the life cities tried to calculate how much memory that is free and failed completely so I looked in ways that regulation wrong and in the end it turned out that the math library used for ECU you had at some Auerbach that was integrated into some other library triggered by that so it was really completely unpredictable of that this would happen this again you can ask questions on so as I said we ran into a problem and that problem kind of explode as I of think to realize what is somehow I'm not 1 a half years ago 2 and you would not just those kind of so that to was was having a very similar problem to what I just described that the with relates to that of for 2 months and fix that fixed the problems for real but really we kind of forget that we could not continue and the feeling that users but we have to do something for real so if I tell 3 we tried to to have more people look at this problem there's we've created some factory maintain that 2 responsible and all these development groups so you get to get some some some cross development groups integration we so we increase the testing coverage for milestones and all that but everything done with people and everyone involved in the process that become continue that like this so with for certain 1 we we invested in in a tool called Open QA that was there evidence low-lying something that we really really really put basically all of all of our efforts in into this too well to make milestones tested by an automated tool it's for tools we continue bit with that but I mean

the but for remaining change the

the development process the around this over so that we now can say Open Look factory is a rolling distribution and is meant for users and the longer development branch and it can actually replace tumbleweed and so we we use the the main Tumbleweed for that through a product of coming out of victory lists so development processes the call factory and the outcome of the product is called Campbell read this isn't distinction that people who reading who reading kT press releases will appreciate but answers that very confusing but where we're going so they the universe that the the check out all of the things I really love in the opening period process they but I mean as I said when enforce father was really popular but it's kind of the remaining to go when we read the main factory use of 4 and the US was really interesting far phonological universe to really tested rolling distribution from opens was so it really we really we see a lot of newcomers to the technical summary hopefully music of business that this will continue and false that is 3 8 that's there so what discussion on if we go back to 8 9 3 is that it will due 12 month of in of having to but we still have not yet than the roadmap for that but we where in discussion at similar to release the open release the sources and the updates of lead enterprise into the build service and then learning what is a hybrid is between the open factory packages and burst packages at the base and we finally releases as they become as we think less interesting for users want newest versions we give them from they will enterprise with disparities but this is really a very fresh discussion and on how we do that and if we do it is still ongoing and was just started None of that theory is role in distributions on of knows get would just be controlled and the opening of the really starting to warm and fuzzy and we are used to be fox from victory as Fox from factory not so interesting because we have the fault of factory we think that making popular releases more boring by having the base for a variety of in

this and that was developed the room this

is beneficial for all of for the that does not want to the work follow version of and so now can become the Rio talk about the new opens the development process and the 1st thing I did the early last year was splitting factory and tool than 2 rings this is I mean the discussion you find several distributions actually to to splits the distribution so we have Turing is 0 as the bootstrap cycle this is really there you can full of any of these packages without breaking any some other it's necessary to talk all these packages to a new architecture to to trigger new to popular new finally the term so we have 96 of them I just read recently that theory has the following the news of cycle through other distributions are other pick me up 96 on top of that we have a ring 1 that this not likely it's really is that you can only do bootstrap that all the time on top of the ring 0 there that it includes than minimal takes insulation with all the tools required abilities this is 900 around polymers changes up and and on top of that we have ring to which contains the the the d the as KDE and Gnome installations and Java and some of the things and then we have everything that is not in the ring that is the rest of the 3 that its currently consists of about 6 thousand packages and who we limiting the impact on on on testing we say we have set it that comes a package we know if it's in the ring and if it's in the inner ring we can only the only required to test that ring and if it's not in the ring we don't care so this is not part of the of the but

it is not part of the has that

things this is uncovered space so recreated the head of Beijing project we have 11 at the moment sergeant but the when the random let us associated to that when the submit requests come into effect during really pick 1 taking project planned through Group 2 packages so they they are integrated together FIL if there's a new Pearl we put the power modulus required for this new up to fund new Russian in the same project if this occurred on is that requires a new which would want to put a little more next to we mean of the stadium projects of what we would call it branches in in gets that you watch together into master when it's ready there there are several information in this in this that's what that we have the review was that means the the the fall of meaningful policies we have the government still here we have an information kit which takes the package on its own if it can be installed on factory we have you that checks of the license changed or if the light of the new packages and we have these projects that you have to really don't I have to prove that all packages to build we we test them I come to the phone they have them when they're tested we're waiting for the mutiny done and if they are acceptable they can go into a factory as as a whole sometimes the we use on a giant on would sitting project becomes unacceptable or we have a failure eigenmodes during building on during festival to with the I should have connected

to ports on sort of at so this is their

staging projects that means there is now a variant Paltan step between submit request coming from the development group and acceptance the factory that is the submissions have to we have to prove themselves before they can be integrated so we have sometimes been here this problem with the broken power that we have but what what is wrong but this is wrong and you know we only have to count of blame and if it's wrong and seen we only have at the end of that some sound the problem you most likely but think it is very unlikely to break of inspectors but you know you know it's very very much less of a problem to to blame the problem to 1 of these that is the proof of the and very often the problem of identify with the with the this OK we take the UK and known occurrences on this but then another thing to predict the image of the thinking of the new for that the problem the peers and then have the more I love in the courts the so and then these bills go to

open QA OpenLook or and are

tested by an automated framework this is I It's will mainly written and all that you this image matching and BNC connections to currently queuing will but also a support for real hardware and so it will

it look at this meeting press not he's and mild bottoms and waits for quality due to become either and all these things that a normal test that would do it has another bit fast here but

it's testing basically that all the basic tools for that you can get fall and cool to system that you can start xterm fatigue can look into as age that you can start Firefox and Chromium

at their office to box

yeah yet that's and we have aid of

these tests run in parallel on open queue love art and this creative you all has a few really

collapses the organizations they

can watch them all night

long yeah what light you which of the more interesting because its sometimes that but if you if you would imagine I would have to do this all the time I found myself manually 5 what was so this tool is really was a

game-changer through to this whole

process we could really say OK the father not as that 1 go in but before we didn't have these 2 little automated testing we could not we required follow sorry at least the stadium projects power I had that had requirements all the packages that are in rings you have to build a new package users mostly problem for for toolchain packages then we have 7 have an open protest project that is a minimum executing it an old a rate 1 and encrypted and the which is very pot because this effective laptop runs put that and if factory would break I wouldn't be able to show that the limitations of a test it's any and all the time and we test and update for that you can update from a previous factory to this new stage of projects and there's no box and focus that all for the Packers dependencies so so this policy as the submissions then go on the right if if emissions then you have reviewing the submissions then you have this the info their open protesting then you have manually reviews and then you have factory integration process and then you have another huge for for that 1 and and I only after all this it goes to user as probably go but so this is the overall probably best what you have a projects here you can see the the process that have the René project the job really I'm fault rings then you have the the the current state of factory source you see that 29 failing vectors of only 100 but these are all not ring of the it they have to renew cues for different teams and here this is the interesting part I want to show you have the Foster version of the the base the last fall change happen this is to in this case is almost all of the of the uniform different then you have snapshot from the intestine than than doing and what will be to to testing and then you have the last Nachula fiction and this is this much of power testing and went out to users and Syria after the current will be very size we have 102 tests running for the whole tumbleweed and 2 of them at at the moment which is really really good from where we came from the my of this means I have but is more but it and 20 in that's what I said he should have more questions the spaceship the fact that in the wake of the no Richard the font very the other fact that no no I that is a victim of which I've just said I can explain open pay more as I said I I did not know what a dozen is unknown to the audience but the question was is open create something specific to open lowest there's a generic framework and the answer is it is it a generic framework that runs tests on we have seen everything that we can talk the and the tests are currently vary as we write them off open-loop but we have me repositories 1 is the tool that runs Curie and all that was caught was alterations then you have opened QA which is that when you iron people and their their their mark start of the stuff that's the thing you are passed the point is to live and then you have the open user-specific tests and

there are no Terence Christmas that Hunedoara tests for follows 21 against then tended to an since early this year that would be created some and read tests so to I think and Retix sick it's a succession the it depends on on on the test you right but the the right the possibilities of this tool are really so this is the so the the

branding is over loose Pacific but I hope you can I'm with that they can't see that on the we have this is also something that might be open Pacific and we just don't know it but we have the machine a distribution this where this idea comes from where it happened a type the so-called also flavor of someone's where rate can have like the the you know the the moment he would have been differently handled the guys detector which also handled differently and then you have to build number and the test the running which is a collection of thing the tests and then you have of the results the one-dimensional something else here these are the build overview if it was finished knowing off there of the latest probably snapshot we have a test people for good at end for X for which is well known and if would 1st invoke a move that we have been we have no 1 that machines we have install only at the that singer as amply that Kadima carrying a laptop through FKTU of 1 of the 4 new he stakes the basics the theory of the media check this that the mentors says that there's that like that and it's all up very early nice cold it's like so this is what I mentioned we have we have whoops we have 1 or 2 we have all been creative front end and we have the tests for of with different on top of different repositories because you can plug in some of the solutions that and actually to the 2 words tests for idea and tests for fedora and tests for the Indian but as they are not maintaining the kind of of the all are some we are not there yet to answer the question this is something where we have to discuss in at it should be possible for they in in general I would say that the branding and new stuff the typical derivative problems are still there for so we need to sort out this but this is as as I mentioned is really too early in the course of of of of of the bills but I wanted to show him the test sorry I will come back so we have the income of something that's for example if if you know the only run of neurological looking that really in the test you write your own the test itself just types clear on the command line and then runs to bar and then once in about script and at the end that's the search screen which requires this effect the which led to image so this this is this is in this reference screens are very important he did not that is this this is the welcome screen knew that has to match and this has to include this area and has to include this reverse number down here does not matter to us to the affirm between so this is really very very generic and I'm I'm really really really looking for work for other distributions to to maintain their tests and help developing the the 2 and so and about a question this is open K opens August 1 gene this all this test though we have like a lot more to build the packages this is what my part of the old this is

all part of the opening of the book so that everyone can use and it has some machines so now where there are it in good times we reached a form of so this brings us back to the question everyone can build profanes less than this theoretically yes but we would actually scaled down his if you use which a not so I'm not sure it would ever finish of Iowa machines if you want to do that from also it would be the person this this this is that for any question with that if a tumbleweed QA step is also automated and this is the 1 I want trying to show here this is the follow of has all over the period for the for the tree the example that we have several test on 1 architecture for tumbleweed we actually has I will argue that just then it's a lot more I will see a lot more on of staging but everything is automated the show and yes yes and also we are very we try and concluded that all we're going to find this could help pages I think that's in here so there is that we have and the whole tool on how to install opened here locally yeah I just wanted to say to to see the remote rendered version so we have and installing guide and we have a writing test to teach that might have gaps but we'll try to many and so we have people who managed to understand so and we have and they have opened factory on 3 nodes where there discussed and so on but new questions I guess really the so then the light at the time of 1 final slide because in all right this 1

so then as I said we only really is when it's tested this also means that if there's a problem that we are not releasing it which means that I'm not losing a snapshot every day which some people already complained about the effect of time was not really a rolling distribution because about all all the time but we are releasing very very often and we're only releasing a reliable snapshot so I think and this is what most people agree to after complaining and explaining a getting it explains its that often enough so we have this done in this process since the 1st of august something and in this time frame we released 40 and 54 snapshot I counted but there are weeks where we really every day and then there are time frames where there's a problem that means we have 14 days all essential short and then after that again 2 weeks where you use it every day so it's not you can say everything that they should be a factory but It's always fresh enough way more the rolling them funding to all these spot that would kind of of a loss the though that I wanted to travel summary slide so this the staging project that we introduced this was really helping a lot by tool to isolate changes and to get developers actually feeling responsible for things they broke because they now have some playgrounds to to to to hack in and integrate that changed and the you to the automated testing we have C. a steadily increased quality we actually track fairly us and we had we used to have like 15 us 13 and we're down to only 2 failures and these are the that's a quite these are corner cases of where the only the KDD diversity only on 32 bits starts the external on the wrong terminal when coming from China and no 1 really understands the problem so we we will fix that to at some point and after that we will never have any regressions again because we know what changes broke and then fed into a process we released referring to a normal and we did not have a single I've found we did not have more than 1 2 read 1 meter 1 really scary enough of intervals final because we could trust the development process for the the the quality of the of the distribution of every day so that was basically a trick so we only released bait us to get user feedback and that word but all were you worked well that's how I would summarize this won't work work so operations that the the 1 behind you was the question of that

their security but there's a fast lane for security updates and that's actually a very good point because we have the the best shell shows check show chop block except in that time frame and we had to create updated we're our previously sure that factory with off road will really often enough to not be a problem but that park it exactly in that time frame so we have to sit and online updates

repository for security opted for factory but we never used it beside the bench from now your question so the question is how long does it take to 2 wheels of entry this book and pay there's actually scalable we have you can plug in more machines in that if the parents depending on what you test

there takes let's see if I can have that

information in this is a key test that to look body 9 minutes we have word has roughly in all of of and we have 8 of them so you know the whole will 100 2 tests so it's roughly like 6 hours 7 of us we could end that what said you could we could have more machines if we wanted to and the effect of the architecture slower or if you test more than offers you have to ask for more but the moon the curious the the

here is really not a problem so this is to see if I can for this Monday at the

the typical features is an image here was

was was bid up so it was really it

will click and pressed the cheerfulness a

senior thesis means literally 1st

moving but their whole insulation take nothing will begin to think about what they're doing is playing tricks we installed I would be my date content of that's really go away information of the what the there we are some of instance of 3 workers of foreign and so we have no you has more and open like this Council of exceeded 6 exceeds the and

their thoughts to however 64

and and 7 and not not yet been so there are at least 3 online platforms and at least 1 polar and from this 1 and cumulative reports as we know but no MIPS long and 68 K of of what I

said the question is if we test for dependencies that a new package would require something we don't want this you know and we test that the rings of the packages and rings satisfied by other ring packages so if the package is outside of the reasons we don't care for the dependencies we only care for it to stop this this is due to put and the

so we only tested the package has all its dependencies also in factory for it ends of the factory and we test that ring packages have all the dependencies in rings before we accept them perfectly I we don't there is no automated way I think you can test for unwanted dependencies so that then it was still the always they will be as always builds in in in in in in the common for each for the Open Build Service always sets up the overture machine of only the caribou dependencies the there it says that the use unwanted dependencies and I'm not sure how to define unloaded now but it again we we have the dependencies within the rings are verified by by having 1 script that one's against the submissions and Everything alter the rings is generally uninteresting to me and the ring this kind of natural talent this packages and yeah sometimes for example he packaged tend to be splitted march together and then a dependency on some new game or whatever but this is you

this today I depend on the key take it to the funeral of Italy I don't have an

automated way to say this is unwanted as people right test cases we can expect them I don't know I have an example and parking OK then and we