Merken

Release Management with Devpi

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
but I still in this problem we have to think about how we do do we get our packages from already well buddhist into our production service and we know use was pictures and what wasn't wasn't taking pictures but we still need some kind of process to do this and so on we have a lot of the problem that we had at least 1 guided something that will mean that Europe and the Middle and somehow step-by-step see how we can improve this process so we somewhere used to have 1 or we had a
guy that had a package let's called package for also with who I was working in 1 2 0 and that was about in this picture and it's all always tooling around this with well the difficult so what he ended up doing to fix a botanist tickets wants to undertake the tarball fix the price inside repackaged everything and then you want to deploy this to production and obviously that's not the way you want to the way you want do things on what tilted tries attach a golden copy to this particular package so that he knew that this was a special 1 and not the ordinary other 1 with the and fixed but and then as you can imagine if you go ahead and let you do all this to stuff like this
and you will get burned rather quickly so and I don't want to blame the developers because we sum over all of us do this we take surpassed the Phillies resistance and this is part of its not what you want you have to invest into tools and processes to someone improve this 1 and then we will now look at what we did so basically we had this problem it's about OK how do we solve this and our solution was
let me say OK we need immutable packages so when you create a python package we want to created only once and then stored forever and then when we give you waiting for this present pickets we want to do right persuasion from all deep repository and so on let me know note sector OK it has this detector and then 5 after it with this text and so we know exactly 1 of the package what what's the total but so ways in this package to to last but not least we want to have those as we have so this means we don't we don't want to use source distributions but we've negative because we use basically just creates an archive of your pricing finance and C code and whatever and your installation you just can't take the wheel of people and pick the wheels for you so this is much safer than a source random what also do is when we installed like weeping everything so we basically sure that not only pictures of this western but also all a crime and ah as you expect I should
example of how we actually do this because on most of all internal to use a open source that is called set of tools or you the reparable around with recall pi scaffold on but it is basically all the same and life an example a repository here it's called so-called it has to get and 1 . 1 1 2 0 and when you go ahead and create you present it applied the beast we'll so we build the what Paulson gossip goes ahead and what the set of 2 let's see and asses have looks at the text so OK let's take 1 of 1 and then it is OK to commit that go beyond this takes a greater was number called I don't want to because interest but 1 and then the number of comments I've made in the text so basically it if you want a unique words and numbers were pictures look at set of of the model basically solve this for you so would from the stuff we've mentioned before we now have the in the foundation laid out we we have pictures but if you need with numbers but now we have to upload them somewhere because it's not enough that and you've created a unique package on your
desktop or on Jenkins or whatever but you have to do something with it and then for us is that we upload those pictures to density and 50 is a picture server for peasant pages and 4 wheels and that its dual public apical and has the great advantage that its competent with high the present or so on you can use it just in the way you would look you have used the but you can run it on your on the whole Honda and you only on and stacks so you don't have to upload Europe pictures to the Internet but can you the local the local that a
short example of this looks like in the 1st line we say OK I want to talk with this that over and this particular you and this index we don't as you uncensored OK problem this package and the package and so on and so on and what it's basically the last line it's really easy to install package an ordinary to command you just specify the index you installed from so on you take the way I look at this so that the server and cancel from so basically from a security perspective or what that's that's it for for this you want available on DVD will cancel
get a where where a website also similar to because get the metadata good on which thinkers have been uploaded so on BSE the diseases that mutation and being that's basically what you get out list of the books and so
we can now look at the entire workflow that we have to build so what he see is we have already developers they take to get so you're getting a great Texan commits and whatever and then Jenkins goes ahead and effectively latest latest let's check out build these packages runs all tests and once all unit and integration tests of pasta on our taken so go ahead and upload these packages to already if the and then we basically store exists and if you forever and then when you want to install something from production we just look at that the install from there and uh and that's how we want to do these things and they ask for some of you in the audience this might all already the workflow that suitable from simply and it is a little bit more difficult and because we have to have a few constraints 1st of all we have multiple themes that I collaborate so with multiple views model outfit that upload stuff and then we want to somehow keep the artifacts of these different teams separate so good example before we use that the you we just had 1 approach here and be on placed all or pictures in a large fold on the and it was similar total chaos because some you upload stuff and by the word and and you know you know who who did what so on for only solutions that we we want to seperate teams a little bit like namespaces that the 1st constraint 2nd constraint is that we have not only present code will be on the ship stuff like Fortran code or NumPy or whatever so stuff that's compiled and we don't want to we don't want compile production so we don't want to have GCC and all our production service and we don't want to spend the time compiling so what we do is we upload binary picked so that the all and from with so we must support so this is and the last thing is that even though we want to keep our teams separated on which they will want to some oil easy to manage the dependency so in my picture pendant repetitive another team and the movements of those pictures we have to somehow provide a solution that's working for all of and then we'll have a look into the death box and there's so we can see how and how we achieve this so
OK up to it if you watch them shortly
back to the example we've seen before it to 2 important things to notice 1st of all we look at the we on life said that you want to upload we want droplet binary packages and those that built on a specific distribution for example on urging him to go ahead and build a package on variance that's why I want to be made but unfortunately the same package when run on deviance because we've compiled against some system libraries that will work and we have some the someone has a support of this but unfortunately in this limit but it doesn't say what kind of so on we have have we need to walk around this when we landed at the user's initiative and this is the 2nd important thing is he's never called on that p is a a little bit of supports users in any system and we cannot have multiple users and multiple indices of in insulin of kind of modeling he's is a little bit like it up with a degree of model uses in each you the kind of model repositories it's the same and if we just with you indices and again you can think of an index just like a back off from your presence OK so how do we use this to major
stuff so what we end up doing is on we created for each team regretted that he uses so they have their own credentials to log in and then for each operating system that is support we create 1 and when we now the packages and Jenkins we looked this
particularly in so when you build and then is suddenly uploaded deviance and when you build and again a we have looked at every in a this work for the spinal pictures but there's also on packages like that you Python packages look bad stuff you don't always want to upload these to all your group job the operating system and it is so we also introduce the so called generic index when you upload this pure Python pictures and then use the future 50 which called on it's inheritance so we can say OK this that means 7 index inherits from the generic and X and then you can see on the given 7 it's all packages that also Waibel this is basically the you for 1 team but as study of multiple teams and so what we do is we create 1 of these users reaching and then have a so-called aggregation explore we call it the platform in and then it inherits from the corresponding genes so if you look at the title index it has a let's say deviance of index and this deviance 7 index inherits from the dead 7 indices of all the different things so in the end when the user wants to install the library or whatever you just write this command line below through the OK to install my package From this inexorable or from this step by and understood by taking this platform and the last part of the operating system you use and then you get the pre-compiled pictures from this particular operating system so this is basically how we and so all of pictures except if you specialties and 1 of these is that a lot of our work and now we have the open source packages these are not really biology by or crafted by our teams that's basically all what's wavelet and open source community that's a genuine regret and whatever and different teams depend on the those results of 2 as 1 so this offer won't work without them but we
have to upload them somewhere and them what we decided OK we symmetry the whole open source community of 1 team in our company and created 1 particular for them again with the different operating system and we now have a company-wide list well pictures stand that we depend on and that we build those using it to which we call the people that will looking the euros you then from there we can install so yeah that's basically the stuff or would be useful for libraries and so on but I this also saying yes somehow
sometimes you have pictures which are not the library but specific application with a specific configuration that's supposed to run in 1 specific mission and you don't really want to share with sample and we do mission learning model so when you have a model for 1 specific customer you don't want to upload it to the opera applied apart because then everybody could use it but they're not supposed to what it would make any sense for them so we also create consume indices which are basically just indices inheriting from the from the but on and everything but can also upload their yeah that's basically 0 index layout how we use it and you can get that some of you think that totally complicated
and somewhat unnecessary but and for us it has been a great success on week Summer introduced that in our company about a year ago within days or even most days a week so I'm everybody has migrated to use so on the adoption ingrained in this 1 year we have uploaded by 10 thousand
packages and documentation troubles we have but there's about 350 in cities and the others have been downloaded about what the downloads accumulated to about 18 terrabytes that's quite a lot for packages but it was able to do all this with only yeah small mentioned this fall for cross and for the good of from the legal system is rather like by here which somewhat great and so but we are now using all of this on Jenkinson DB Service everywhere and then we somewhat have the problem is that goes down most of what you will about course can go home for the state because that's when they commit stuff that takes a self 1 run because they kind look the packages for basically if this thing goes down we can we can work so on which to invest how do you make stable how do we only give it up all the time and this so we use so there this is not a single set of so we just look what processes are running themselves so are there at the front we have an injection some of the recommended set up but what we now do we use the replica and master who and this is a constant that the which someone saying you can think of it as a that's a transparent patch you should talk to the replica it can directly observe the packages and if you want to install but if you want to upload the package the replica transparently talks to the master of the master of the performed state change upload to package and finally replica conserve you request basically everything for us this is a great advantage slip along with that we wanted to become the master we can simply go ahead disable the master of the replica still so the group press will come down to the engine in the end will pass into the replica replica will allow people to install packages if an apple someone tries to upload a picture this will work but at least you can surf traffic in the 1 in the back of tongue we had to make the master methods and works again so we can somewhat to 0 downtime update updates or because for installations and also have when we have to do my 2 and not that that the they have to export of the state from the master grade you must import all data again and this also case where we can use it to set up so we just disable the master creating new under the replica fields of interest now this is a single set up but you still have the problem OK this single host might so we read somewhere having extended this to multi modal the idea is basically the same you have to have the reverse protein from this which basically routes incoming traffic to 1 host to the other and then on each chose began the replica in a single month we're talking to this is a plot of the 1 last but that's not a problem case because we will basically circuit the same guarantee have every before because in the whole state style it's alright because the still there to such traffic using vertical if the beagles don't we can still serve traffic and also and pictures and Frosty 7 traffic part of this important we always wonder located pictures because if I'm the kind of packages they all it is only do this during daytime so we also can think so whole set up during office hours but the installation path also test look at nite so this is basically a great fast because of yeah we assume all we we don't need really need from there we don't need to set nite to fix this yeah was some of these how we used as fiat the general structure of the shot repetition of what what you've seen so as fast as that we created with pictures so binary packages on we give them a unique where numbers using we then upload all this to decades is 1 with 1 indicating operating system is 1 factor that I missed before reflect all over the features as non volatile this means you can upload new packages to them but you can't override existing ones and you can't delete existing work on so this is important because when we yeah we can vary only great packages and a user doesn't has to have to feel that what the fence or something and installed again you get other words and or whatever but he's always should get the right 1 on the last thing is we create uses for intelligence and the so yeah that's basically it and in Europe and this set 1 the 1st 3 steps might be enough for you for now but in the future to sing about OK it has to be highly well below you can look into the optical future and use it for the last so but 1 last trip for you if you decide OK on this is somewhat interesting to that we have learned that might help you 1st of all you want to set set of a test so that users can use across many cases where people graded in breach of new artifacts and as you know jenkins yes the 1st Shinkansen the 1st 1 will always succeed so you get many broken pictures and then the people had to come to us and say OK can you please delete this package because they are someone messed up the worsening whatever but all this index of a volatile the candidates themselves have to come to us and the somewhat overhead and then you can solve the system failed filter give so let's say that you test set 57 just install them just somewhat up and then the other thing is you it is also advisable to not let the users use that Klein directly from it has some important common alignment in the DFT using a shared environment or when you're using stuff like it assumes and people tend to forget so if you inclined rapidly conformal fix those arguments for use with they can so on that's basically the run through profit of the candidate answers thank you for your attention on 1 of focus now you
can come wide opens and after school after be Henry
questions yes if you were talking about those uh the group for all the dependencies from the open source community that can best be work like up proxying uh repository for them by so if I a request on you my fact is that of 17 hasn't been bullet and I target despite the the loaded and passive solar for later right that he has a mirror future so by default you wouldn't need this process index how we use it would afford afforded the area Bishop specific use an index called the root slave party and then you get all packages form pocket but little by default so you don't have to do anything manually it's just as we decided OK we want to pet repackaged all those packages as wheels and most of them are not available as we'll unpack also so we somehow say if you want to upload them specifically and it also has the advantage if you sum up to this meant that this menu some home in your seventies OK I want to explicitly say that these should be on my desk and that of the company that's on the use of the service so we have to look at the licenses we're using and so on we were not allowed to use any effect all of the picture and that the internet but have to look at the licenses what some of these you Amities OK so long as we sort of this License check when you say OK we set this picture this wouldn't do this large white list and then I want to do that and upload them automatically so we have this step in between but you don't need it you can buy full use the popular in future and thank you the 2 as indication when installing packages so I assume that the time that the arisen available publicly so the like in the Internet of Services and actually installed from it and giving us indication during in solitary apples exactly because I think you upload file packages so that everybody should be able to install them and that's that's a problem somehow this understanding our company only internally so it's not available from the Internet arm this is somewhat feasible for us because we don't want stuff only open cloud but only on our own computers in our understanding of this work fast on everybody form you have passwords Minneapolis stuff you can use other users and the group's falsification or just uses you created you set on that is so that works on but during install there's no authentication there so by default if you know when index and user and a package let's there you can install on that the current so the only way to the basic counselors something like that from a T P when trampling based authentication Woodward that's what we use that's thought maybe there's a events that better solution and there that might work but and I would I would be interested talk to you afterwards so but for now we have basically open for thanks for the talk so what you mentioned earlier you like they used to build wheels and armor in your pocket was stories but also used mirroring feature so is there any way that I can have like a local and that's what packages that aren't exposed still use the mirror I can build the wheels for small not filed results that works on because a circuitous route index and what you can do it you can in when you get your entity can inherit from this pattern so for the whole you will see you packages and all from the internet and if it's any side OK when this specific packages that we you read pictures that we have uploaded to you index 1 and then you basically get the rest of the that you pictures and this 1 is a weird but and this 1 thing you have to look out for on that someone is security issues because of the few few it's a future when you upload a picture that's where Unix and also in the internet difficult the head and says OK I'd want look at at the internet for this package anymore because otherwise some guy might go to it has gotten upload a picture you brother taken with the highways and they would give this so there's the widest future and if explicitly say I happen this 1 of the we also to look for other stuff on the internet so I cannot have been version upload it's a multi foreign G 5 did something other than to rebuild it said that that in front of that media support the simplex that something I 1 to deployed version of production and I want to know that version 2 . 0 examples only has a set of packages that I want us to be able developed so unfortunately get question what can you repeat it please please support natural of indexes by default knows that future but you can somehow rebuild it yourself I mean this is the so-called American back to its steady pushed so you can say OK on this particular indexes perspective this version and I want to move to another index without modifying the so you can basically do this actually like became known while this package to another index and then that can be modified this is what you would you want yeah maybe an explanation why we don't use this feature because here we have some guys that can be that likes to automate stuff that it would simply automate this this should push command and then there wouldn't be any advantage so we said OK we don't need to because they will automated anyway at the end of the day job any workers and be at
Prozess <Physik>
Server
Vorlesung/Konferenz
Biprodukt
Metropolitan area network
Mereologie
Besprechung/Interview
Softwareentwickler
Biprodukt
Computeranimation
Distributionstheorie
Videospiel
Dokumentenserver
Open Source
Eindeutigkeit
Zahlenbereich
Packprogramm
Code
TLS
Computeranimation
Informationsmodellierung
Menge
Klon <Mathematik>
Wort <Informatik>
Versionsverwaltung
Server
Computersicherheit
Indexberechnung
Keller <Informatik>
Ikosaeder
Computeranimation
Dichte <Physik>
Homepage
Internetworking
Last
Perspektive
Automatische Indexierung
Server
Gerade
Softwaretest
Binärcode
Nebenbedingung
Web Site
Bit
Namensraum
Sichtenkonzept
Quader
Güte der Anpassung
Mailing-Liste
Data Envelopment Analysis
Biprodukt
Code
Computeranimation
Integral
Metropolitan area network
Informationsmodellierung
Multiplikation
Diskrete-Elemente-Methode
Einheit <Mathematik>
Unordnung
Server
Installation <Informatik>
Softwareentwickler
Umwandlungsenthalpie
Distributionstheorie
Videospiel
Teilnehmerrechensystem
Bit
Dokumentenserver
Indexberechnung
Physikalisches System
Binärcode
Computeranimation
Metropolitan area network
Last
Multiplikation
Informationsmodellierung
Minimalgrad
Automatische Indexierung
Programmbibliothek
Inverser Limes
Indexberechnung
Tropfen
Varianz
Resultante
Beobachtungsstudie
Offene Menge
Open Source
Gruppenkeim
Indexberechnung
Ausnahmebehandlung
Systemplattform
Computeranimation
Systemprogrammierung
Multiplikation
Wavelet
Automatische Indexierung
Prozess <Informatik>
Netzbetriebssystem
Mereologie
Vererbungshierarchie
Programmbibliothek
Indexberechnung
Operations Research
Umwandlungsenthalpie
Offene Menge
Open Source
Indexberechnung
Kartesische Koordinaten
Mailing-Liste
Physikalisches System
Ausgleichsrechnung
Computeranimation
Informationsmodellierung
Symmetrie
Automatische Indexierung
Stichprobenumfang
Programmbibliothek
Indexberechnung
Konfigurationsraum
Differenzenrechnung
Server
Prozess <Physik>
Gruppenkeim
Zahlenbereich
NP-hartes Problem
Diskrete Fourier-Transformation
Computeranimation
Eins
Gradient
Service provider
Physikalisches System
Multiplikation
Reverse Engineering
Proxy Server
Netzbetriebssystem
Eigentliche Abbildung
Installation <Informatik>
Operations Research
Datenstruktur
Phasenumwandlung
Softwaretest
Parametersystem
Anwendungsspezifischer Prozessor
Indexberechnung
Plot <Graphische Darstellung>
Physikalisches System
Fokalpunkt
Teilbarkeit
Office-Paket
Patch <Software>
Software
Datenfeld
Menge
Automatische Indexierung
Rechter Winkel
Mereologie
Injektivität
Digitaltechnik
Client
Versionsverwaltung
Programmierumgebung
Aggregatzustand
Resultante
Prozess <Physik>
Natürliche Zahl
Familie <Mathematik>
Versionsverwaltung
Gruppenkeim
Computerunterstütztes Verfahren
Computeranimation
Internetworking
Systemprogrammierung
Multiplikation
Bildschirmmaske
Perspektive
Prozess <Informatik>
Mustersprache
Vorlesung/Konferenz
Passwort
Indexberechnung
Wurzel <Mathematik>
Default
Haar-Integral
Schreib-Lese-Kopf
Umwandlungsenthalpie
Soundverarbeitung
Dokumentenserver
Open Source
Computersicherheit
Gebäude <Mathematik>
Routing
Biprodukt
Elektronische Publikation
Ereignishorizont
Simplexverfahren
Isolation <Informatik>
Menge
Flächeninhalt
Automatische Indexierung
Hypermedia
Digitaltechnik
Server
Authentifikation
Streuungsdiagramm

Metadaten

Formale Metadaten

Titel Release Management with Devpi
Serientitel EuroPython 2015
Teil 44
Anzahl der Teile 173
Autor Erb, Stephan
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20207
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Stephan Erb - Release Management with Devpi Devpi is an open source PyPi-compatible package server. Its versatile features make it the Swiss Army knife of Python package and release management, enabling anyone to shape a custom release workflow. In this talk, I will detail how we use our company-wide Devpi installation in order to share a large set of packages across teams, deploy binary packages to our application servers, and mix and mash open source packages with our own. With Devpi being a critical part of our release and deployment infrastructure, I will also cover our high- availability setup and how we perform major version updates with minimal downtime. While this talk is not meant to be an exhaustive introduction of all available Devpi features, it can offer insights on how Devpi can be used at a larger scale.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...