Add to Watchlist

Rethinking packaging, development and deployment


Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Rethinking packaging, development and deployment
Title of Series EuroPython 2014
Part Number 86
Number of Parts 120
Author Kožar, Domen
License CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/19964
Publisher EuroPython
Release Date 2014
Language English
Production Place Berlin

Content Metadata

Subject Area Computer Science
Abstract Domen Kožar - Rethinking packaging, development and deployment In Python, we're trying to solve packaging problems in our own domain, but maybe someone else already solved most our problems. In the talk I'll show how I develop and deploy Python projects that can be easily mixed with non-Python dependencies. "Nix" will be demonstrated to replace technologies in our stack: pip, virtualenv, buildout, ansible, jenkins. ----- Python is often mixed with other languages in development stack, nowadays it's hard to escape any JavaScript dependencies. If you add some C dependencies such as GStreamer to the stack, packaging becomes a burden. While tweaking our packaging infrastructure will make things better, it's hard to fix fundamental problem of packaging with current ad-hoc solutions in Python domain. Using Nix for about a year gave me an insight that solving packaging problem at operating system level (bottom-up) is a better approach. For example, wouldn't it be cool to have "virtualenv" implemented inside your package manager, so you could isolate also non-Python dependencies and not just Python packages for your project and not worry if system was updated? We'll also show what benefits do we get by using the same tool for development and deployment and how little we have to do to deploy our application.
Keywords EuroPython Conference
EP 2014
EuroPython 2014
yeah so basically all on to talk about some of the hot topic at the moment and we have 1 in 40 minutes of better start on the talk about a tool sets that have been using for the last year and a half called makes from and how it applies to the whole stack of packaging development deployment on basically like using
Python we'll mode it's a quite a depressing topic but is getting better and 1 of the main things that I really really really hate about it is that we have this set about high that is dynamic and whatever you do we have to run these dynamics create but at least from the egg full command to get something out of it and that's why for example we don't have all dependencies in the pipeline and so on and so on and for example no just community has Jason simple adjacent folic esthetic you write it down and and you can easily parts packages and stuff with it but there's a whole there is have for just 6 meted out of 2 . 0 which specifies basically the
adjacent into that of a packaging and hopefully people will then generate this how put it into a distribution with with Python source and we will have said current the time available it's in the draft mode kernels when when will this will get upstream but there is hope and the 2nd point is we have a lot of legacy infrastructure which is kind of connected to the set of that by but all there is now Python packaging authority group that's working on news and all the contributors and I think the servant lost with the work few and then there is this scary to a part of of the problem is there's desert non-Python dependencies and this is this is the problem that every community tries to solve by you know building an infrastructure to package but all the other stuff that not just Python and we all share this goal but maybe maybe it's time known to
to to look out and find take something else in all of nowadays Cholesky factor is the inevitable you you have always shell scripts that individuals and you know we can either build all the time the tools in Python 2 processes just could dependencies and so on and so on or we can take the jealous Click on tools and the use of it and we need a tool that will actually make heightening job scriptwriter for altercation and then we might even auditing so next
project was basically started 11 years ago are and it was developed by a local those stars as part of his PhD and the peace talks about dependency hell and and how to approach that and this was done in university intuitive in Holland in functional language department so they do is to take functional and dysfunctional model thinking and applied to the packaging problem on it turns out it's it's a really really really fits the problem on so it's at the nexus the things it's itself but his manager and it's also the language which we also comics expressions in which a very minimal language is basically a configuration files plus lambda functions and of the real rate of of other herself on its lazy evaluated that something that's we're kind of much used in the part community that much and you have to to get used to the whole thing that when you actually something is actually touched to then it's actually related and that is really a lot of power for a configuration which I will show a little
bit later on it's a stand-alone package manager you can in solids on on on any policy search system the official support we have 14 notes mac and PHY busy that could be it could work on Windows if a company will sponsored everywhere but currently the support is visited is contingent on so what is a purely
functional language shall give a very vague description in
in sense of the softer package so basically the software package the output of a function that is the humanistic and it only depends on the function of inputs without any side effects so when we describe the packages in makes the the need to that of that you put on those files that's the only thing that should affect the package and nothing nothing else on we we call this security things and so there is the for example package and this is the prefix with a package would get installed and especially its story is like a flat repository of all the packages and then you see the cryptographic hash how long and on and on the name of the package and the Virgin tired package is stored in a separate folder and because we want these results the outputs which to be to be deterministic we want to make it intervals so the whole mix stories mounted as only just to be sure that nobody will touch and all the time stamps are said 2 units so they times of close 1 and so on and so on come and this this has to see there is basically the head of all the inputs so if we if this theory was that it so that the results depend on on the inputs and then if we have to then we can uniquely identify a package so this is
connections University I hope you can see it but this is like an example engine expected Hollywood package and makes it a little bit simplified at but we currently have in the repositories so at the
top you see this is our anonymous land the function are gets higher order functions or the those are just some of the functions passed in and then we call those found in the environment make duration which is the main domain of the function that does all the heavy lifting and in that way the see psycho college attribute sets and expertise with the dictionary and we present name version and then we tell it's where to go to the of sources are we tell the dependencies which is called the inputs something flags and then some description about the packet I and and then all this is basically passed through to about 3 that goes to different phases and he knows what to do with the storage of data and what you see here this is basically that's what gets attached to these are all the inputs to be 0 and an engine makes this is all the information we need and of course there is a dependency graph of the of of the packages or organize themselves it and so on and so on are also written in this so this is like a quick
example how whole powerful mixes on if you look for if you go
and be back if you look at the of this of this file we we we want all right the things basically the the land the function of the top and the metadata about package so we want to give user of this distribution over or Beckett's repository the power to to change anything so that
the top the top line basically all rights so don't delay and the
function with something new that's what we fell asleep on so
basically the overrides the land the land the function and we can we can say OK let's let's take another as selling feed into and then we get the new engine package which which addition over NASA sellers and move and we can all right Federation itself and for example at the bottom example I have the source and we can see for example the engine leaks from get on and this is to see what you can do in the user's face right because sometimes you you have to change what upstream so some just on the extended distribution this is like an industry from security point of view people will like go crazy but you cannot escape and sees nothing that much and you can run it yourself but basically this is the easiest way to install it and because everything is sort section IX you can just remove flesh and you don't have your package manager anymore and we also have to be removed during the profile and you for users the
because of this is this is basically where where everything comes together and there's a lot of things to explain around how exports so although we say that we have so actually accessible or endings have the packages instances version and fire so somehow we need to get this file system here he that's where used nowadays right and that basically is then joined together in the end user environment which you can see on the right on and this is this is basically your environment with all the wineries our libraries are stored and on the stage being specially and so on and on and because we have this set of packages would be called that we can have multiple of those 1 on the system so this is what we have also called profiles and things like that so what we have then been so I will talk later about makes us which is distribution deals on top of the package manager and we have a system for of them there which is basically you're distribution then the each user gets a profile and you can define it can create profiles on the projects and but it's probably has its own life cycle of Hollywood install packages inside an upgrade and and the and and so and basically then the the profile also has a whole history of what what's changed so basically all when you install package you would get the user environment with the binary insights and the profile would get like and another a version number on in the history and basically the the last thing that you actually do something to the package manager is that same length of next slide next profile defaults will then at the end of alteration change to the new user environment and because the links are atomic in in the policy 6 that means that we have atomic operations of installing operating in an installing packages and then is on the left you see that user then yet has its own profile and that means that users without through so X is the can and packages for themselves of course you can visit this if you want and so on and so 1 of the really cool features is that's what need see is source and binary based collection of packages this is very unique and the way that this is actually works is that we have so so-called to build farm called future online review of all the packages of the the packages there and basically the because the has basically provides the but basically unique you uniquely identifies the package you can go as the hijra server although you have a package with his has and if you don't has this package which will affect the binary and if not it will go and compile it and this is something that companies that are then used to to to instead of history their own servers and have their own like basically continues integration tools for a full in the packages on and there is no all and the next 1 . 7 I think there is also support for SSH so you can
do the same thing to the SSH protocol not only http the so just and this
is going what but let's try it out the that's a little bit of that and the so basically like this is the excited so it's a little bit different and if you only use needs but like my binary then points to division 2 divisions binary that is stored inside the next quality let me show you next door just wanted to see that there is a bunch of stuff in there so that this is this is the whole thing all and for example if we look at the if we look at the link here of the
and you will see that all the dynamic and responds to precisely what package in the next and that makes it very deterministic but you you know for sure that it should be bit on 2 different machines you will get the same result if you use the same sorts of things packages so all right let's
look at the highest in the of course we have also a collection of Python packages and we we have these have the function called your Python packages and the of wrapper around make duration that's knows about these details and set of tools our networks and will be quite the same and then the make the relation and this is like for example Hollywood package will provide tendencies sort the 2 data and builds but bycatch will know how to run the set up by built and and set up I installed at the right faces on you can
subject inside expected repository how it works just like 200 lines vertical implementation so when you have a lot
of packages for example and I also do long development we have like 250 packages on you don't want to do that by hands on so there are basically 2 2 tools for this up items in each is basically just so goes there grabs the probable that's the hash and spits out the template that is very generic and if there is something really like on Titan dependencies then you have to fix that on your own and there is the schools tool called by PE techniques which will also be we will be working on doing this sprints there's quite a lot of the developers here on that tries to handle all the edge cases and automatically basically fetch packages from piping and then generates Usenix packages of 4 years or and we have these tools for knowledge we have about makes and we have our ah and songs right but let's face it let's move to development right so wouldn't it be cool if we had actual like each other environments but on the left on the on the layer of the package manager and not just for Python sulfur also you would activate environment and you would get like geese and all the non-Python basically dependencies and tools available and that's what each of us what so basically
Hannuksela words is that build all the dependencies of your it's really source of all the information it has those instead of actually go and building this package it will get you into the shell that it's in the shell that it would actually be the package so you have everything there are available on and there is a cool well what really high because it's also meant to this but it is a cool features that you can say that you're not building any packages have source to and then it just provided inputs and using the shell and you get these dependencies available in your shall opt for example and in this right these works on on a point system so this was system so you can this to developers and they will get always the same environment with the same the exams and so on and so on there is also a flag on a LocalTalk so by default make shall inherit your current environment and you will have all the tools available and secure basically means that it will not do that it you will have only this was available released in the build inputs so what's I'm
sorry for the whole something
so this is basically all activated shallow mean I and it is formal lab what was for the goal and all from from he drove but which flaking so recently I have for to
available well I tend to see that so far yes and I have
it available and if and you to your here and indeed it will say
it's not available because it will not in here but and so this is 1 way to write for that you have all the tools in your so
dissenting this is by using this
service so Jake to actually installed of media course on sentence because it just didn't want to model with Python there so just use the whole next state stack of packages and shell and then I have everything available and run neutral and insulin is the fact that so the single most of you have a Python package
basically clear the so this is for
example a Python against of this is 1 project that it like that have G streamer and energy by us in all kinds of things that hard to take it normally with Python and and there is a the school teacher recall we have like we have a variable called each shell so we actually run the shell this will be true and we can act but it's dependencies in this case and you only build the packet those dependencies will not get into the derivation but
so so that we get OK we have
set of packages how could we extend this idea of functional language to to the whole operating system and build a distribution on top of it and it turns out that he has this works pretty nicely and when you think of it like a configuration file so basically just 1 file and software package is a bunch of files known differences that you're going distribution will packets author for you and in the configurations is what you will know when you write yourself or change some default on but makes makes it is basically the language that we have now so you can use this language on both sides my so excited so basically is the status of this user's status approach to configuration so for example in public expense and have the have also declarative configuration of and from that that's the back basically they still executes a step-by-step imperative the Czech entities of the knowledge and then it all started and so on and their lot of edge cases to cover here on so a lot of errors that you can hit on 2 and here is the the way it works is if something changes I will show later an example of for 2 to assist in the process and if any
parameter to that system the process changes and as you know that it has to restart or results that's process so it all what down to these data going through these functions in x and and when something changes it will do updated so here you can see like a minimum configuration and just like a configurable on and so you you would dancing excite revealed Swedes and it would activate and get the machine to the state so so
1 of the things that's also going to mention here is that makes us is basically was from
the beginning and you want go and change of figuration files by default have 1 file specify it's a wedding 21 years in the state of Michigan she said to be and you executed and then we have a tool that basically those provisioning of cloud service and so on and that on so for example if we wanted to use periods so which I'm using my day job basically you we would important to the full the next father using the for for last so that the project is already packaged with then we would say rights tests and we would like to production in need of file to be explored and write text is basically a function that will write configuration files to the next sort and then we will be declaratively specified OK we have processes assist in the process of that should start with the period being surf friend has the production in the file and for example if the production unified changes here then of these hash of real change of this service and it's almost has to be lower restart it and so on on and then of course on top of that we want to use a provision right on and this is like the minimum example how to talk to them provision exists machines being found itself you specify area of for example this is the 3 a few machine so we have like about all running a parties serving some static files this is like doing the physical configuration and then we have the logical which is basically where we want to deploy it also data back and his visual balls to give you on the light of memory and then I have like a studio heads because the Csub supports like Amazon and its mirror and now also google compute images this experimental 1 and look at my then you would say creates the courageous configuration and and and intellectually provision the ritual box and so you would have a pattern mining individual books so uh
I don't really have a for this because it's gonna take awhile
to actually show it but Justice Justice for the whole state so when I would actually deployment projects then I would have 3 files wonderful that makes which is for them at the development and on the building of of this project and developing example machine which defines the physical and the physical state of machine and individual box OK and then also the history it's for on and if we look at it at all those files is falls and so
that's true but basically they had just like like it's my name sources current directory and appendices improvements are from and this is this is for example all configuration of machine that would launch chromium for screening just more than sort of the so the topics they say OK imported packaged then enable each server enable Display Manager window manager and sets set up and the best manager will provide overall command which is basically weights 3 seconds and then run chromium what's local 80 80 and a lower you see again the configuration of the system the service after periods very simple example and the bottom see how we define and declared to be a user called guess that we use for the Producer Forwarder chromium radical interfaces and it's this is basically that the whole configuration of the machine and so you can see like you packets the package was about 10 lines this is about I don't know about 100 lines or something 150 and the future walks is like a few
lines also and this is and this is the whole packaging development and deployment study which is then used to to actually and but still try to
and now it tries to download a basically means maintenance yeah we can wait for a little while but is there is no point so basically would have done Thursday mention that all the dependencies are configured images fit around the mutual blocks copy all those snakes so packages inside and then activate and you would get the full screen chrome which we the primitive occasion when the we're having the
2nd excessive ranging in Slovenia bladderwrack aren't running from the 2 23rd and 22nd of August so that this is a great opportunity to me the develop corresponding talk to them the or there could also will probably also be there the core developer and this is this is basically the image from the lecture on just to to
shameless flux I wrote a blog post on a little bit more into detail while awaiting when exerts tries to solve this problem in a little better way than the order of solutions and I had a talk that force them lots of makes less than it was more focused on excessive there's a video on youtube if you want to watch it and causal take it out and so the oracle and you're welcome on the notes to stop by and say hi and read community be so
questions here 1st the all I want to learn more which the yeah and so that's quite interesting and by the node makes so as before so it looks like it might public migrate and everything in a controlled way and that makes them all runs little books why that like widely adopted so so far and what it can do out that 1 of the differences were pros and cons and so on so the question is if widely why is it not that popular yeah that yeah comparison to like and all the established tools which are of course very different and have different itself and so on so of course it would be also 1 approach to this kind of problems and the short answer would be we need more marketing people had the longest would be that's an excellent now for about a year or 2 we have also make stops and NHL and so on and now I think finally this is a mistake is ready to be used and 1 of the the biggest accompanies his logic blocks and law and hell about 100 service provision with this on and and the community is really going the Haskell community basically uh there were a few blog posts in the last few months explaining how people develop Haskell which makes us and it's bent very moral and love love that the same happens to to the Python community if we actually want to solve this problems and i think now is the time that you know this is we really see a lot of a lot of new users and you can see that and mainly cytosine everywhere and I hope that we get the whole this we get to the same point it looks really interesting from the he said that it supported on public systems that include match at the end we have we have quite a bunch of all unhappy home X uses the same music so as the core of vectors thinking but we don't have that much the is the users yet are it's more work just 1 thing we do is not clear what does the character the benefits all this other person just within this from the the part of the benefits of all it should be using our own stock in developing makes using mixed them with Python it's hard to get in our company we got our development environment which was the kind of switch to the continuous development environment but not the only for deployments to use the distance but also for continuous of developed development for itself each time a developer recurrence like In a 2nd fingers have a development environment for the and no virtualization right it's just your system of it's kind of a lot different ways quickly getting started with new projects that was a major boost for complaints hi I regret great told I few questions 1 is that you mention briefly binaries find that do you provide like dust makes itself right binders would you expect that 1 1 of these fighters for the place of the next to like Roy all the binary storage of something is so that the hijra project please you can host of herself and then it will build the binaries of your customized packages for a proper projects in a company and you can point to that to the official huge and urogenital just as the both for binaries and fetch them it will have an and developed 1st in the minus more like a security related events that you find the you know things that would never happen you find about open SSL or something and you get like so that companies like the fire normal accident say that any of the books I get new open a cell which has the same API competitive the only thing I need to do is that just receptor services for them to end up same so that was the nearest worship so called you because I saw that you mean specifically to those like a certain words software produce all this makes the this just the the building everything works as so now this this is 1 of the problems that we so basically if you if you change the openness to sell them he drew history compiled all the banners this moment like the last time that was the whole window but as a cell and I think it took like 1 day or something on that of course unacceptable on but we have now an option recall so I think it's called the security of basis something unique size and basically there is a heck around so as that is the Hessian not change so you don't have to recompile everything so you say the original the regional library was open set as a cell and the new 1 is openness of cell of above a blind and that will replace all everything that uses this openness the salivary to model and review of everything that's needed on and that way you can really fat really quickly you know object your server and because is using makes less if we also know that which which sort of processes for using Open SSL angle and restart the and the a hash of over of roughly about depends on its inputs which is can available programs are so the hash for 40 package events on the inputs which say for pure Python package would be and by the itself and so so you can rebuild that and at the same
versions but is that is there anything that also ties in to version of and the next 2 chain that used to build it because if there was a new feature introduced in the next dual-channel something is that is our basically rebuilding a back like 2 years ago something exactly as it was at times safer basically if that makes up the greater something yeah but that's like the whole makes is upgraded separately on so it doesn't affect the next chain but everything else down to the GCC and you leave and so on it's basically a dependency them off of here the title of bycatch mentioned that as a a question on steps that so and I believe it needs to change so that could build a binary will change change the office environment that's track but if the so that if they if that behaves like changes something like that and you rebuild it is standard version of Unix to change used in a box that just says that as it stands is not exactly what the dependency among a month the questions ahead of a lady with a solid sphere containing and what questions the the nearest slides I was working definition when you the dependency but where are they defined like this so basically this is kind of like 1 just how word the reason is tied to the main on and the name basically at the origin is not important on what in unique so all because that's just the beginning to later on and when and when does the expected just repository changes you would just get the the new practice of an entire there of course there is the version names but still we don't do any like detection or something to that of all the versions but if if you should be fine examples so I want some specific like the stream of words and so it was part of the name yeah so when we have like when you have existing which you have version 1 and so the the region before 1 then we have basically 2 packages are and then you can figure out which 1 you want to use arms and you can always override the source and gets another version if you want to change the upstream default for years project or server whatever a what is the difference between needs and Dr. how that's and so basically tries to isolate the environments from your system right our and provides a very nice a guy on top of that was makes basically tries to solve the packaging problem and configuration problem so these are not like I think those 2 things go together you can use nature's inside the document you want on course we're also using the adopted to solve the packaging problems and providing a huge binary blob but that's another discussion of the makes you don't have to do this you don't have this problem on but it's still nice to have you know those light freight containers 2 experiments on that and so on that's very short answer called both men and 1 and 1 question you know just what do you do use needs a new web development because you showed a lot of stuff of all the you always dependencies and to always package dependencies and even if Biden dependencies let's say a right it's for the vacant but in our company we have a lot of struggles oncogene and deploying the services with a local let's say the for example JavaScript with polar and so on so called sneaks of like that are I know that you can use to declare your own sources and you can be jealous sources but the do you have any for example jealously to the repository and followed the doesn't apply to the package so that for example by conical defines the those jobless people like libraries and so on because this is always a crucial problem did that can for example can handle their own dependencies and it's fine you can do you your own body with good people repository and it's fine but knowing the deep people and border for example that the struggle and how those make stuff like that but that's that's that's exactly when it sounds really good so we have a tool laboratory makes I think it's called to generate some of our packages to generate expected use from our upstream in than expected as repository and then he would go in your project do the same for all the extra stuff that you want and basically then it's knows all of also of all of both sets of packages and you have those available and then you have all the patent appendices available and then use them to make the relation and the child to to develop on that and it will make it that it will explore those packages for you to use are it's really hard to explain this result an example on but there are there is a blog post if you if you go around you will see our it's useful knowledge packages from always think it's but it's same thing I mean just front and on and this is exactly where in its relations so when you have to combined to choose thanks to that he prefer
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation


  505 ms - page object


AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)