Managing Python Environments with pypi2nix

Video thumbnail (Frame 0) Video thumbnail (Frame 8376) Video thumbnail (Frame 16752) Video thumbnail (Frame 18269) Video thumbnail (Frame 19297) Video thumbnail (Frame 23539) Video thumbnail (Frame 24485) Video thumbnail (Frame 37469) Video thumbnail (Frame 38274)
Video in TIB AV-Portal: Managing Python Environments with pypi2nix

Formal Metadata

Managing Python Environments with pypi2nix
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
This talk wants to give an overview over the usage and development of pypi2nix, a tool to generate and maintain python environments with nix. pypi2nix is already a viable tool to generate and manage python environments with their respective dependencies inside of nix. The first part of this talk will be focused on the current functionality of it. This includes features and shortcomings. The second part of the talk is about nixpkgs-python and how we use pypi2nix to maintain semi-automated self-updating sets of python packages. Finally we will discuss how pypi2nix could be used to maintain python package sets inside of nixpkgs in the future.
Point (geometry) Scripting language Beta function State of matter Code Execution unit Set (mathematics) Online help Solid geometry Mereology Code Metadata Formal language Mixture model Pi Thermodynamisches System Internetworking Phase transition Software Bus (computing) Information Configuration space Installation art Parsing Information Expression Binary code Data storage device Metadata Mereology Instance (computer science) Word Configuration space Right angle Window Library (computing)
Meta element Twin prime Inheritance (object-oriented programming) Multiplication sign View (database) Home page Mereology Food energy Rule of inference Product (business) Revision control Freezing Derivation (linguistics) Mathematics Cache (computing) Root Thermodynamisches System Interpreter (computing) Touch typing Flag Energy level Integrated development environment output Overlay-Netz Electric generator Theory of relativity Information Expression Mereology Line (geometry) Variable (mathematics) Derivation (linguistics) Category of being Thermodynamisches System Integrated development environment Personal digital assistant Repository (publishing) Function (mathematics) Radio-frequency identification Calculation Phase transition Interface (computing) Planck constant
Service (economics) Cache (computing) Error message Personal digital assistant Interpreter (computing) output Error message Compiler 2 (number) Social class
Standard deviation Meta element Home page Function (mathematics) Information privacy Variable (mathematics) Neuroinformatik Uniform resource locator Peer-to-peer Interpreter (computing) Flag Information Multiplication Library (computing) Touchscreen Electric generator Computer file Sound effect Derivation (linguistics) Latent heat Arithmetic mean Thermodynamisches System Computer configuration Repository (publishing) output Software testing Right angle Quicksort Hacker (term) Regulärer Ausdruck <Textverarbeitung> Inheritance (object-oriented programming) Directory service Product (business) Revision control Cache (computing) Thermodynamisches System Integrated development environment Configuration space output Default (computer science) Slide rule Inheritance (object-oriented programming) Information Expression Semantics (computer science) CAN bus Radio-frequency identification Revision control Gastropod shell Library (computing)
Personal digital assistant Home page output Bit Library (computing) Derivation (linguistics)
Presentation of a group Group action Overlay-Netz Run time (program lifecycle phase) Multiplication sign Insertion loss Function (mathematics) Information privacy Mereology Variable (mathematics) Neuroinformatik Uniform resource locator Medical imaging Mathematics Bit rate Computer configuration Interpreter (computing) Core dump Set (mathematics) Cuboid Information Extension (kinesiology) Library (computing) Position operator Overlay-Netz Electric generator Web page Gradient Sound effect Instance (computer science) Type theory Latent heat Arithmetic mean Thermodynamisches System Process (computing) Fluid statics Repository (publishing) output Software testing Right angle Product requirements document Game controller Twin prime Regulärer Ausdruck <Textverarbeitung> Robot Directory service Rule of inference Machine vision Product (business) Revision control Pi Cache (computing) Integrated development environment Spacetime Software testing Configuration space output Maß <Mathematik> Condition number Default (computer science) Data type Addition Default (computer science) Standard deviation Slide rule Patch (Unix) Expression Debugger Binary file System call CAN bus Word Basis <Mathematik> Mixed reality Revision control Gastropod shell Library (computing)
Overlay-Netz Multiplication sign Set (mathematics) Diallyl disulfide Revision control Latent heat Process (computing) Basis <Mathematik> Function (mathematics) Set (mathematics) Software repository output Software testing Quicksort Automation
I'm going to present to use Iverson Gerda he contributes through makes packages occasionally and his internet and co-developer through by writer makes in a state of the Georgia's with so many cartoons and here you have to use a bus German but thanks and but in yeah I wanna talk about so private who makes an expected despite that and the the goal of this tall is basically to give you an overview about what these 2 words are doing and also that ask for help than that of its parts more on that later yeah 1 of 1 of the pipe idea to begin with but yet you push away to stop quite packages this Eckart probably everybody who were used by of 1 point said could install package X and but yeah that's how all this was done unseasonal everyone basically changed it through you snakes and states and yeah there are a lot of packages on paper the ethyl the player for the this uncritical of we you probably won't be able to support all of them but yeah food for this that's right yeah Piton packaging have some issues that I don't know the if you like if you ever written in Python package you probably know that you have to write the set up of pi to learn that you make your Piton codes into a package Ngô those set up a pie is just a pions instance so yeah you can put any code in there and a lot of people do that like if it's Windows for better know if so who checked for certain C libraries and do something and if they are not there and that like that so there's actually no way to do what no wait and generates who gets 0 comma decimal 7 pence information before you run a set but but it appeared that gives some meta-information about what's is store at dependencies when you install solids though when you leave a package bears new metadata we use by the way of the binary Hall Medal in heightened packages and you can in there you can find some information about what was done what was installed and it received later that we passed detected that kind of information yeah and there's another problem there because it's not a compiled language there's no configure things so anything so that if we won Obaidullah Piton why Romans we also have to make sure of it when we call a white you do all imports so in 1 units that the linear part is correct said and yeah then yeah the boring sets we the we metadata and because it's not such as it is their best what's up there with Atle come by makes if anybody knows that probably and not all node to make so clearly we have to do a lot more we have to go there too it was set to get where we want we do that and me stages and yeah 1st stage is basically just to install we've beta and environment replicas that would be the good and also you the instances is unique was doubling the pipe packages and in this environment which we create just the mixture of yeah we run story that it's they each to his despair to extract the metadata probably right fellow yeah yeah know licensing solar dependencies there all and that kind of stuff end of H-2 we also calculate the check sounds that we put in our to be generated expression that move yeah that you and stage 3 is just a rhythmic expression from that that here on its expression with a what they it would that we generate pipe to
have 2 parts what three-part maybe you know of course we have the machinery in place to it's pipe packages with the twin from packages but mostly you this gesture it's the generated expressions for downloading and cleaning White packages as you would find them in next packages physicking because we use just the pipe package from the G and as an overwrites parts which is similar to a nobody views of overwrites the possible though right that they to give you related to to to but so it's like fish information the pipe tunics fight for example dependencies that's IT tunics was made at the GA but because they energy kept property in the pion package so yeah and that that should a flag states only reflects especially environment variables that should be sent on the time and we use the fixed-point calculation similar to the overlays and so on I will later show how this might look the will there was just to give you that to show you the stuff expressions like this is part of the generator package that for senior verify that rule that ended it it just looks like you would expect next this is an example for overwrites fortunately the lot and overlay so we can do stuff like packages of height and then the top because our status aggressive use in this case we need the pine packages and not expected to know now but here I mean that level kind of birds you get the idea we just overwrites the derivation to remove some at here that the we don't wanna have in our set up a prize for example version Planck put it because we wanna take care of this the only yeah but nothing special I guess you know if you've ever oral and their relations there are no comes the fun part like them and on yeah I prepared to make 5 and the 1st look for that this is a big enough to read yeah most people what people people were prepared probably you know what about requirements but 60 but just to summarize its basically you of polymer like requirements that OTC phase UG there to specify what should go into parts of the development environment as tightly tendencies or even a production in the production environment and performances you just lists the packages you line separating basic you can put a modern just pick a change and also version paintings but we're going to and with the company or you could reference Git repositories of and so here we also have to resort to make them so this part and then we can generate expression with high between X and this is basically an example in for this case of of root quickly go over the flakes alone I B 2 makes supports pipe to a 3 and the 1st leg like the line of the just as it use by 3 and the mind of our take is similar to the pit lane thick there specifies the requirement 65 as should be used to generate an expression the bearish new flair lets the user specify next packages that should be available at the time was that in that time and the wireless the lower case is just the levels of touch the yeah that's that's true that's all so if nothing in here and now we won amigo required but she she finds the nothing special just there's everything in there and not generate the requirements the next Friday
the yeah service takes the Eitel could seconds or minutes because it has a compiler some stuff in this case revered and its Maybachs just like 1 of the more famous so it's parsing errors for heightened and I chose this example to demonstrate that we can handle the dependencies yeah and that he you can see yeah just we just run follow the yeah I don't know really with related it's maybe I don't know somebody question so how many you can do that in the meantime sorry and we saw it and get a 1st
class your use of if we can read We use overwrites or I like that you created
for a specific environments and yes there is a way through and that is the way to the but if it's just show is that and there is a flag
of its lets you would specify additional rights the finally no on the screen of their parents anyway and this is the best digital rights that lets you specify in the yeah any override private so you privacy created you can even get you overwrites fired from you and it's all get repositories of where the effects of the revision that is currently it lost all you can specify in another revision so there is that there is a way to reuse overwrites endeavor talk about this later this L axonal library also depends on the fictional to right how how can degenerate to find a dependency do you have to specify mentally or here that's a movement that during the interdependence next the means you will you specify yeah you need to examine the and the big them into when X SHE it would just be there for every heightened patents because we cannot but in general the will that information about which package exactly uses a scene and so we just the giver designated input toward applied pipe packages in the whole separate just generated the Python packages can be derived the dependencies right from the pit it output OK I will continue no the with the devil but we just demonstrated just generated the requirement thoughts the next flight the and and purport like this which subsequent computer shows show shows said within the course of the sort of I don't I don't know the use of the generate expression products and then the but it and I mean in this
case that minutes has no quite dependency so the propagated bit inputs this it's and you can in in and he youth ability make I mentioned earlier the common bit inputs on but if them into because if you and this would be true for all the hype packages that we would have put them there but and just to show that it's and
on works with make good
interpreters of that I have this speed science because I promise it has of computer so you have to believe me that Mix bots yeah that's too quickly you start up to partner the printer and important from yeah work the long OK let's go a presentation or his own you see what works seeing that we can generate by the vitamins and that we can use them but there's a lot of that doesn't work follow that and and this is especially you'd tests and there's a big issue because we we have to support tests to make this a viable option may be to use an expected of and even not like this was just a necessary to break the 1 used in production we don't support which the pipe II instances so if you have a private type II the so you cannot use a typewriter makes tool like selective need the police from of vision type I and your custom pipe and this is usual in his to my knowledge kept has no way of telling the user for where it gets the other packages Of course we could positives standard outputs yeah we always do that because that would break every time it we don't wanna rely on the front end of it to generate our expressions that yeah and if anybody wants to do that in really to a conjecture or core meanings yeah polish the you know and I want to talk about expect despite and expected is a collection of Piton packages that generated pipette tunics and there's a get-out repository CORDIC expect Piton which or calls and yeah it's semi-automated said updating and what that means the out or explain this later it offers binary substitutions of because the we push to the binary right age the and it is also serves as a library for common rights and yeah a semi-automated set up at the effect of that package of yeah we also went up of privacy I and there's basically con job running every you know it's which executes pipe to mix and generates expressions of which can be used by the user ends at it could the it's changes that made because many because of a Python package update upstream or whatever to this repository that them which provide inserts them like over grades the funeral will use some bots yet not looking like but that I found everything like the place of the jingle I think it's called jingles will placards no extensions by now but we plan on extending the and for the organizing group that make sense like you would make you would have put parts of the pie test with it work its extensions in 1 package said because you have to make sure that this work together and this is how we organize well especially to mitigate like really lead times there ends here different political pardon 2 and 3 mean this will be an issue for too long that's but for now it is and here it's the library of control rights it means like they already a predefined overwrites that it but is that you can use and that pikey all us that might best as the overwrites lakebed Towards automatically that overwrites from its picketers tighten that they're in your next expression this is this is also like it it depends on the on the on the revision of that was current when you executed this command so that it could generate makes it special rate in the future even that even if we update something that might break you your expressions what's still work and this is not only for the data but it is planned to be the default behavior small here are the women what we wanna do is that with expected Pinewood provide an easy way to install your Piton packages and for that we would like to use overlays maybe like with the but is intended to have 2 and this already the condition in there we go so that you could use but here as we saw yesterday it's of overhauling and that is and we could generate a steady pipe I from our expression that would be also really nice to have a word for additional generation talking images or whatever because if you have a type II repository with only the roses you really need for all the job you don't run into trouble with like version capsule whatever their yeah I know to enable tests we need to rework pine and was structured the it you know if the possibilities no and so this current you know where each rule this and to differentiate between the 2 inputs for package like test inputs especially high test and runtime inputs from the way we generate or expression so we have to figure out something there on the yeah so no the the future we would very much like to use an automated twin expected just to generate Python packages of for basic injury consumption will idle for packages that uses dependencies animal that only by its we did don't support this now like yeah don't support don't don't expect us to true integrate red with your expenditures means for now because we're just not there yet so we would like to support that box yeah for the reasons I already explained that's the beatific and so what
we could do instead of generates a set of overlays and often that's true Linux uses this would probably be like what these intrusive way to get Peter to know the tool and how it works the yeah the input request side about would be nice to have but there's probably a lot of people think OK on on the
that was my talk any questions with the job the question so there is a new door could there people I can't see the thing so it's it's like a it's using proprietary and so the prime requirement that makes on the ground the DAT CSTEE and they generate their people find the clock which should have liked that like the dependency of the roads and the height of the or the back so that makes sense said to be used in part in and it's sort of a shearer commended it to be used by Python and OK and yet they still didn't have we look into this it extended until locus that quantitative VN last version so it's just wondering what you select which packages to put into the expected would duly built dependences of these free so it is a good ro and 10 was OK I need some time pictures from my work so I put it in its because despite and so it was basically derived from what I and what news for all of this and signal but so there's no formal specification of what goes into what kind of package sentence packages Python and you know but we don't for proposals of course kind thank you FIL