Shipping ready-to-run Python apps without the need to install Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 131 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/69500 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 202423 / 131
1
10
12
13
16
19
22
33
48
51
54
56
70
71
84
92
93
95
99
107
111
117
123
00:00
Mobile appService (economics)SoftwareUnicodeCore dumpComputing platformModule (mathematics)Software developerComputerMathematicsPhysicsInformation technology consultingProduct (business)Interface (computing)DatabaseServer (computing)Revision controlBitCore dumpCASE <Informatik>Software developerWebsiteNatural languageComputer animationLecture/Conference
00:51
Product (business)CASE <Informatik>DatabaseInterface (computing)Server (computing)Revision controlComputing platformIndependence (probability theory)Distribution (mathematics)Run time (program lifecycle phase)Modul <Datentyp>CodeExtension (kinesiology)Local ringArray data structureLink (knot theory)Time evolutionOpen setEmulatorGeneric programmingRegular graphOrdinary differential equationOptical disc driveData typeTotal S.A.Standard deviationServer (computing)Product (business)Client (computing)DatabaseWindowMobile appFreewareComputer fileExtension (kinesiology)Revision controlComplete metric spaceGastropod shellSoftwareBitDerivation (linguistics)Distribution (mathematics)Run time (program lifecycle phase)Multiplication signWeb 2.0Scripting languageWebsiteStandard deviationFile Transfer ProtocolMiniDiscComputer-generated imageryCASE <Informatik>Compilation albumBinary fileInternetworkingSuite (music)Directory serviceLibrary (computing)Different (Kate Ryan album)CodeModule (mathematics)Software testingOnline helpInterpreter (computing)Polarization (waves)Physical systemBytecodeLink (knot theory)Open sourceSpacetimeInteractive televisionBinary codeResultantStudent's t-testFreezingLipschitz-StetigkeitoutputComputing platformProjective planeConnected spaceMechanism designExecution unitArray data structureUniformer RaumSingle-precision floating-point formatDrop (liquid)Dynamical systemComputer animation
10:04
State diagramRun time (program lifecycle phase)Data typeTotal S.A.Standard deviationDefault (computer science)Modul <Datentyp>Shared memorySoftware testingTurtle graphicsParsingCryptographyType theoryComputing platformIntegrated development environmentIndependence (probability theory)Computer fileMobile WebPersonal digital assistantDistribution (mathematics)CodeRevision controlCompilation albumModule (mathematics)Software developerCore dumpLocal ringVariable (mathematics)BuildingTime domainVirtualizationConfiguration spaceComputer configurationVirtual realityRevision controlArithmetic progressionAreaIndependence (probability theory)Computer fileRun time (program lifecycle phase)Multiplication signCASE <Informatik>Software testingMedical imagingStandard deviationModule (mathematics)Online helpSpacetimeTurtle graphicsType theoryCodeCuboidWindowCartesian coordinate systemBitInterior (topology)Library (computing)Binary codeSet (mathematics)Link (knot theory)Compilation albumDemo (music)Wave packetComputer-assisted translationSoftware repositoryOpen sourceSoftware developerSingle-precision floating-point formatExtension (kinesiology)Mobile appInclusion mapCompilerSymbol tableData compressionBuildingComputing platformFreezingMathematicsMiniDiscBytecodeProduct (business)ParsingTotal S.A.Line (geometry)DecimalComputer animation
19:03
Computing platformSoftware testingBuildingTime domainCompilation albumPersonal digital assistantPseudodifferentialoperatorSlide ruleWeb pageProduct (business)InformationVideo game consoleFunction (mathematics)CodeBlogCovering spaceFilm editingMobile appMultiplication signSoftware testingDecimalLipschitz-StetigkeitWeb pageBinary fileQR codeDirectory serviceModule (mathematics)Interface (computing)Computer fileLink (knot theory)Standard deviationScripting languageRegular graphMiniDiscCodeComputer animationLecture/Conference
22:57
Electronic data processingMedical imagingTerm (mathematics)BitRun time (program lifecycle phase)HTTP cookieLibrary (computing)Right angleModule (mathematics)DampingBinary codeLecture/Conference
23:50
Linear programmingStandard deviationModule (mathematics)Library (computing)Open setBinary codeLecture/Conference
24:15
Computer fileProcess (computing)Directory serviceFunction (mathematics)Video game consoleSoftware testingRevision controlLine (geometry)Default (computer science)Context awarenessBootingSymbol tableOpen setLecture/ConferenceComputer animation
25:21
Data acquisitionCartesian coordinate systemRight angleLaptopCodeLibrary (computing)Stack (abstract data type)System callComputer virusLecture/Conference
26:17
Multiplication signRight angleLink (knot theory)Semiconductor memoryOrder (biology)Module (mathematics)Structural loadComputer fileCodeLecture/Conference
27:05
Operations support systemLecture/ConferenceComputer animation
Transcript: English(auto-generated)
00:04
Yeah, thank you for the introduction I'm not here not to talk about euro Python, but instead talk about Python so a bit of speaker introduction I've been around for ages. I don't really want to you know, tell you how long I've been using Python
00:23
I've been very active in the in the community. I've been euro Python society a chair and fellow and PSF founding fellow and Python core developer Unicode, for example, or the DB API. That's all on me
00:41
and There are lots of other stuff that you can read on the website there What I want to talk about is a use case that we in my company genetics. We ran into in 2008 so we had developed a product which consists it of a Client application that you could basically download as a package from PI PI and a server application that you had to install on the server
01:05
providing database connectivity It was called MX or DBC connect server and we allowed this to work on Windows and on Unix and for Windows we had a good solution for that for the server side So we just used pi to XE at the time
01:22
But for for Linux we didn't or for Unix in general we didn't find anything that was suited for this now Of course, we could have just said, okay Please install Python on the Unix server and then you install our package and it should work But the realities on Linux are typically different and other Unix derivatives
01:42
As well because every single vendor basically customizes the Python installation in one way or another and so Supporting all this as a small companies next to impossible So we wanted to make sure that the Python installation being used on the server side Was basically managed by us and so we wanted to ship Python the complete Python installation along with the product
02:05
So these are the or were the business requirements that we had back in 2008 Everything should have should be you know, completely independent of the OS Python installation We wanted it to make it very easy to distribute very easy to to put on the server
02:23
we wanted to support mostly Linux, but also other app platforms like Mac OS free BSD and In those days, I think we also supported Solaris And so we thought how can we do this because we didn't want to ship a hundred megabytes worth of files to the to
02:42
the client so I remembered an even older project from 1998 That I had started which is called MX CGI Python. How many of you know what CGI actually is? Not many anymore. So that was basically the way to do dynamic websites in those days
03:02
So in case you wanted to to put some you know Let's say a web shop or so on on the website on the on the internet Then you had to basically rent some space at a web poster Put your code there and and then hopefully it ran The only way to to do this was to upload files into a CGI bin directory on
03:25
an FTP site that you had And of course in those days Python was not as popular as it is nowadays So pretty much all the web servers that you could rent only supported pearl and maybe shell scripts So what we did is we wanted to put Python there and we had kind of like a similar situation
03:46
We wanted to make it easy to ship the Python to that web host or and so what we did is we basically thought it would be a good idea to wrap Python up into a single file just upload it use a shell script to make it executable and
04:02
I put it into the CGI bin directory to have Python as a runtime and This actually worked so we kind of hacked ourselves into the web posters and Lots of other people thought it was a great idea. And so we got lots and lots of help and support for various different platforms
04:21
all kinds of different Unix systems And we supported all the the popular Python versions at the time, of course after a certain while the web hostess started to realize that Python is actually a thing and so they started supporting it as well and we basically did not have to support MXC GI path anymore, but
04:42
This turned out to be an excellent recipe for for our problem that we had in 2008 so How does this work or what's the what's the recipe the secret sauce behind all this? So if you want to try to put everything into a single file, you have to do multiple things One is that you have to take the interpreter which you normally get as a binary
05:03
You have to add all the C extensions that come with the standard library and Link those statically into that binary so that you have those available and then you need to find out a way of how to get the Python code into that binary as well and the Way that we did this is we we use the tool for this
05:24
To basically compile all the Python files into bytecode and then put the bytecode into C arrays Compile them and then link them statically into that binary So the end result is the single binary that includes the the Python code of the standard library
05:40
plus all the C extensions and The tool we used for that is called freeze that is actually a tool that Guido himself wrote in 1994 so it's it's really really old. It still exists in the source distribution of Python I don't think many people actually know about it It's gotten some recent use for the import lip
06:04
that we have in C Python because the input mechanism in C Python was rewritten from C in Python and then This freeze tool was used to then basically do the same thing freeze the bytecode and then put it into the binary so
06:21
This is a great tool. It worked great on our Linux it Mark Hammet also made it work on Windows But we didn't want to use it on Windows So We had a solution for our problem. So we use this MX CGI to Python and and then basically shipped our product with this technology then of course
06:41
Because we saw that it's actually quite a successful way of distributing software We thought we might be able to take this a little further So we wanted to have a command line available for this for this binary Now the the thing is that if you freeze something and you compile it you don't have a command-line
07:01
Interface available that you can use so we had to basically Reimplement the Python command-line interface in Python and then again freeze it into the binary and make it work like that It's a little bit slower but it's a whole lot easier to maintain that the than the C version that you have in C Python and
07:22
Then over time, of course, we also wanted to have interactive Use available. So the REPL we added zip file support for the whole thing We added support for pip and setup tools and all the other nice things that you have in the packaging space We even made it possible to compile C extensions using that single binary. Of course you need to include files
07:45
We saw we had to ship those For for those cases where we wanted C comp C compilation to actually work But then you can just use for example pip and have pip compile things on the fly for you when installing stuff and then in
08:01
2012 at Euro Python, of course, we open sourced everything is eugenics pyran So, where are we today with eugenics pyran? It's today it's open source License it's Apache license It's it's free. Of course, like most of the Apache software. It's an almost
08:22
Complete, you know drop-in replacement for the Python runtime It's available for Python 3.8 up until 3.11 And I'm going to try to work on the port to 312 at the sprints So someone wants to help or maybe dive into this then please you'll come join me
08:40
the executable sizes are Pretty small actually, so you have a complete runtime in between four or six megabytes on disk and It works on Linux and also on on Windows if you use WSL It used to also work on on Mac OS and free BSD and some other Unix systems
09:01
But we haven't looked into those ports for the more recent versions anymore because we simply didn't have a use case for that It doesn't support native windows, but we could probably make it work as well And we've actually had lots of requests for this especially from teachers and other people, you know trying to
09:21
Show to to pupils or to students how Python works and make it easy to install It includes most of the Python standard library of course we stripped out a few things that we don't need for for shipping products like for example the test suites or a Couple of you know lesser used modules And it does support all the Python C extensions that you have so you can you can have numpy run
09:45
You can have pandas you can have polars All the packaging stuff works so it's pretty complete and this is the how it looks like when you open the shell and Look at the file sizes. You see this pyran 3.11 there. That's a compressed version
10:01
It uses upx the compressor for basically stripping down the The file and compressing it into an executable that decompresses on invocation the standard one that you see that the 19 megabytes that is uncompressed and
10:21
Depending on use case. Let's say if you want really fast startup times Then you should probably use the standard one because the upx one does take a bit longer to start up Because of the decompression that's needed and then of course we also ship the debug version which has all the debug symbols from the C compiler included in case you need that and
10:42
You run it just like standard Python and then you get a prompt you can do stuff in the rebel so I Mentioned what's what's in the box? I mentioned that it most of the Python sound libraries in there and I decided to basically also disclose What's not included of course because you know out of fairness
11:02
Like I said, the test models are not included Some other modules that we don't need are not included like intro PIP. For example, you can install that if needed You know using get pip dot py for example TK intro we don't have in their total we don't have in their parser tab nanny
11:21
We remove this to chills because it's been deprecated anyway And it's going to be removed and I think three to thirteen and Some of the other modules that are currently deprecated in 311 already because we don't need them And I don't think that many people are using them anymore There are also a couple of modules that we don't link statically again simply because we didn't need them one is the dunder
11:48
Decimal module some of you may may think well, why not decimal? We're not using decimal and and the decimal model is actually quite a huge chunk of code C types is not included because we again we don't need it read lines not included because of licensing issues
12:06
And you know the other things They all work if you install the dot SO files and everything can be configured So if you need these things, it's well possible to to put them into the binary as well
12:20
Right, so where are we now in? 2024 of Course we want things to work a bit differently than in 2008 we want to make it possible to to actually use pyran to to ship pyran applications, which basically means a single binary that has the pyran
12:42
Runtime plus your application and we might want to make all of that as simple as possible So that's what we're currently working on and Then once you have all that you have some great use cases for ejxpy run a great use case For example is putting it into a docker image because it's so small
13:01
If you come if you think about it, if you combine combine this with the Alpine images, then you get really really small Image containers that you can use to to fire up a Python runtime in your dark installation of kubernetes Something else that we thought about was having a
13:21
Truly OS independent virtual environment replacement So what you can do or what what we want you to be able to do Rather soon is to just take a complete virtual environment and put that let it let the the the tool Compile everything into this, you know single file or maybe just a small set of files
13:42
Like I said training and education is a great use case Because you want to make it very easy for people to just get up and running and same for demos plus There are probably lots more use cases that we haven't even thought about yet So where do we want to go from here? We want to basically create a small community around it because it's been open open source since 12 since
14:05
2012 But We did not have it on github yet The reason being that we use a mono repo and add eugenics and so it was not that easy to to just you know Put everything up online What we've done now is we have extracted everything from from our mono repo put it on github on July 1st
14:24
As version 2 5 0 and we're basically continuing development there So if you want to help feel free to just go to the get up get up Search for eugenics pyrun and then you can you can start, you know submitting pull requests or or file issues there
14:41
Now some advanced topics because I was saying that we want you to be able to run Python applications with this The way to do that is very easy if you have a Python application that is pure Python it's literally just creating a zip file with your Python code and Then using the Unix cat command to
15:04
Combine the pyran binary and your zip file into a new file and then make it executable and run it And that's it. That's your application so it's really really easy and You can even skip most of these steps by using the zip app module that you have in the standard library
15:25
No compilation is needed. This is it. It's simply concatenating two files The other thing like I said is if you want to customize pyran, so for example, you want to include extra Libraries or extra C extensions that we don't have in the standard set. It's also very easy to do that
15:45
You you don't need to be a court f to do this You do have to have some knowledge about how Python is being compiled though So you need to know how the freeze tool works how it finds modules how it puts everything into the binary And you need to know how to configure modules in C Python
16:05
And because all of this is still too complex we just said that We're just going to skip all this and we're going to create a command line tool for you to use so basically you can just enter the commands and then you know use a few options and The command line tool will basically take care of everything. So it'll provide ways of installing pyran updating it building it
16:27
We want to make it very easy to just create these pyran apps And we want to make it very easy to create these pyran virtual ends This is work in progress. It's not been released yet. So we're still working on it
16:41
Of course with every single Python version that gets released We have some extra work to do right because we need to port it to those new versions and sometimes things are easy Not many things change in in areas where we have to actually actually go in and then make changes to see Python
17:01
at other times It's actually a lot of work. So three dot eleven for example was way more work and way more complex than anything else We did before That was because a lot of things in the in the startup and a lot of things in the internals of C Python Changed in those areas that we used and so we we had to basically put a lot of effort into making it work
17:24
And you can also see that the pyran size changed a lot between three dot ten and three dot eleven And this is mostly because the bytecode in C Python Has become rather Well verbose, let's say
17:40
It's actually grown by about 40% between those two versions now the good thing though is that you know you invest a little bit more space on your disk and then you get a much faster C Python and That's basically on Mark and his team So Another nice thing is that this is mostly on us because of course we have the skills to do this
18:04
So you don't have to worry about it. Now what we like maybe you to contribute to all of these things is Some help in certain areas where we don't have the necessary deep skills Or maybe we don't have the necessary, you know platforms or infrastructure
18:21
So we would like some help with building and testing which should be fairly easy If you have the main expertise like say on on Mac OS or especially on Windows Then we would definitely like to tap into that. So please come and talk to me if you have skills in that area We also want to explore cross compiling to make it possible to compile for example on Linux and then
18:44
Have everything run on Windows. There's a great tool called zig CC. We want to explore for this And of course if you know about new use cases that we haven't thought about we'd like to know as well so You're welcome to to contribute and we hope you like the tool
19:03
We hope you like the community of that. We're going to build around this and yeah This is like the main takeaway from all of this, you know, never Never forget about the stuff that you've done in the past Always keep remembering the tools that you might have you know cut in your cover and then try new things never stop to learn and
19:24
Right, this is it this is the the resource page I put all the links up here the QR code goes straight to github and I have a few minutes left. So maybe I can show something Can you see that? Yes
19:43
The problem is I can't see it Right what you have here. This is a This is basically the the build directory where you where you built pyran and the way it looks on disk is
20:02
Basically like a standard Python installation. So you have a bin and include and lip directory The bin directory is pretty much what I just shown except that this also has some other stuff in it because we're using this Testing and as you can see pip is installed setup tools is installed a couple of other packages are installed because we're using these for testing
20:23
All of this works just fine You have the include files available. So if you want to compile something then you can just compile things You don't need this for for regular use of pyran and as you can see here
20:44
These are this is not This maybe Yeah, these are the SO files that we don't have directly in the binary But you can you can if you you know reconfigure the the setup files. You can put those in there as well
21:02
And as you can see the where's the decimal one decimal one is here So it's 1.6 megabytes just for decimal and if you don't use it, I mean why put it in there, right so But like I said, all this is configurable we're going to make it very easy to use and
21:21
then Just to show you how easy is to create these Clean here to create these pyran apps. So in this example, I use an old script I had Called xre which is essentially like in the command-line interface to the remodel
21:45
So it's really old code It's a huge huge hack I did at the time Yeah from 1996 ported it to Python 3 of course all this nice code and
22:04
If you want to use it, then of course ideally you just want to call it, right? So if you want to turn this into one of these pyran apps This is what you have to do and here I'm using the zip app module so you just invoke pyran over here and
22:24
Then use the zip app to create a zip file Which you can see here app dot pyz and then you can catenate both together and make it make it executable and then runs right This is how it looks
22:41
And that's all I have to say Thank you. Thank you very much mark. I don't know if there are any questions. Oh, we already have someone there I first want to give you this. This is a little cookie of
23:02
Gratitude for you And then the first question, please. Yeah. Thank you for the talk. It's a really interesting topic for me. I wanted to ask Since right now you don't support standalone binaries Could you go a bit more in detail about what the runtime dependencies are because I think as a bridge would be nice to
23:21
Have an image that has just those dependencies that you could copy into from the from the yeah In terms of dependencies you just have You mean the the binary dependencies? Yeah So essentially I could build it with like base Python image and then do like a multi-stage build and copy it into a scratch image with nothing and it would still run
23:47
Right. You basically just need You need a few libraries, of course that you need to for compiling the modules in the standard library So like visa visa to for example or open SSL you need those things
24:03
and Then once you have those you can basically compile everything and then the resulting Dependencies that you have on the binaries they are pretty standard. So they look like this you have to use the Not this one. I think this doesn't work. No, I have to go back to the boot one
24:28
Because UPX basically removes all the symbols so LDD doesn't work on it But if I go here and then here
25:01
These are the dependencies that you have on the binary So pretty standard stuff. You have lipsy you have to busy visa to you have SQLite of course open SSL and the rest is just on a Python send at seal it Thank you, okay another question from the from the other microphone
25:23
Hello. So what if your application is a Jupyter notebook that runs with custom code? Is it possible to package it as well? You would probably have to first turn the Jupyter notebook into a Python array and I think there's a tool for that I'm not very much into the data science stack. So I don't really know what it's called. But
25:44
Once you have that tool then yes, you can definitely take that and then compile everything into a single binary Not our question over here Thank you. Amazing talk Your solution pyran seems similar like beaver
26:05
Package I know about advantages disadvantages From if I compare pyran and beaver, right? I don't have experience with that particular tool, but I do know other tools that you know use a similar approach
26:21
I try to basically pack everything into a single file most of those tools though they they go and and basically put everything all your Python code into a zip file and then This tool can actually go and then compile everything into these into C and then link it directly to your binary
26:41
the advantage with doing that versus Having a zip file is that the OS can can much more easily Access the code. So the startup time is much better You don't have all this I own going on in order to load Python modules because it's already in RAM Or memory mapped, right?
27:04
Okay. Thank you very much. I don't think we have any more questions. So once again, give it up for mark