We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

The Catch in Rye: Seeding Change and Lessons Learned

00:00

Formal Metadata

Title
The Catch in Rye: Seeding Change and Lessons Learned
Title of Series
Number of Parts
131
Author
Contributors
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Python has a lot of packaging tools and what better way to solve the problem than to add another one. Rye is in many ways just another way to try to fix Python packaging, but it's also happy to not exist if something takes its place. This talk goes over all the lessons learned from writing it, what has lead to it's creation, how it's built internally and what we can do to just solve the problem once and for all.
Steady state (chemistry)MathematicsProjective planeShared memoryElectronic mailing listData miningSoftwareElement (mathematics)BitGroup actionMereologyCodeProgramming languageMultiplication signComputer animationLecture/Conference
Software developerSoftwareFormal languageProgrammer (hardware)Cartesian coordinate systemCodeOpen sourcePoint (geometry)SpacetimeText editorMultiplication signQuicksortCohesion (computer science)BitSet (mathematics)Arithmetic progressionHuman migrationSoftware developerBuildingFormal languageLevel (video gaming)Lie groupComplete metric spaceSoftwareComputer animationLecture/Conference
WindowData storage deviceVideo game consoleBeta functionComputing platformInstallation artMultiplication signBitFrustrationMereologyWebsiteLine (geometry)Binary fileData managementScripting languageQuicksortRule of inferenceUniform resource locatorAnalytic continuationDifferent (Kate Ryan album)Virtual machineCASE <Informatik>Pauli exclusion principleRevision controlProjective planeTheoryComputer animationLecture/Conference
Lattice (order)Software maintenanceTheoryPersonal digital assistantStandard deviationSoftware frameworkLaptopInheritance (object-oriented programming)WebsiteIntegrated development environmentQuicksortContext awarenessRight angleProjective planeSoftware developerOpen sourceRow (database)Installation artPhysical systemDemosceneStandard deviationNeuroinformatikOpen setComputer animationLecture/Conference
MereologyCuboidNeuroinformatikWebsiteRevision controlRepository (publishing)Order (biology)Projective planePoint (geometry)Computer animationLecture/Conference
Software frameworkMereologyInstallation artRevision controlComputer animationLecture/Conference
Time seriesComputer fileRevision controlRight angleLatent heat2 (number)Computer filePoint (geometry)NeuroinformatikProjective planeMultiplication signLecture/ConferenceComputer animation
Virtual realityData managementTotal S.A.Revision controlDemosceneBitComputer fileMereologyData managementVirtualizationProjective planeSynchronizationLecture/ConferenceComputer animation
Computer fileBuildingPoint (geometry)Projective planeVirtualizationQuicksortElectric generatorRevision controlComputer animation
MereologyResolvent formalismGoodness of fitCodeRoot2 (number)Revision controlDifferent (Kate Ryan album)Software repositoryMultiplication signLimit (category theory)Text editor1 (number)NP-hardLocal ringBuildingProduct (business)Projective planeUtility softwareStandard deviationVirtual machineRepository (publishing)Shared memorySoftware developerSingle-precision floating-point formatIntegrated development environmentModule (mathematics)Physical systemBitDot productSoftware testingAdditionResultantOrder (biology)Computer fileReal numberInternetworkingRing (mathematics)MathematicsInstance (computer science)RepetitionThermal expansionCategory of beingAsynchronous Transfer ModeVariable (mathematics)VirtualizationOpen setFile formatImage resolutionCASE <Informatik>AstrophysicsFlock (web browser)Computer animationLecture/Conference
Bit error rateTelecommunicationWeb pageExtension (kinesiology)Repository (publishing)String (computer science)Tape driveCodeDeclarative programmingComputer fileMereologyMetadataDynamical systemNeuroinformatikWindowLibrary (computing)Revision controlComputing platformResolvent formalismInformation securitySubject indexingSoftware maintenanceData storage deviceProjective planeInstance (computer science)InformationMultiplication signLocal ringRight angleUniform resource locatorSoftware developerPauli exclusion principleElectronic mailing listWritingAbstractionQuicksortStandard deviationMassNetwork topologyShift operatorDifferent (Kate Ryan album)Phase transitionHuman migrationFrequencyUniverse (mathematics)Default (computer science)BitSoftware bugPortable communications device
MassSingle-precision floating-point formatRevision controlImage resolutionStandard deviationPauli exclusion principleBinary fileTerm (mathematics)Uniform resource locatorGoodness of fitMereologyInstance (computer science)Different (Kate Ryan album)ConsistencyWindowCartesian coordinate systemQuicksortPoint (geometry)Right angleComputer filePauli exclusion principleMultiplication signProduct (business)Revision controlSingle-precision floating-point formatBinary codeStandard deviationBitExtension (kinesiology)Module (mathematics)Form (programming)Software bugContext awarenessData conversionGroup actionElectronic program guideSoftware testingComputer programmingInformationMetadataResolvent formalismFunctional (mathematics)Link (knot theory)Physical systemBuildingMusical ensembleProgrammer (hardware)Integrated development environmentSubject indexingMeta elementImage resolutionInternet service providerPortable communications deviceRepository (publishing)Data loggerComputer animationLecture/Conference
Integrated development environmentInformation securityBitFirewall (computing)Queue (abstract data type)Lecture/Conference
CASE <Informatik>Design of experimentsRevision controlOpen sourceSoftware maintenanceQuicksortVideo gameDifferent (Kate Ryan album)Point (geometry)Multiplication signRandomizationComplex (psychology)Interpreter (computing)Binary codeFormal languagePhysical systemSoftware testingDirection (geometry)Virtual machineInternetworkingMultiplicationInformation securityIntegrated development environmentInstallation artSource codeManufacturing execution systemSoftwareLecture/ConferenceComputer animation
Transcript: English(auto-generated)
Cool. I'm quite happy to talk about Python Packaging for once. I don't know if you know what Rise, but it was a little project of mine that I started in April of last year. And it is yet another tool in a long list of tools for Python Packaging.
So I want to just share basically how it came to be, what did I learn from it, and maybe what we can do as a community with it. So I will not go into myself too much now. You might use some of my software, you might not. But the key part is that I actually built a lot of it 10 years ago and it just didn't
do that much Python anymore. But why? So I got a little bit burned out on Python software. And I also started to pour a lot of my time into Sentry, which is the company I work for.
And through that company and also through this Python 3 transition that I didn't enjoy all that much, I started looking quite a bit into Rust as a programming language. And of course it was modern and new and everything, so I kind of fell in love with some elements of this. And so I brought a lot of Rust code in the first couple of years at Sentry.
If you're not familiar with Sentry, I'm not going to bore you with it, but it is entirely written in Python with some Rust code in it, obviously. It is an application monitoring tool. And I have spent 10 years working for this company, basically, at this point, which is the longest, by far, that I've worked on anything.
And through that, I still stayed, even though I didn't write that much open source Python code, very much in contact with Python day to day. So quick detour to Rust. The tagline of Rust is a language empowering everybody to build reliable and efficient software.
It's a little bit of a lie, because everyone will not enjoy Rust. It's very complicated compared to Python. But what is interesting is that you're really surprisingly productive writing Rust code if you sort of sit in the kind of space of Rust where Rust is actually productive.
There's a lot of stuff that probably you would really hate if you have never used Rust before, but there's a certain set of it that's really good. And the main reason a lot of people, myself included, like writing Rust code is the developer experience is really great. And to give an example of this, syntax autocomplete in most editors works really well.
Packaging is solved for once. There's just a lot of stuff that works really well, and it does it to such a high level of quality and cohesion in the community that it just sets like, oh, this is possible. Like you can build stuff like this. And also what drawn me to it was the language really strongly encouraged backwards compatibility,
but it also had a very strong urge to enable innovation and progress. So it's sort of, when I started using Rust, there was actually a huge fear of Rust making the same mistake as Python 3 migration did, where it's like, how can we prevent this
from happening? It was like a really big thing at the time. What affordances do you have to put into the language to make a future Rust 2 reality without everybody having to rewrite the code or doing a painful migration? So I want to take you back in time a little bit to around 2014. I guess I mentioned I picked up Python, Rust roughly at the time of Python 2 was still
very active for me. Of course, a lot of people moved to Python 3 already. But professionally, I used Python 2. And what is important is cargo wasn't a thing yet. Cargo is this sort of tool that is a packaging manager.
You can think of it as like PIP for Rust. But it slowly came to be a thing. And there were four or five competing tools at the time in the Rust ecosystem for what the best package manager should be. There was an earlier cargo, there was a later cargo, there were a bunch of other attempts. But if you use Rust today, this is just one tool.
And I want to cite PEP 20. There should be one, and preferably one obvious way to do it. Rust has achieved that. Python didn't. And this is the continuation of this. No case is special enough to break that rule. Packaging definitely isn't a special case. In theory, there should be one and very one obvious way to do it.
And so I saw the light in the Rust community, packaging doesn't have to be painful. And not just this packaging solve there, it also has a bunch of really other interesting things. Like, you never have to think about any more brew install Python.
Have you ever... I don't know how many people use Windows here, but if you ever use the Windows 11 machine and you write Python to console, the Windows store pops up and tells you that you can download Python from the Windows store too. Like, there's so many different ways in which you can download Python, there's just one way to download Rust. And any version of Rust installs the same way on any platform. And I can have one project that runs an older version of Rust, and another project
runs a new version of Rust, I can test the beta version of Rust, it all comes automatically. And it's the same experience on Windows, Mac, and Linux. Part of my frustration with Python packaging over the years has actually been teaching it to other people, and not Python packaging, but Python.
And even as you write a tutorial for Python packages, like, how do I install a thing into something, you have to make one line for Windows, because the activation script is in VM slash script slash activate, and on Mac OS and Linux is VM slash bin slash activate, and also with a dot in front. Like, there are all these sort of differences, like, it's all different locations and
so forth. Anyways, long story short, at Sentry, we have a tool. It's open source, but we really can't use it for anything. It just manages your Python developer experience for you. It installs dependencies, it installs Python to your system, it uses PIP tools behind the scenes. And the problem of Python at Sentry was basically solved for the most part.
But I also had side projects, and I still wrote a lot of side projects in Python, and so for me, I had this little tool, never to be published, that just did the same thing as we had at Sentry, just for my own uses. And in 2023, I talked to a bunch of people, like, that every company has their own little
way of running PIP tools, and I felt like, okay, maybe it's nice to show people what kind of crappy thing I sort of hacked up that makes it work for me. And the reason I never publish this is this, right? Everybody's aware of this XKCD. If there are, like, 14 competing standards and yet another one that rules them all,
you're going to have another one of those. And so when I published RAI, I actually announced it with precisely this comic, because I said, I do not want to be yet another Python packaging tool that sort of piles up on top of that. And funny enough about XKCD is that there's even one that specifically addresses Python packaging and Python installations.
So this is a problem that's well known out there, right? So my only goal was to dominate, and if not to dominate, then something else would dominate, right? I just want it solved. I have no desire that my tool is going to be a solution for this problem. And so keep that in mind. I'm not going to, I'm not here to pitch RAI on you, right? What I'm here to pitch on you is this idea that we should solve this problem.
If you haven't used, sorry, I just want a quick show of hands how many people have actually tried RAI. Very few people, this is perfect, because I'm going to annoy you for, like, two minutes with a very short recording of what it does. So, when you go to the website, you can curl RAI to your computer.
It will just ask you four questions if you want to use the new fast way to install packages or the slow old way to install packages. And it automatically downloads Python for you, and it's there to run. Basically at that point, you can keep using your other Python that you have, but you can now go into any repository, into any folder, you can create a new project
there with RAI in it. And it's going to use the version of Python, it just downloads it. It automatically installs your packages, it auto-manages your virtualenv, you don't have to do anything anymore. You just start Python, you don't have to activate the virtualenv, you just get access to the stuff. It just works magically out of the box. So that's the experience that I had for a while and that I wanted to show people
like this is great. That's my version of this. I like this idea. The biggest part about this was actually for me originally to get Python installed in the first place. Because I'm a Mac user. It's not as bad anymore as it used to be, but a very common experience that I had is what I installed a new version, like brew install Python, and then random other tools
stopped working or everything updated, or like I accidentally installed something into the global Python for no good reason, all of a sudden, like, I don't know if you've used this much, but there are different ways to install Python, framework builds, non-framework builds on Mac, it's very complicated. So what I wanted is like, you never have to think about this anymore.
And there's a tool that we use at Sentry called Volta, and if you use Rust, there's a tool called RustUp, which does this. Volta does it for JavaScript, RustUp does it for Rust, which is, you just install it once and then it fetches, like pyenv does, Python for you, but without compiling, it's pre-compiled. And so you can ask Rai which pythons are available, if you want one that's not there,
you can just say like, I want it, a couple of seconds later, it has downloaded it, and then you can directly select a specific version of Python, it also reads you a Python version file, and it gives you the right version of Python. I like that, because I don't have to think about this anymore, it's just there. And if you don't have it yet, it tells you exactly what command to run to get it.
The next thing I really like is log files. I want that if I have a project on my computer that used the version of whatever package I had at a certain point in time, that I give it to my coworker, they're going to have exactly the same version of this.
Rai, disclaimer, does not have log files, but it has a requirements.log file, which looks a little bit like one. Behind the scenes, you use a tool called UV to install packages, and so it guarantees that everybody has the same version of these packages installed. And it does it automatically for you, you don't have to think about it, it manages those. Last part, virtualenv management, I've shown this a little bit, but basically, whatever
command you run, so if you run Rai at Django, it will add it to your pyproject.toml, but it will also automatically install it into virtualenv. So you can see, it is as a dependency in the pyproject.toml, and if you wipe away your entire virtualenv, it will recreate it.
If you run py in this folder, it automatically has the virtualenv enabled, if you run Rai sync again, you get it back, even if it deleted it. It's pretty straightforward, and also thanks to UV, it's very fast. Okay, so that's what I wanted, that was the experience that I wanted. It's actually not that hard to build the initial experience, it's very, very hard
to actually make it work well. And so this is the little disclaimer here, is that most of what Rai is doing at this point is not Rai at all. There's a company called Astral, which raised money for Python tooling, it might be familiar with one of the tools that they built called Rough, which is a linter, and
I talked to a bunch of people after I sort of built Rai, and said like, hey, can someone please build something like this, and make it really great. And I talked to Charlie from Astral a lot about this, and they independently started
working on a tool called UV, which today basically does virtualenv creation, pip installation, pip syncing, log file generation, that sort of stuff, but it doesn't have the end-to-end experience. And I told him, I will donate my project to you, do whatever you want with it, I have
no desire that it survives, you can rename this, you can do whatever you want with it, and if for whatever reason you don't like what Rai was doing today, I will also commit myself to making a version of Rai that's like the latest, the last version of Rai ever, that's just 100% dispatches to V under the hood, so that it can stop existing, right? So I mentally got prepared to this idea that my tool is there to die, and hopefully be
replaced by someone else. That's really what I want. I want to never have to think about Python packaging anymore, that is sort of the way I'm going about this. So the question here is, does Rai work? Yes, there are issues, and so it does work, but there are issues, but surprisingly enough,
all of the remaining issues are not technical challenges, they're actually a problem all of us can solve together, which is we have to rally around fewer tools and more opinion about how stuff has to be done, and there are a bunch of standards we might have to adjust a little bit to iron out the last bits and kinks that are in this whole thing.
And it can be roughly defined in the following order. Def dependencies, which is the concept that I have a package that might depend on some things in addition for running tests of a local development that I don't have in a production usage, for instance, if I want to make a Docker build.
Local dependencies, which are dependencies that I have to a checkout of another Git repository on my same machine. So I think pip install editable, so I might in production use depend on a published version of this package, but doing local development, I want my five different packages in different
folders without having to run pip install all the time. Then the concept of workspaces, which is basically just an evolved version of this, which is if I have a mono repo that consists of 20 different packages, I don't want to set up 20 different packages for development, I want to check out this mono repo and just have the whole thing magically work.
Then there is pyproject.toml, which comes with a bunch of limitations that I think stand in the way of innovation, let's put it this way. And then mypadp, which is a single version resolution, which is you can only, the resolver is required to come to a single solution for what version to use for a dependency.
And that's a problem if that solution cannot be found. So if you have part of your project that wants pydantic1 and part of your project that wants pydantic2, you are out of luck. Python cannot do that today, but I think it is solvable. And the last part is good Python builds, which is there are some limitations to how
RY fetches Python from the internet today that I think we should fix or could fix, and I want to pitch you on this idea that we should advocate for this and maybe throw some money into the ring. Okay. When I started RY, the biggest problem was the resolver, which is how do you actually figure out which package to install?
It's not fully solved because clearly a lot of more work has to go into UV, but UV as a tool, I mean, of course, I like what the Asper people are doing, but UV is a very, very good tool. It is not perfect, but it is very quick to install packages. On a hot cache, like if you have used this project before, I have a project that takes
I think one or two seconds to install, whereas PIP tools before took over two minutes. That's of course a little bit of a corner case, but there are a lot of packages in the ecosystem like Bodo or PyTorch, stuff like this, which it can take a really long
time for PIP to come to a conclusion of what is the right package to install, and UV is very optimized of doing this very quick. So that problem is solved, and I think that is very nice, mostly solved, I would say, but it is not anymore the biggest problem. The biggest problem now for me is actually depth dependencies. Every tool that we have today in the ecosystem, so we have PDM, we have poetry, we have,
of course, Rai, UV, plenty others, have their own suggestions of how to deal with depth dependencies, and that is a problem because it is a tax that everybody has to pay all the time. So for instance, if I open VS Code in a new checkout, I cannot just run format, run the
code formatter, because how does VS Code know which code formatter to run? Like there's no concept of that, and there's no place where I can say like, oh, I know this project uses rough, so I need to install rough as a tool into this thing.
VS Code has a very complicated system for installing dependencies for these tools, and every editor would have to copy this. There are different ways to run these depth dependencies, some of which are fully isolatable, so if I run black, which is a code formatter, or rough, which is a lint and code formatter, these ones can be isolated, which means I can put them into new
virtualenv, they will never import my code, perfect, I can reuse it. Others cannot work with isolation, pytest needs to import my code, so it has to be co-located with the other code, and others are sort of in a mixed bag where like flake8 sometimes can be co-located, because it cannot be co-located, because it might import custom linters from the project, sometimes it can be.
And so we have no concept of declaring these dependencies, so everybody invents one. I will not say like what solution I would like, because I don't really know, I also don't really have a strong opinion, all I know is that there should be one solution, and it should not be too flexible, because what I learned in Rust, which is very nice,
is in Rust, I can check out the repository, and I don't have to configure anything, I open the editor, and the editor knows while I'm in development, I install all the dev dependencies, if I run docker build, I don't install any dev dependencies. And so that understanding of like there's just two modes, and this is how they work,
means I don't have to configure anything anymore, because all of those tools agreed on these very basic conventions, which is a very, very nice property, local dependencies. That is a very hard problem today, everybody knows about pip install dash editable dot
dot auto folder, and then you get the thing installed there into a virtual end for thought, having to reinstall it with every code change, the problem is there's no standard for doing that in a pyproject.toml file, and there is no way in which this works reliably, every tool again has their own solution, hatch uses curly braces with variables, PDM uses
percent project root environment variable expansion, some versions of pip allow you to do pip install dot dot slash in the requirements txt file, some tools that pip cannot
do that, they require file slash slash a bunch of other stuff, and at the end it's a very confusing experience, and so some people know how to make it work, others don't, end result is just a very disappointing experience. And it really stems from the fact that we don't have a standard of saying like this is how dev dependencies should work. My proposal that was thrown into the ring is that the way dev dependencies work, local
dependencies work, you really only have them during development. So what you could do is you could say like I actually depend on the real package name, but while I'm developing, here's a separate file sitting around somewhere that says like this package called other slash step is actually checked out in this other path,
and whenever you install the virtual end, don't do the thing where you go to PyPI and download it from there, but actually go over there to this other folder and do an editable installation there. It still doesn't solve this problem entirely that we need to find a way to also reflect this in this dependency array, so that requires some changes that haven't been done yet,
but it will get us somewhere. And I think that will actually greatly benefit the way local development works. What could help even more is this concept of workspaces. RAI does support this today, so you can, I don't know if you can see this on the
in a mono repo, you can say RAI show, and it will show you every single package that's in this configured workspace, and they all share one virtual end. So that has the benefit that they can cross reference each other, so I can have my main
monolithic code depend on my like GRPC utility module to talk to my microservice somewhere over there, and I don't have to deal with multiple virtual end, I don't have to deal with multiple editable install, it just magically works. But again, it's a proprietary extension, right?
Like I edit another stupid standard on top of our ever-growing ecosystem of Python standards, and that really is not great, right? That there's a tool dot RAI dot workspace, and the same way as a tool dot PDM dot local dependencies and all that stuff, it's just, I'm fueling this burning pile of Python packaging tools, and I don't want to do it, I want that we think like, okay, maybe this
is a thing worth solving, let's get together and try to do it in a way that we don't add another tool to it. So in RAI workspaces, as much as I like to say that this is a solved problem, don't work well enough, they're definitely very ripe proprietary, and I run into a lot of
challenges with this local path thing. It's hacked together with duct tape and string and glue and terrible solutions. But when it works, it's a really magical experience, because you just check out the repository that consists of like seven different dependencies, and you can just start working
immediately. No complicated stuff. So why is it so hard to solve some of those problems? Like why can you not just go in and say like, hey, I have this idea for an extension, let's make a PEP out of it. Part of this is that this is actually not a technical problem, it is not really a problem
of like going in and making it a standard, it's actually coming to an agreement of how it should work. And I will put a finger on I think the part that I think is the trickiest. I forgot to write down which PEP it is, but I think it's PEP 508 or something like this. There's a PEP that says like, how do you declare dependencies?
And this is how you do it, there's like a project dependencies, and then every single dependencies is a list item. And this list item is a very complicated string. Of course, you can just write Flask in there, and it will depend on some abstract version of Flask. But you can write things like, I want to depend on more ITER tools in the version between
4 and 6 if the Python version is 2.7. But if the Python version is greater than 2.7, then I want more ITER tools greater than 4 or something. 2.7 is probably like a weird Python version here, because nobody uses it anymore, but these sort of really complicated dependency declarations, they happen all the time.
And they are the reason why resolvers are sometimes so hard, working so hard to come to a conclusion of which version to use. And the problem with this dependency array is that there is no place in the string to attach extra information. And that's actually a big problem.
For instance, let's say you want to have a project depend on Flask, but you don't want to depend on a version of Flask that's on GitHub, sorry, on PyPI, but you want to depend on a version of Flask that's on GitHub. This is one of the best things that happened to the Rust ecosystem, is that I can depend
on a third-party dependency, and when I have a bug in there, rather than going and saying I depend on the one that's on PyPI, I make a fork of the dependency on GitHub, and I rebind in my requirements.txt file, or in cargo Tom will Rust speak, my dependency from what used to be on PyPI on GitHub.
So I can immediately start fixing the bug, can use the bug-fixed version, every developer will automatically pull in the Git version, and it can still work as if nothing has changed. You can sort of do the same thing with requirements.txt files, but you cannot do the same thing with PyProject Tomo files today. It is not permissible, or at least in most tools it will not work to have this Git
URL in there as a local dependency. So what I would like to have is a better way to declare dependencies in this PyProject Tomo file, so that we can have this extra information attached. And one of the reasons it would be very nice to have this extra information attached is that, for instance, if you work in a company, some packages are on PyPI.
Some packages might be on your internal index. So what I would want to do is to say this package comes on PyPI by default, but this package is my company proprietary other package, so I want to say look only on that index. That would solve a bunch of security problems we have today. Or I want to say, I have this really cool tool that I wrote, and I want to put some
extra information on there so I can run some experiments. For instance, I would like to do auditing on my dependencies. If I could make a marker on my dependencies, like this passed my security check on day XYZ, then I could write a tool that actually does auditing on my dependencies. But all of this is contingent on the fact that I can store extra information on it.
And I would also need this for better supporting local checkouts and stuff like that. There are other issues with PyProject Tomo. I don't want to go too much into this, but there are 20 different ways. Again, we like to do things many times. You can have dynamic metadata where instead of looking into the PyProject Tomo file to
find the version, you have to run some Python code. There's so many countless extensions by different tools. There are 20 different ways. I think there was even talk about this, of defining licenses in PyProject Tomo files. It's a very complicated thing, and there's some opportunity to simplify this.
This is slowly being sorted, but it's still not entirely solved. This is this concept of portable locking. If you have an M1 Mac, you might depend on different versions of the libraries than you do on a Windows computer, and so the resolver has to come to different conclusions depending on which platform it is on. UV and write support universal locking now, similar to poetry, but it's kind of buggy,
and it's not very documented yet. And this problem is actually very hard. The reason it is hard is not so much because it's technically hard. It is also technically hard, but it is because we have as a community decided that it's a great idea to have these really complicated dependency trees. And so there are certain packages that if you start depending on them, they will just
make your resolver really slow. And I'm not saying like we as a community should punish those packages and say like, hey, doing something stupid, stop doing that. But I think that if tools were a little bit more like telling a user of why the installing doesn't work or why it's hard to resolve, but pointing at the problematic
package and say like, hey, this thingy over there keeps doing stupid things with dependency declarations. Maybe you should talk to the maintainer nicely and tell them there's a better way to do it by being like more specific with the dependencies, right? Because that would actually make the whole thing much better over time.
This would be the hardest thing to solve, but I do think it is solvable, which is to allow multiple versions of one library to be available. If you depend on version greater one and version lower than one simultaneously in your
project, it is uninstallable. You cannot come to a conclusion of what the right version of this package is. JavaScript doesn't have that problem. Rust doesn't have the problem either. Why do they not have the problem? They don't have the problem because they say like it is actually okay for two separate
versions to coexist in your dependency tree. So when you have dependency Flask, depending on, okay, Flask doesn't do it, but like say you have your project that depends on pydantic1 and you have this other library somewhere in a tree that depends on pydantic2 already, you don't have to say like, well, I cannot
solve this because clearly they want different versions. I can actually say this one sees the newer version, this one sees the older version. Now I don't think it's a good idea in general to do that because then you have this massive dependency tree of like 20 different versions of pydantic in your tree. But for transitionary periods where I as a library offer are moving my library from like
1.0 to 2.0, I might want to temporarily say like I'm this migration phase where I am willing to subject my customer or my user to having two different versions of this in the tree. That would make these massive ecosystem shifts like pydantic2 to pydantic3, pydantic1 to
pydantic2 be a much easier move because I can have both of the things at the same time. There are challenges with this, right? Because the APIs don't really support it, there's just modules which is a bunch of really weird stuff that stands in the way. But I actually think that it is very doable, it is technically solvable. For just Python imports, even 13 years ago I had a version of this that worked.
It's only running in Python 2, so you need an ancient Python to test it. And it doesn't solve C extensions, which is a little bit of a problem now because we have so many of them, like NumPy, PyTorch, pydantic, like all of this stuff that's a lot of Rust
and C extensions, but I actually do think that it is solvable. If we agree as a community that it would be good to support that, at least in limited form, I think we could make it work. Last part, good Python builds. There is a PEP called PEP 711, which proposes a standard for distributing Python binaries.
That is not accepted, PEP, and if I want to provide things like RYFETCH 3.8, where is Python 3.8 coming from? I'm telling you, it's not coming from Python.org. Why does it not come from Python.org? Because the Python you can download on Python.org is not qualifying for the needs of what RY
has, which is automatic installation. I can place it at a random location on the disk, and long-term I would love that RY could also just create a reusable package of just my application plus Python that can throw in a, maybe not USB stick, but somewhere where I can give it to another person
that never used Python in their life and run it. So what it needs is a portable Python build. So how does RY accomplish that if you can't find this Python Python.org? It goes to GitHub, it goes to a repository by a person named Indy Greg, his actual name
is Gregory, I forgot his last name, and he is the one person in Nebraska, he's not really in Nebraska, but he's this one person that builds every single Python version for every single architecture, and I can download it from there. So I can get 3.8.2 for RM64 on that thing.
And I can get it for Windows, I can get it for Linux, I can get it for all of that. But it's run by a single person. And the thing is, it's not just RY depending on this. I know that there are companies who gave up making consistent, essentially for instance, consistent Python experience, and we also use those Indy Greg builds.
So it's getting to the point where some people are discovering the benefits of this, but it's sitting on the shoulder of one person, which is not ideal. So that is a thing that I think if I can only wish for one thing, it would actually probably be this. Like give me the Python as a binary I can run and use and automatically install.
It would be really nice. However, long story short, I'm actually very optimistic about the whole thing, because I feel like we're very close to solving it. Technically, I feel like Python packaging at this point is mostly solved. And not because of RY, but because of all the work that went into this whole thing
over the last couple of years. PyProject.toml, great improvement that was set up with Py. Metadata available for resolvers to use, tremendous improvement. That stuff like UV actually can install dependencies that quickly has been the work of a lot of people that made this metadata information available on PyPI.org.
Generally speaking, the improvements to the build system infrastructure underneath the hood has enabled a lot of this functionality. Wheels are there. Like a couple of years ago, wheels were still not widely used. So all of the bits and pieces underneath are there to make all of this work. What actually stands in the way is us as a community.
I feel very strongly at this point, and again, I will very clearly say that I do not think that RY has to be the solution for this, or maybe even shouldn't be the solution. But I think it would be great for us as a community to say, like, hey, we actually want this one experience.
That is, I go to Python, I download this meta thingy, and that's all I have to do. It gets Python to me, it installs my packages, it can publish on PyPI.org, maybe it can publish on a private index, there's one command to run the tests, there's one command to create my virtualenv, and I don't have to read 20 different guides anymore and 15
different tools to make it run. And I feel like we are very close to that. We are very close to that because there are so many tools that sort of showing up as being like the blessed tool underneath the hood that we can start using. RY never implemented the resolver itself. RY used PIP tools. Because at the time I felt like PIP tools was the best tool to use for that.
And now I think UV is a better version of it, because it's the same thing, just faster. But RY didn't do anything at all. It was a crappy version of a little Rust program that just cobbled together stuff from other people. And so all these tools are there. We just need to package them up in nice reusable thing and say, like, this is the way for
you to run it. The last thing I'm going to say on this is this is sort of a lesson that I learned in multiple times over the years, which is when you point at the floor of something, there will always be someone addressed in it. I do the same thing.
Like, if someone points at the bug in a thing that I built, I'm like, all right, I got this. I will fix it. That's a little bit of a problem, because that implies that when someone tells me, like, this is a problem, and I feel like, ah, I'm going to fix it, and it's like, that's the last time I'm going to talk to you, maybe I don't actually have enough context to fix this problem. Maybe the problem is bigger than I am.
And I see this a lot with Python packaging. We have I don't know how many different groups of people that are trying to solve Python packaging. And whenever I see conversations about, like, hey, it would be great if this does this, everybody runs off and solves it on their own.
Log files was an interesting discussion recently, everybody was like, oh, I know how to fix log files. And so we have seven different log file standards. Like this cargo or cargo for Python, three different people working on it on their own. Hedge tries to do it. Rye, of course, tries to do it.
UV tries to do it. So we're all guilty of the same thing, which is, like, everybody feels for themselves, like, I know exactly how to solve it. And what actually I think the problem here is that, like, this is not a problem of technology, this is not a problem of finding solutions, this is a problem of creating the necessary desire that this would be great and sitting together in some sort of productive kind
of environment and saying, like, hey, what should this look like so that we can have the same sort of experience that you might find in Rust and else. And so I, again, guilty myself on this because I didn't talk to everybody about it. But I feel like there is, for me, a nice outcome is, like, if multiple people in the
community, myself and Cruz, are like, you know what, I've worked X amount of time on this tool here. I'm actually okay with it slowly being replaced by something else. And we all work together on making this thing work a little bit better. So that in the end, a new Python programmer stepping into the world of Python finds
a very consistent experience everywhere and never has to think about any of this anymore and just can focus on solving actual problems and not Python packaging. And that's basically it. I will throw up some links if you want to play with it.
I feel like it shows a little bit where Python packaging could go. And I really hope that maybe in a year or two from now, we all feel like now we can worry about other things that maybe are more important than packages. And if there are any questions, I'm happy to answer.
For the questions, please queue in the microphones that you see on the two of the sides.
Okay. That was a long walk. But anyway, so what do you think about the security implications of this? Like many people would say just putting in cURL and downloading something and running
it. It's a bit of a security flaw. And also, how does that work in a corporate environment where maybe you don't have a firewall access to actually download from the public? Yeah, so you don't have to cURL to Bash anything. You can go on GitHub and download the thing directly. In general, I think that there is actually not that much wrong with, before I started
cURL to Bash, because the actual problem is that everybody just nearly willingly downloads random packages from the internet to begin with. Setup.py, pip install can run anything on your machine.
And what we should actually probably be figuring out that was a panel on open source sustainability is figuring out how to make the life of individual maintainers really well so they are not the people you have to worry about. Because if I'm an overworked open source maintainer, the biggest risk I'm going to pose
on you is my personal security, where someone might get access to my package and republish it. So I would worry a lot less about, hypothetically, if the Python Software Foundation would provide this Python installer, that that is compromised compared to, I don't know, your random package from Nebraska being attacked.
And so I think it would work just fine in corporate environments, because you can at any point in time do all the things that it downloads manually from another source. You can rehost all of the binaries if you want to.
Just go quickly. So when installing a Python interpreter, the requirement to have a version pinned version of the Python interpreter is a real requirement, but it applies across languages.
So is it realistic that every language, and even things that aren't languages, are going to have to have a system like Calgo or Rai, or is there more mileage in tools like SDF and Miis that install tools across languages with version pinning, but don't
do anything with the packages? So I came back and forth on this, and I'm now strongly believing that this should be solved language by language independently, because the needs of those languages are so different, and it's very hard to have a one-size-fits-all tool that deals with
this externally. Just as an example, Tox is a great tool. It can run tests against multiple Python versions at once, and so it in itself wants to find different Python interpreters, and if it now has to work with, like, how do I do this on Nix, how do I do this on something else, it sort of permeates this complexity everywhere.
So I think it's easier to say, like, well, we're going to solve it for Rust, we're going to solve it for Python, we're going to solve it for JavaScript, and most of those ecosystems have already done that. All right, thank you, sorry, we're out of time.