Multi-build Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 33 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/54693 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
2
3
5
6
7
9
13
14
18
19
21
22
24
26
28
29
30
32
33
00:00
Revision controlModul <Datentyp>Virtual realityMultiplicationBuildingMacro (computer science)Musical ensembleProgrammschleifeSingle-precision floating-point formatSystem callConfiguration spaceLatent heatProjective planeMacro (computer science)Revision controlModule (mathematics)IntegerNoise (electronics)Module (mathematics)Computer fileDot productMultiplication signInterpreter (computing)TrailBuildingGoodness of fitDirectory serviceAnalytic continuationSoftware testingWebsiteInstance (computer science)MultilaterationThermal expansionLoop (music)Focus (optics)Binary fileSet (mathematics)BitStreaming mediaPhysical systemSoftwareDistribution (mathematics)Standard deviationRow (database)WordNumberLie groupEmailLibrary (computing)MereologyRing (mathematics)OvalIntelComputer animation
07:06
FingerprintSoftware testingMacro (computer science)Installation artComputer fileReading (process)Complex (psychology)RootFunction (mathematics)WebsiteInformationSource codeMetadataMacro (computer science)Revision controlModule (mathematics)MultiplicationComputer fileWebsiteComplex (psychology)Installation artFingerprintSoftware bugBuildingSoftware testingSingle-precision floating-point formatGame controllerBinary fileModule (mathematics)Distribution (mathematics)Exterior algebraSheaf (mathematics)MetadataPlanningHash functionOpen setFormal languageProjective planeSoftware developerCache (computing)Source codeGroup actionDescriptive statisticsScripting languageTouchscreenCloningPoint (geometry)CASE <Informatik>Sinc functionMessage passingVariable (mathematics)Integrated development environmentOvalSystem callComputer architectureLibrary (computing)Object (grammar)Sound effectMiniDiscUtility softwareThermal expansionUnit testingFitness functionPiElectronic mailing listNumberError messageCellular automatonArc (geometry)Independence (probability theory)Parameter (computer programming)Common Language InfrastructureBootingComputer animation
14:06
VideoconferencingVirtual realityHypermedia
Transcript: English(auto-generated)
00:10
Good morning, good afternoon, or good evening wherever you are in the world and welcome to my talk about multi-build Python here in this beautiful 2020 continuation
00:23
So first a little bit about me I live in Sydney Australia surrounded by wildlife that wants to kill or maim me and I have roughly 20 years experience with packaging software Most of that is Debian based, but a lot of the concepts remain the same across just disciplines of distributions. I
00:45
Started with Python in roughly the year 2001 and I quickly fell in love with it So I've been following along with it from the early days of 1.4 I've been at SUSE now for coming up to five years and I've spent just over a year on the Python packaging team
01:02
with a few others So I'm going to assume that you have some familiarity with the RPM build system How macro expansion works and a tiny bit of knowledge about OBS? So which version should we ship? Tumbleweed currently ships the full stack for Python 3.6
01:24
3.8 and 3.9 So this means we make every effort to fix build failures for leaf packages in for those versions We also currently ship Python 2.7 in Tumbleweed, but we're trying to remove it and Disentangling it is quite hard work
01:42
We don't build any leaf packages for 2.7 And a lot of up streams are shifting their focus away from it obviously The main version of Python so that is what you get when you run user bin Python 3 is currently 3.8
02:00
Leap 15 ships 2.7 and 3.6 as its fully supported versions So we build packages there for 2.7 and 3.6 We do ship later versions But only with the interpreter pip and setup tools and so we don't build leaf packages for those later versions
02:22
So there's this disconnect between Python base and the Python package Python base ships the actual binary and The parts of the standard library that don't require external dependencies Because it's contained in ring 1 The Python package contains everything else
02:43
Because some modules require external C libraries and things like that That's a little bit of a lie too because Python base requires Bluetooth, but we get around that Python devel includes the header files like python.h and we of course build documentation in the Python docs package
03:02
Python as a bare word in our in in package names means Python 2 whereas Python 3 has the version name the version number in the name such as Python 3.8 devel or Python 3.9 base So the macros I'm going to talk about are contained in this package
03:24
Their standard RPM macros and that means they're written in Lua and not Python so I'm going to skip over the internals So I introduced this back row first because pretty much every other macro uses it We set this macro in the tumbleweed project config and the leap project config
03:46
To which versions of Python we want to build modules for So this is currently set to Python 3.6, Python 3.8 and Python 3.9 for tumbleweed If you don't specifically override it, that's what you'll get
04:02
you can override it if you wish and If you only want to build for a specific version of Python you can set this to Python 3 and we call this single spec So you'll only build for whatever the main version of Python is when it's built So some complex packages do this like TensorFlow
04:24
So Python build and Python install these two macros both just basically call setup.py for each version Python build calls setup.py build and Python install does the same thing and calls setup.py install
04:41
So Python expand loops over whatever is defined in percent Python so you can run things for each version of Python for instance you may wish to remove something in each site package directory or run a custom test runner or run fdupes or something
05:01
I include this macro fairly early because a lot of the macros that I mentioned further on down the track use this internally So from time to time you may need to know what version of Python you're building under So you can for example on the build require on a module
05:21
for one version of Python only So these two macros help with that so Python version returns the minor version so 3.6 and the no dots variant returns a whole number so 36 So skip Python is a definition and not a macro
05:42
Set it to one and that will instruct other macros such as Python expand that they should skip this version So for example percent defined skip underscore Python three six one You could specify it multiple times, but any more than two
06:01
You should probably think about setting percent pythons And you can't use this definition to switch to single spec because if you do then the package you're left with will Contain that version and not be called for example Python 3. It will be Python 3.8 something So Python module we use this in build requires
06:22
It will add on Python dash for you and will also loop over every version in pythons so build requires Python underscore module requests will install Python 3.6 requests Python 3.8 requests and Python 3.9 requests
06:42
This can also get a little awkward when modules already have Python in the names, so you get things like build requires Python module Python dash foo and Due to the way this expansion works you can specify Python module devel Which will pull in for example Python 3.8 devel
07:04
So this line noise exists in a lot of spec files It defines an expansion for Python module if it doesn't exist so that it will pull in Python foo and Python 3.0 foo If your spec file is really only for tumbleweed and it won't go anywhere older
07:22
Then you can you don't need this you can remove it But it also doesn't hurt because it will only take effect if Python module isn't defined So Python sub packages Python files this will expand your source package to Python X name
07:41
So for example Python 3.9 Django where it includes the version number the macro Effectively generates new spec files for each version it encounters and along with its friend Python files And that expands the percent file section I've listed these macros together because you have to use them together if you if you have one without the other you'll get errors
08:07
So since a fair number of packages ship entry points that are installed into user bin This means we need to specifically handle that case when we build packages for multiple versions Since we're not able to ship the same file in multiple packages
08:24
Twisted is a really nice complex example of this because it ships a lot of entry points So we we use clone to clone the script which moves it From script name to script name dash version, and then in the file section we mark it as an alternative
08:43
Don't forget to Require post update alternatives as well as requires post on update alternatives And be sure to add Python install alternative and it's opposite Python uninstall alternative to the post and post on sections
09:01
Don't worry if you missed all of this it will come up later So Python site lib, Python site arch there expand to The firstly to the arch independent site packages So distutils calls this purelib as in modules that are pure Python
09:23
And that's most often used because most modules are pure Python and the arch dependent one distutils calls this platlib and you'll use that for packages that build a shared object a .so file or use something like CFFI
09:41
So these testing macros of Python Pyunit test and its architecture cousin They set the Python path and environment variable and then call Python minus M unit test discover Which discovers tests and then runs them percent pytest and its arch dependent cousin percent pytest underscore arch
10:04
Sets the Python path And then calls pytest to do the same Some packages still call the last one on the list But upstream setup tools has deprecated that use of setup py And so you should update them to use the first macro
10:23
But also you shouldn't feel the need to be constrained by these macros only to pick on twisted again it for example uses a trial in its check section to run tests So Let's let's put this all together. So we have a simple example
10:42
Python crayons. I've trimmed down the spec file so to fit on screen I've dropped things like version release description and the prep section So you can see we only really require Setup tools Python module and the rpm macros package to build it
11:01
So this package also contains no tests so there's no check section but this shows how Python module Python sub packages Python build Python install Python files and The Python site led macros all work together in one spec file so if we build it it gives us the source rpm and the three binary packages for each Python version and
11:27
If we pick on Python 3.9 crayons, you can see it ships the pycache files The egg info the crayons module itself It's readme and the license So this is a more complex example Python Raven. I've had to trim it down a lot more because it's a lot larger
11:47
You can see that we're using version qualifiers in Python module boot requires And this is the update alternatives requirement that I mentioned before So build is very simple here
12:00
But during install we need to clone the Raven CLI tool and run FJUPS During check we export an environment variable for Django's benefit And then we use the %pytest macro and as you can see we can pass command-let arguments to pytest using that a macro
12:20
So here we install and uninstall the alternative and we also mark it as an alternative in the files section as a bonus example from Python talks We can see here if guards for a build requires We can't use Python module here because that will expand against pythons, and we can't do that since
12:45
Import lib metadata only exists for Python 3.6 We also need to check for both Python 3 base and Python 3.6 base, so we handle both tumbleweed and bleep Since this may be backported to either distribution
13:03
So future plans that we have we're discussing allowing user control over which version of Python is run for user bin Python 3 but this gets it been complicated and May in fact Not work out well for things like single spec and things like that, and we'd also very much like to remove
13:24
Python 2 from tumbleweed But it's pretty firmly wedged in there with with a lot of things unfortunately still requiring it So would you like to help you can of course file bugs on any packages you can send us a submit request to our development projects in OBS, which is
13:45
Devel languages Python or its sub projects, and we have an ISE channel on LibraChat, which is percent hash openSUSE Python and Thank you for listening are there any questions