A Tale Of Python C Extensions And Cross-Platform Wheels
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 115 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/58839 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 20214 / 115
1
3
19
25
31
34
36
38
41
42
44
46
48
52
60
61
62
65
69
73
76
82
83
84
92
94
96
103
110
113
00:00
Extension (kinesiology)MathematicsGoogolWide area networkSoftware engineeringMereologyModule (mathematics)Extension (kinesiology)Library (computing)Functional (mathematics)Default (computer science)Directory serviceData conversionComputer fileObject (grammar)Error messageShared memoryElectronic mailing listSystem callLipschitz-StetigkeitMultiplication signParameter (computer programming)Computer programmingString (computer science)IntegerView (database)Software bugFile formatExecution unitDoubling the cubeCodeDynamical systemData managementLinker (computing)Sheaf (mathematics)Variable (mathematics)Ocean currentArc (geometry)Physical systemFormal languagePointer (computer programming)Cellular automatonSingle-precision floating-point formatWeb pageFunction (mathematics)Ext functorEmail1 (number)ImplementationException handlingCyclic redundancy checkCASE <Informatik>Projective planeType theoryUtility softwareBitMacro (computer science)CuboidTerm (mathematics)Cartesian coordinate systemWebsiteStrategy gameNumberInterpreter (computing)BuildingEinbettung <Mathematik>Connectivity (graph theory)Set (mathematics)Link (knot theory)Group actionStapeldateiMusical ensembleRoutingBlogMatching (graph theory)Computer configurationExtreme programmingInstallation artGastropod shellComputer architectureVirtual machineSoftware developerInstance (computer science)Mobile appIdentifiabilityRevision controlTable (information)Matrix (mathematics)Medical imaging2 (number)Social classKeyboard shortcutSelf-organizationPattern recognitionDifferent (Kate Ryan album)Interface (computing)Endliche ModelltheorieTouchscreenProcess (computing)Latent heatSummierbarkeitSource codeField (computer science)Token ringSubsetWindowProbability density functionDistribution (mathematics)Integrated development environmentOrder (biology)Scripting languageHash functionFrequencyLine (geometry)Configuration spaceSemiconductor memoryPoint (geometry)Programmer (hardware)Computing platformOperator (mathematics)AutomationGoodness of fitComputer fontInformationFlagData structureContent (media)ResultantCountingData storage deviceFinite differenceTime zoneOperating systemSpeicherbereinigungProgramming languageCross-platformOpen sourceWritingMemory managementStandard deviationVisualization (computer graphics)Rule of inferenceRepository (publishing)Letterpress printingRun time (program lifecycle phase)Maxima and minimaGeneric programmingOnline helpCase moddingPattern languageStructural loadUniqueness quantificationoutputMeeting/Interview
00:45
Meta elementExtension (kinesiology)Module (mathematics)Library (computing)BuildingGroup actionProbability density functionIndependence (probability theory)Menu (computing)Scripting languageModul <Datentyp>World Wide Web ConsortiumGamma functionGEDCOMDesign of experimentsBuildingExtension (kinesiology)Library (computing)Probability density functionModule (mathematics)Functional (mathematics)Parameter (computer programming)2 (number)Interface (computing)Keyboard shortcutIntegerMacro (computer science)Endliche ModelltheorieMereologyType theoryCodeEmailComputer fileElectronic mailing listWebsiteInformationObject (grammar)FlagWritingSlide ruleMemory managementResultantAdditionCountingImplementationData storage deviceCASE <Informatik>NumberFunction (mathematics)Operator (mathematics)Multiplication signStapeldateiWrapper (data mining)Single-precision floating-point formatWeb pageData conversionString (computer science)Error messageSystem callSpacetimeInstallation artExecution unitInstance (computer science)Utility softwareTable (information)Interpreter (computing)Data managementPattern recognitionIdentifiabilityVariable (mathematics)File formatLine (geometry)Pointer (computer programming)Repeating decimalProgrammer (hardware)AutomationPoint (geometry)Computer fontComputer programmingCartesian coordinate systemConnectivity (graph theory)SpeicherbereinigungSet (mathematics)BlogExt functorOperating systemPhysical systemRevision controlJSON
08:08
Extension (kinesiology)Front and back endsPhysical systemLibrary (computing)Structural loadProcess (computing)Directory serviceIntegrated development environmentVariable (mathematics)Order (biology)Read-only memoryStandard deviationBuildingMereologySubsetSelf-organizationLevel (video gaming)Configuration spaceLibrary (computing)Dynamical systemFunctional (mathematics)CodeBuildingExtension (kinesiology)Single-precision floating-point formatWeb pageDefault (computer science)Function (mathematics)Macro (computer science)Data managementComputer configurationVirtual machineInstallation artVisualization (computer graphics)Software developerGastropod shellMultilaterationProjective planeServer (computing)Group actionPhysical systemComputer architectureMedical imagingScripting languageWindowProbability density functionDifferent (Kate Ryan album)Revision controlSubsetLatent heatCASE <Informatik>Instance (computer science)Module (mathematics)Order (biology)MereologyIntegrated development environmentVariable (mathematics)Letterpress printingInformationLinker (computing)String (computer science)Computer programmingRun time (program lifecycle phase)Directory serviceCartesian coordinate systemElectronic mailing listOcean currentSet (mathematics)Computer fileMessage passingTerm (mathematics)Type theoryBitConfiguration spaceWebsiteLink (knot theory)Multiplication signProcess (computing)Semiconductor memoryExt functorSource codeStandard deviationEmailFormal languageFinite differenceParameter (computer programming)Front and back ends
15:30
Library (computing)Extension (kinesiology)Modul <Datentyp>Mach's principleRead-only memoryDirectory servicePhysical systemVariable (mathematics)Integrated development environmentStandard deviationOrder (biology)Structural loadModule (mathematics)Hash functionDistribution (mathematics)Group actionRevision controlPasswordConfiguration spaceBuildingMatrix (mathematics)Gastropod shellField (computer science)Exception handlingRevision controlExtension (kinesiology)Process (computing)Directory servicePhysical systemInstallation artToken ringFunction (mathematics)Software developerVisualization (computer graphics)Library (computing)Electronic mailing listSubset1 (number)EmailProjective planeModule (mathematics)Functional (mathematics)Computer fileFormal languageExt functorStandard deviationData managementView (database)Ocean currentCodeWindowScripting languageSheaf (mathematics)Group actionUniqueness quantificationDefault (computer science)Computing platformParameter (computer programming)Variable (mathematics)Integrated development environmentGeneric programmingOnline helpCASE <Informatik>Pattern languageHash functionSet (mathematics)Resource allocationTable (information)Multiplication signAngeordneter KörperMultilaterationSource codeLink (knot theory)Maxima and minimaSemiconductor memoryStapeldatei
22:53
Memory managementExtension (kinesiology)Modul <Datentyp>Distribution (mathematics)CompilerHash functionKey (cryptography)Extension (kinesiology)CodeType theorySource codeDistribution (mathematics)Lecture/Conference
23:32
Bit rateInferenceLink (knot theory)Social classLibrary (computing)Revision controlWindowMeeting/Interview
24:55
Lemma (mathematics)WritingGame theory
Transcript: English(auto-generated)
00:07
All right, let's welcome our next speaker, Vinayak Mehta. Vinayak is a software engineer and an active open source contributor. Hi, Vinayak. Hi, can you hear me? Yeah, I can hear you.
00:21
I still remember your talk from the last year about command line applications and interfaces was a really good one. Thanks. Yeah, so today Vinayak is going to talk about writing Python C extensions and cross platform views, very nice. So over to you Vinayak, good luck. Thanks.
00:46
So, hey everyone, I'm Vinayak. And today we'll talk about how we can write Python C extensions and package them into wheels so that they're installable by users on any major operating system straight from PyPI. This is a basic outline of the doc.
01:01
We'll write a basic extension module using the Python C API. Then see how we can do that using PyBind 11, which is a package that offers seamless operability between C++ and Python. After that, we'll briefly look into shared libraries and dynamic linking and see how we can build wheels for our extension module and finally automate wheel building using CI build wheel. The goal of this talk is to give you
01:21
a good starting point for how you can go about building wheels for your extension modules. The day begins at the Recur Center, which is a self-directed educational retreat for programmers in New York City. Last year in August, I started my RC batch with the goal of removing a platform-specific dependency from Camelot, which is a Python library that can help you extract tables from PDFs.
01:42
The dependencies used to convert a PDF to a PNG so that Camelot can do line recognition and identify tables on a PDF page. The problem with this dependency, GoScript, which is a large PDF interpreter written in C, is that it isn't available to install from PyPI. Users need to install it using the system package managers like apt, pacman, or brew, et cetera.
02:02
And on Windows, they need to go to the download space to get the EXE installer. And then hope that Camelot, which is installed from PyPI, works seamlessly with GoScript. In the past, there have been instances where users would install the incorrect GoScript version for their system architecture, leading to segfaults. While Camelot would not be able to find the GoScript
02:20
executable on the path, leading to an import error. This led me to search for a pure Python PDF to PNG converter, which could be installed directly from PyPI. I wasn't able to find one in pure Python, but I found a C++ library called Poplar that has a CLI utility called PDF to PPM, which could let me do this. I was able to write a wrapper on Poplar called PDF to PNG,
02:41
which takes in the path to a single page PDF, converts to a PNG, and then writes it to the given output path. I was also able to build wheels for every major operating system, and then include PDF to PNG as a requirement for Camelot, so that it can be installed directly from PyPI. In this talk, I'll describe all the steps I followed to package the PDF to PNG C extension module,
03:01
which depends on the Poplar shared library. On a side note, if you're interested in other things I've worked on during my RC batch, or what it feels like to do RC virtually from another time zone, you can check out my blog, where I wrote a blog post for every day of my batch. So C Python, the reference implementation of Python, as the name suggests, is written in C.
03:20
It exposes a rich API, using which we can extend Python's functionality. The main use cases are writing extension modules, so that we can use C or C++ modules in the Python interpreter, and embedding Python as a component in a larger C or C++ application. We're gonna look at the first use case in this talk. We're gonna look at how we can write an example module,
03:40
which just lets us add two numbers. So we start by creating an example.c, and including the python.h header file, which will give us all the functions in macro definitions needed to use a Python C API. Since all Python objects are treated the same way by Python, the programming language, they're also represented by a single C type, by C Python, the reference implementation.
04:01
In C Python, everything is a Py object. This is what a C function to implement our add Python callable looks like. It takes in a pointer to the self-py object, and a pointer to the args-py object, which will contain all the arguments that we pass to this function. We then define two long integers, a and b, and since we get all the args that we pass into this function in the args-py object,
04:22
we can use a py-arg-pass-tuple function to pass those parameters into individual variables. You can think of it like scanf from C. Here we are passing two long integers from args, and storing their values in a and b. L is a format unit for the parameter that we want to parse, and there are two of them, ll, because we are parsing two long integers.
04:41
The string after the colon and after ll is used as a function name in error. L is not able to parse our args and return false. We need to return null to raise an exception and pass it on the call stack. Then we define two py objects, p and pb, and then use a py long from long function to convert c longs a and b
05:00
to two long py objects p and pb. And finally, create another py object called r, which stores the result of the py number add function on p and pb. In the end, return r, which is the result of the addition. But before we do that, we need to use pydecref to decrease the reference count for p and pb, the two py objects that we created.
05:21
This will ensure that p and pb are de-allocated by Python's garbage collector. If you're writing an extension module using the Python C API, you need to be aware of Python's reference counting model for memory management so that you do not leak memory. You can check out the resources in the last slide to learn more. After creating our add function, we need to create an array of py method dev structures,
05:41
which will define the list of functions that our module exports. In that list, we specify the name of our, the callable that we should, that should be available in Python, the C function, a flag that says our C function should be called with self and args, and finally, our function docstring. And after that, we need to create a module definition struct,
06:01
which holds all information to create a module object. Here, we can add the module name, its docstring, and the list of functions exported by the module. Finally, we need to create a module init function called pyinit example, which needs to follow the convention pyinit underscore module. And call the py module create function inside it with a reference to our module definition struct.
06:22
And that's most of our C extension code. To package our extension, we'll use setup tools. We start by importing extension on setup tools, passing the name of our module, the part to our C code, and then call the setup function with this ext.modules list. And we install it using pip. Pip will pick it up, use setup tools to build the extension, and install it in our site packages.
06:42
And if we import it in the Python REPL, we can see that pip used setup tools to build a shared library for our module. And voila, we should be able to use the add callable now. We can also store its return value in another Python object, since we return it from the function. Now let's look at how we can create
07:00
the same extension module using pybind11. Pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa. We can use it to easily create extension modules on top of existing C++ code. So this time, we create an example.cpp and include the pybind11 header file instead of the python.h from before. We create an add function,
07:21
which takes in two long integers, a and b, and returns the sum, which is also a long integer. And finally, we use the pybind11 underscore module macro to initialize a module. We pass in the model name as the first parameter, and the second parameter, m, defines a pybind11 variable, which we can use as a main interface to bind our C++ functions. We then define the module docstring with m.doc
07:42
and add all the C++ functions that we want to use as Python callables with m.def. The first argument for m.def should be the name of the Python callable. Second argument should be the reference to our C++ function. Third argument should be the function docstring, and the rest of the argument should be the parameter that the Python function accepts, in this case, a and b.
08:01
And that's most of our C extension code using pybind11. Now we get to a set.py, where we again create an ext.modules list after importing extension on setup tools. We pass in the name of a module, the path to a C++ source file, path to the pybind11 header files using pybind11.getinclude, and specify the language as C++.
08:20
In the set.py, again, we call the setup.function with the C ext.modules list. And finally, we need to create a pyproject.toml, where we define all our build requirements so that pip can install and use pybind11 at build time. We also need to set setup tools as a build backend. Finally, when we install our extension module using pip,
08:40
which will then, it'll then use setup tools to build it and install it in our site packages. And if we import it, we can see that it built a shared library out of our extension module and put it in the site packages just like before. And it works like before too. With pybind11, it's easy to write an extension module or wrap an existing C++ library
09:01
to create a new extension in terms of not having to change existing function parameters and return types. The API to write an extension is also a bit less verbose compared to playing with pyobjects. With the Python C API, we add two first-person coming parameters, convert them to pyobjects, and then finally return one too. We also have to raise appropriate exceptions,
09:21
but pybind11 can automatically take care of all of that for us. In the case where I wanted to wrap the PDF2PVN popular utility, I could just take it C++ code with a set of files that identifies in the main function and turn it into PDF2PNG.cpp where I change the name of the main function, pass it into parameters, the paths to the single page PDF and then output PNG
09:41
and initialize the module with the pybind11 underscore module macro. And after building it for different operating systems, I could access the C++ convert function in my Python code like this. Let's look at how we can build extension modules that depend on shared libraries. But before we do that, let's briefly look into shared libraries and dynamic linking. To understand shared libraries and dynamic linking,
10:02
we'll go through an example of how a program might run on Linux. Let's say we have a C program which prints a string called hello world. When we compile it with GCC to get an executable and run it, this is what happens. The program sees that it needs the code for printf from the C standard library called libc.
10:20
And since it does not have it, it asks the dynamic linker LD for the path where libc is stored. The dynamic linker looks for it, finds it in one of the default paths and gives it to the program which can then finally finish executing. Here libc.so is the shared library and the whole process of finding it is dynamic linking. Shared libraries are loaded into memory
10:40
the first time a program that requires them is executed. And if during that time, another program that requires them starts executing, it can just reuse a copy that was already loaded into memory. On Linux, shared libraries follow the libc.so naming convention. For example, libc, libpng and libpopular. This is a shared library search order on Linux.
11:00
LD first looks into one of the default directories, then looks for all the directories listed in its config file, then looks for any parts on the LD library path environment variable and so on. You can check out the full list on its man page. On Windows, shared libraries are called DLLs or dynamic link libraries and have the .dll extension. The Windows dynamic linker looks like this. It does not search for a DLL
11:21
if a DLL with the same name is already loaded into memory or if the DLL is on a list of known DLLs for the current Windows version. Otherwise, it'll search for a DLL in this order. It'll look at the directory from which the application was loaded, then the system directory, then the Windows directory, then the current directory, and finally, the directory is listed on the path environment variable.
11:41
Let's try our own simple shared library on Linux. Let's say we have three different files, mod1c, mod2c, and mod3c, which contain three functions that print some strings. We can compile all of them into a shared library called libfunc.so by passing the shared option to GCC. Now let's say we have a program which expects to call these functions one by one at runtime and then print hello world.
12:03
We can get an executable for it and link it to libfunc.so like this. And when we execute it, we get an error. It says that the dynamic linker can't find any libfunc.so. Yeah, that's because our current directory, where libfunc.so is located, is not one of the default directories that dynamic linker looks into. We can temporarily fix this by adding the current directory
12:21
to the LDN library path environment variable like we saw in the Linux search order. And yay, it works. All the strings that we wanted to print have been printed to stdout. Now let's see how we can use that information to package our extension with shared library dependencies into wheels. In part one, we look at building shared libraries that our extension module depends on.
12:42
You might not need to build one because most C or C++ shared libraries that you want your extension to link with might be directly installable through a system package manager. But there could be some instances. For example, you might want to drop a large private C or C++ code base, in which case you might need to build a shared library first. The build instructions could also vary widely
13:01
for each C or C++ project that you're looking to build. We look at the tools we can use to build our shared library on each of these operating systems. And we'll go with the examples for building popular and linking it with PDF to PNG. Let's start with Linux. Each Linux restore has its own package manager and its own latest version of shared libraries.
13:20
That's a problem because you can easily get version mismatches when you compile an extension with specific versions of some shared libraries on one Linux restore and then install it on a different Linux restore which might have different versions of the shared libraries. To solve this problem, the PyPA came up with a subset of shared libraries that are available on all major Linux restores. If you compile and link your extension with a subset,
13:42
it is guaranteed to work on many Linux restores, thus calling it Metal Linux. According to the Metal Linux 2014 spec, this is the shared library subset you can link with and then stop worrying about these libraries not being present on your user's Linux system. PyPA pushes Metal Linux Docker images for different versions and architectures to QA.io.
14:01
So you can pull the image for the architecture that you want and run a Docker container where you can play around with building your shared library. In the case of Poplar, we need to install these packages using yup, yup, and then build a libpoplet.so-shared-library which is what we need to link with PDF to PNG. We'll put all these steps in a shell script so that we can reuse them later
14:21
when we are looking at automating wheel builds. For building a shared library on macOS, we either need a Mac computer which runs macOS or we can use FastMac which gives us SSH access to a macOS server on GitHub Actions for six hours. It is very useful for debugging potential build issues. On macOS, we can install packages using BLEU and then build a shared library like we did before.
14:44
And then again, we put all those steps in a shell script which we'll reuse later. For building a C or a C++ project on Windows, we need a Windows machine. And we'll also need to install Visual Studio 2019 Community Edition with the Python native development tools. Windows doesn't come with a system package manager.
15:02
So one option to install the dependencies that we need is vcpkg which is a C++ library manager. Hayesoft can use vcpkg to install the shared libraries that we need. And since vcpkg won't install shared libraries on the default search path where CMake could find them, we need to specify the vcpkg installation group directly to CMake.
15:21
vcpkg is also installed by default on the GitHub Actions Windows runner. And this environment variable is also available there which will be helpful when we are looking at automating wheel builds. Again, we put all those steps in a batch script for reuse later. After installing shared libraries or building them on their own like we just did, we need to make sure our extension is linked
15:40
with them when it's compiled. We can modify our set.py to do that. If the extension is built on Linux or Mac OS, we'll create a list of directories to search for poplar at link time. We'll also need to create a list of library names that we want to link with. In this case, it's poplar because we want to link with libpoplar.so. If the extension is built on Windows,
16:00
we'll also need to add the vcpkg installation to directly to the search path and create a libraries list with the names of all the libraries that we need to link with. There are a lot more libraries in this list on Windows compared to the last slide because we're using vcpkg which doesn't install the dependencies in a standard search path. Unlike Linux and Mac OS, we're yum and blue handle that.
16:21
And if your C or C++ project needs some header files, you can create a list of all directories where they might be present. These are all the ones that poplar needs. And at the end, we also specify the Python 11 header file directory using the Python 11.getinclude function. We can then create a list of ext modules where we pass in the name of our extension module, the path to the C++ source code we want to build,
16:42
the include directories, the library directories, the libraries themselves, and specify the language as C++. Finally, we call the setup function with our list of modules. And when we install it using BIP, we should be able to use a convert function from our extension module in our Python code. We can also create a view for our extension module using pip wheel,
17:00
which should create a wheel in the current working directory. If you look in one of these views, you'll see that our extension module has been built and placed into the wheel, but the shared library set depends on aren't there, which means that if we ship it to PyPI, users who install it would also need to install the shared libraries using their system package manager.
17:21
This brings us to our next section on how we can bundle shared libraries inside our wheel so that users don't need to install them separately. On Linux, we can use audit wheel, another tool that PyPI has put out. It can check if the many Linux views that we built are compliant with the shared library subset that we saw earlier, and also bundle shared libraries which are not in the subset into the many Linux wheel.
17:43
Here's how we can use it on the wheel we saw earlier. We run audit wheel repair with output directory and the wheel we need to repair as inputs. In our case, we temporarily add the popular build directory to LD library path because it's not one of the default directories LD can look into to find lib popular.so.
18:00
And if you look at the wheel again, we can see that all the shared library set that our extension needs are nicely bundled into the wheel and their names are also mangled with a hash for uniqueness. On macOS, we can use delocate, which works in the same way as audit wheel. We can list all the dependencies for our wheel using delocate list steps and then run delocate wheel with output directory and the wheel that we need to repair as inputs.
18:23
Again, we add the popular build directory to the di-ld library path, which is the same as LD library path, but for macOS. And again, we can see that all the shared libraries that our extension needs are nicely bundled into the wheel itself. On Windows, because of the DLL search order, we could basically place all our DLLs in the same directory
18:42
from where our extension loads, which is the first path in that search order, and we should be good to go. Which means that we can copy over all the DLLs from the vcpkg installation directory and specify the package data keyword argument for the setup-rules.setup function. So we make a minor tweak to our setup.py by adding a copy-dlls function, which finds all the DLLs
19:01
in the vcpkg installation directory and copies them over to the directory where our built extension will be present. If the platform is Windows, we call the copy-dlls function and add the star DLL pattern to the package data like this so that DLL files are included in the wheel that we built. And if you look at the wheel, all the DLLs are present there, and our extension should work.
19:22
In case where your build process puts DLLs into another directory, which is not the same directory as your built extension module, you can use a windll.loadLibrary function in your modules in a .py to load the DLL before anything else happens. And since Python 3.8, you can also add a DLL directory to the DLL search path using os.addDLL directory.
19:42
And bundling shared libraries with generic names like we just did, we could run into and contribute to DLL help. If you remember the Windows DLL search order, the system won't search for a DLL if a DLL with the same name is already loaded into memory which means it's possible that the DLL version set is shipped with a generic name, might not play well with DLL versions that other wheels ship with a generic name
20:02
and vice versa. Depending on your particular scenario it needs, you can go the DLL mangling way where you mangle the names of all the DLLs like order field does and then update their import tables to reflect the newly mangled names. To do this, you can write a script which unpacks your Windows wheel, recursively looks for DLL dependencies for the extension module
20:20
in the vcpkg installation directory, mangles their names using their sha-256 hash and copies them into the same directory as your extension module. It should then modify the import tables for your extension module and each DLL using a library called macho macho-maggler. And finally, it should zip everything again into a view. This is what the Windows view looks like after those steps.
20:40
You can see that the DLL names are mangled with their sha-256 hash and their import tables are also updated with those mangled names. You can also check out Dell wheel which works like audit wheel and allocate but on Windows and does these DLL mangling steps that I just described. Finally, let's look into how we can automate wheel builds using CI build wheel and GitHub actions.
21:01
CI build wheel is a useful tool which can help us build wheels for all platforms with minimal CI configuration. This is how it works. We create a GitHub workflow which will get run on each push of attack. In the environment variable section, we specify the cibw build variable with cp3 question mark which should build wheels for all Python 3 minor versions
21:20
on Linux, MacOS, and Windows, except Python 3.5, which will be specified in the cibw or discourse skip variable. We can specify all the commands that should run before building our wheel on Linux using the cibw before build Linux variable where we pass in the shell script that we created earlier. And then we also specify how our many Linux fields should be repaired with audit wheel using the cibw repair wheel command
21:42
Linux variable, which contains the same command that we saw earlier. You can do the same for our macOS build script and specify how we want to use Delocate to repair our macOS wheels. And we do the same for our Windows build script, then specify how we want to use Dell wheel to repair our Windows wheels. We then define a matrix of operating systems
22:01
we want to build wheels on, check out our GitHub repository, install Python. On Windows, we also need to set up the developer command prompt for MS Visual C++, which we can do with this GitHub action. We then install cibw and run it like this where we specify wheelhouse as an output directory. We do the same for Linux and macOS like this.
22:22
And in the final step, we upload the build wheels as a build artifact, which we can then download and manually upload to PyPI. Or just add another job, which can do that for us automatically. The first step would be to download the build artifact from the last build job and then upload it to PyPI using the PyPI publish GitHub action from PyPI.
22:41
We just need to configure our PyPI token as a secret and we should be good to go. And when the build and upload job finishes, we should be able to see our wheels on PyPI. And that's mostly it. When I was playing around with PyPI 11 in Google, that Julia Evans wrote some years ago,
23:01
I tweaked it a little bit so that it resembles a PyCon Australia mascot curly boy and added various types of food emojis that it can eat. And I built wheels for Linux and macOS so that it can be installed just using PIP. If you're interested in a source and the build process, you can check it out here. Here's all the code that you can check out to learn more.
23:21
And here are some awesome talks you can watch to learn more about Python's key extensions and their distribution. Thank you for listening. You can reach out to me on these links. Thanks a lot Vinayak, it was really a nice talk. We have two questions, I'll read them out for you. Let me put them on screen as well.
23:40
Pybind 11 infers versions 11 of C++. What about the newer versions of C++ like 14 and 17? I think I didn't show the Pybind 11 build-ext class. That's in one of the examples that they put out in their GitHub organization, which lets you add other C++ versions too.
24:02
So I can post a link in the room later. Sure, that would be great. Hope that answers the question. The next question is about how to disable shared library DLLSO discovery on Windows Linux. Oh, I'm not sure about the question. Why do you want to disable their discovery?
24:24
Okay, I think this can be taken offline then. So yeah, that was a really nice one. I see people typing in the matrix channel. So if you have any more questions, please feel free to reach out to Vinayak
24:41
on the Breakout Optiva channel, and he would be happy to answer them all. Thanks once again Vinayak, I really enjoyed it. It was a pleasure listening to you once again this year. Thank you. Bye. Bye.