Python as the keystone of building and testing C++ applications
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 15 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21083 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
SoftwareComputer hardwareLink (knot theory)Uniqueness quantificationCartesian coordinate systemBuildingLink (knot theory)Computer hardwareMusical ensembleSoftwareSoftware engineeringVideo gameLecture/ConferenceComputer animation
00:43
Electronic visual displayDemosceneLaptopWorkstation <Musikinstrument>DigitizingCartesian coordinate systemWindowTouchscreenCodeNuclear spaceElectronic mailing listString (computer science)Video gameNetwork topologyDemosceneSoftware frameworkMusical ensembleElectronic visual displayCategory of beingLink (knot theory)Execution unitComputer animation
01:24
Scripting languageGame controllerMIDISoftware developerLink (knot theory)SynchronizationCartesian coordinate systemMusical ensembleBeat (acoustics)outputGame controllerWritingMIDIScripting languageRemote procedure callComputer animation
02:02
Scripting languageGame controllerMIDIContinuous integrationData managementBuildingStatistical hypothesis testingOpen sourceComputer fileObject (grammar)Physical systemVisual systemCodeRepository (publishing)DisintegrationComputer virusComputer-assisted translationCarry (arithmetic)Software testingConfiguration spaceMaxima and minima3 (number)SineInformation managementAsynchronous Transfer ModeMusical ensembleOrder (biology)Visualization (computer graphics)Software maintenanceComputer fileModule (mathematics)Unit testingScripting languageBuildingFunction (mathematics)Physical systemRepository (publishing)Video game consoleParameter (computer programming)Continuous integrationData managementConfiguration spaceDivision (mathematics)CodeSource codeCompilerElectric generatorLinker (computing)Projective planeCartesian coordinate systemWindow1 (number)Object (grammar)Default (computer science)Flow separationBitSoftware testingView (database)Total S.A.Sinc functionStatistical hypothesis testingCompilation albumStructural loadLevel (video gaming)Computing platformLibrary (computing)Product (business)Different (Kate Ryan album)Content (media)Open sourceSystem callSoftware frameworkRevision controlExpert systemProcedural programmingConfidence intervalMereologyDivisorVideo gameComputerExecution unitFreewareCorrelation and dependenceFrame problemMotion captureProcess (computing)AutomationHyperlinkComputer configurationHill differential equationSocial classAbstractionWordBinary codeArithmetic meanMultiplication signSymbol tableRoundness (object)AnalogySimilarity (geometry)Moment <Mathematik>Binary fileData storage deviceParticle systemLink (knot theory)Lie groupLatent heatLecture/ConferenceComputer animation
10:04
Statistical hypothesis testingBuildingComplex (psychology)Physical systemTotal S.A.Computer fileScripting languageSystem callExtension (kinesiology)ArchitectureLibrary (computing)Directory serviceElectronic mailing listWeightFingerprintDreizehnMulti-agent systemArmData typeParameter (computer programming)Parity (mathematics)InfinityChi-squared distributionInterior (topology)EmulationRing (mathematics)Software engineeringComputer-assisted translationGroup actionComputer architectureChaos (cosmogony)Functional (mathematics)Default (computer science)WordScripting languageFunction (mathematics)Projective planeParameter (computer programming)Extension (kinesiology)Shared memorySet (mathematics)Physical systemModule (mathematics)Multiplication signException handlingTypprüfungSequenceError messageParsingMathematicsProcess (computing)ExistencePrice indexRepository (publishing)Directory serviceCompilerSinc functionSoftware testingLatin squareWindowWeightSoftware developerPoint (geometry)Complex (psychology)Cartesian coordinate systemFormal grammarVideo gameLatent heatTouchscreenLine (geometry)Descriptive statisticsMassForcing (mathematics)Monster groupRight angleElectric generatorRow (database)Software bugBuildingType theoryJust-in-Time-CompilerComputer fileStandard deviationConfiguration spaceCodeVideo game consoleArc (geometry)ResultantConsistency2 (number)Metropolitan area networkComputer animation
18:00
Default (computer science)BuildingLatent heatDisintegrationRepository (publishing)Scalable Coherent Interface3 (number)ParsingInclusion mapGrand Unified TheoryParameter (computer programming)Software testingCAN busArmRule of inferenceParameter (computer programming)ParsingBuildingSystem callScripting languageComputer fileType theoryDefault (computer science)WordFunctional (mathematics)Logical constantBitModule (mathematics)Content (media)Programming languageProjective planeConfiguration spaceRepository (publishing)Physical systemCartesian coordinate systemComputing platformLink (knot theory)Water vaporOntologyParsingTheoryHeuristicVideo gameLine (geometry)Latent heatShared memoryMereologyComputerCodeBasis <Mathematik>Different (Kate Ryan album)Lecture/ConferenceComputer animation
21:22
Latent heatDisintegrationSoftware maintenanceInclusion mapScripting languageAxiom of choiceBuildingExtension (kinesiology)Mechanism designSingle-precision floating-point formatCAN busUniform resource nameProjective planeExtension (kinesiology)BuildingPhysical systemScripting languageParsingFunctional (mathematics)BitMathematicsModule (mathematics)MereologyReal numberComputer fileINTEGRALLatent heatSoftware maintenanceElectronic mailing listLevel (video gaming)Parameter (computer programming)Context awarenessFrame problemNumbering schemeSpacetimeDemosceneSlide ruleClosed setEndliche ModelltheorieTouchscreenSystem callComputer animation
25:30
Uniform resource nameExecution unitPhysical systemDivisorAttribute grammarDefault (computer science)Group actionFunctional (mathematics)NP-hardEndliche ModelltheorieModule (mathematics)Video gameComputer configurationExtension (kinesiology)CASE <Informatik>Vector potentialOrder (biology)Network topologyMultiplication signIdentity managementMoment (mathematics)Projective planeData structureCorrespondence (mathematics)Point (geometry)TupleLibrary (computing)Electric generatorComputer fileRadiusCodeScripting languageRootQuicksortObservational studyCompilerNumbering schemeLine (geometry)Object (grammar)Interpreter (computing)Food energyShared memorySoftware maintenanceArithmetic meanRepository (publishing)Student's t-testVisualization (computer graphics)BuildingIntrusion detection systemWrapper (data mining)Electronic mailing listDirectory serviceStructural loadSoftware developerInheritance (object-oriented programming)Computer animation
Transcript: English(auto-generated)
00:00
Okay, so we have Alan Martin speaking about how to build C++ applications with Python as the core, kind of, so please introduce him and I'll give you a big applause for him. Thanks.
00:28
Hi everybody, I'm Alan and I work as a software engineer at Ableton in Berlin, Germany. Let me introduce you to what we do. At Ableton, we make Live, Push and Link, software and hardware for music creation and performance.
00:44
First, we make Live, but you can see on the laptop screen here. Live is a digital audio workstation for creating musical ideas, turning them into songs and performing on stage. On a technical level, it's a C++ desktop application built for Mac and Windows from
01:02
a pretty old code base, 16 years old. Then we make Push. Push is a USB device which becomes a powerful electronic music instrument when combined with Live. It embeds a multi-color display at the top that renders a Qt Quick scene powered
01:22
by Qt, which is a C++ application framework. And then we make Link. Link is a technology that keeps music applications and devices in sync by synchronizing their beat. It is available to app developers as an iOS SDK.
01:42
Now you might ask, but what does that have to do with Python? Well, we do use Python at Ableton. First, we use it to write remote scripts for MIDI controllers. These MIDI controllers are devices like Push that musicians can use to interact with Live
02:03
in order to enhance their music creation workflow and to perform. Then we use Python scripts to automate continuous integration and releasing new versions of Live. And last but not least, since it's the topic of this talk, we are using Python to
02:23
build and test C++ applications such as Live. So let's see how we are building Live with Python. In order to build C++ applications, you have to compile C++ source files and sometimes also C and Objective-C source files.
02:44
And so you compile them into object files, binaries, which then are linked by the linker into libraries and executables. When building Live, more than 1400 files are compiled to product 88 libraries and 8
03:00
executables. Fortunately, there are tools to help you build C++ applications so you don't have to call the compiler and the linker manually yourself. These tools are called build systems, and you might already know the ones I've listed here. Unfortunately, most build system tools support only one platform, and we want to build Live
03:27
on Mac and Windows. And actually, Live used to be built from MEC files on Mac and Visual Studio projects on Windows, and it was a total maintenance nightmare. So we started using a higher level tool that would abstract most of the platform differences.
03:46
This tool is GIP. GIP stands for Generate Your Projects, and it's a meta build system. A bit like CMake and QMake, it's a multiplatform tool that generates files for platform-specific
04:01
build systems. It can generate MEC files, Ninja build files, Visual Studio projects, and Xcode projects. And it is written in Python. Nowadays, it is mostly used to build Chromium, Electron, which is the base of GitHub's
04:20
Atom text editor, Node.js, and the V8 JavaScript engine. And it is also integrated in the Live repository to actually build Live. Here is the Live Git repository, a very minimalistic view with just what matters for that talk,
04:41
and at Ableton, we use GitSum modules to integrate third-party code, or to reuse our own code across several projects. And here, in order to build and test Live, the Live repository contains build system, which is also an Ableton repository as a GitSum module.
05:02
And there's also a root-level GIP file named Live.GIP. Now let's see the content of build system. So build system itself is a Git repository, and it contains, first of all, the GIP repository
05:21
as a sub-module. So we have some kind of sub-module ception happening here, but it's manageable. Then it contains Python scripts to build and test C++ applications. There are three top-level scripts. Configure.py is responsible for calling GIP with the right arguments.
05:45
Then build.py calls the correct build system tool, either MSBuild, Ninja, or XcodeBuild. And then run.py calls C++ test runners or Qt-specific tools.
06:01
Going into the full details of how Live is built would take way too long for this talk, so we're just going to use a simple Hello World example to see how the build system scripts do their work. So here's the Hello World project, really simple for now.
06:21
It obviously contains a copy of the build system repository, otherwise that wouldn't be the build system Hello World. And like in the Live repository, it has a top-level GIP file. The C++ code to compile is spread between a source folder and a test folder, which
06:40
also contains a copy of CATCH, which is a C++ unit test framework. Now let's have a look at Hello World.GIP. Here, two targets are declared, Hello World and Hello World test. They both will produce one executable, built from one source file.
07:03
So pretty simple, pretty easy. You might find the GIP syntax familiar, and that would make sense because, well, it's a Python dictionary, actually. The content of Hello World.CCPP is straightforward, classic C++ Hello World.
07:22
And in test Hello World.CPP, we use the CATCH test framework to just write this really dummy unit test that always passes. Now let's build that. So to build and test the Hello World project, we first call the configure script.
07:41
It indicates what it's doing and for which configuration. So win is the platform, it could be Mac, but I'm running on Windows. 64 stands for 64-bit, could be 32. And DLL is the linking style, it could be static.
08:01
Configure.py is, as I said, responsible for calling GIP, which loads the Hello World.GIP file and generates Visual Studio files, since we are running this example on Windows. The generated files end up under the IDE folder in a subfolder which matches the configuration.
08:24
In a second time, we call the build script. Based on the configuration, it prints what is about to happen. Then it calls MSBuild, since it's the default on Windows. I'm a bit repeating myself. You can see in the MSBuild console output that two executables have been built.
08:44
Hello World.exe and Hello World Text.exe. Build.py finally prints what happened, which is really convenient when there are dozens of targets with hundreds of files and you don't want to scroll everything, like when
09:00
we're building live. The build artifacts end up under the output folder, here again in a subfolder based on the configuration. There's one subfolder more called debug, because when calling build.py, we can choose between debug and release.
09:21
This division into subfolders based on the configuration allows us to simply build and test several configurations of the same application in the same repository. And now we can run the Hello World executable. Hello World.
09:41
To finish that example, we call the run script with the CppTest comment, and it will basically call every C++ test runner from the correct output folder. By convention, only executables ending with test are treated as C++ test runners,
10:04
thus the Hello World.exe won't run here, only the Hello World Test.exe. Run.py also prints a summary of which test runners have passed and which have failed, which is again pretty convenient when you have dozens of them and you don't want to
10:24
read all the output. So now we've seen how build system works in a very small project. But now let's talk about what it means to use it with complex applications, such as live. So compared to the Hello World project, there are a lot more files in the live repository,
10:45
so it takes much more time, like really much more time to build and test, but it works in the exact same way. The only thing to do is simply configure, build, and run. Nothing more.
11:01
But don't get fooled by the apparent simplicity. The fact that there are only three scripts to call doesn't guarantee on itself that these scripts are easy to use, maintain, and extend. In fact, these scripts could have been developed in a totally chaotic way, and it would be a mess, not only for developers who work directly on them, but also for
11:26
developers who are using them to build and test their applications. And developers really don't like to work with a mess. So I mean, you may have guessed, we didn't develop our build system in a chaotic way.
11:43
Thanks, really. Otherwise, I wouldn't be giving a talk. I would still be in Berlin hiding from my angry colleagues. So no, everything's fine. Instead of chaos, we embraced consistency. The three top-level scripts, configure, build, and run, share a common architecture.
12:03
They only use modules from the Python standard library, principally arc-path, logging, and sub-process, and they follow the same sequence of actions. First, they parse the arguments, including finding the project .gp file, since it's the
12:26
And then they just do it. More than just a common architecture, these three scripts actually share a common design. This design is made of design principles.
12:43
And by implementing these principles, the scripts become easier to use, maintain, and extend. Now, right now, I want to share with you three of these design principles. They are generic, so you should be able to apply them to your own scripts.
13:03
First principle, fail early, loud, and clear. I don't really think I have to explain why, but we'll go through an example just to be sure. Here is a small script. We declare an argument parser with one argument, target, dir.
13:24
Then we get the target directory actually out of the arguments, and we count how many entries there are in that directory, and we print it on the console. So, when running this script on the HaloWorld project, there are six entries.
13:47
Everything is fine. Now let's try to do the same thing on a non-existing directory. Well, exception. So we get an error, which is actually a platform-specific error, it's a Windows error.
14:06
So my colleague working on Mac won't even see the same thing. Here it's pretty easy to trace back where that error comes from, there are not so many lines in that script, and it's pretty obvious what happened, but if you have a really complex
14:24
script which does a lot of things, it's really helpful to explain to the user what is wrong is what you typed on the command line. It's not a bug in the script. So how do we do that in Python? Well, the trick is to check the argument type as early as possible.
14:44
So here I have a function which basically takes a path, figures out whether that path is a directory, if it's not, it raises an argument type error exception with a comprehensive
15:01
message, otherwise it just returns the path. And we use that function in the type of the argument. So now in passing the arguments, the arg parser will actually call that function, and if everything is fine, nothing changes, still the same output, the user sees the result.
15:25
But if the folder doesn't exist, then we actually get an error message which is clear, because now we know it's something wrong when passing the arguments and not a bug in the script. So fail early, loud and clear, just to make your scripts easier to use.
15:46
The second design principle is support custom defaults. So you might be wondering, custom defaults don't really go together well. Let me explain. I've showed you that JIT can also generate ninja build files.
16:05
Ninja is not the default build tools on Windows and Mac, but we support it because it's pretty fast. So if I want to configure the HaloWorld project with ninja, then I call the configure script with minus minus ninja, I get the output with extra ninja indication.
16:27
Then I call the build script with minus minus ninja, and I get the ninja output, and then I can run the HaloWorld and everything is the same. It's the same C++ code, and the compiler code should be identical.
16:42
But I'm an easy man. I don't want to tap minus minus ninja every time I call configure or build. And I do that a lot every day. So to solve that issue, we've created a way to have custom defaults.
17:01
So in a file, which again is a Python dictionary, users can write what they want to be given to the arc parser as the first argument. So these arguments go at the beginning, and then whatever they put on the command
17:20
line goes next. So they can always, on the command line, override anything that is in the .rc file. So there's one .rc file per user in the user folder. Then there can be one RC file per build system repository.
17:43
So when you have several working copies, it can be useful to have different settings based on the working copy. So this file is get ignored, it's not checked in the build system repository. And then, as I said, the command line arguments at the end takes precedence since the arc
18:02
parser reads the arguments from left to right. So now, with the content of the RC file, including minus minus ninja when calling configure.py and build.py, I can just call configure and call build, and I don't have
18:23
to type anything else. Now the third design principle, do not integrate project-specific features. So the build system repository is completely live agnostic. It doesn't know that it's used by live.
18:43
It can be used actually to build any C++ applications. If we have a look at the build system repository and how it's composed, we saw there's JIP, they are the three main scripts. And to share code between the three main scripts, there's a Python module which contains
19:08
functions for argument parsing, like the existing JIF function I've just shown, functions for getting the default arguments based on the platform, and function to figure out
19:23
where is the top-level JIP file. So there are some heuristics, so as you've seen, I don't have to specify where is the helloworld.jip or the live.jip file when calling configure and build. So now for this example, let's imagine there is a fourth script, because configure, build
19:44
and run are pretty big, pretty complex. So let's just go with a simple arg parser. Here is the content of the simple arg parser. We import arg parse, obviously. We import constants from build system, but then we can use to define, to specify the
20:07
default value of the first argument, which is the platform, then we have the word size, and then we have the linking. And this script only prints the content of the argument namespace, nothing fancy. So let's call it just to see how it looks.
20:23
Here we go, just prints, linking is DLL, platform is win, and word size is 64, since these are the default values. Now, let's go back to the helloworld project. In the helloworld project, I would like to call this script with, I don't know, a
20:46
specific argument that I really would like to have. So here we could imagine that we want the helloworld to change the language, so instead of saying helloworld, it would say bonjour le monde, or hola el mundo, or whatever.
21:05
So we give the language on the command line. Obviously that won't work for now, there's no such argument. So now, I might be a bit sneaky and ask, can we add that to build system, please?
21:22
No. We do not integrate project specific features, ever. Because it adds maintenance cost, and it leads to API breaking changes as soon as another project starts using that feature, and then the original project decides to change it.
21:44
So no, instead, we make it easy to write script extensions. So here we're back to the simple arg parser script, with some space, because I'm going to fill the blanks.
22:00
First thing we do is include another module from the build system, then we create the script extension, I will come back to that part in a few slides, but basically we create the extension and then afterwards we use it.
22:21
So we have a function called extend arg parser before, which we put before defining the arguments, and another function called extend arg parser after that we put after defining the arguments. So this is the only thing we have to write in the build system script.
22:40
Now let's see how it works in the top level projects. The first thing they have to do, so the first thing HaloWorld has to do here, is create a Python module named build system extensions. That's the convention, it has to be like this, and in that module, there should be
23:07
one file, at least for now, called simple arg parser, and this file matches the name of the extension that was created. Here we define only one of the two functions, so here we define the extend arg parser after,
23:23
and now when we call the script with our specific argument, it works! No errors. So how does that black magic happen? Well, first, as I told you, there is a script extension Python module, and in that
23:49
Python module there is a function called getScriptExtension. When calling that function, we pass two things. The first thing to pass is the script name, and the minimal API.
24:03
So the minimal API will define what are the functions that the extension has. And now we go into the dirty bits. So who has ever done the first line?
24:20
Import imp. Okay, less than ten. So this is the low-level way of importing a Python module. The first thing you have to do is to find it, sorry, using a list of paths where that
24:44
module could be and the name it has, then if it has been found, you can actually load it. Then there is a bit more craft to handle closing files and whatnot, but what you have to keep in mind here is that now we have a function called find and load module, and
25:04
this is now really useful to create the script extension. So creating the script extension for real. So we're back to describing that getScriptExtension function. It takes two arguments, the script name and the minimal API.
25:22
So the first thing to do is to find the build system extension module. So the one I told you, this is by convention, it's how it works. And we basically do that by just going up on the tree structure and listing all the
25:42
parent folders. So we start at the HaloWorld repository and then just go up, up, up, up until the root of the drive. And if in any of these folders there is a folder named build system extension,
26:01
actually not just a folder, a Python module named build system extensions, then it is loaded. And if we found it, then we can try to find inside it a module which has this desired script name. If nothing has been found, that can happen because maybe the user, the HaloWorld
26:22
project or another project hasn't defined any extension. The script extension is just a dummy empty object. And then we iterate over each attribute of the minimal API. And if the script extension doesn't have that attribute yet, we just set it.
26:41
So that way, the script extension that is written by this function is complete. So the script that is using this object doesn't have to check whether the attributes exist and it can just rely on this minimal API being complete and, yeah, just work.
27:04
To wrap it up, we're building live with Python. And to echo the title of the talk, Python is indeed the keystone. Since without Python, nobody can work on live.
27:28
About the design principles, the thing to remember is that having some is good. It helps you write better code. But knowing and sharing them is better. So I invite you to come to me and to your fellow EuroPythonista and share
27:45
whatever design principles you are using in your code and in your scripts to make them better. So then, all together, we can make the users of our scripts happy. Thank you for your attention.
28:05
We actually have, like, a lot of time for questions, so raise hands, please. Yeah, so just a small question. I was wondering, why did you decide to include the build system as a model instead of
28:21
packaging and deploying it? We have kind of a history at Ableton of having everything in the repository. I mean, almost everything. So we obviously don't have the C++ compiler, we don't have the Python interpreter
28:46
itself, but all the code we run, we compile and we use, is inside the live repository. So I think the submodule is... So then that means that if you want to pick something like a hotfix or anything
29:00
for your libraries, you actually have to update your submodel, right? Yes. How is the experience going that way? I mean, I'm not saying that's bad, but... No, no, it's fine. We have done way more acrobatic things. Read some modules. Hands up, please.
29:22
Someone else. Did you think about using WAF or Scons instead of building your own build system? Also Python-based. Yeah, I know. So the thing is, WAF and Scons call the compiler directly,
29:46
but we are really attached to using certain IDs like Xcode. I know that Mac developers really like to use that tool. And unfortunately, Xcode only works if you have an actual Xcode project.
30:03
So JIP being able to generate project files that really look like the native projects was a big plus. And then the build system repository is merely a wrapper around JIP.
30:24
More questions? The talk is being recorded, so... Dumb question, but did you look at CMake?
30:42
Yes. Yes, we looked at CMake and we actually have a colleague who is one of the main maintainers of CMake. The thing is that when the transition was made from the Mac files and Visual Studio that were checked in the live repository to JIP, at that time, CMake was still pretty
31:07
annoying. But nowadays, it's a viable option, let's say. More questions? I have one myself.
31:21
How do you actually do... Do you support dynamic loading? Like, you have gone through the packages and going up, like, recursing the directories to the parents, looking for modules, but how do you actually set up the order in which they are executed?
31:42
So here, basically, the all-parent-folders function returns a list or... No, I think it's even a generator, but anyway, it's the list of the current folder
32:00
and then its parent folder and then its parent folder until the root. And just because of the order in which the paths are, the... We have to go back here. The find module function, which takes all this path, just does it for you. So you just have one extension.
32:22
You can only look for one file. Yeah, so the first build system extensions Python module that matches is the one loaded. And that's the default behavior of Python for any kind of module. Have you actually looked into the...
32:42
With setup tools, you have, like, entry points, and you can register, like, extensions, if you want to have a look. If you have seen Track, for example, or PyTest, they load the functions like setting up extension points, so that you can actually extend through those points and, yeah, more dynamic.
33:05
Although it doesn't look like you need it, but... Yeah, actually, in the end, we were pretty happy about the number of lines it is. It is small, actually. Okay, please. Thank you, Alain.