Writing unit tests for C code in Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 88 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21089 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201688 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
Formal languageUsabilityPower (physics)Software testingExecution unitSoftware frameworkCryptographySummierbarkeitInterior (topology)Social classModule (mathematics)Structural loadSource codeMountain passLibrary (computing)CompilerInclusion mapComputer-assisted translationFluid staticsHexagonStrutComplex (psychology)Glass floatImaginary numberType theoryCompilation albumReal numberPreprocessorClefNumberCASE <Informatik>Default (computer science)Function (mathematics)Object (grammar)Local ringData conversionCrash (computing)Process (computing)CodePersonal digital assistantKolmogorov complexityWrapper (data mining)Source codeDeclarative programmingFunctional (mathematics)Perfect groupMultiplicationBitComplex (psychology)ImplementationEmailModule (mathematics)Type theoryMathematicsCompilerFunctional (mathematics)Computer fileNetwork topologySummierbarkeitPreprocessorWritingMultiplication signCache (computing)ParsingMereologyObject (grammar)Content (media)Interior (topology)String (computer science)CASE <Informatik>CodeSoftware testingDampingResultantLine (geometry)Structural loadSystem callSoftwareData structureUnit testingExecution unitError messageDifferent (Kate Ryan album)FirmwareParameter (computer programming)Normal (geometry)Constructor (object-oriented programming)Order (biology)2 (number)MicrocontrollerMultilaterationLibrary (computing)Attribute grammarReading (process)QuicksortSingle-precision floating-point formatNumberSoftware frameworkElectronic mailing listFormal languageIntegerAlgorithmIntegrated development environmentLimit (category theory)Level (video gaming)Social classContext awarenessField (computer science)Software developerGame controllerCombinational logicAbstract syntax treeUsabilityInterface (computing)Complex numberFehlererkennungscodeNeuroinformatikOnline helpDebuggerProcess (computing)Power (physics)Universal product codeHigh-level programming languageAssembly languageHookingCorrespondence (mathematics)Electronic signatureProper mapLink (knot theory)Direction (geometry)Wrapper (data mining)Exterior algebraStatement (computer science)Abstract syntaxControl flowRegulärer Ausdruck <Textverarbeitung>Interpreter (computing)1 (number)WordCalculationPosition operatorConcentricRepresentation (politics)Physical systemRow (database)Sheaf (mathematics)Parametrische ErregungArrow of timeSurvival analysisTraffic reportingData managementMetropolitan area networkArithmetic meanArithmetic progressionBasis <Mathematik>Incidence algebraBoundary value problemMoment (mathematics)BlogEndliche ModelltheorieInsertion lossIterationAbstractionSerial portPattern languageStudent's t-testSet (mathematics)VideoconferencingCrash (computing)Point (geometry)Euler anglesPairwise comparisonPresentation of a groupPhysical lawContinuum hypothesisElectric generatorScripting languageLocal ringConnectivity (graph theory)AdditionProduct (business)Bit rateParticle systemDatabaseCommitment schemeCycle (graph theory)Term (mathematics)Binary codeRight angleIdentity managementMatching (graph theory)Computer programmingSemiconductor memoryComputer animation
Transcript: English(auto-generated)
00:00
Welcome everyone. We'll have a talk by Alexander Stephan about writing, testing C code with Python. Please welcome Alexander. Thanks for joining the session.
00:21
I work as an embedded software developer, so I write firmware for microcontrollers mostly. Unfortunately, this is mostly C code, not yet Python code. Though recently we've ported microPython to one of our controllers, so somehow we're getting better. Before I start with the talk,
00:41
I'd like to know a bit more about you, to your experiences with unit tests. If you've written any unit tests in any language yet, please raise your hand, so get an overview. Okay, great, that's most of you. And who of you has written unit tests for C code? Probably in C then. Okay, that's probably about half of you.
01:01
And last question then is who enjoyed the experience? Especially if you compare it to writing Python code instead. Well, a single guy, yeah, perfect. So then maybe I can show you a more fun way to write unit tests for C code. Now you might wonder what my motivation is for that,
01:21
and some of it can probably be summed up with this quote here, that the C language combines all the power of assembly language with all the ease of use of assembly language. So with C, you've got control of everything, and you can control everything, but you usually also have to control everything. You need to do everything yourself. There's little support from the language,
01:42
and for the testing stuff, you probably don't need all this power. You're not constrained with resources. You don't have that performance requirements that you might have in production code. So you could actually use a higher level language to make it easier for you to write your test code. Don't do everything in a low-level language like C.
02:04
Now let's look into that in a bit more detail. If you write unit tests for C code with C code, then there are some good things. You've got the same language everywhere, so as a developer, you do not need to switch context between different languages, different styles, different syntax. And it might also be good for a lazy developer
02:21
who only knows a single language. And of course, if you're working in an embedded environment like I do, then you could be able to run your unit tests on the target device, or at least on a simulated device, so that if there are bits and pieces of your code, for example, implemented in assembly, you can also test those.
02:42
But it's also a bit limited in some ways. I already told about the limitations that the language offers here, so you can only use C constructs, which are not as powerful as Python constructs, for example. You need to write much more code than you could in a high-level language. But you're also limited by what the framework
03:01
has to offer you. And if you look up unit testing frameworks for C code, there are tons of frameworks out there, but most of them are very basic. They don't offer advanced features that you might be used to when you look at unit testing frameworks that are offered, for example, for Python code. So there are a few frameworks only that offer mocking, for example.
03:21
And in the end, you're also limited by what the ecosystem has to offer. For example, we would like to test some cryptographic algorithms in our implementations, and of course, you can call into OpenSSL to verify some calculation, but it's not really that easy, and it might be nicer to do that in Python.
03:46
Now, maybe we can do better than that, and I've prepared a few examples to show you how unit testing C code with Python would look like. So the first example is the most basic thing I could think of. You've got a single function in our C code
04:02
that just adds two integers and returns the result. So this is the header file, the public interface that we want to unit test. And this then is the implementation of that function. It just adds the numbers and returns the value. And if you write a unit test for that, it could look like this. So as usual with Python unit tests,
04:22
you've got a test case class as a container for all your test cases. The single function in there then is your test case. We've only got one here. And it's rather simple. It loads in the source code that I've shown you before, creates a module out of that, and then has an object on which it can call the functions that is defined in this module.
04:41
This function returns the result, and we can assert that the result is really correct. Now, you don't see any C code in here, and no construct that really do anything with the C code from before. The only mentioning thing that you see is the name of the module, the parameter for the load function, and this is where all the magic happens.
05:01
So let's look into that. The load function here consists of three steps. First, it loads the source code on the module, so it opens the C file, it opens the header file, and reads out the source code. And then it uses CFFI to build a Python module out of that source code.
05:21
There are three calls that you need to make on the CFFI object for that. The first call, the cdef call, will tell CFFI what interface it has to export to our Python code. So we pass in the header file contents that defines the public interface. We want to test that, so CFFI needs to generate the interface for us.
05:42
Then with the second call, we need to tell CFFI about the implementation of the function. So we pass in the source code here, and the last step then is for CFFI to actually build the module that we want to have, so it runs a C compiler in the background, builds the module, and in the end, does the last step, we can import that module
06:00
and return it to our test case. And that's really all you need to run this example that I've shown you before. Now I've got three more examples that all build on this implementation, so I'd like to quickly ask whether there are any questions for this example already, so that you can better understand the following examples.
06:42
You mean if you had more than one source file? It works now, yeah, because if in this example source file, I have some sort of includes and dependencies to other source files, and how do I cope with that? I have to compile them also and link them somehow? How does it work?
07:01
Yeah, I've got some more complex examples with multiple files and with external dependencies, and we'll show that later. Okay, any more questions? Otherwise, I'll continue with the second example. And the second example, still rather basic, we've got again a single function that you can call multiple times, it will just add up all the parameters
07:21
that you pass into it and return the current sum. This is its interface, and this again, the implementation. So now we've got a global variable that we use to sum up everything, the function just adds to it and returns the current value. And the unit tests now look like this. To make matters a bit more interesting, I've implemented now three unit tests, not only one,
07:42
and so that I do not have to repeat this load call in every test case again, I use the setup method, which gets executed before each test case is run, it will load the module for the test case, and then the test case can access the module just as before, could call the function there, assert that the results are correct.
08:02
But if I were to run this test case with the load function that I've shown you before, it wouldn't work, and why wouldn't it work? Well, in the source code, there's this global variable there, and the load function that we had before, it just imported the module at the end, and if you know a bit about how importing works in Python, those imports are cached,
08:21
so if there are multiple test cases running, the first one will actually import the module, initialize the global variable, all the other test cases will just get the cached import back, and it won't be initialized again. So the assumption of the test cases, that the sum always starts with zero, doesn't hold here, and so the test cases would fail.
08:42
Now, there are several solutions to this. I'm just going to show you the simplest one, and that looks like this. The load function is still the same, just the first line with the comment has changed or it got added, where I generate a random name for the module. So this avoids all caching by importing, essentially, a new module
09:01
every time this function is called, which might not be the most performant solution, and it will also use more memory, but it avoids nicely all the problems that you could otherwise have with caching old data. For this, I use the UID module, which just generates a random unique ID and appends that to the file name,
09:22
which is then used as the module name. All the other code in here is the same as before, so each test case can still load the module and get a fresh copy every time. You could also implement that in a different way, and when you've imported the module, just reinitialize it every time, but that would take more code, so I don't show it here.
09:44
Okay, then example number three, and here we are getting to multiple files now. Since all the other examples so far were very basic, it was just a single C file and a single header file, now we take at least a second header file, and we want to do some mathematics with complex numbers, so we define our own structure for that.
10:03
That has just two fields for the two parts of a complex number for the real part and the imaginary part, and we have that in one header file, and then we want to implement a function that uses this type, so again, we use the example of addition, adding two complex numbers and returning the result,
10:21
and we can implement it like this. We just add both parts together and return the result at the end. Now, the test case for this, again, doesn't really need to know much about the C code. We load the module as before, and you don't even have to deal with the complex type that the header file declared somewhere.
10:42
When you want to call the add function, you just pass in the lists here, and CFFI will automatically generate structures for that so that the C code is happy and gets the correct results. And also, the result of this function call is a nice Python object where you can access the parts of the structure
11:04
with normal names and can assert that all these results are correct. But again, from this example to work, we can't use the previous implementation
11:22
of the load function because in the previous implementation, it just looked at the source file and the header file of the module that we want to test. It doesn't really know about the other header file that we also need. Now, if you remember the source code, you could say, yeah, well, the other header file got included into the module's header file,
11:40
so it should be present there. But unfortunately, CFFI cannot deal with these include statements. So what we need to do is we need to run some kind of preprocessor, like the C preprocessor over the source code so that there are no more include statements in there, no other directive that CFFI doesn't understand. Otherwise, it would throw an error.
12:02
And this is done with this preprocess call in here. Again, there are multiple ways you could implement that. I've chosen to just run the GCC preprocessor over the source code and get back the results. Then at the end, I've got one large string that contains the content of both header files, and CFFI is happy with that.
12:23
Now, for the last example, it gets even a bit more complex because now we have some external dependencies. In this case, you can imagine you want to program a microcontroller, and maybe the vendor of the microcontroller
12:40
provides you with a nice library like this here, where you can read GPIOs using simple function calls. The vendor has chosen to implement different functions for each GPIO that you can access. So he provides you with a library that has this interface here. But maybe in your code, you'd rather like to use this interface. You only want a single function call
13:01
and a parameter to select the GPIO that you're interested in. Now, you can implement that in your own code. You just look at the parameter, call the appropriate function, and if you get a parameter that you cannot deal with, you'll return some kind of error code. And now, this is the code that we want to cover with our unit test. We don't want to test the vendor's library,
13:22
so we don't want to use the read GPIO zero or one calls here. We probably couldn't use them in the unit test because they might access some registers of the microcontroller that aren't there in our test environment, so we somehow need to replace those calls with our mock functions so that we can run a test case
13:40
that knows what the GPIO values are. The test case for that looks like this. The first change that you'll notice to the previous implementations is that the load function now returns two values, not only the module as before, but also an FFI object. That's part of CFFI's interface,
14:01
and we'll use that in the first test case to replace the C function that we don't want to use with the Python implementation. So we define a function that has the same name as the C function we want to replace, and we tell CFFI, hey, when the C function gets called, please use this Python implementation instead.
14:21
Don't use the C implementation that you might find somewhere. And so the Python implementation just can return a fixed value, and the test case can call the function that we want to test with the correct parameter and see that the value that it defined before is returned in the end. The second test case for the GPIO number one,
14:42
it does the same thing but using a different construct. So in this case, we don't want to really define a function, but we want to use a mock object like you might be used to from the unit test library, and you can do just the same with it. You configure your mock object to return a value when it's called,
15:02
and then tell CFFI, hey, this is not a function, but it's just something else that you can call. Use that in place of the C function, and then the test case again works, can call this function, and at the end, you can also use the assert methods that are provided by the metric mock function.
15:22
And in this case, again, we need to modify the load functionality. This is, again, for comparison, the old implementation, and we need to add some more code to that for this example to work. There's three changes here, all again marked with a comment. The first change is that it's not sufficient anymore
15:40
to just process the header file for the module, but we actually need to process all the header files that are included in this module. So it just uses a regular expression to collect all the include statements, then runs that through a preprocessor, and as a result, gets one large string again that contains all the include statements,
16:00
all the contents of the include files of our module. The main work then is done in the next two lines where we need to tell CFFI which functions we want to replace with Python code and which functions are implemented in our C code. So the first line just goes through the source code
16:21
and looks for all the function definitions so that we know which functions are implemented by our source code, and the second line then goes through all the includes that we have, looks for all the function declarations in there, and whenever it finds a function that is not implemented in the source code, it will tell CFFI, hey, please insert a Python
16:42
implementation in here that we can replace later. The functionality is all there in CFFI, we just need to prefix the function declarations with this extern Python plus C statement, then CFFI will know, okay, I need to generate some code for that, and this will already make the compiler happy,
17:02
it will find a reference for this function so it can call it, and we can later replace it with Python code. And in the end, the last change is, as I said before, that we now need to return this FFI object also from the load function so that the test cases can tell CFFI about the implementations that we want to use.
17:24
Now I'll show you in a bit more detail how this step in the middle works, where we analyze the source code to find the function definitions. This is based on PyC parser, and this is the first part that collects all the function definitions. So PyC parser will analyze your source code
17:42
and will build an abstract syntax tree out of it, so you can later warp this tree with a class that's already provided, and whenever you hit a function definition, this wizard function here is called, it will get the node out of the tree and can just ask this node, okay, what is the name of the function?
18:02
We'll add this to a list, and so in the end, once it has walked through the whole tree, you've got a list of all the functions that are implemented in the source code, all the names of the functions there. And this is then used in the second part, again based on the PyC parser module, where we actually parse all the include contents
18:24
into an abstract syntax tree, and then tell PyC parser to regenerate the corresponding C code from that, so that we can modify some bits of that. And PyC parser already has support to regenerate code from the tree, and we just hook into that,
18:40
and whenever we see a declaration for a function, this is then again the visit function for declarations, we look at the declaration there, and see whether it's a function declaration, and if it is, and the name for this declaration is not in the list of functions that we found in the source code, then we'll just prefix it with the extern pysin plus C statement,
19:03
so that when CFFI again parses the source code, it will know what to do with these functions. Okay, this was the last example that I wanted to show you, so to sum up, I want to talk quickly about some of the drawbacks that this approach might have, if you're used to other approaches,
19:22
and one of the main drawbacks is probably that if you use this code, as I've shown it to you, if your C code does something bad and tries to access a null pointer, for example, then it will also crash the test process, because the code actually runs in the same process, there are no boundaries between it, so when your C code destroys something, your test will crash,
19:41
you won't get any nice error reports, and you might not like that, so one solution to that problem would be to run each test case in a separate process, and have one main process collect all the results, then if one test crashes, just crashes the single test case, the main process can still report on the errors, and all your other test cases will continue to run.
20:03
This might add a little overhead, of course, because now you have multiple processes running that need some more computing time, but at the same time, you can also run your tests in parallel, so if you've got multiple cores, it might actually be faster in the end than running everything in serial,
20:21
and another big problem might be that debugging your test cases gets harder now, because you've got a Python process that calls some C functions, that again might call some Python functions, and where really do you debug that? You can attach a debugger to your Python test cases, but that won't help you much once you enter C land,
20:40
you won't see what the C code does there, or you can attach a C-level debugger, so that you can see what your implementation does, but then you have to deal with all the C calls that are done by the Python interpreter internally, and that you need to skip somehow, so it would be nice, of course, to have some maybe better integrated solution here,
21:00
some combination of two debuggers, one for the Python side, one for the C side, but smoothly hand over control once you enter the other part. Or one could also argue that since we are talking about unit tests here, if you really need to debug your unit tests, maybe you could also think about simplifying your code,
21:20
simplifying your unit tests, or even the implementation, so that you don't need to debug them in order to find a problem, but so that you've got unit tests that really can tell you where the problem is when something breaks. But to end on a positive note, if you're going to remember something from this talk, I'd like you to remember that writing the test cases
21:43
is really simple, and no matter how complex your C code looks like, so you've seen all the examples that I've shown you, the test cases look pretty much the same, because all the complexity that you need to care about is hidden inside CFFI and the wrapper code that I've shown you here. As a test case author, you don't really need to deal with that.
22:02
You just can concentrate on writing your test cases, and you need to solve the hard parts only once, have it in a generic part of the code, and never look at that again as long as it works. So thank you for your attention.
22:23
If you have any questions? Can you run tests from Python on a compiled library, for example, like from a build binary code?
22:43
Can you import that in CFFI? Yeah, that is one of the main use cases, actually, for CFFI, that you can interface from Python to existing libraries so that you can build a nice Python interface for libraries that already exist without needing to reinvent them. So that's, of course, possible. This approach was more meant to test the source code
23:02
so it passes in the source code, not a library, but of course you can also tell it here, use the existing library. And you could probably also do this, could you do this trick with mocking, or like put alternative function or function definitions
23:22
in a loaded shared object or something like that? Do you think that this is possible? Well, it depends. If the function that you want to mock is not part of the library but would be part of another library and you don't link against that library, then it should be possible because then you have to insert your own implementation of that function anyway for it to compile.
23:41
But if you want to mock a function that's part of the library that you want to test and it's implemented in there, you can't really replace it because it's part of the same binary and the code will just call the function in there. You can't really take it out and insert another implementation there. So you cannot switch out the binary code, okay.
24:09
If I would refactor my C code, say change the name of the function as a signature and forgot to adapt my test, how easy it is to spot the mismatch?
24:22
Do I get the proper error message or does it just crash? No, CFF, I will tell you if you want to call a function that doesn't exist, that well, there's no such attribute on the module, you'll get the usual error codes for that. If you change the type, it probably depends a bit on how compatible the old type is to the new type.
24:40
If you maybe change an int to a float or something like that, you might even not need to adapt your test cases. Even if you pass in an int, CFF, I will just convert that to float value then for your call. But if I would use a different struct name or so, it would detect that. Or the order of the. Can if you change the names so that are not compatible,
25:01
you get an error message. If you have, on the other hand, a structure that's completely different, a completely different name, but has the same types in there, then you probably won't notice. If you change the complex number structure, for example, and just switch the order of the fields, you won't notice that when you pass in the parameters. You will only notice that
25:20
then when you test for the assertions in the end. No more questions? Yes? Can the CFFI module can be also used for the C++ code?
25:41
For what, please? C++. For C++, I think it's not completely supported now, but there's the main CFFI developer, Armin, in front of the four of us. You can ask him about new features.
26:03
Short answer is no. Okay, thank you for your attention, and thank you, Alexander, again. Thank you.