Debugging in Python 3.6: Better, Faster, Stronger
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33702 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Code division multiple accessIntelIntegrated development environmentSoftware developerSoftwareDebuggerSoftware developerDebuggerComputer programmingLecture/ConferenceComputer animation
00:28
Statement (computer science)LoginDebuggerContent (media)Function (mathematics)Physical systemEvent horizonLetterpress printingBuildingCellular automatonRange (statistics)Tracing (software)Computer programmingFunctional (mathematics)Line (geometry)DebuggerVariable (mathematics)BuildingEvent horizonFunction (mathematics)CASE <Informatik>Group actionStatement (computer science)Software bugFlow separationMultiplication signNumberArc (geometry)MereologyObject (grammar)Ocean currentLetterpress printingReal numberVideo gamePhysical systemCalculationParameter (computer programming)InformationState of matterString (computer science)InfinityLoop (music)System callType theory2 (number)DivisorUniqueness quantificationAutocovarianceNetwork topologyAnalytic continuationWordFunctional programmingRobotDistanceCategory of beingElement (mathematics)Point (geometry)CuboidControl flowComputer configurationZustandsgrößePower (physics)Software framework
06:58
Performance appraisalPauli exclusion principleCodeObject (grammar)Content (media)Tracing (software)Performance appraisalFunctional (mathematics)Pointer (computer programming)Object (grammar)Alpha (investment)CodeException handlingCASE <Informatik>Functional programmingLine (geometry)NumberIterationInformationWordExtension (kinesiology)EvoluteParameter (computer programming)Term (mathematics)System callFunction (mathematics)Computer programmingWritingTheory of relativityState of matterDebuggerDefault (computer science)Interpreter (computing)Insertion lossMathematicsField (computer science)Bit rateOrder (biology)Pauli exclusion principleLetterpress printing
10:10
Function (mathematics)Performance appraisalDebuggerBuildingCodeMaxima and minimaStructural loadLine (geometry)Functional programmingPerformance appraisalFunctional (mathematics)Computer programmingAnalytic continuationDebuggerBytecodeSequenceParameter (computer programming)Mechanism designOperator (mathematics)InformationLine (geometry)CASE <Informatik>Constructor (object-oriented programming)Presentation of a groupVariable (mathematics)Event horizonBitCodeModule (mathematics)Wrapper (data mining)AdditionStatement (computer science)ResultantSource codeHydraulic jumpStandard deviationBuildingPoint (geometry)NumberIterationAuthorizationSlide ruleBit rateObject (grammar)Set (mathematics)Control flowMusical ensembleTheory of relativityElectric power transmissionInsertion lossZustandsgrößeWeightImplementationMereology
16:01
DebuggerPerformance appraisalTracing (software)Object (grammar)CodePauli exclusion principleThermal expansionStrutSpacetimeFlagPrice indexGraphical user interfaceContent (media)DebuggerTheory of relativityFunctional (mathematics)MereologyPerformance appraisalComputer programmingLine (geometry)Field (computer science)Multiplication signDampingCodeSpacetimeFlagDefault (computer science)AdditionInformationPauli exclusion principleFunctional programmingObject (grammar)Phase transitionCore dumpNumberPoint (geometry)CASE <Informatik>Loop (music)Bit rateTraffic reportingEvolute
19:34
Performance appraisalTracing (software)Video gameTheory of relativityReal numberResultantExistenceComputer programmingProduct (business)2 (number)DebuggerOpen setIntegrated development environmentMultiplication signSampling (statistics)DivisorTerm (mathematics)RobotOpen source
20:53
Performance appraisalMaxima and minimaPersonal digital assistantJust-in-Time-CompilerPauli exclusion principlePrototypeSource codeDebuggerBytecodeComputer clusterTwitterCASE <Informatik>Extension (kinesiology)Statement (computer science)Object (grammar)Order (biology)Theory of relativityFunctional (mathematics)CodeMultiplication signProjective planeWritingPauli exclusion principleIntegrated development environmentPerformance appraisalLibrary (computing)Mechanism designNumberLimit (category theory)Flow separationLoginBitPhysical systemImplementationAuthorizationSystem callPatch (Unix)Field (computer science)CompilerResultantSoftware testingSoftware frameworkParameter (computer programming)Variable (mathematics)Software developerComputer programmingSource codeMathematicsMachine visionBeat (acoustics)EvoluteSubject indexingDivisorElectronic mailing listFunctional programmingPointer (computer programming)QuicksortMoment (mathematics)Presentation of a groupInteractive televisionRow (database)Data conversionExpressionJust-in-Time-CompilerCustomer relationship managementTerm (mathematics)Level (video gaming)Execution unitQuantificationComputer animation
Transcript: English(auto-generated)
00:04
Hello, my name is Elizabeth Ashakova and today I'm going to talk about some interesting new opportunities appeared in Python 3.6. First, let me introduce myself. I'm a software developer at the JetBrace company.
00:21
I'm working in PyCharm team. I work on debugger in PyCharm. And I've come here from St. Petersburg. When we write programs, unfortunately we always introduce bugs into them. And there are many different ways to find these bugs. For example, you can just simply add print statements.
00:41
Also, some people prefer logging. In fact, it's the same print statements but with the ability to turn it on, off, or add some options. But there is a big separate group of tools named debuggers. Debuggers are much more complicated than logging because they allow users to pause the program in some place.
01:04
They allow to execute stepping commands. So to monitor the program execution line by line, watch the variables values and do something else. But unfortunately, there are many people who prefer print statements and logging to debuggers.
01:23
Why such people exist? The answer is quite easy. Because debugger is rather slow. On average, in big real life programs, it's usually almost 30 times slower to run program under debugger than run it without debugger.
01:46
I think it isn't breaking news for you because everybody knows that debuggers usually slow down program execution. But why does it happen? And what can we do with that? Today in my talk, I'm going to answer to these questions and we will learn how to build Python debugger and how to make it faster.
02:08
Let's start with tracing debugger. It's named tracing because of tracing function. Python provides a standard way to set this system tracing function. It takes three arguments, frame, event and arc.
02:24
Frame object contains the information about the current state of the program. Event is the string representing the event which appeared in the program. And argument, the argument of this event. Let's define very simple tracing function here.
02:44
It prints the line number under the execution and event which arrived to our program. Let's see how it works. For example, we have very simple function foo and we define this tracing function.
03:02
On the first, we will receive event call on the line one because we called function foo. After that, we receive event line on the line two because line two is executed. After that, we receive two events line again on the lines three and four.
03:23
And after that, the output high bop will appear in our program. After that, we will receive event lines again on lines three and four. And then execution goes to line five and we receive event return on the line five.
03:42
That means that we are leaving the current function, we are leaving the current frame. Okay, how can we build debugger based on this function? Debugger consists of two parts, breakpoints and stepping comments. Breakpoint allows to stop program on some special place
04:02
and stepping comments allow to execute comments, monitor program execution line by line. And we can implement both these parts of debugger with our tracing function. For breakpoints, we can inside our tracing function,
04:24
we can check the current line number. And if its line number equals breakpoints line number, we understand that we need to pause program in this place, so we call some breakpoint function which pause our program.
04:40
In fact, it's continue in infinite loop until program, until user continues program execution. And for stepping, we can use our tracing function too. We just check the type of event which arrived to our program, to our tracing function and handle it in different cases.
05:01
Okay, we built our tracing debugger, it works, but we tested it just with a very simple program. But what if we consider more complicated program? For example, this function calculate, it sums number from zero to the seventh power of ten.
05:21
And let's define a very simple tracing function, it in fact doesn't even print anything, it's just return itself to continue tracing in the current frame. And let's run our function calculate. If we run it without debugger, it takes about one second to execute our program.
05:43
But if we run it with our tracing function, it takes already almost seven seconds. Our tracing function is very, very simple, but we called it on every line of our program. And if, for example, we have in our tracing debugger three breakpoints,
06:02
so it means that on every call in our tracing function, we need to iterate through loop with three elements, it takes already almost twenty seconds. So it takes much more time, the program becomes almost twenty-five times slower.
06:21
And let me remind you that the tracing function was very simple. And this small experiment shows us that, explains why running under debugger is much slower than running program without debugger. And the main problem with our tracing debugger
06:42
is that we call our tracing function of every line of the program. Okay, let's remember this problem, and let's consider a small story about Python 3.6. As everybody knows, Python 3.6 was released half a year ago.
07:01
It has many cool features, and one of them is new frame evaluation API. It was introduced in PEP 523, Python Enhancement Proposal, and PEP 523 allows to specify per interpreter function pointer
07:21
to handle the evaluation of frames. And also it adds a new field to the code object to use it by this frame evaluation function. Okay, it sounds a bit tricky, but we will consider an example. We'll try to write our custom frame evaluation function.
07:42
This is frame evaluation API, this is in fact C API. So in order to use it, you need to write C extension, but for example, you can write Cython extension, like we do it in PyCharm, and for better readability, I will use Python.
08:01
But in fact, this code is written in Cython. Okay, we're defining our custom frame evaluation function. It takes two arguments, frame object, which we've already seen in tracing function, and exception flag. We have frame object, so we can get the name of the current function.
08:22
We can get the line number of this frame. So let's print this information and call the default frame evaluation function. We don't want to change program's behavior, we just want to print some interesting information. And let's see, and we need to define this
08:40
to call this custom frame evaluation function, and let's see how it works with example. We have three functions, first, second, and third. The first one calls the second, and the second calls the third. And when we run this program, we get this output.
09:01
That means that our custom frame evaluation function was called on the line one, when we entered the function first, on the line four, when we entered function second, and on the line seven, when we entered function third. Okay, it works, that's great.
09:22
And also, from this example, we learned that our frame evaluation function was executed while entering every new frame, and inside this frame evaluation function, we have an access to frame object. So, to the code object as well.
09:41
Okay, we know about this new cool Python 3.6 feature, and you remember that with tracing debugger, we have problem that we called tracing function on every line of the program, and if we, what can we do with that?
10:01
We can remove tracing function. But in this case, our debugger will stop working, so we can't just remove it, but we can replace tracing function with our custom frame evaluation function. And let's try to build frame evaluation debugger,
10:21
debugger based on custom frame evaluation function. As you remember, every debugger consists of two parts, breakpoints and stepping comments. Let's start with breakpoints. When we had tracing function, we had a complete mechanism to monitor program execution,
10:40
because in every line, we know all information about event, about line number, but in case of frame evaluation function, we don't have such mechanism. We have only frame object, and we need somehow insert breakpoints into this new frame which we are entering, and we can do it another way.
11:03
We can insert breakpoints code right into frames code. So, for example, if we have very simple function maximum, which returns the biggest value of two arguments, how can we return the breakpoints?
11:22
For example, if we want to insert breakpoint on the line three, that means that we want to insert some breakpoint function call right before the return statement. So, after our modification, the result will look like this. Before this, before returning the value a,
11:44
we want to call our breakpoint function and suspend program, and we want to wait for some user comments. Okay. How can we insert one piece of code into another piece of code without changing the source code? We want to modify bytecode.
12:05
Let's use standard module this, which shows the Python bytecode in a human readable presentation. For example, for our function maximum, the bytecode will be like this. This bytecode is generated for line two,
12:24
this for line three, and this for line five. As you can see, the bytecode for line four wasn't generated because if-else construction was replaced with this pop-jump-if-false operator. Okay, our bytecode. In fact, we're not interested in what's going on inside our bytecode
12:44
because for us, it's just a sequence of operators, with or without arguments. Each operation has its offset, these even numbers, and arguments, which can be absolute or related jumps. For example, here we have an absolute jump
13:02
from the operator pop-jump-if-false to the operator with offset 12. Load fast. Okay, and we want to insert our breakpoint's code. Okay, we can just take sequence of bytes and insert it into another sequence of bytes.
13:21
But we can't do it just without changing anything because as I've already said, we have some jumps, so some references from one operator to another. And when we're inserting our code, we need to update some arguments, offsets, because all operators after breakpoint,
13:42
they go down and their offsets will be increased, so we need to change references to them from the other operators. But when we do it, our modification will be done because the resulting code will be the original code but with the additional calling to breakpoint function.
14:05
It sounds a bit scary, but in fact it is 200 lines in Python. We just have to write it carefully and it will work. Okay, now we know how to insert breakpoint, but we need to decide what to insert.
14:23
It's quite easy to answer because for our breakpoint we can create some simple wrapper and its bytecode is shown on the right side of the slide, so we're just calling global function. Before the bytecode modification,
14:42
we add this global function to the frame global variables dictionary, so we can quickly just call it and inside this function we can do anything we want, add some additional debugger functions and we don't care about it because for us it is just calling some global function
15:02
and it's quite simple. Okay, our breakpoints are ready, but we still need to implement stepping in our debugger. There are two ways to implement stepping. Of course we can insert temporary breakpoint on every line of our program,
15:20
but in such case we will return to the previous situation when we called tracing function on every line of our program and it slowed down our program significantly. So we won't use this opportunity, but we will use all tracing function.
15:41
When user wants to execute some stepping command, we enable all tracing function, handle events in our program, and if user want to resume program execution, we just remove this tracing function and continue program execution until the next breakpoint.
16:02
Okay, now our frame relation debugger is ready. We are looking forward to try it. Do you remember this slow example with function calculate? And as you remember when we ran it with tracing debugger, it became almost 25 times slower.
16:23
So what about running with frame evaluation? Yes, with frame evaluation it runs almost as fast as without debugger. It happens because we are not calling tracing function on every line of the program, and we just call this our breakpoint function once
16:43
and continue our program execution like without debugger. So that's why it works so fast. But let's consider another example. What if we add some additional function inside loop,
17:01
for example function foo that doesn't do anything, but that means that on every step inside our loop, we call this function. After that, sad news, that our frame relation debugger becomes slower, much, much slower because we return to the previous situation.
17:25
When we enter every new frame, we call our frame relation function, and we do some checks inside it. So we again return to the situation when our program becomes slow.
17:40
Maybe PEP 523 can help us again? Yes, it can. As you remember, it consists of two parts. We have already used the first part. We defined our custom frame relation function. But also there is a new field which appeared in code object,
18:01
this co-extra, a scratch space for code object, and we can store there some information. And for our frame relation debugger, we can use it quite easy. We can mark frames without breakpoints. So when we enter frame and we know that there are no breakpoints there,
18:23
we add a special flag to this frame, and we know that we don't want to do any additional checks here. We can just return quickly default frame relation function, and it will work quickly. So we know all the functions without breakpoints,
18:43
and we can skip them very quickly. And in this case, yes, we were right. After that, debugger becomes faster in all cases. We don't depend on such situations like in example two.
19:00
And I want to emphasize it again that how PEP 523 helped us. The first part helped us to define our custom frame relation function, and the second part helped us to mark frames without breakpoints and quickly skip them during debugging.
19:23
Okay, our frame relation debugger ready, and it works in different cases, but what about real results? Real-life results exist. Frame relation debugger was implemented in PyCharm 2017.1.
19:45
There is also PyCharm Community Edition, which is free and open source, so everybody can download it and try how it works. And it works in production. It isn't just a total example. It's a real debugger in the integrity development environment.
20:01
And for one of our benchmarks, for example, if it took about 20 seconds to run program under debugger, before frame relation API, we added some Cython speedups,
20:22
we write some bottlenecks of our programs in Cython, and it gave us, it increases debugger's execution. It takes almost six seconds to run program with this debugger. But frame relation improved debugger's speed significantly.
20:45
And it became almost 80 times faster. And the only thing that I can say after that is that frame relation rocks because it gave opportunity to dramatically improve debugger's performance.
21:02
And of course, it has some disadvantages and limitations because such debugger is a bit more complicated. You need to implement bytecode modification. You need to write C extension in order to use this API. At the moment, it works only with C Python because this API was implemented only in C Python
21:22
and is available only in Python 3.6. So I can say that frame relation debugger may be yet another reason to move to Python 3.6 because most likely such quickie debugger or some other tools will appear in many IDEs or developer's tools.
21:47
And also, you might be inspired by my talk and you might find yet another use cases for custom frame relation function because, as I've already said, we used this frame relation function
22:02
in order to insert breakpoints code into the original code. But in fact, we can insert everything. For example, we can insert some functions for logging. Just imagine you can enable logging for all your program without changing source code.
22:21
So there is no need to write any log statements during your program. You can just define this frame relation function and your logging will be enabled in your program. And you can disable frame relation function and logging will be off.
22:42
So I believe that such use cases exist because originally this PEP was created by authors of Microsoft's Pigeon project. Pigeon is a just-in-type compiler in C Python
23:01
and they use custom frame relation function in order to generate jitted code and use call extra field to store this jitted code. So this PEP wasn't implemented for debuggers. Originally it was implemented for JIT,
23:22
but successfully we used it in debugger. That's why I believe that other use cases exist and we need just search for them and try to implement them. Let's move to Python 3.6. It's a really cool release.
23:41
And let's find these use cases. If you want to watch the whole code base of today's example, you can check out my project on GitHub. Also, as I've already said, it's included into PyCharm Community Edition. So you can watch the source code of PyCharm 2.
24:03
And also I moved a bytecode modification to a separate library on PyPI. So you can just install it and use it. Everything is ready. There is a small to example. There is library for bytecode modification. So everything is ready for your experiments
24:20
and I hope that all of you will, just right after talk, you will try to do something interesting with new frame evaluation API. Now I'm ready to answer your questions. Feel free to follow me on Twitter and ask anything you want about it.
24:42
Thank you. We have time for small questions, please.
25:03
I really appreciated presentation today. I didn't know about that. I'm wondering though, PyTest used to rewrite the bytecode at startup to add this kind of modification to the bytecode. What is the reason to do that when we enter in the frame
25:23
instead of redoing it at the startup or whenever a user interacts with the debugger and sets a new breakpoint to remove an older one? We can't do it in startup because when we start our program, we don't have access to the frame object.
25:41
The main advantage of using this frame evaluation function that when it is called, the frame object is one of its parameters and you can have access to the code object, to the frame object, to global local variables and you can change it. But in a startup, you can't get all the functions,
26:03
for example, from your program. Of course, you can change source code but it doesn't look like a good idea for creating debugger.
26:27
Hi, thanks for the talk. Can you use this to monkey patch C extensions? Could you repeat this? Can you use this to monkey patch C extensions? Monkey patch C extensions.
26:41
Which is normally not so easy but Python code you can quite easily monkey patch but maybe with this thing you can go deeper. Would it be an opportunity for, I don't know, testing or mocking frameworks to do this at the C extension level? Maybe it is possible. In fact, I didn't try it.
27:02
But you can try and tell us about your results. I'm not sure. I don't know the answer. Unfortunately. Last question. So you were inviting people to create new use cases
27:21
but what will happen if everybody writes to co-extra? There is in frame evaluation API, I didn't have enough time to mention it, but there is mechanism to multiple usage of this frame evaluation API
27:42
when you use co-extra. In fact, you use it by index. You can see the number of usages in order not to intersect with other systems who use this co-extra. So they are stored separately. Thank you very much. Give her a hand.