Debugging in Python 3.6: Better, Faster, Stronger

Video thumbnail (Frame 0) Video thumbnail (Frame 698) Video thumbnail (Frame 2228) Video thumbnail (Frame 2940) Video thumbnail (Frame 5682) Video thumbnail (Frame 9920) Video thumbnail (Frame 10584) Video thumbnail (Frame 15255) Video thumbnail (Frame 18581) Video thumbnail (Frame 24034) Video thumbnail (Frame 25321) Video thumbnail (Frame 26598) Video thumbnail (Frame 28144) Video thumbnail (Frame 29342) Video thumbnail (Frame 30001) Video thumbnail (Frame 31330)
Video in TIB AV-Portal: Debugging in Python 3.6: Better, Faster, Stronger

Formal Metadata

Debugging in Python 3.6: Better, Faster, Stronger
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Debugging in Python 3.6: Better, Faster, Stronger EuroPython 2017 - Talk - 2017-07-10 - Anfiteatro 1. Rimini, Italy Python 3.6 was released in December of 2016 and it has a lot of new cool features. Some of them are quite easy for using: a developer can read, for example, about f-strings and they can start using them in their programs as soon as possible. But sometimes features are not so evident, and a new frame evaluation API is one of them. The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames. It might not be evident how to use this new feature in everyday life, but it’s quite easy to understand how to build a fast debugger based on it. In this talk we are going to explain how standard way of debugging in Python works and how a new frame evaluation API may be useful for creating the fast debugger. Also we will consider why such fast debugging was not possible in the previous versions of Python. If someone hasn’t made a final decision to move to Python 3.6 this talk will provide some new reasons to do it
Intel Software developer Debugger Software developer Software Code division multiple access Integrated development environment Computer programming
Group action Divisor Debugger Multiplication sign Uniqueness quantification Letterpress printing Login Variable (mathematics) Computer programming Computer configuration Statement (computer science) Cuboid Video game Statement (computer science)
Point (geometry) Ocean current Functional programming Building Functional (mathematics) Divisor State of matter Debugger Robot Multiplication sign Cellular automaton Control flow Parameter (computer programming) Content (media) Mereology Distance Event horizon Computer programming Number Power (physics) 2 (number) Element (mathematics) Zustandsgröße Software framework Analytic continuation Physical system Information Autocovariance Building Debugger Infinity Range (statistics) Letterpress printing Line (geometry) System call Category of being Type theory Word Loop (music) Event horizon Personal digital assistant Function (mathematics) Network topology Object (grammar) Physical system
Functional programming Debugger Network topology Line (geometry) Tracing (software) Content (media) Computer programming
Functional programming Functional (mathematics) Code State of matter Insertion loss Function (mathematics) Parameter (computer programming) Content (media) Computer programming Field (computer science) Number Mathematics Performance appraisal Bit rate Term (mathematics) Object (grammar) Extension (kinesiology) Alpha (investment) Exception handling Default (computer science) Theory of relativity Information Debugger Code Line (geometry) Tracing (software) Evolute System call Performance appraisal Word Pointer (computer programming) Personal digital assistant Function (mathematics) Pauli exclusion principle Order (biology) Interpreter (computing) Iteration Object (grammar) Writing
Point (geometry) Bytecode Functional (mathematics) Presentation of a group Debugger Source code Maxima and minima Parameter (computer programming) Mereology Event horizon Computer programming Number Mechanism design Zustandsgröße Information Building Weight Constructor (object-oriented programming) Code Line (geometry) Performance appraisal Personal digital assistant Statement (computer science) Object (grammar) Resultant
Point (geometry) Functional programming Slide rule Functional (mathematics) Implementation Structural load Line (geometry) Debugger Control flow Set (mathematics) Insertion loss Parameter (computer programming) Data dictionary Event horizon Computer programming Number Performance appraisal Bit rate Operator (mathematics) Authorization Hydraulic jump Theory of relativity Building Code Bit Line (geometry) Electric power transmission Sequence Personal digital assistant Function (mathematics) Iteration Musical ensemble
Functional (mathematics) Performance appraisal Theory of relativity Multiplication sign Line (geometry) Tracing (software) Traffic reporting Computer programming
Functional (mathematics) Performance appraisal Loop (music) Theory of relativity Multiplication sign Tracing (software) Computer programming
Point (geometry) Functional programming Default (computer science) Functional (mathematics) Code Strut Code Thermal expansion Price index Tracing (software) Evolute Mereology Field (computer science) Number Performance appraisal Graphical user interface Personal digital assistant Object (grammar) Pauli exclusion principle Core dump Phase transition Spacetime Object (grammar) Flag
Functional (mathematics) Theory of relativity Debugger Real number Code Content (media) Mereology Performance appraisal Bit rate Personal digital assistant Object (grammar) Pauli exclusion principle Video game Resultant
Theory of relativity Divisor Open source Real number Multiplication sign Robot Sampling (statistics) Tracing (software) Computer programming Product (business) 2 (number) Performance appraisal Integrated development environment Term (mathematics)
Functional programming Presentation of a group Just-in-Time-Compiler Code Multiplication sign Source code Execution unit Parameter (computer programming) Computer programming Mechanism design Mathematics Personal digital assistant Software framework Data conversion Extension (kinesiology) Physical system Source code Theory of relativity Software developer Moment (mathematics) Electronic mailing list Bit Variable (mathematics) Order (biology) Quicksort Prototype Resultant Writing Row (database) Bytecode Beat (acoustics) Functional (mathematics) Divisor Patch (Unix) Quantification Maxima and minima Machine vision Field (computer science) Number Twitter Performance appraisal Term (mathematics) Authorization Energy level Software testing Just-in-Time-Compiler Debugger Expression Projective plane Interactive television Evolute Limit (category theory) Subject indexing Pointer (computer programming) Personal digital assistant Customer relationship management Pauli exclusion principle Statement (computer science) Object (grammar) Library (computing)
of my name is Elisabetta scholar and today I'm going to talk about some interesting new opportunities idea in Python 3 . 6 1st let me introduce myself I'm a software developer at the gym the Jeffries company I'm working
in by turned to you and welcome to Bulgaria by chance and I come here from saying that is I want to write programs so
important that we always introduced boxes to them and there are many different ways to find this box for example you can just simply add reinstatement there are also some people prefer logging in fact it's the same print statements but with ability to turn it on all 4 at some options but there is at the except group of tools named dividers dividers are much more complicated and then logging because they unknown users to force the program in some places they unknown to execute stepping commons so to when a door that the program ecution line-by-line a watch their variables linear this entity do something else but unfortunately there are many people who perform restatements and longing to dividers wiII such people exist the answer is quite easy because the
binder is rather small when I reach the unique life problems they use usually almost 30 times slower to run a program under divided and then run it without divided and I think it isn't breaking news for you because everybody knows that and by year's usually slow down problems ecution but why does it happen and what can we do with that
and today later I'm going to answer these questions and will learn how to build Python debugger and how to make it faster let's start with tracing divided and its name tracing because of uh tracing functions appointment by and provides a standard way to set this system tracing function and it takes 3 arguments event and our framework and object it's that contains the information about the current state of the program events that is the streamed representing that event which if appeared in the program and I will mention the argument of this event uh this defined very simple tracing function he'd moved here it means that the aligning number of that under the execution and event which you right to our program so let's see how it works for example have very a simple function from will and we define this tracing function on the 1st will receive an event called on the line 1 because we called function food after that receive event lined up on the line to because line to is ecuted after that receive 2 events line again underlines the very after that that an outcome to hide what will appear in our program after that we will receive them and lines began on lines 3 and 4 and then the execution goes to line fine and we receive events return on the line 5 that means that we're leaving the current function to the current frame
OK how can we build by debugger based on this function divided consist of 2 parts breakpoints and step increments that point along this is going to stop program on some special place and staffing covariance along to the there is a huge comments which are programs line by line and we can if people implemented boffins parts of divider with our tracing function Our foregrip where this week on the inside of a trace function we can check the current line number and if its line number equals breakpoints my number when distance that we need to was problem in this place so we call some point function which uh also program in fact it's continue in an infinite loop until program until user continuous problem execution and for step and we can use our trees function dual we just check the type of event to reach arrived to our program to tracing function and tenderly each different cases but it will be a lot of attention by the heat words bot which tested just with a very simple problem but what if we consider the more complicated program for example this function late each uh some number from 0 to the 7th our power of 10 and let's define uh very simple infiltration function is in that doesn't even bring anything it's just return himself to continue tracing in the current frame the and let's run our function calculates if you run it without divided it takes about 1 2nd to execute our program but run it with our tracing function it takes already almost 7 seconds all interesting function is very very simple but of the cold heat on every line of our program and the for example we have in our tracing debugger 3 break points so it means that on every call In our tracing functions we need to iterate through the loop with 3 elements the takes already almost 20 seconds so the property takes uh much more time the problem becomes almost 25 10 solar and let me remind you that the tracing function was very simple and this small experiment a shows us that that explains why running and the debugger is much smaller than running from your without divided and the
main program without the main problem with our trees and value is that we call our tracing functions of every line of the program OK let's remember this problem and let's consider a small story about 560 everybody knows by the the deceased was released in
half a year ago it has many good features and 1 of them is you'll frame
iteration a guy but it was introduced in that 523 Python Enhancement Proposal I'm not an the that 523 loss to specify the interpret per interpreter function pointer to handle the evaluation of rates and also the it as a new field to the court opted to use it buy this frame evaluation function OK let's do it sounds a bit get but will consider an example will try to write our custom frame function and this is frame of evaluation that this is in fact a C API so in order to use it you need to write C extension but for example you can write something state is Cython extensions like Newton by term and for better readability I will use Python uh but in fact this code is written in the size OK will defining our custom premonition function but it takes 2 arguments frame object which we've already seen interesting functions an exception flat uh in it would have frame object so we can get the name of the current function we can get the line number of this frame so let's bring to this information and to call the default frame relation function what does he want to change problems behavior we just want to bring some interesting information and let's see and we need to define this that to call this uh custom promotion functions and let's see how it works with examples but have 3 functions 1st 2nd 3rd I 1 call you in the first one was the 2nd the 2nd cause for and when we run this program will get this output that means that our custom frame relations function was called on the line 1 when they enter the function 1st on the for when we entered function 2nd on the line 7 when we entered the OK 8 words that's great and also from this example but we launched that our for emotion function was executed while entering every new frame and incentives for evolution function we have an excess to frame of uh so for to the code objects as well OK we know about this new cool Python the 6 features and it's you remember that with tracing debugger left half problem which we call tracing function on every line of the program and if we do what can we do that we can remove tracing function but in this case alpha divided will stop working so we can just remove it but
we can replace tracing functions with our custom frame evaluation function and let's try to build frame
emotion divided divided based on custom frame evaluation function and you remember every divided consist of 2 parts the weights and step increments let's start with the points out when we had tracing function we had complete mechanisms to when it or program persecution because every line we know all information about event about my number but in case of frame which function we don't have such mechanism but but we have only frame object and we need somehow insert points into this new frame which we're entering and what we can do it another way we can insert new points called right into frames so of for example if you have a very simple function marks 0 which adorns the biggest value of 2 arguments about How can we return the points so for example if we want to insert this point on the line for me that means that we want to insert some breakpoint function call an event before the return statement so after our modification the result will look like this but before they had before returning a value h we want to call our great point functions and suspend program and want to wait so for some users comments looking How can we insert 1 is the thought into another piece of quot without changing the source we want to know the modified by 2 but let's use the standard quantities which shows that have a bite and bite in if you human readable presentation for example for our function looks the about what will be like this this byte code is generated for line to this
voluntary and this line 5 yeah as you can see the bite quot line for was an generated because the else construction was replaced with
this fall jump false that the rate but there are other but what In fact not interested in what's going on inside our bank because rest is just a sequence of operators with or without arguments the iteration has its offset these even numbers and arguments and which can be absolute or related jumps for example here we have an absolute jump from the greater for jump follows to that operator with offset well not fast OK and we want to insert power grid points court OK we can just take a sequence of bytes and insert it into another sequence of bytes but we can't do it's just move without changing anything because as I already said we have some jumps so some references from 1 operator to another and when we in our court we need to update some arguments of sense because all operators after break point they goes down and their of sets will be increased so we need to change references to them from the other operators but when we do it Our modification will be done because the resulting quality will be the original court but with that additional calling to function it sounds a bit scary but in fact it is 200 lines in by we just have to write it carefully and that would be it work it OK now we know how to answer on that point but we need to decide what to so it's quite easy to answer because for our requirements we can create some simple author and his but could will is shown on the right side of the slide so we're just going global the name we just calling global function as before they're Abiteboul modification we add to this global functions to their frames global variables dictionary so we can quickly just call it and insight find this function we can do anything we want some additional divided functions and we don't care about it because for us it is just going some global function and it's quite simple what something Our points and rating but we still need to complement stepping in our divided there are 2 ways to implement step of course we can insert temporary breakpoint on every line of our program but in such case will return to the previous situation when called chasing function on every line of of a pronoun and each was new its slow down our program significantly so we want to use this opportunity but we will use all tracing function when user wants to music some stepping comment we enable all tracing functions handle events in our program and the fuser 1 to resume problems ecution we just remove this tracing function and a continue problems ecution until the next that point OK now our
frame relation to values ready we're looking forward to tried and remember
this slow example with function calculate and as you remember when
we ran it with tracing divided it became became of almost 25 times slower so what about running with Freeman relation yes I with free Malaysian each almost as fast as without it happens because they're not calling tracing function of underlined on every line of the program and we just call this our report function once and uh continue our problems ecution like they're like without the so let's might work so fast the but let's consider
another example in a what if we add some the additional functions inside and moved on for example function for it doesn't do anything but the thing is that at every time a step inside our loop we call this function after that said
news that our frame relation to buy here becomes slower much much smaller because we return to the previous situation all on on every what when we enter every new frame we call our free function and we and do some checks inside it so we stick we again returned to the situation when our program slow maybe that 523 can help us again
yes it can as you remember it consist of 2 parts and we've already used the 1st part we defined our custom for evolution function but we also called there is a new field which appeared in the court object this core extra x is greater phase of for code object and we can store their some information and for all the 4 emotion device here we can use it quite easy we can mark frames without points so when we entered frame and we know that there are a number of points that we had a special flat to this frame and we know that we don't we don't want to do any additional checks here we can just quickly default for emotion functions and it will work with so we know all the functions the points and we can keep them quickly the and
In this case yes will right after that divided becomes faster in all cases uh we don't depend on such situations like in Example 2 and so on I
want to emphasize it together they had uh hollow that 523 helped us uh the 1st part helped us to define our custom frame function and the 2nd part helped us to mark frames without that finds and weekly it's keep them during debugging and OK our frame relation to buy rating and it works different cases but what about real
results real life results exist the frame relation to binary
was implemented by 27 2 . 1 but there is also the term community edition which is free is and open source so everybody can download it and try how it works and it works production it isn't just show example it's real divide yeah in the integrated development environment and for some of
our before 1 of our best friends for the sample program uh if it took about 20 seconds to run program uh were under divided but we have before promotion of value we I edit some size speedups we write somewhat on exophora programs inside and and it gave us uh it increases uh what dividers institution the taste I almost 6 seconds to run the program with is divider bot frame of relation uh improved to buy years the significantly and it became almost 80 times faster and the only thing that I can say after that is that's frame revelation rocks
because it's give opportunity to dramatically improved debuggers performance and of course it has some
disadvantages and limitations because the size of ITER is a bit more complicated you need to implement a byte code modification unit to write C extension in order to use this a guy at the the moment it works on the received by and because it is this a that was implemented only in Survivin and is available only in Python from just 6 so I can say that frame relation divide yeah maybe uh yet another reason to move to Python for the 6 because most likely such we could divide there some other tools will appear in many ideas or divide developers tools and also you might be inspired by my talk and you might find yet another use cases for customer relation functions uh because as I've already said that we used this promotion function that in order to insert breakpoints uh quot into the original court but in fact we can sort everything so for example we can insert some functions for logging in just imagine you can enable logging for all your program without changing source caught so there is no need to were there right any uh log statements during your problem you can just define this for emotion function and your loading will be enabled in your program and you can disable for evolution function and a logging will be used so I believe that such use cases exist because originally this that was uh created by authors of Microsoft's vision project a beat it is just-in-time compiler source for uh in C Python and and they use custom promotion function in order to generate and use it caught and uh I use call extra field to store these didn't court so this back of wasn't implemented for dividers originally it was implemented 4 G bytes yeah the successfully and we used it in divider that's why I believe that other use cases exist and we need just search for them and try to implement them so let's move to buy the 6 it's really cool released and let's find these use cases and if you want to try it is to watch the whole could base of the today's examples you can check out my project on the top also by several orders that it's included into by term community edition so that you can all which is in the source code of question tool and also I moved a byte code modification to separate library and by the i so you can just instillate and use it everything is reading there is a small town example there is a library for a bite quantification so everything is ready for your experiments and I hope that all of you will have just right after top you will try to do something interesting with new frame relations API now I'm ready to answer 2 questions feel free to follow me on Twitter and ask anything you want about it the thank you thank you and more and the the appreciated presentation to the union of the thing it is all about that I'm wondering goal of like this used to ride a bike hold up stock of 2 of these kind of a vindication provide what would use the reason to that when we entered the fray in instead of doing doing it at the start of or whenever user interactions the by the and set the record pointer to remove all the 1 uh we can do it because uh when we starting program we don't have ccess to the frame object and the main advantage of using this promotion function that when it is cold and the frame object is 1 of these parameters and you can access have access to the code due to the frame object to global and local variables and you can change it buys in starts out you you can't get all the functions for example from your program of course even change source code to fight it doesn't look like a great idea for creating divider yeah so I had this little can you system of the extensions yeah of how could hear this list and then use this the monkey patch he extensions and the policy would enter really not so easy but I think you can quite as we move on but maybe with this thing you can go deeper would be an opportunity for a lot of testing or mocking frameworks to do this at the extension level mn mn maybe it is possible 0 in fact I didn't try it but you can try and tell tell us about your results and and sort of I'm not sure I don't know dances conversion expression um so you well a inviting people to create new use cases but what will happen if everybody writes a lot of cool and extract yeah there is an in-frame relation and I didn't have enough time to mention it but there is a mechanism to multiple usage of this frame of relation at time when you use the core extra infected you uh usage by index you can see the number of usages and in order to not to intercept with other systems we use this score extra so they are stored separately it your gene room theft