Python Debugger Uncovered


Formal Metadata

Python Debugger Uncovered
Title of Series
Part Number
Number of Parts
Trofimov, Dmitry
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place

Content Metadata

Subject Area
Dmitry Trofimov - Python Debugger Uncovered This talk will explain how to implement a debugger for Python. We'll start with setting a simple trace function, then look how it is implemented in modern IDEs like PyCharm and Pydev. Then we go further in the details and uncover the tricks used to implement some cool features like exception handling and multiprocess debugging. ----- Presentation describes how to implement debugger for Python and has 4 parts: * Tracing Python code Explains how to use trace function * Debugger Architecture Explains which parts consists of a modern full-fledged debugger. * A Bit of Details Explains how to make code to work for all python versions and implementations, survive gevent monkey-patching etc. * Cool Features Explains how to implement exception handling and multiprocess debugging
EuroPython Conference
EP 2014
EuroPython 2014
Metropolitan area network Computer animation Lattice (order) State of matter Debugger Software developer Debugger Code Quantum
Metropolitan area network NP-hard Computer animation Code Sheaf (mathematics) Task (computing)
Computer animation Code Blog Gender Debugger Cuboid Function (mathematics) Resultant Local Group Hypothesis
Standard deviation System call Hecke operator Presentation of a group Divisor Code Multiplication sign Event horizon Formal language Maxima and minima Gamma function Computing platform Condition number Metropolitan area network Computer icon Process (computing) Software developer Debugger Functional (mathematics) Local Group Computer animation Personal digital assistant Module (mathematics) Interpreter (computing) Reading (process)
Ocean current Point (geometry) Standard deviation Computer programming Context awareness System call Computer file Line (geometry) Debugger Range (statistics) Division (mathematics) Control flow Primitive (album) Online help Parameter (computer programming) Function (mathematics) Mereology Event horizon Formal language Number Prime ideal Causality Insertion loss String (computer science) Smart card Video game console Uniform space Exception handling Metropolitan area network Constraint (mathematics) Forcing (mathematics) Expression Debugger Line (geometry) Variable (mathematics) Functional (mathematics) Frame problem Radical (chemistry) Event horizon Sample (statistics) Computer animation Function (mathematics) Cost curve Normal (geometry) Iteration
Point (geometry) Metropolitan area network Computer programming Divisor Code Lemma (mathematics) Expression Line (geometry) Set (mathematics) Frame problem Number Maxima and minima Subject indexing Computer animation Video game console Game theory Data type
Point (geometry) Metropolitan area network Email Process (computing) Thread (computing) Debugger Debugger Visual system Virtual machine Interface (computing) Control flow Event horizon Frame problem Connected space Revision control Computer animation Network socket Telecommunication Right angle Remote procedure call Communications protocol Computer architecture
Computer programming Thread (computing) Digital electronics Open source Debugger Cloud computing Parameter (computer programming) Event horizon Type theory Computer network Integrated development environment Metropolitan area network Process (computing) Validity (statistics) Computer file Expression Code Line (geometry) Frame problem Connected space Message passing Computer animation Network socket Telecommunication Computer network Dependent and independent variables Cycle (graph theory) Communications protocol Data type Separation axiom
Point (geometry) Computer programming Context awareness Computer file Debugger Line (geometry) Control flow Mereology Weight Computer font Event horizon Number Video game Computer network Integrated development environment Metropolitan area network Process (computing) Scripting language Consistency Content (media) Code Bit Line (geometry) Functional (mathematics) Frame problem Message passing Event horizon Computer animation Network topology Network socket Function (mathematics) Cycle (graph theory) Data type
Metropolitan area network Computer programming View (database) Interface (computing) Binary file
Metropolitan area network Software bug Multiplication sign
Implementation Line (geometry) Multiplication sign Control flow Parameter (computer programming) Tracing (software) Event horizon Smart card Video game console Integrated development environment Information Gamma function Conditional probability Traffic reporting Uniform space Condition number Exception handling Newton's law of universal gravitation Metropolitan area network Computer icon Multiplication Real number Forcing (mathematics) Expression Java applet Ext functor Functional (mathematics) Inclusion map Pell's equation Event horizon Oval Business & Information Systems Engineering Hill differential equation Ideal (ethics) Data type
Metropolitan area network Computer animation Hand fan
Metropolitan area network Message passing Computer animation Smart card Line (geometry) Smart card Text editor Line (geometry) Event horizon Functional (mathematics) Frame problem
Standard deviation Group action Thread (computing) Code Line (geometry) Debugger Scientific modelling Client (computing) Parameter (computer programming) Mereology Coprocessor Formal language Revision control Natural number Smart card Haar measure Subtraction Multiplication Condition number Form (programming) Metropolitan area network Multiplication Standard deviation Constraint (mathematics) Process (computing) Touchscreen Debugger Line (geometry) Functional (mathematics) Inclusion map Arithmetic mean Computer animation Order (biology) Interpreter (computing) Queue (abstract data type) Library (computing) Row (database)
Debugger Gradient Software developer Multiplication sign Source code Moment (mathematics) Branch (computer science) Field (computer science) Revision control Computer animation Linker (computing) Repository (publishing) Arithmetic progression
Point (geometry) Bytecode Metropolitan area network Computer programming Dataflow Product (category theory) Projective plane Line (geometry) Client (computing) Formal language Computer animation Video game console Condition number Exception handling
Ocean current Computer programming Process (computing) Product (category theory) Computer animation Observational study Integrated development environment Personal digital assistant State of matter Electronic mailing list Instance (computer science) Exception handling
so next day because Dimitri to remove and his talk is Python debugger covered and Dmitri is said that development that quantum ideas that the meeting hi my name is today I will speak about right demography 1st introduced myself and to the stock I will forge brains and the last 4 years by developed by chance and debugger is someone there other things that I do and often when I see people that I developed of applied and bodily state while it's kind of hard to complex but I'd like to show you this this development abiding debugger is quite an easy
task so there is no rocket rocket science here but in the 1st place so why do we need to buy section as Bryant going
down the road of his book dividing
this twice as hard as right and according those places therefore if you write the code as there is possible you are by definition not smart enough to the budget and what we do normally we all these are all from but
I think rather always write on code as clearly as possible and as a result we have blogs so that well I output and then we need to find those box and they have problems the only thing that can pictures save us that is good about and we'll look to implement 1 so there are a lot of white and barbarous by but I'd thereafter they
divide them into the groups the first one but antibodies that got implemented in Python itself and those it by sham debugger and by by actually showing divided use a fork of pi divided was for 4 years ago and it gained a lot of new features and now we're in and we're strange situation when we do well in these are gender Varga separately and far below the mean dignified devotes separate separately and exchanges thesis and Bridgeport features
but we we now in this some kind of process tools establish situation but I'll tell me about it the end of my presentation so but antibodies submitted 2 the most advantage of such devices that they are platform independent they can run and once you've identified by prior fire white and that's what that is because of their reading your Python but the problem with this device that they can be broken by user code because the the same fight interpret you can just write something like a clearer overseas modules and to debugger will evaporate from the memory and so it won't work minimum uh the 2nd group of debuggers are those that are implemented in C and uh the main dividers our the in DB and they work only for but they don't
interfere with cortisol so they were better for such cases like a democracy due and all these to the court in but that is actually not a problem for by debuggers written Python because all those cases can be solved but you need to to make something for that so how to implement Bijan divided uh actually minor languages provide developers of debuggers some kind of API to develop divided and divided and has been different in this case it's a price and 1 function to develop debuggers functions called sees trace and if we look into the bike and conditions we see that you can set trace functions and it will be called every time you get any event in the Python interpreter and the
events can be a cost function execution of line or a returned from function exception or use some of the kind of the binding so that they can be calls the function returned from the functional 1st the exception so when we want to implement our toboggan receivers documentation and we realize that this is the only 1 function that we can use we can get it is scary um because it's variance as sort of primitive and we feel comes very constraint but there's been all constraints that restrict duty so as I show you uh the causal that fact that we constraint a lot we can implement a lot more features than actually exist in normal languages like Java and current C + + and so on so trace function how can we use it she will we implemented a simple trace function that is a function with the 3 arguments the 1st 1 is frame that actually is the top all call stack of this is the context in which has a local variable so they went with with that we get in some parts parts of dependent on meant and we bring just which went get and on beach line again the line that you can access from the frame so we would also use models such and then we have some simple called retreat from within the range and to bring to the deletion also a summary of arithmetic expression atlas look what we have so after we have set out to its function
on the 1st line that will be line number 9 it should be it will the line number 15 but we have all of the function this is declared like lumber and 9 we have a and then we have executions before line number 10 they have execution before line number 11 and after that we got to the 1st output then duration continues we are again on the line in the den and the lion number 11 and we get the gain our output and on the 3rd at iteration we have an exception so but institution terminates and you see that we have in these short program we have all 4 quite and events that are possible all wines exceptional so eventually that shows us that developing debugger with the help of the trace functions possible so let's developed simple a Council divided of to test it we use a simple program that is simply the defense see if that's their function that gets an as a parameter and bring all the primes between 2 and an and here are out of divided this picture is only 23 lines and it works I'll just tell you about how it is implemented and I'll show it and were it has 2 main parts the 1st part these 3 functions as we've seen before we can we get events and to here except just bringin all the events that we have a break point there is only 1 break point and if the and execution equals to 0 this breakpoint line we do a simple thing we just read the input from and these into we handle comments from on so there are 3 possible answers the forcefulness frame if you get it we just print all variables the 2nd context C for continue inspired by GTB and we continue to the to the next institution of the breakpoint and any other string is just treated like an expression that is evaluated and the 2nd part of this a simple debugger is in the morning part we should get here from the the command line arguments in the line of the breakpoint file to be but and then we set out to function as a trace function and execute this far so let's look how it works so you
see we execute our simple divide we set point the line number 6 and 2 we passed defense of by as of simple programs so what about a stop somewhere and do we uh type frame and we see that we have a favorable so this out in year our multiple sets that is empty we continue and 2 we get to the sound prime we print don't frame it game here we see that our index is 3 and multiple set is all even numbers so we can continue continue continue until the if we don't want to avoid I read all the frame we can look now that doesn't work bring we can bring to just the thing things that we are interested in so 1 of we have our simple Council divided just implemented in in 23 lines of 20 code but these the divided it did
not a full-fledged it's it can be used to make sure it's not very convenient because we consider only 1 requirement and normally uh we run our program and we need to please breakpoints somehow interactively and actually here you can pass while to as the expression of the hot hand will process and it's not very convenient to read all frames in the canceling the you we need some kind of 3 for
that of something of you saw all thinking about that we came to the idea that we need a visual device and if we need visual debugger we need some multitasking and for this we need some kind of architecture and the subject can look something like that the left you see Our or a debugger interface and of the right you see the bite and process that is being developed
communicate between each other within connections that allows us to run them on different machines and to allow remote debugging for example and L. debug interface sends break points and commands to the Python process and gets back events threads frames evaluated will use so on and the by the process to handle this communication we need 2 threads for reading the right and we use the rest here because we don't need actually performance uh and use a lot of problem for us but we need the process was work we need to do these things to work on Titan and all versions of Python so we just use threats and it looks good and he would have read afraid right of freedom to those use forever so are because they feared talk about communication then we need to find a protocol
the protocol for this communication with will be quite simple every message is just a line and separated by by
why inspiration and to all the data inside this line is divided by then separation the 1st 1 for the the 2nd 1 common type and then we have 4 different arguments that depend on died so common types can be set breakpoints resume get threads get frames valid expression and for example for the message uh get thread we we we generate some ID and the response for that it will be of it will have the same idea and these will make us to know that this is a response for these were request so this is very where simple protocol was very powerful dictionary so we can get rid of responses for our request we can get not but if we want to get responses we can get exactly the same responses for this event that you want so it's very simple and very powerful so what was on the side of Heidi uh we assume we we will not go into details will focus on the Python court but we assume that I decrease of circuit for us and declines discrete that is being developed with the command line and processes of the subject to address and the FIL the passes the simple program as argument on divided so let's look at how our liberal will be and it's quite simple also the main port looks like that we In should divided and 1st of all we make a so connection so the characters the simple crater and connected it is already opened on the side of the idea the next we initialize on network communication it's very simple you just created right of hereditary the threat of Southern How them the company logo so How can look very different it just In the cycle until killed 3
data from the socket and find some lines spiration find fly inspiration it thinks that it gets the wall message that can be passed the past pass messages just with Leighton eaten by tabs which that those 1st element as as a deal the common the 2nd of type of the content and we have would this and our process q the writer is implemented by the same so what's next the next is a bit more interesting Veronneau programs to do that it fills the trace all such we set out trees functions and then we have erp weights for a commentary from idea to start because this point we need to be sure that all all the data from ID has just arrived the breakpoints assets and when we get to a common from idea to start we execute and the most interesting part is trees function is actually also a simple we handle here of life events the taken from the frame the line number and the file name and see you have breakpoints points for this file here we have break points of this file and if we do have for a break point for this line then we just send a message to already that we need to suspend and to be too in this point In a cycle for a message from idea so execution is suspended here we don't don't execute consonants we just wait for message from idea to resume and if you don't have break points of these files but we don't
they this context because it will optimize a lot alterations so sold if I can show you how it looks like font is a bit small but I think it's OK if you have our
uh simple program the and we do about it and to
read just stops upstream this says it looks strange discrimination 48 so actually that that's the we just implemented 2 views of the body that communicates with interface but we leg now
were important features the first one is conditional breakpoints its ability to go to a set of let me close how to
demonstrate that don't know I I see that this is the 1st time OK can show on with the OK so
we need to implement conditional breakpoints exception break buttons stiff always have been to much think we need to make it work on white and do not for 2 3 4 and we were like to implement multiples buying and I'll show you now that is also very simple so how do we implement conditional breakpoints we just enhance all uh trace function that we the if we have any condition condition is just biden expression that is related to true or false see Heaven condition expression we evaluated and if it is true this is false we don't don't stop and this very well so what we have conditional breakpoints exception breakpoints but to trace exception breakpoints so we need to store traces to handle somehow all exception events and we do that were a simple we get our exception time from arguments and we see if we have an exception report for these exception type if we do have was suspended if we don't have we don't suspect so that being force must been and run to align these functions are very simple I had like to show you whether you would
I need to know what this smartest thing to show you but you so steps supply 1 but so
stepping do is just going inside the function executed Staford's Stephen of the execution of a function and going the next line and just must have been 2 is just a the possibility to step into 2 of the selected function and go to line is actually a going to the specific line that you you can select a new editor so how's it implemented it's very simple is that of a simple step into is just as you installed on the next line step it's just step into what we stand on the next line in the same frame we remember 0 which frame wasn't when we receive a step forward the message then with event and using a go somewhere inside and when the top this research those stack frame where we the still there and smart city into it's
just the thing to do but we stopped and the line of the selected function under underlines just a temporary breakpoint which we remove after we have reached its so these 4 for future sigh
implemented Justin I don't mean trouble flights
each so what about support of light in the form an old version of 510 all interpreters how to actually this is not the best part of the code because when you need to support old versions you quote of worries fast genes the alliance like these you need to handle all differences and standard library but litigation you going you can have collected in only 1 file and that this we will not spoil the rest so lots crosses the border and that is that that that I like the most because these features shows us how constraints that we haven't quite and only 1 API functions to implement debugger allows us to make something something better than in different languages for example you can add about multiple processors and John easily but bite and you do it is demand demand nature and due to the fact that we implement all by hand also stimulus and if you go to the by elaborate the condition again we see that all the only processes in the end user was the function of the voice model like that's exactly where and parliaments and there are about a dozen maybe 1st of all of the fork function is executed normally and then some of some of the of this function 1 of these functions so what we can do it by then we just Monday question and we do that in this way we yeah take it voice model function and replace it with all New Exec functions and in always a function we call the original function that with botched arguments and the question of violence is very simple if a despite having executed we believe it and then and debugger screen in front of the real arguments and hosting in order to already have and what has happened in in in practice is that our new process that is it about to launch it 1st client inside the divided among connects to the ID and then the debugger recordings accused of this and you process so we have a debugging of a new process like the body of the new thread action what we have learned today we just saw that it's very simple to trace but called and it's where it is simple to make a simple parts of the body and also it's where is simple to implement a real visual device but what for I
think encourage you actually to also contribute there are a lot of features that can be implemented in this field have and they can be implemented by you will with health I don't say edges that we give up that we are starting to develop divided by this we actually make a lot of work but if you help to solve your data problems that will be greater and the sources who where the got the first one is the link to
debugger In by uh opensocial was during the 2nd 1 is the link to by barter in some detail there is 1 moment that I'd like to tell about a now there is a work in progress there is a merged version of by shaman pointed to by the uh the repositories already created is culture by deaf Dutch divided it has no quality of because the president was created just for the last week and it has some some development branches but stay tuned to uh in the short time or we get merged version of my death and questioned about their with all the union also a different features we that have both
of the IT and also the conditions there so what would possible to contribute to this project and to learn how it is all implemented so that's all you could you can just I will start
with a simple question is there a console client for your point about the region toward higher aware of anything that not yet but I think that after we establish this notion required that we only make 1 is never 101 and having considered data watchpoints and how hard would just be accomplished collected language we considered to implement that's was really has not government evaluated the performance problems that can which can be there I think it definitely worse to trying whom but cannot to say nothing about of cereal production conditions of the feature is possible the to modify the flow of the program can skip the execution of lines or suppress exception of pictures here it's not possible in spite of it's it it is this is partly because partly possible so you can fax byte code
with you get bottom it won't work in all cases as as follows so for instance uh no I think now that you can answer suppress the the exception that is the raison not caught i can question so it's nice when you run your programs in development environment and you can run the museum modern about when you're on problems in production they are usually not instrumented but they still fail sometimes and you want to troubleshoot and about them so what the current state of biotin dividing or the and instrumented processes so all of that sexually the first one stay tuned hope alright so it's not yet there but that it is the 1st on the list OK any more questions many a case study of the the end of the session and so thank you to make use


  802 ms - page object


AV-Portal 3.9.2 (c7d7a940c57b22d0bc6d7f70d6f13fde2ef2d4b8)