RevDB, a Reverse Debugger
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 9 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/62300 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
2
8
00:00
DebuggerRange (statistics)Module (mathematics)Demo (music)Computer fileCountingLoop (music)Object (grammar)Electronic mailing listDebuggerLine (geometry)Computer programmingMathematicsExpressionPoint (geometry)Interface (computing)Software bugMultiplication signCASE <Informatik>Web page2 (number)BitDemo (music)Context awarenessMereologyProjective planeRevision controlIntegrated development environmentImplementationStandard deviationReverse engineeringForcing (mathematics)PiLogicTransport Layer SecurityBeat (acoustics)Error messagePixelFigurate numberHookingCountingMusical ensembleArmRow (database)CodeType theoryLetterpress printingNumberLecture/Conference
09:48
Principal ideal domainDemo (music)Computer fileModule (mathematics)Port scannerGEDCOMFunction (mathematics)World Wide Web ConsortiumMobile appMultiplicationExtension (kinesiology)Modul <Datentyp>CodeRead-only memoryDebuggerComplete metric spaceEvent horizonProcess (computing)TrailDesign by contractComputer programLoginReverse engineeringString (computer science)CodeLine (geometry)Wrapper (data mining)Interface (computing)Point (geometry)Computer programmingRow (database)Multiplication signMobile appDifferent (Kate Ryan album)NumberComputer fileSign (mathematics)Module (mathematics)WindowFunctional (mathematics)Extension (kinesiology)Revision controlProcess (computing)Semiconductor memoryElectronic mailing listInterpreter (computing)Ocean currentThread (computing)System callDebuggerOnline helpObject (grammar)Local ringVariable (mathematics)Letterpress printingMeta elementAttribute grammarControl flow2 (number)Level (video gaming)outputDeterminantRaw image formatCrash (computing)CodeError messageDressing (medical)Pointer (computer programming)SimulationBitMatrix (mathematics)Physical systemOrder (biology)MereologyMultiplicationTerm (mathematics)Green's functionCASE <Informatik>Lecture/Conference
19:36
Complete metric spaceProcess (computing)TrailComputer programDesign by contractEvent horizonPairwise comparisonFormal languageMassPort scannerSoftwareDebuggerCodeVideo trackingModule (mathematics)Computer fileMultiplication signFunctional (mathematics)Object (grammar)Point (geometry)Software bugNumberLogicRow (database)Normal (geometry)Bit1 (number)Slide ruleDebuggerComplete metric spaceStability theoryRight angleProjective planeDatabaseUniform resource locatorReal numberFormal languageBootingCodeData compressionType theoryComputer programmingVariable (mathematics)PiSoftwareLetterpress printingFamilyCondition numberPlanningOrder (biology)Canonical ensembleInterpreter (computing)Arithmetic meanLattice (order)DivisorReverse engineeringLecture/Conference
29:24
Computer animation
Transcript: English(auto-generated)
00:00
So this talk is about reverse debugging, now what does that mean? So first a bit of introduction, I'm Armin Rigaud, I'm part of the PyPy project.
00:22
So the PyPy is 14 years old, other implementation of Python, who focuses mostly on performance, as in if you have a Python program, like a Python 2.7 program, you can try to run it in PyPy instead of the standard C Python, and then maybe it will be five times faster.
00:43
That's the goal of PyPy. But this is not what I'm here to talk about. I'm talking about RevDB, which is a kind of a hack that I did by modifying PyPy.
01:01
So it means if you have a Python program that runs on the standard Python, well chances are that it also runs on PyPy, and if it does, then you can use RevDB on it. Okay, but what is RevDB? What is a reverse debugger, generally speaking?
01:23
So here I have an example of a very large uncomplicated program. Keep in mind, please, that this is a very large uncomplicated program for the purpose of this talk. So this has a bug.
01:47
So it will fail the final asset, sometimes in the loop. Can anybody see why?
02:16
Now it's just a list of Xs, and the X.value starts at five here, here it's incremented
02:28
to six, here it's incremented to seven, and then we assert it is seven. But this fails sometimes. And it fails because here I put one X object twice in the list, so I have a big list of
02:46
101 objects, but there is twice the same. So it means here it will be incremented twice for this special X object. So here it will fail with an eight for just one object.
03:05
So you can imagine that this is an example of some very large uncomplicated program in which the same problem occurs, but obviously it's the kind of problem that you will have to track down for one week or something. So let's keep that in mind.
03:21
Now what can you possibly do? Well if you have a reverse debugger, here is the reverse debugger. I'm at the end of the program. So it is something that has an interface that looks a lot like PDB in case you know it. So I can print the value of X, well it's some object, but what is X.value, seven, what?
03:50
Eh? What? Wait a second. Yes, okay. Forget about what I just did.
04:03
Now here it crashes, the X.value is eight, okay? Now the question is, if I'm here, I see that the X.value is eight, yes thank you, why is it eight? What could have occurred in the past to get the value eight here?
04:20
Well, what occurred in the past, now I'm going to use magical common backward step, and now I'm one step before. So here X.value was seven, so it was seven just before I ran the line X.value plus equal one. Okay?
04:40
Well, so far so good. Okay. But then what occurred with this X.value, at this point X.value should be six, right? Because I should have incremented it only once in the loop above. So what occurred? So I can actually watch it. Now I'm going to watch this, yes, okay, sorry.
05:04
I need to print X, so X is an object called $4, you see here at the start of the line. So I can print $4, that's the same object, and $4.value, it's seven, and then I can watch
05:20
$4.value. This means I put a watch point on the value of .value of this exact object, $4, okay? So what can I do now? I can back continue my program.
05:44
So now I'm back in the past at the point where my $4.value changed. So it says here $4.value changed, basically. So what is it now? It's six.
06:00
Okay? But I need to go back again to understand what is going on. Ah, now I'm hitting another time the same line. So now something strange is going on. Why am I seeing the same $4.value changed by the same line twice in my program?
06:21
That is, I mean, from how the loop looks like, it's obviously because X is twice in the loop, in the list, LST. Like what if I count them? Yes, there are two. My object $4 is twice. So if I print the list, it is a long list of objects, okay?
06:46
Here in this example, I've reduced the number, I think, but you would see a list of 100 objects. But importantly, you see that this list is now called $6. So I can say that $6.count of $4 is 2.
07:04
Now why is it 2? Who puts twice the same object in the list? Let's watch it. And let's forget about the first watch point because I'm no longer interested. So now I'm watching changes to the expression $6.count $4.
07:25
And I can back continue. When did this expression change? It's here. So here I get the point that adds the same object twice, well, another time in the list.
07:40
So now I've figured out where the bug is. So it's really, well, think about this kind of playing around and, well, it would have worked just as well if the program was a huge mess instead of just one page. It would have found the exact same point.
08:05
So now I hope that you're sold to the ideas that reverse debugging is sometimes useful. Okay. This is a question that I will leave for later. How is that possible?
08:22
First that you did not cheat. What I showed you is not a pre-recorded demo or anything. It's something that really works, and it's something that really works for large programs. It is a tool that I developed because I had the bug.
08:40
So yes. At one point, oh, no, I have this bug. I will spend one week digging around. So instead of spending one week digging around the code, let's make a reverse debugger, and two months later it took me only one hour to solve the problem.
09:01
Thanks. Yay. Okay. A bit of context how I can get this kind of reverse debugging going.
09:22
So here I have a demo. That's the demo that we have seen. So if you run it, you see the assertion error. You can run it with PyPy instead of Python. You get very much the same thing because PyPy is just a variant of Python.
09:43
But the point is that you can use a special version of PyPy, and then I use an environment variable to tell it to write a file that I'm calling log, okay? So what occurred just now is that it ran the complete program, but it also wrote into the
10:06
file log, a log of everything that the program did. So you get this file log that is not as huge as we could fear. It's maybe a few megabytes per second of execution.
10:22
Which is okay nowadays. Okay. So here I got the file log at the end. And then the point of reverse debugging is that once you get a log of an execution where
10:41
something buggy occurred, then you can replay the log as often as you want. So here I have the file log. I'm going to keep it safely in a corner. And I'm using then another tool, for example, rubdb.py, and this gives me the prompt that
11:00
looks like PDB. So it's a prompt where I can do things like that and go to the point where it crashed and enter my command and look around and keep and restart. And the point that is it's always the exact same recording that you can look around.
11:24
So yes, the kind of features that you have, well, you have seen them most. I mean, you have next to go to the next line, B next to go to the previous line, step and B step. And the difference between next and step is whether we want to enter function codes or
11:45
not. And then you have continue and B continue, finish to run until the current function finish and B finish, which means of course run backward until the current function was called.
12:00
You can print something with P and watch something. You can use this with a special, I mean, this is just the dollar zero is just an extension of the syntax. So like dollar zero dot something.
12:21
Yep, that's useful mostly for watchers because what you want to watch is typically the exact value of this attribute of this exact object here, but then you cannot really name the object in such a way that works at any point in time.
12:41
As in you cannot use a local variable name, for example, because as soon then as you do continue or back continue, well, these are local variables going to go out of scope. So what you do instead, if you watch based on dollar zero, dollar one, and so on, which really means this object here, even if we go back or forward in time.
13:05
Yes, Anna, well, and then you have a few more commands that are more common to debuggers like break, break points on functions or file and line numbers and so on and so forth. And I mean, can break on a line like the watch points are triggered if you go backward or
13:27
forward when you reach that line. Yes, well, with this kind of example I showed on a very small example where we can really
13:40
follow what's going on, but on a larger example what you want to do is to write down in a separate file, write down the things that occur in your program and keep that file ordered by time because you're going to go back and forth all the time and then you don't know anymore, you're getting very easily completely lost.
14:03
So keep your file ordered by time and the number here, the number on the left of the prompt before the dollar sign is the time precisely, like if I do B step, it's decreased by one, so keep your log, keep your file sorted by this time.
14:25
Yes, you have a built-in help command with a list of all commands, et cetera, okay. So what works, what works right now, like I mean, you can use it basically, you can
14:42
run any Python code that PyPy can run, yes, okay, that's a lot of Python code. You can run it on multi-threaded apps. If you record what occurs in a multi-threaded apps, then when you replay the recording, you see the same, well, the same interleaving basically that was recorded.
15:06
How is that possible? Well, with the global interpreter lock. You know the Python and PyPy have global interpreter lock, which means that actually only one thread can run at a time with well-defined switch points. So with this in mind, it's not very hard to do the recording and replaying in multiple threads.
15:32
More interesting, if you have a big program that use tons of C extension modules, you can actually use rough DB, and you can go over the steps that would, I don't know, print
15:46
something to the screen or do whatever IO, et cetera, like call this function in this module that is going to compute the determinant of this huge matrix, and you can go one step
16:05
after the other on the Python level. Of course, you cannot, well, it's not combined with a debugger that would be able to do the same thing in C, but then you can still debug on step-by-step on the Python level.
16:23
Here's a small warning is that when you print, when here, I print X, for example, it gives me this string. This string is a wrapper of X, I mean, X.repper basically, okay? But the point is that, well, for a lot of objects, we cannot call the wrapper method
16:46
if the wrapper method is going to do, I don't know, to do things like send inputs and output or call things, we'll call things that would be in C, et cetera. So if you try to print an object that rough DB cannot print use, you will get the error
17:05
or might attempt to do input-output or access raw memory. What does not work so far, the main thing that does not work is long-running programs.
17:21
So by long-running, I mean more than a few minutes. So if you have a program that crashes randomly, well, there are two cases, either it crashes randomly, but only after at least one hour of execution, then it does not work, then you're lost.
17:40
Or it crashes really randomly, and then you can run it for five minutes, interrupt, running for five minutes, interrupt, running for five minutes, interrupt, et cetera, leave that running the whole night, and then if you are lucky, the next day you will get a crash that occurred after just a few minutes of the program starting.
18:02
The other thing that don't work nicely so far is anything using the stacklet, greenlet, or gevent interface.
18:22
Anything where to really understand what's going on, you would need to track several process instead of just one. Windows is not supported, this supports POSIX basically, I mean it works on OSIX and Linux, and it's probably very easy to adapt to other POSIX systems, but Windows is harder mostly
18:44
because we use a lot of fork internally, I will come to it later. Python 3, this was an example running in Python 2. Now Python 3, well, this is done, so as I said, this is a special version of PyPy, okay?
19:07
Now a version of PyPy that would run Python 3, like Python 3.5, is getting ready, as in we should have the first beta out in two weeks or something, is kind of getting there.
19:21
Now a version of Python 3 that contains revdb is easy too, because the revdb work was done, well, as everything in PyPy, it's done at the meta level, so it means it's done by, well, it's not done by going manually through the whole interpreter and adding logic everywhere.
19:50
So it's done in a way that will be easy to do with Python, PyPy 3.5 too, okay? So it's not done, but should be done soon, okay?
20:05
Now what I've just shown is reverse debugger for Python, now reverse debuggers are something with various names that are called omniscience debugging or historical debugging or just
20:21
backwards debugging, it is actually something that exists for the C language, for example, you have traditionally undo db, gdb, and more recently rr, both of them are kind of working only on Linux.
20:42
For Python, we have some experiments too, but they are not really the same thing, epdb, pod. Okay, well, this looked cool, why is it not more well known as a technique, basically?
21:02
Well, there are several problems, one problem is that it is often a cannon to take down a fly, okay? Not always, sometimes it's really useful, but most of the time, no, I mean, like I spent two months doing revdb, then spent one hour fixing one obscure bug in PyPy, and I've not
21:26
used it anymore. I think at one point I tracked it down because I was too lazy to run the normal pdb or something, but yes, I mean, it is an extremely useful tool, very rarely, yes, you also have issues
21:53
like performance, like if you run a normal gdb or pdb, you expect things to be mostly
22:00
just as fast as without a debugger, but here it's slower, as in the recording really has an overhead maybe two or three times, which is fine if you don't use it too often. Well, there are more issues that revdb does not have, I hope, like other programs that
22:26
are doing reverse debugging tend to crash, I can say from experience, tend to crash a lot, and also they tend not to give a reliable history, as in you go to some point, you go back in time, you print stuff, but what you're getting is complete nonsense, it is not what
22:47
should be printed at this point, it's just nonsense, so you have to guess if the debugger is now telling you the truth or not, and well, also, there is also a bit of a non-technical
23:03
issue is that these reverse debuggers, they actually tend to be proprietary software with very restrictive licenses, like undo db, well, I don't know if I should do, if I should publicize undo db or not, and in which way, because, well, yes, it's a great tool, we have used
23:24
it in the past, but it is appropriate software with restrictive license, you cannot just install it on your laptop and run, so these are issues that revdb does not have.
23:40
So yes, this I explained already, the reverse debugging is a canon, but sometimes, hopefully really you will need a canon, yep, okay, so with this end of the talk, I have some extra slides that go a bit into the how is it done side, if I have some time, we'll see, maybe
24:08
I'll start with the questions, and also, also, we are, well, a bit of publicity here for the next sprint, we're doing a sprint on pipeline in general, but that may include
24:20
revdb or not, depends on who is here and what the interest is, so it is going on in Lezans, and everybody is welcome, so if you want to come for the weekend or for the complete week, it's fine, please contact me, okay, you have the URL for revdb, thank you.
24:49
Thanks, do we have any questions, we have some time left, yes, here in the front, wait
25:04
a second, okay, thank you, so it seems like this kind of data that you collect in that database that should compress really well, probably, I mean, I didn't try to compress
25:22
it because mostly the main meeting factor is recording time, five minutes, so the problem is that if you record for one hour, the problem is that every replay is going to boot in one hour as well, so it can get very quickly annoying, so if you limit yourself to five
25:46
minutes and it takes one gigabyte, it's fine, it's just a temporary file in a corner, right.
26:02
Any other questions? Can you watch a local variable, can you watch a local variable there? Because you watched the global thing, you have to watch objects really, so if a local
26:27
variable, you mean, right, well, here for example, I print x and then it's a $0, then
26:40
I can watch really $0 or $0.value, right, so if it's x is a local variable in some function and you start watching it, you will only see it in the one execution of that function, because in the next execution it will be a different object already.
27:00
Yes, but you can't basically, you cannot, you cannot say watch and use a local variable simply, you have first to print local variables that you're interested in and then watch the numbers $1, $5, $6, so you're always watching an object. But only this one is, like, it's not smart enough to figure out that I mean every time
27:24
I call this function and then whatever is there. No, no, no, you, okay, that's more advanced features, but they are not. No, no, I'm just, I'm just curious if that works. Yes, yes, I mean, here it's really watch points which are really, really about, now I want
27:43
to look at this object here. All right, okay, thank you. You want to ask something else? No, I got a question for you. Now he's talking about advanced features. What's the future of this project? Does it stay an experiment or will it evolve?
28:03
It works. I mean, yes, it works, that's it. Just because you said you used it once, so that doesn't... Yes, I mean, I've used it as an example on toys and everything a lot, definitely. Okay, and because you said there's going to be a Python 3 release probably, so you're
28:25
still working on it, I guess. Yes, I mean, well, what I mean by that is that it is not a lot of work. I mean, it's really a magic layer on top of PyPy, and then as PyPy itself evolves, like, will grow to support Python 3.5 or 3.6 in the future, et cetera, then, well, the same
28:47
magic layer would work. Okay, but other than this, there's no future plans? Sorry? Other than this, there's no future plans? No, not really. I mean, yes, as I said, yes, I could spend more time adding conditional breakpoints and
29:03
everything, but, well, also what's the point at some point, because, well, it's really a tool that is meant to be used once a year or something. May I ask the next speaker to come up and get ready? Do we have more questions?
29:21
Nah, doesn't look like. All right, let's thank Armin once more.