Inspect (Or Gadget)
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33721 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017103 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
SoftwareIntelArchaeological field surveyCodeKey (cryptography)CASE <Informatik>Lecture/ConferenceMeeting/Interview
00:30
Stack (abstract data type)Process (computing)CognitionFunction (mathematics)Library (computing)Data typeEquals signAlgebraic closureSource codeSocial classObject (grammar)Parameter (computer programming)Personal digital assistantString (computer science)Type theoryoutputModule (mathematics)Interpreter (computing)Presentation of a groupCodePoint (geometry)String (computer science)Endliche ModelltheorieType theoryParameter (computer programming)Functional (mathematics)Object-oriented programmingSource codeInformationCharacteristic polynomialObject (grammar)Attribute grammarSoftware testingArithmetic meanFormal languageMereologyIdentity managementDifferent (Kate Ryan album)Latent heatElectronic mailing listInstance (computer science)CASE <Informatik>Library (computing)Open sourceSummierbarkeitService (economics)Condition numberFlow separationMetropolitan area networkComputer configurationDescriptive statisticsWave packetMultiplication signStatement (computer science)Computer fontVector spaceNamespaceComputer programmingFunctional (mathematics)Electronic signatureSocial classCategory of beingElement (mathematics)Sound effectLine (geometry)Demo (music)BitProcedural programmingSystem callLimit (category theory)Physical systemInterface (computing)TwitterNumberSoftware developerOrder (biology)Run time (program lifecycle phase)Stack (abstract data type)Complete metric spaceBuffer overflowSoftwareConstraint (mathematics)Core dump
09:23
Type theoryoutputUniform resource nameElectronic signatureString (computer science)Dynamic random-access memoryLatin squareExecution unitTwin primeInstance (computer science)Type theoryParameter (computer programming)Error messageModule (mathematics)Library (computing)outputPoint (geometry)Validity (statistics)Representation (politics)NumberIntegerFitness functionStructural loadFunctional (mathematics)Electronic signatureRegulärer Ausdruck <Textverarbeitung>Expected valueBitCodeLine (geometry)Default (computer science)Latent heatAttribute grammarSoftware testingConsistencyMultiplication signConstraint (mathematics)Object (grammar)Population densityNormal (geometry)Internet service providerRight angle1 (number)String (computer science)Condition numberComputer configurationPosition operatorDistribution (mathematics)MereologyContent (media)CASE <Informatik>MaizeMoment (mathematics)Extreme programmingState of matterMeasurementSolitonArithmetic meanView (database)Medical imagingPhysical systemNetwork topologyPhysical lawSeries (mathematics)Execution unitMarginal distributionTape driveHidden Markov modelXML
18:17
String (computer science)outputType theoryVarianceMetreImage warpingFunction (mathematics)Interface (computing)Polymorphism (materials science)Meta elementComputer programmingIntegrated development environmentCode refactoringImplementationSocial classCASE <Informatik>Functional (mathematics)Keyboard shortcutResultantInstance (computer science)Order (biology)Line (geometry)Parameter (computer programming)Object (grammar)Type theoryPolymorphism (materials science)Interface (computing)String (computer science)outputElectronic signatureOnline helpRepresentation (politics)CodeDifferent (Kate Ryan album)ImplementationSoftware maintenanceWordAttribute grammarGoodness of fitMultiplicationMultiplication signPresentation of a groupExpected valueMereologyMetaprogrammierung2 (number)Point (geometry)BitState of matterModule (mathematics)Latent heatFunction (mathematics)Data storage deviceFormal languageRun time (program lifecycle phase)IntegerView (database)Error messageValidity (statistics)InformationPortable communications deviceRight angleCondition numberSoftware testingSound effectINTEGRALSystem callComplete metric spaceNetwork topologyCellular automatonPrisoner's dilemmaOffice suiteComputer architectureDialectMedical imagingSpacetimeWeb crawlerWebsiteMetreInformation securityStatement (computer science)Computer programmingXML
27:10
Performance appraisalCodeModule (mathematics)Mathematical analysisEndliche ModelltheorieLecture/Conference
27:34
Mathematical analysisStaff (military)Functional (mathematics)outputTheoryData structureState of matterComputer programmingSource codeCodeComputer filePerformance appraisalSemiconductor memorySlide ruleCASE <Informatik>Fitness functionModule (mathematics)Group actionNetwork topologyAddress spaceVolume (thermodynamics)BitFormal languageMathematicsFunctional (mathematics)String (computer science)Ocean currentDivisor1 (number)Lecture/ConferenceMeeting/Interview
Transcript: English(auto-generated)
00:05
Thank you, Dugal. First, a quick survey. Who knows what introspection is about? Could you just raise your hand? Okay. Who has already used it in his own code? Okay.
00:20
For the people who have not raised their hands, don't worry, you are not alone. It was my case a few months ago, and that's why we had this idea of talking about introspection. How did this start for me? We were working with different people on writing a test for hiring a Pythonista.
00:43
We had already tested for the languages, but we had known for Python, and we saw that it was important to have such a test. Among the tests, we wanted to verify that the candidates could write specific type of functions, typically like a generator.
01:00
We wanted also to enforce, typically, the use of list comprehension by asking to modify a code by changing just a line. The problem is that this test is, of course, automated, and that we wanted to check if the candidate had indeed followed those constraints. And all the things we had are the three lines
01:22
that you see there. So roughly, we could import a module called answer, including the function that the candidate had developed. Then, of course, how could we do that? No idea. So who should you ask? Of course, your best friend. And who is the best friend of any software engineer?
01:43
Stack Overflow. And of course, we found the answer there, and the answer is, oh, just use the library called Inspect. Just call getSource and isGenerator, and then you have the possibility to answer that original question.
02:02
What is exactly introspection? So let's go back to dictionary and ask. Okay, introspection, two possible meaning, one in psychology and the other one in object-oriented programming. Okay, let's select the first one. And the first one is just ability of a program to examine at run time the type or property of an object.
02:25
The ability of a program to examine at run time the type or the properties of an object. Hey, that's cool. That's cool because in Python, everything is an object. A function is an object, a class is an object,
02:41
everything is an object. Isn't that cool? So you can, in fact, go and look everywhere in the different element of your code. In fact, I was already using introspection without, in fact, knowing it. You probably recognize that snippet of code.
03:01
It's the code which is used in order to separate the code which should be executed when you are running your module directly or when you are importing it. And the difference is made just by looking at that specific attribute called name, which is the name of the current module
03:21
or similarly, the current namespace. So by using that, I was, in fact, using introspection. But now let's look at what the standard library can offer and let's start by the built-in. So this is a list of functions which is part of the standard library which can be considered as being introspection.
03:43
You have things like ID to get back the exact identity of the object you are looking at. DR which is providing the list of attributes and methods, type which is providing the type, as attribute get attribute, relatively explicit, is subclass is instance callable,
04:02
providing you information about a specific characteristic of your object, as well as different functions for accessing variable depending on the scope. The standard library provide also a lot of attributes. There is a really huge list. There was a lot of effort done by the core developers in order to provide a uniform interface for the object.
04:24
And if you look at the documentation of the inspect library, you have the full list of all the attributes you can call. Typically, if you want to get the documentation of a module, a class, or a method, it's the same way. What does that mean in our case? So you have a function, stupid function,
04:42
nothing important there. What can you get from it? You can ask, of course, for the type and you get the type as a function. You can ask for the name, so you could ask what the interest of asking the name of something I am providing. Well, this can happen when you try to associate
05:02
a different name to the function. So it's important to be able to get back to the original name. You can, of course, get the documentation. You can get the module it is part from. You can ask all the methods and attributes that that specific function has. So this is a list.
05:21
But the Istana library provides even more than that. It provides a dedicated library called Inspect, whose objective is to inspect live objects. It's a pretty big library. It's around 60 different functions, but providing four main services. First one, type checking. Is it a module, is it a class, is it a method, et cetera?
05:41
Getting the source code, get the module, get the source file, et cetera, no problem to ask that. Inspecting classes and functions, what is the signature of a given function, what are the different parameters inside the signature, and examining the interpreter stack. We are not going to discuss about the last one. I refer you to the presentation of Alessandro Molina
06:03
from Monday on piton by code and introspection for that specific point. If you want to know more, there is, of course, as usual, a pretty good documentation of that model. Let's go back to our example. What does that mean we can do with that library?
06:24
We can get the exact parameter, inspect.signature. You provide the function, and you look at the attribute parameters, and then you have another dict which contains the different parameters part of the signature. Of course, you can ask for the complete source code of the function.
06:43
Okay, that's very nice. We've seen a lot of functions that we could use, but in practice, what could be the use of such a stuff? And so, it's time for a demo. Thanks. Now we know a little bit more about introspection. More importantly, we know how to do basic things.
07:02
We know which function to use, for example. But what about doing a bit more? Do we doing real things and having a bit more fun? For example, we could check that our docstrings in our code are always up to date. It can be very interesting because you communicate
07:22
to the outside world with your documentation, and the guy who will use your library will see your doc and will know what parameters to call your function with. So, it's important to have a documentation
07:41
that tell the exact number of parameters to give and what type of parameters to provide. As well, when we will be able to get valuable information from our docstring, maybe we could check that the value provided at call time are indeed of the right type
08:01
according to what's documented. Let's begin. First, we have to agree together about one syntax for docstring because they are actually different syntax that we find in open source libraries. Let's take this one.
08:20
I take, I write colon, param, then an optional type name, then the name of the parameter, then colon, and followed by an optional description. What is relevant for our use case here is to be able to retrieve the whole list of the parameters, names, and also if they have types documented or not.
08:45
Okay, here is how you just get the documentation string. Nothing new here. You just invoke the attribute doc. If you have the documentation string, here it's how it looks like.
09:02
You just have a string, the row, original string, and if you have no documentation on your function, you will have none. And I try to do that here. The attributes return none.
09:25
Okay, we have to extract the parameters. Here is a piece of code that is not very funny. We just have to parse the doc. I have nothing very interesting to learn to teach to you.
09:40
Just create your regex for your syntax if you don't have already a library to do this. Here we extract, I don't know if you see if I select, but here I'm looking for the keyword param in the documentation string. Here I'm getting the name of the parameter
10:00
and here is the optional type. Then I will just return all of the coppers containing the name and the type of each parameters. But just let's run it to see how it goes. I have, I can see here my four parameters as expected.
10:24
A, B, C, D. I can see that C is assigned to the type, to no type of parameters. Yeah, I get a none, which means there was no documented type. And for the other one, I get just what was written as type name in the doc and that's all.
10:42
Nothing very smart here. Yeah, okay. Now we have to do a little bit more. If we do validation afterwards, we need to load the type. We don't just want to have the name of the type.
11:01
Why? If you just wanted to use the built-in type, which gives you the type of an attribute, you could check the representation of the type, which is its name, fits the type name in the documentation. But if you want to have a number as input
11:21
and you write this in your documentation and you provide an integer, integer is a number, but the type name of an integer is not number. So I have to load the actual type that was written in the documentation and then check if the input provided fits.
11:42
And so it's instant for that, but you know that. I also need another thing. It's to import, being able to import a module because in my doc string, you saw that I was referring to a custom type called example type
12:01
that I was just defining just above in the current module. So I have to be able here to import the current module and use the function get that just as above to load example type from the current module instead of from built-in, for example, as it was above.
12:22
So here I can see that it has effectively loaded this type, example type. And here, a basic function I will have to use at some point is instance to take that value, for example, three is actually an int,
12:41
as I told us before. Let's put all these little pieces together. A function that will take the name of the type and optionally, not optionally, but name of a module and I will optionally use the module
13:01
to load the type from this module. But if there is dot in the name of the type, I don't have to because you know that if we have dot in the name of type, then it's a fully qualified name and I just don't need anything else to know from what module it has been,
13:23
I mean, in what module it has been defined. So just we see that I just use the built-in function I used before, get that, this instance and the import module function from import lib. And now I can see how it behaves.
13:43
I'm able to load a type from the module main, which is a current module. And here I try to load a type I didn't actually define in the module name, in the module main
14:02
and I can see that I have a proper error saying that this is at least not a valid type. If we just look at this module. Now let's put again what we just did together
14:20
and find a function that will very simply just extract what we are able to extract from the documentation, get the name and this time load the types from the type names we get from the docstring.
14:41
And here it can be used. It's very similar to what we saw before, but this time for each parameter name we have an actual type here, the type function and here still none for C which has no type constraints.
15:02
And you can see by the way that the built-in types and the custom types have been imported without any problem. So the goal for reminder was to be able to verify that the docstring is update according to the signature,
15:21
the actual signature of the function, what parameters it takes. So now the second point is to be able to get the signature. The good news is in Python 3 we have a function that does that very, very easily. It's called signature and it returns
15:40
an object of type signature. If you get a representation of the signature of the object, subject signature, you see between the parentheses the parameters that your function takes. And what is useful among others
16:03
is a parameter called parameters inside this signature object which is interrable. Actually I think it's a read-only map, I mean a dict, read-only dict. And you can, for example, get a number
16:21
of parameters very intuitively. You could also get the default value of a specific parameter by its name, so it's very easy as well. You just give the name and use on the parameter object the attribute default.
16:41
And here I'm expecting to see 42 because it was a default value I gave to parameter D in my signature. And now it's a big piece of code but it's really not to be afraid of. Just read the doc and it'll be enough.
17:01
Basically we just want to check that if we provide a default value in the signature, it has the right type if the type is provided in the docstring. So we are just checking that the docstring is consistent when it comes to the types and the default values in the signature.
17:21
It's quite straightforward actually. We just use what we did before, getting the parameters names and types, getting the signature and comparing them. Here I ran what I just did on my function, my testing function, and I'm expecting to not raise,
17:42
I mean this function is supposed to raise an hour if documentation is inconsistent. So here I can see it's consistent. For the example, of course I provide another function that is badly documented.
18:00
You can see that it takes only three parameters and I just copy paste in the docstring of the other function which had four parameters. So obviously the last line here which tells that it takes parameter D is not relevant and here when I run the function,
18:22
check the connected parameters and that function, it tells me that it's wrong because signature takes just A, B, C. The function just takes A, B, C and documentation tells about A, B, C and D present B. Okay, so the good news is we just achieved our first goal.
18:42
I don't think it was too hard. So now I can verify that wherever in my code I want to check that, I can check that docstring is up to date. Side note, docstring is a doc that is assigned
19:03
to a function but also to a class or method or other things actually. So you can do that everywhere if you want. So second goal, yeah I'm sorry. I show that the second point is yet to be done
19:21
and now it's a bit more at runtime. It's more an actual introspection because I really need now to know the actual state of my input at runtime to be able to check it against the expected types according to the documentation.
19:41
So we see here I'm invoking the function called signature provided by the module inspect so nothing new here. But now I'm calling another method on that object which is called bind. Very simply bind, you just give it all the arguments you would call your function with
20:02
when you want to just call it. And bind will tell you that this value would be bound to this parameter and so on. Just look at the result. Here it tells that other func as I provided as first parameter
20:23
will be bound to the parameter A because A is a first parameter and then I explicitly invoked bind with the parameter C which was actually the third parameter and you can see that there is no mistake. It correctly bound the value two to the parameter C
20:42
and the instance of example type to the parameter B. So it's no magic but it's very easy to have that and not have to implement it yourself. Now that we have that again and again we just create a function that invoke what we did before.
21:04
I'm able to get the names and types from my documentation string. I'm able to get the signature and now I'm able at runtime to bind the values that are given to the function to the parameters
21:21
and the parameters to the documented types. So with this again it's not very big deal just it's a decorator. If you don't know what the decorator is, very simply here my use case of a decorator is I want to add a check step before doing the rest of my function behavior.
21:45
I don't want to touch what will do my function but I want to decorate it in order for it to just do a call, to do a check on the input before doing the rest. So the usage is that on the first line
22:01
I just add the decorator to any function that is defined just below. My function can do whatever I don't care. Here it does nothing and I just add the decorator to tell at least my input should be validated first.
22:20
And that is what I will do here. I'm calling the function that I was decorating just before the function still has the same name and when I call it despite the fact that this function was not doing anything I have an error because the decorator added a step to that which was a validation step on the input
22:43
and I also have a valuable information to work with because I know that my mistake when calling this function was to provide an integer as parameter on B which was expecting actually an instance of example type and that's it.
23:04
We achieved our two goals and so well done. Okay, then we promised to try to answer the question when should we use introspection?
23:23
It's a tough question. The general principle should be relatively clear. You should be only using it when you don't know the information at compile time. Well, that's the principle. No in practice, we don't pretend that we have the truth but we have tried to identify some good candidates
23:44
for places where introspection could be used. The first one is about exploration, learning and debugging. It's the idea that introspection allow you to have an in-depth view of what is happening with your object and that's very nice if you want to learn typically the Python internals
24:03
or to better understand what is happening inside your code. The second one is related to IO. That's typically the case when you receive from outside an unknown object and that you want to adapt the behavior of your code depending on the specificity
24:20
of the object. On the output side, it's more about when typically you want to serialize any objects in order to log it or store it for future use. Then in that case, you have to adapt your behavior also depending on the specific type of the object. The next one is when you are missing so much
24:40
the polymorphism of other languages or typed languages. So you would like to have a single function which is processing whatever type of object that you receive and in that case, of course, you will have to adapt what is happening inside your code
25:00
to the specific type and specificities of your object. The next one is if you are missing also interface and typically in an architecture of plugins, then you are supposed to use classes or modules that you don't know and where you would be interested in checking that well, indeed, I have all the methods
25:22
I need or I am expecting. And the last one is close to the first part of the exercise of UG is about metaprogramming. It's all the things around your code like I would like to check some convention, I would like to check some coherence within my code.
25:41
I want to be able to do autocompletion when I am at command line. I would like that type of things where you can also use introspection. Just three words of warning. The first one is about performance. Of course, when you just access an attribute of a class,
26:02
it has almost no cost, so it's fine. But you could have some cases where introspection is more expensive. And also, as it's a good practice in programming, don't check multiple times the same things through your code. So check it once when you receive the object from outside.
26:23
As far as portability, depending on the Python implementation, some objects could be not inspectable. So that's something to take into account if you want your code to be working on the different Python implementations. And finally, maintainability. Depending on the ID you are using,
26:40
it could be, you could receive more or less help when trying to do refactoring. Because, of course, everything is just a representation like a string of classes when you are calling them. That's all for us. We would like to thank you for attending this session. And, yeah, by the way, Kritio is hiring
27:01
if you are interested in working on really big data. Thank you. Okay, we have about four minutes for questions if there are some questions.
27:24
So I have one question. So when you're doing introspection for some of the examples you have, it means you're executing the code when compared with using the AST module or some other way of evaluating static analysis. I was just wondering if you combined the two approaches
27:40
or, I don't know, for example, validating the doc streams, you could do either way. Yes, true. Actually, we created a slide for showing how AST works and how it's easy to create a visitor. And first it didn't fit in the 30 minutes.
28:02
And it was also very, it's another subject that I really, really, if you do AST you're not introspecting because you are analyzing the various structures, the static structure of your program. And in fact, when you visit the AST in Python you have to load a source file.
28:20
You're not inspecting the actual code that is running. You really are loading something else even if it's the current file you are evaluating. But you are loading something and creating the AST and visiting it. And you don't have the memory state of the current execution. You don't have the state of the data that is going through your functions.
28:43
So it's not introspection actually. But when it comes to doc string, yes. The first example was to check something that's truly aesthetic. And it was just to show here that a way to check it was to use introspection in this very case because it's very easy with Python
29:02
to just load your module. And because you just loaded it you have something. You are rendering. You just have a state and you can see the state of a function. So it's just a way to do this. But the second example, the one where you check the inputs
29:20
is the true, I mean the pure theoretical example of introspection of course.