Using and abusing Python’s double-underscore methods and attributes
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 94 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21104 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201694 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
Electronic data interchangeAttribute grammarNormal (geometry)ArmAriana TVLipschitz-StetigkeitObject-oriented programmingConstructor (object-oriented programming)Representation (politics)Time zoneCodeOperator (mathematics)String (computer science)Object (grammar)Maxima and minimaFile formatComputer fileModule (mathematics)Line (geometry)System callType theoryScalable Coherent InterfaceSet (mathematics)Key (cryptography)Data typeInformation managementCloningSocial classLarge eddy simulationRandom numberSoftware testingProgrammschleifeElement (mathematics)Range (statistics)OvalLie groupGrand Unified TheoryMIDIIntegerFunction (mathematics)Online helpArithmetic meanMenu (computing)Computer-assisted translationSummierbarkeitRule of inferenceException handlingCASE <Informatik>Block (periodic table)Ewe languageMetropolitan area networkSpecial unitary groupSequenceNumberParameter (computer programming)TorusPoint (geometry)Price indexLengthProgramming languageOverhead (computing)Reading (process)Boolean algebraOperations researchFingerprintVirtual machineJava appletLevel (video gaming)CoprocessorMereologyExecution unitFirst-order logicBinary codeShift operatorInfinityRing (mathematics)Streaming mediaDemo (music)BuildingPerturbation theoryMoving averageInformationLetterpress printingContent (media)Attribute grammarType theoryIntegerLetterpress printingOperator (mathematics)Element (mathematics)Data managementQuicksortRegular graphFunctional (mathematics)Interior (topology)Multiplication signLatent heatKey (cryptography)Range (statistics)NumberPoint (geometry)Functional (mathematics)ProgrammschleifeSoftware testingRepresentation (politics)Object-oriented programmingSequenceImplementationSocial classInheritance (object-oriented programming)Information overloadStatement (computer science)Address spaceElectronic mailing listLengthCodePearson product-moment correlation coefficientComputer-assisted translationBitParameter (computer programming)Order (biology)Hash functionRight angleContext awarenessSystem callEquivalence relationData dictionarySet (mathematics)Constructor (object-oriented programming)Module (mathematics)InformationString (computer science)Ferry CorstenCASE <Informatik>Loop (music)Subject indexingPresentation of a groupShared memoryDirectory servicePoisson-KlammerSampling (statistics)Open setTerm (mathematics)Online helpWritingSlide ruleWordComputer virusComputer fileBlock (periodic table)Random number generationData modelDefault (computer science)Cache (computing)Uniqueness quantificationUniverse (mathematics)Instance (computer science)Video game1 (number)Shape (magazine)Object (grammar)Different (Kate Ryan album)SpacetimeRevision controlData conversionAreaSound effectNumeral (linguistics)ArmEndliche ModelltheorieGenderRankingFrequencyDirection (geometry)Interface (computing)Streaming mediaNormal (geometry)Control flowBuildingMereologyPrice indexBit ratePositional notationLevel (video gaming)Rule of inferenceThresholding (image processing)Pattern languageWikiSource codeSoftware repositoryIntegrated development environmentText editorRandomizationHypermediaLine (geometry)Network topologyVector spaceWeb crawlerDisk read-and-write headSource codeJSON
Transcript: English(auto-generated)
00:02
Hi. I'm Anjana Vakil. Hello. Hope everybody's enjoying the week so far. I don't know about you guys, but I've seen a lot of slides this week, so today I figured I'd do something a little bit more experimental. No slides. I'm just going to show you some code. Some really, really silly little code examples.
00:24
Please do not take them seriously. Don't write code like what I'm going to show you today. My hope is that they'll illustrate some of the fun special dunder or double underscore methods here, which are also called special methods, magic methods, but I like the term dunder the best, so I'm going to call them dunders.
00:43
As I mentioned, I put up all these examples in a little repo, vakil-a-slash-dunders. What I'm hoping is that at the end of the talk, to have a couple extra minutes to discuss with you guys, I'd like to do this more interactively if people have other dunder tips and tricks that I am not able to cover,
01:04
because there's too many wonderful things to fit into 20 minutes. I would love to have people discuss them afterwards, and possibly even after the talk, if you can contribute to this repo, add stuff to the wiki, open issues, start discussions, and if you have fun little code samples that you want to add, there's a directory in here, sharing is caring,
01:24
where you can put in whatever you want, and file a PR and I'll put it in there. So, that's what I'd like to do today, because dunder methods are super fun, if you ask me. So, what are the dunders? Well, they're these special methods and attributes surrounded by double underscores,
01:43
which is why we call them dunders, and some of them are our best friends, right? So, everybody probably uses dunder init all the time, it's our basic constructor method. So, here I have got a custom class, I'm going to call it a stringy int, and it's going to be a weird kind of number.
02:01
I'm constructing it with this dunder init, I'm giving it an attribute called value, and then another dunder everybody probably is already super comfortable with is dunder str, maybe there's a better way to pronounce that, I don't know, where we can have whatever kind of string representation we want. Usually it's just like the value of the object, but this one's going to be more exciting.
02:21
It's going to have zelmagad. So, this is why you should never live code in presentations. So, if we run this little module, we get to use another dunder that's super fun, which is this beloved if dunder name equals dunder main block,
02:45
which, as you all probably know, is run only when you run the python module itself, and not when you import it into something else. So, that's cool. Let's run a little interpreter, and I've set up some stringy ints here, so we've got one is a stringy int object, and two and three.
03:05
Awesome. And if I print them, since I've got my dunder str, zelmagad, I have fun string representations. And so, another dunder that we all probably know and love is what is giving me this weird python-y, object-y looking string representation here,
03:25
and that's dunder repper, which, if I uncomment this here, is supposed to be another type of string representation that's more the code object itself. It's supposed to be more unique to the object. In this case, I'm just going to have it be more boring
03:40
and just print the string of the value. And so, we can see that when we quit this and do it again now, now, when it evaluates one, it prints out just the value, so I don't have to look at all this gobbledygook addresses and whatever. Okay, so far, so boring.
04:00
We all probably know and love these dunders. What about other fun stuff? So, when I have regular integers, I can add them or multiply them. That's because the built-in int object has these fun operator dunders, like dunder add and dunder mole, which are used by the plus and the star asterisk operators
04:24
to perform these mathematical operations. And I can overload them in my own classes by just implementing these methods. Super cool. So, let's see here. What happens now when I have my one and my two, and usually, if I add two integers together,
04:43
I get, you know, reasonable things, like three. But if I add my stringy ints together, ah, it's giving me an integer that's just mushing them together like strings. This is a completely useless class. You probably don't ever need to implement anything like this. But the point is, you can, because Python magic.
05:00
So, what if I want to add a stringy int to a regular int? Hmm, doesn't work. The operand type is not supported. That's because it's looking for a dunder add function on the integer to the left of the plus operator that works with a stringy int object, and it doesn't find one. So, if it tries to find the dunder add method on the left-hand side object,
05:24
but doesn't find something that works for the right-hand side object, Python will also try looking for these special R methods, which are like the sort of opposite, implemented for the right-hand side object, like, for example, dunder rad,
05:41
which is possibly the best-named method in the world. So, this one will be called on the right-hand side object as sort of a fallback if it can't use the regular dunder add on the left-hand side object. So now, we can see that if we try one plus one,
06:01
aha, it works now. We didn't change anything about the built-in int, but we used this dunder rad, and it is super rad. I'm sorry, I can't help myself, you guys. Okay, one other operator that is a little bit different
06:21
is this equal equals, right? We all know and love it. One is one, and the stringy int one is itself, but it would be cool if we knew that those two things were similar somehow. So, I can do that by implementing dunder eek, and there's equivalents for all the, you know, less than, greater than, et cetera, et cetera.
06:41
In this case, I'm gonna try to make it work for anything that I can intify. So, again, don't write code like this. It's just an example. So, now, if I try, okay, one is still itself, and my stringy one is still itself, and now, hopefully, oops, wrong direction,
07:01
aha, now it knows how to compare these two different types. So, the dunder eek function also is a bit special because it's also used for, for example, making objects hashable so that it can be used, like, as keys in a dictionary. So, if I have a dictionary d right now,
07:23
if I want to make an integer a key, that's no problem. We do that all the time. But if I want to try using one of my stringy ints, it says, oh, no, it's unhashable. Well, we can fix that, of course, with a dunder called dunder hash.
07:41
Oops. So, if I implement this dunder hash function, what I want to do is return a unique hash value for whatever this object is. In this case, I'm going to do kind of a really silly one, which is just returning the integer value itself. Ideally, you'd have something better than this.
08:03
But the important thing is that you don't want to implement this kind of thing on an object that shouldn't be hashable, like a mutable object. You don't want that to be a key in a dictionary. But these are not mutable, so I can use it. So, let's see now. If I have my dictionary d, okay, I can still use my regular integer keys.
08:24
Let's try now. Oh, I overrode it. Sorry. Aha! Okay, so now I've got... It's not complaining about the hashability. And because of my dunder repper function, it's difficult to see this. But if I do...
08:43
For key in d keys... It really is harder to type when you're up here, right? Uh... Let's print the type of each key. Ah, okay, I see I've got one int and one stringy int, so it's working. So we can now use this new custom type
09:00
as a hashable type for dicks. All right. One other thing I wanted to talk to you about with these stringy ints is a fun little dunder that's an attribute, actually, not a method, which is dunder slots. So, dunder slots is a bit different. When I have an object like a custom object, usually I have a dictionary, dunder dict,
09:25
that stores all of the attributes for that object. So that's what allows me to do, like, one dot value and get something out. And if I assign something new to the object,
09:42
if I look at the dict, I see, aha, it got added to this dictionary. It's just a regular dictionary. You can mess with it however you want. But the thing is that dictionaries take up space. And so whenever you create a new object, Python gives you this dunder dict for all of the object's attributes.
10:00
And it might be the case that for an object like stringy ints, you know that it's never gonna have more attributes than value. It's only gonna have that one. Or maybe you just have a small set of attributes. And if I'm gonna be creating, like, millions and millions of stringy ints, creating all those dunder dict dictionaries could take up more space than I want to use. It takes up also a little bit of time.
10:21
So what I can do is use this declare this dunder slots and name out all of the attributes that I want on my object. In this case, it's just value. And what that does is prevent the dictionary from,
10:41
the dunder dict dictionary from being created. So if this is right here, if I try now accessing the dict directly, ah, it doesn't have one. Does it still have its value? It does indeed. Can I add a new attribute? Uh, oops.
11:04
Nope, I can't add extra attributes. So it's basically constraining the size and shape of this object in a way that if you're creating gajillions of objects, that might, that efficiency might actually come in handy. So I thought dunder slots was pretty cool when I heard about it earlier this year.
11:21
All right, so, let's see. Time flies when you're having fun with dunders. There are a lot of other fun dunders that we can use to make container objects, for example. We already saw how we can make objects that are kind of simpler, you know, like numbers, numeric types.
11:42
But what if we want to make things that have contents? So, for example, let's say I want to make a list, but I find lists really boring, because, you know, when you add things to them, you append things, you know they're gonna show up at the end, and when you index things, you know that you're gonna find the right object for the right index. What if we want a crazy list,
12:01
where there's just an element of randomness? So, in this case, I've made a silly little object called a crazy list, which, um, I've got my dunder init, I've got my dunder repper, I'm adding a little append method, just because all good lists need one, but instead of, you know, appending things to the end of this self.values list, I'm just gonna insert them at a random place,
12:23
because why not? And so, again, I'm using my dunder name thing in here to, uh, run some code right when I run the module. Let's see here. All right, so, I've got an L object, and, uh, it's got some elements in it.
12:42
I want to find out how long it is. Oh, no, it has no length. That's not good. What should I do? Probably use a dunder. In this case, the one I want is dunder len. Uh, this is what's called by the built-in len method. So a lot of the built-in methods that Python, uh, that we were used to using in our super-beautiful Pythonic code,
13:02
they depend on these dunders, and one really simple one is dunder len. Um, so in this case, instead of telling you actually how long the list is, I'm just gonna sort of give you a random number that's somewhere in the vicinity of its length. Super useful, right? I hope you guys noticed the word abusing in the title of this talk.
13:22
So, uh, now, if I try to len my object, aha, it gives me a completely wrong number, but at least it gives me a number. Sweet. Okay, um, so there's another... Let's see, I'm gonna try and skip ahead here. Uh, what if I want to do
13:40
a for item in my list? Uh, you know, print the item. Oh, no, it's not iterable. That's not good. How do we fix it? We fix it with a dunder. Yay! In this case, um, dunder iter. I skipped dunder bool here. That's used for, like, if L. You guys can get the idea.
14:01
But dunder iter is probably, uh, more important if you're trying to make some kind of sequence object or something that you should be able to use in for loops and that sort of thing. You're gonna want to implement this. And, um, how it works exactly is a little bit more complicated than some of the other dunders, but in this case, I'll just point you to the documentation because we're running out of time. And, um, in this case, I'm just gonna kind of yield
14:23
a random element from the list in the completely wrong, uh, range of-of-of the number of elements that may or may not be in my list. So, really useful dunder iter function here, but hopefully you get the idea. So now, uh,
14:42
let's try to do our for i in L print i. Ah, okay. So I have a little thing in here that shows when dunder iter is getting called. It got called for that for loop, and it's doing something really useless, which is, uh, not only is it printing the wrong number of items,
15:00
but it's also just printing question marks sometimes because mystery. Um, but the point is that we can use this now in for loops, so if I wanted to define a really super useful dunder string method that, uh, uses a for loop to print out all the things in the-in the list, I could. Okay, um, what about, so usually in a list, like L, I want to be able to get,
15:22
you know, a certain element by using, uh, this bracket notation, but it doesn't support it. If only it did. If only we had implemented dunder get item, which lets us, um, pass an index or a key to these brackets for our object. Um, and so,
15:40
depending on whether you want your object to be, uh, indexable using integer indices, or you want it to be, like, a keyed item, like a dictionary, you can define this method to, let's say, look for a certain type, like only integers, or to-to just handle anything, which is what I'm doing here, is I'm not even caring about what the key is or the-the index is,
16:01
I'm just gonna accept it and just give you a-a random, uh, item from the list, no matter what you ask me for. So, I have my L. If I want, uh, a certain index, yeah, sure, whatever. Uh, yeah, oh, it's totally working.
16:20
Great. And what if I want to try to access it as if it were a dictionary? Sure. No problem. Yeah, we'll just give you random things from the list. The point is, if you were to make an actually useful dunder get item method, then, boom, bada-bing, you've got a-a dictionary or an indexable sequence, whatever you want.
16:42
Okay, and there's also a dunder set item for the equivalent, which, uh, you can imagine for setting a certain item at a certain index or, um, setting the value of a key. Okay, um, one other dunder that I want to talk about with containers is dunder contains. So, um,
17:00
if you have, uh, an object where you're gonna be wanting to test for membership, like, you know, if X is in L, um, if you don't have dunder contains implemented, what you'll see is if I do, uh, let's say, uh, I want to see
17:20
if 2 is in L. If you don't have dunder contains implemented, it's actually gonna use the dunder iter and go through everything in the list and see if it finds, um, something with the... something in there that's the thing you're looking for. Uh, that can be a bit slow, depending on various features, like, for example, how far close to the beginning of the list, uh, the item you're looking for is.
17:42
So if you implement dunder contains, this can be, doesn't need to be, but can be, um, a faster way of testing for that membership. So depending on what your use case is, if you need something that you can really quickly decide whether, uh, something is a member of, like a set, let's say,
18:00
um, dunder contains can be a good idea. So now we see that if I ask, okay, is 3 in L? See how my dunder iter didn't get called? I had that little print statement there. It's using dunder contains first. Okay, so, a crazy list that's completely useless, but nevertheless showcases the magical container dunders.
18:21
One last thing I want to show you, and this is probably my favorite, are some fun function dunders. So, um, I have a little function here called add. It's super boring. It just adds two things, spam and eggs, whatever. Um, but because Python is magical,
18:41
since I have this, uh, docstring in here, I can, uh, if I ask Python to help me out with add, it tells me, okay, cool, the contents of that docstring and information about the, uh,
19:00
the function itself. Sweet. And, uh, when I add two things, you know, it does what the function says to do. But, what if that's not cool enough for me, and I want to hack my little function on the fly? So this is probably something you should never ever do. But if you were a cat, you wouldn't care about adding two numbers together. You would want your human to give you more
19:22
tuna. So, what I've got here is a little function which makes use of some fun function object dunders. Dunder doc, which contains this docstring. I'm gonna change it to something more cat specific. And dunder code, which is actually the code object, the content, the functionality of your function.
19:40
You can actually mess with that. You can replace it to be, for example, the contents of another function called more tuna, which instructs the human to give more tuna. So now, if I, um, if I have my regular add function, uh,
20:00
okay, add's still good. Add, it still works. Great. If I now catify it, and I try to add two things together, it actually changed the functionality that's attached to this add name. And similarly, the docstring
20:20
is different and useless. Um, this is something that you probably don't really ever want to do. But the important thing is that you should know that you can. And so if you ever see this kind of messing with going on, be really careful. Um, all right, last thing. Hopefully I have time for one more dunder. Uh, the, um, the with
20:41
keyword works with two special dunders called dunder enter and dunder exit. And this is what allows us to define a context manager. So basically, what happens is this dunder enter method is called whenever we enter a with block. And it can set something up for us. Like,
21:01
for example, if you use, like, with open file, right, it'll do some things like read in the file object, and then when you exit the with block, it calls this dunder exit method, which can do something useful like closing the file object. In this case, we're going to have it do something less useful, which is it's going to catify a function on the enter, the dunder enter, and it's going to
21:21
uncatify the function by replacing the original, uh, dunder code and dunder doc with the boring human code and the boring human doc on the dunder exit. So, um, what we should see here is that now, if I have my
21:42
regular add function, it works. But if I do with cats in charge of add, and then I try to add two and three, um, and let's do something else, too. Let's, uh, let's just call the help, and let's add, I don't know, four and five and six, sure.
22:03
So, what I'm doing on the dunder enter is catifying this add function, so we should see that any add calls I make in the middle of this block are the catified version, but when I exit it, I'm sending it back, so any add calls I make after this block should be normal again.
22:20
And just because maybe you guys want something that's actually a tiny bit useful out of this doc, um, I'm also putting in a timer here, so I'm timing when the cat rain begins, when I go into the dunder enter method, and then I'm, I'm logging the time of when the cat rain ends, when I leave in the dunder exit method, and I'm printing out how
22:41
long the cat's ruled for. So this is an example of something you might actually want to do, is write your own timer, for example, to time an arbitrary block of code. Okay, so let's try it out. Ah, okay, so it called help, and it's meowed, and we saw
23:01
that it called add twice, and I'm actually using the numbers as the arguments to determine the r's on purr here, and then it told me how long it took, uh, how long the cats were in charge. So, just a little example of some things that you can do with these context managers, depending on what you're trying to do with your code, it can be a really useful pair of dunders.
23:23
Alright, so, um, that's all I wanted to tell you about the dunders that I find cool. I would love to hear now if people have other ideas for dunders you find cool. And just before we kind of open it up to everybody, um, I just wanted to point out that the, um, the documentation for the Python data model
23:42
has information about all kinds of dunders that you could possibly want to know. Um, so, if you're curious about any of these, yeah, check that out. Alright, what do you guys think? Dunders to share? Thank you for your talk.
24:13
Um, if you overwrite dunder all in a module, then you overwrite what can be imported. So you can just say,
24:20
ah, all is just these three functions. Okay, uh, dunder all is not in this thing, but that sounds really awesome. So if you, if you underwrite dunder all, you can overwrite what's been imported in the module. What can be imported from the module, that's cool. So is that something I should be able to access right here?
24:43
Like, is it a... Try it. No, okay, uh, is it something that would be in the vars, like in the... I've only ever written it. Okay, no. Where would it be? Anybody? If you put it in your
25:00
module. Just anywhere? Should be a list? Okay, so if I want to say that only, we can only import catify, like, would I do it like that, or would I use the object itself? Okay.
25:22
Does the order matter? Does it need, does catify need to come first? Okay, sweet. Let's see. So if I, no wait, sorry. This is what would happen if I import from that. Okay, so let's try import
25:42
from catification import add shouldn't work, right? Oh, okay. Alright, alright, alright. So if I do from catification
26:01
import star, then add should be aha. But catify is a thing. Cool, thanks. Anybody else want to share stuff? Yeah, and it'd be cool if
26:20
whoever just mentioned that could like put a note in the wiki or add a little example. That'd be awesome. Sweet. Anybody else want to share something? Well, if you're looking for donors that you can do
26:42
crazy things with, the ones Aren't we all? The ones you look for is donor new. Donor new? Yes. Which, if you want to do crazy things normally you would return your object instance, but you can return whatever you want. So for instance, if you return the integer 42, instantiating
27:01
your class will get you the integer 42. Sorry, if I run the... If you in your donor new in a class, if you put return 42 at the end, instantiating that class will get you the integer 42 and not an instance of that class. Okay, so if I, in my stringy int,
27:21
if instead of an integer I want, whatever integer I wanted, I just want always the answer to the life of the universe and everything, I could do donor new self whatever, is it called with a value? Not self. You get the class. Sorry, no, yeah, no. CLS? Yes.
27:41
And you get the value also. So it'd be like this? Yes. And I could just return 42? Exactly. And so now, if I run this, I should have I had like one,
28:01
should be 42. Exactly. Awesome. Very cool. Thank you. And if you do type on one, it'll say it's an int. So tricky. Thank you. Yeah, if you could put that in the repo, that'd be awesome. Anybody else?
28:21
So something I actually find really useful is if you still live in Python 2 and you don't have the LRU cache, yeah, I live in Python 2, sorry about it, and you can use the underscore, underscore, sorry, the donor missing, you use the donor missing, you get the cache out of a dictionary in five lines of code.
28:40
Sorry, could you say that again, maybe more slowly? So if you inherit from dictionary, and you implement donor missing, whenever an item is not found, you can specify a function. And that's a cool way to implement a cache. Awesome. So, like, I imagine that would also be useful for things if you want like, default values, like a
29:00
default dict or something, would that work? Very cool. Donor missing. Yeah, if you could add that too, that'd be awesome. You can implement method call and donor call to call object as function. Right, okay. So if I have like, an integer shouldn't
29:21
usually be callable. You need to delete method new. Right. Let's replace it. Okay. And it takes what self, right? And any arguments, can we like, just take however many we want? Will that work? And I don't know, I'm just gonna print
29:42
like, yay, dunders! And return I don't know, 42. Okay, so now if I have one, okay, it's my number, and it's still hopefully a stringy int.
30:01
Cool. But I should be able to call it. Yay, dunders! Sweet, donor call. And you can implement method donor get, donor set, donor do and donor delete. Right, right. So yeah, we saw like, some of the
30:21
getting and setting, but there's a bunch of other dunders, like delete and whatever, which you can do. Special cleanup code, or whatnot that you need. Very cool stuff. Thank you. I don't know how we are for time. Are we out? Alright, well thanks everybody for sharing.
30:47
And yeah, if anybody wants to contribute to my little dunders repo, I'm hoping that it can be like a conversation starter. So go for it. Thank you.