A Taxonomy of Decorators: A-E
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44986 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201819 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
NP-hardBoom (sailing)ImplementationSocial classFunction (mathematics)CodeSheaf (mathematics)DebuggerHelixParameter (computer programming)Algebraic closureUniformer RaumMetropolitan area networkElectronic signatureSocial classFunctional (mathematics)Algebraic closureString (computer science)Letterpress printingException handlingSoftware testingSinc functionSoftware developerLine (geometry)MathematicsCASE <Informatik>Point (geometry)Multiplication signParameter (computer programming)Web pageSign (mathematics)CodeRevision controlHypermediaMathematical analysisResultantImage warpingComplex (psychology)Type theoryDebuggerVideo gameFuzzy logicBitUniform resource locatorElectronic mailing listQuicksortFluid staticsSet (mathematics)Constructor (object-oriented programming)Reading (process)Category of beingPattern languageSheaf (mathematics)Numerical taxonomyOrder (biology)MultiplicationSoftware design patternWrapper (data mining)System callRight angleComputer animation
09:45
ImplementationFunction (mathematics)Communications protocolLibrary (computing)Real numberProgramming languageHybrid computerNormal (geometry)Formal grammarKontrollflussControl flowRandom numberPlug-in (computing)Image registrationNP-hardCodeRule of inferenceSource codeSample (statistics)Boom (sailing)Normal (geometry)Fluid staticsCategory of beingRule of inferenceImplementationParameter (computer programming)InfinityCodeProgramming languageNumber1 (number)Mobile appRun time (program lifecycle phase)Functional (mathematics)Electronic mailing listSource codeCommunications protocolError messageTypprüfungWeb 2.0Pattern languageQuicksortExterior algebraObject (grammar)Instance (computer science)Control flowSocial classAttribute grammarBytecodeAsynchronous Transfer ModeMixed realityFunctional programmingKeyboard shortcutLine (geometry)Query languageCASE <Informatik>Multiplication signLevel (video gaming)NamespaceSpacetimeInheritance (object-oriented programming)Library (computing)Plug-in (computing)Event horizonSoftware frameworkContext awarenessSampling (statistics)SequelEndliche ModelltheorieGreatest elementSoftware testingDataflowAbstract syntax treeComputer animation
19:29
Parameter (computer programming)Keyboard shortcutInternetworkingNumerical taxonomyImplementationPlug-in (computing)Image registrationNP-hardAlgebraic closureFunction (mathematics)Control flowParameter (computer programming)Water vaporCache (computing)Structural loadNumerical taxonomyLibrary (computing)BytecodeRight angleMultiplication signUnit testingMereologyStatement (computer science)Term (mathematics)Level (video gaming)WritingDebuggerNamespaceCommunications protocolExecution unitElectronic mailing listCASE <Informatik>Standard deviationMixed realityType theoryFunctional (mathematics)Slide ruleImplementationBlogContext awarenessAsynchronous Transfer ModeData managementSocial classTwitterKey (cryptography)Abstract syntax treeQuicksortDegree (graph theory)Module (mathematics)Hybrid computerPoisson-KlammerCodeInformationCategory of beingGreatest elementCodeMusical ensembleSound effectComputer animation
29:14
Computer animation
Transcript: English(auto-generated)
00:05
Thank you. So a taxonomy of decorators. First, let me introduce myself. I'm Andy Fundinger. I've been a Python developer since Python version 2.4.
00:21
At the time all the books came out, they said Python 2.2. And then they had a little mark that said, newly revised for Python 2.3. And there were a couple pages in the back that said, once Python 2.4 comes out, here's the new stuff you'll get to use. So that's where I started with Python,
00:41
lo those many years ago. You'll see some notes on older things in these notes. Everything we're talking about today works in Python 3 or such older versions as you might happen to still be running. What is Bloomberg? Bloomberg is first and foremost a technology company.
01:05
We sell financial data and analysis. We're also a media company. We have major locations in New York, London, San Francisco, and all over the world. Now, what is this talk?
01:21
This talk is about creating a taxonomy of decorators. We use decorators, but describing them gets a little bit fuzzy. We don't really have established categories, so I thought I'd try my hand at throwing out some categories and saying, maybe these are good categories to talk about decorators in.
01:44
I'm hoping with a common set of terminology, we can discuss these more easily. Sort of like with design patterns, we can say that's sort of like a singleton pattern. Even if we're not implementing a singleton pattern, we don't want to implement a singleton pattern.
02:00
We're not using it, but it at least is a data point. This talk is intended for intermediate developers who can write decorators, but maybe aren't sure when or why, and architects who may need to work across teams to implement and manage their design.
02:20
We will be going over the syntax of decorators, but I'm not attempting to teach how to write decorators in this talk. I'm going over it just so that if you need a quick refresher, you've had a quick refresh. That said, we are going to go over without the at sign, function decorators, class decorators, decorators
02:42
with arguments, and decorators with it written as classes relatively quickly. If you've taken other talks on decorators, you'll know that could be a talk right there. Remembering, of course, anything in a decorator,
03:00
you could just replace it with the code that's in the decorator and put that code in the function. There's no special magic for decorators. What they're generally doing is inserting a section of code of arbitrary complexity using a single line, and generally causing both the debugger and our eyes when we read the code
03:24
to skip over that line for better or for worse, which is a lot of what I'll list as the problem with any particular kind of decorator. So, basics and tasks, going back to our first decorators,
03:40
static method and class method, these came back before decorators. In Python 2.2, we got static method, and we wrote them like this. When we wanted a static method, we took a method we already had, we ran the static method function on it, we assigned it back to its own name, and it worked.
04:03
In Python 2.4, we got some special syntax to do this, and that worked too. We could also write our own, so we could write our own, say, something like this,
04:21
and this just creates a nonsense function that does nothing and returns that. And of course, by the time we got to Python 2.6, we could decorate our classes. We can go ahead and run our nullit decorator on a class.
04:43
If we do that, we run it. This will actually work. If you run it, now your class constructor returns true. That's not what you want it to do, but it does work syntactically.
05:02
Then we can look at decorators with arguments. Now, the trick for a decorator with argument is that it's not doing anything special. They're simply calling the function that you have written with the arguments that you have given. That function is returning a decorator. That decorator is then being used
05:22
to decorate the function that follows it. So here we have a multi-arg, which is a quote-unquote decorator with arguments. It takes an argument mult. It returns an actual decorator, deco, that takes a function, which returns an actual wrapper,
05:45
which is the wrapped function. If we go ahead and see that in use, at mult arg three, we decorate print x. Print x does nothing but print x. If we print x of one, well, then it kind of makes sense
06:01
because it prints three, which is three times one, and all that math makes sense. If we do print x of hello, well, three times hello is hello, hello, hello, because we can multiply strings. Whether we want to or not, we can.
06:25
And off we go. The thing to notice here is that we actually have two closures going on. One is for the argument mult, and another one is the function has its own closure, func.
06:44
Okay, now for reasons that I've never been able to understand really, we normally write our decorators as closures. It may be a couple less lines of code and a couple more lines of unclarity.
07:01
There's definitely no particular benefit to this. You can definitely write your decorator as a class, and if you're writing a more complex decorator, I will suggest you do so. They look like this. Init takes that function and really ought
07:21
to do something with it, and then you'll need a callable class because you have to return a callable. So that should do something with that function. So in this case, we store it away. When it's called, we're just gonna print whatever arguments were passed.
07:41
It's a nice little tracing thing, and this does what it says on the tin. We decorated rand with trace it. When we run it, we print the arguments. We run the result, and life moves along,
08:00
right along to our actual taxonomy. So the first item in our taxonomy is our argument-changing decorators. The argument-changing decorators are one of the most common types of decorators, and probably the most common decorator
08:20
that you will use when you go to write an example of what a decorator does. I think half the examples I just used were argument-changing decorators. Argument-changing decorators add or remove or change an argument when the function is called. They're messing with the arguments.
08:44
They're changing the type as the arguments go through. Or they're changing the return value because that's really functionally the same as an argument. What's the problem here? Well, the problem is if you actually
09:00
just look at the function, put together the arguments, call the function, it immediately fails or subsequently fails or does not do exactly what the plain reading of the code says it should do. So if you're trying to test this function, you're going to have to do whatever it takes
09:21
to set up the decorator in order to make it actually drive the function in the proper way. An example of this actually comes right from PyTest. PyTest tests always have no arguments except PyTest mark parameterize.
09:43
We'll go ahead and create some arguments and fill those in for you. So these arguments aren't there. The decorator takes them out. Of course, writing these is pretty simple. We often write these as examples,
10:01
but here's one just so we see what we're talking about. Here's a func with name decorator that tells the function what you called it. So in this case, the first argument to the function becomes the name of the function. So function self-aware gets its name and another argument when it's called
10:22
and does what it says on the tin. So moving on to B, the binding decorators. Binding decorators implement the descriptor protocol to change how functions behave.
10:40
The standard library has a few of these. We saw static method earlier, class method and property are two more. The problem here is they create an alternative to instance methods and attributes and things that your teammates are familiar with. They create new or they enable new language patterns
11:01
that maybe are better for other languages where you got them from. Maybe you should leave them there. Or they might actually cause an error to occur sometime other than when it should have occurred. You can use these for lazy execution. Lazy execution will cause your errors to occur sometime other than when
11:21
they would have otherwise occurred. That's why it's lazy. SQLAlchemy has one of these in the hybrid properties, which if you access it in one context, it'll give you a value. If you access it in another context, it'll give you a query. It's kind of cool, but it's definitely not doing
11:42
exactly what you would expect. Here's a particular implementation of it just to show playing around with it. Normally our normal methods in Python, if you call it, if you access it from a class,
12:02
you can get it and you get an unbound method. Fine. It's not necessarily useful, but you can get it and you can bind it later. Maybe you don't want people to bind your methods later. Maybe that would ruin your day.
12:20
I'm not sure how, but maybe you have a very complicated inheritance thing and you don't want people to pick up a method and rebind it somewhere else. So what this does is this is an instance method. It cannot be accessed from the class. If I decorate something with this, it cannot be accessed from the class because it implements the descriptor protocol
12:41
and if you access it from the class, it raises a type error. So simple method and normal method are exactly the same, except simple method is decorated with the instance method decorator. If you use them normally, no one can tell.
13:02
If you pull the normal method and access it from the class, you get the unbound method. In Python 3, that's a function object. Python 2, it's actually an unbound method object.
13:22
If you try to do the same thing with simple method, you will get an error. The tricks you can play here can be much deeper. I was seeing one at an open space where they were actually using this to trigger events to a UI on access.
13:44
When you changed a value, it was giving a chance for an arbitrary collection of functions to mutate the value that was being set before it was being set and call back arbitrary numbers of things to do arbitrary things based on the fact
14:00
that a value was being changed because there was a decorator. So that's binding. C, control flow decorators. Control flow decorators are, in my experience, kind of the other most common decorator.
14:22
These are the ones that change whether a function will be called and how many times. The problem, well, you don't know whether it's gonna be called. So if you're looking at a debugger or if you're trying to figure out when did my error happen or why will it happen,
14:43
well, now you don't know. It might be zero, one, or many executions of a function. Our common example here is the retry decorator from retrying. We probably use this a lot
15:01
and we probably just use it a lot more and hope that it makes things work. We can write our own. That's not so bad. Infinite retry just says as long as we're getting a runtime error, go ahead and try it again, and again,
15:20
and again, and again. And it looks like that. Descriptive decorators are ones, are kind of one of these ones that people may not even think about existing or existing as a thing. This is one that someone told me didn't exist until I wrote one out for them.
15:44
What these do is they add the decorated object to some sort of collection. That collection will serve some other purpose like documentation, dispatching, or plugins. The problem is once you've done that, you don't know why you did it or where that collection is
16:02
or how that's maintained or maybe how you'd get it out of there or any of that stuff. We'll go back to pytest for another example because pytest does a lot of decorators. Pytest Mark does this one. Look, I just marked it as a web test. What does that do?
16:25
Look, it's an app route. What does that do? Where do I find that? Look, here, I'll write my own. Here's a QA decorator. It's gonna maintain a global list
16:41
of all the things that I should really QA. We actually do have quite a few of these. Most of your frameworks, if they're following any sort of the flask model, will use something like this and register something.
17:02
It's not an uncommon pattern, but it is an under-recognized pattern. Oh, yeah, there's my QA list. Now, one that is uncommon and fortunately uncommon, it's the execution decorator. These are the ones that read the code that's been decorated
17:26
and they may totally reinterpret it to be something else. Under problems, I'm not even gonna try to list these. The code you wrote is not what will run. It might be just analyzed for dependencies.
17:44
It might have the AST swapped out. It might be recompiled under different rules that aren't really Python anymore, which is exactly what's happening here. Cython has a mixed mode. In the mixed mode Cython,
18:01
you can decorate things to be Cython or not. It's perfectly reasonable. How else would you mark it? But that's exactly what's happening. It's no longer Python inside this decorator. Here's an implementation in case you ever wanna write one.
18:21
This one goes ahead and gets the source off the function that you got that you decorated, splits off the lines, throws away the zeroth line because that's at replacer, I know that, takes the first line because that's a function name, takes everything else, runs it through the search and replace,
18:40
execs the whole thing in global's namespace, takes out of the global's namespace and returns that as a new function. Not even the best way to do it. Don't worry. So here it is without the decorator. Here it is decorated for B. Down the bottom, that's 36.
19:04
And it gets even more fun if you decorate it for replacing A with A times three because now the value returned becomes 14, but it also found that other A in sample. There are some more practical tools for this.
19:22
So you can do these things, I'm not gonna say safely, but with controlled danger. You can manipulate this at the byte code or AST level. They just don't fit quickly on slides is the main reason I'm not showing those today. And that's how some of these do it.
19:45
So here's our taxonomy in conclusion. We have the argument changing decorators, the binding decorators, the control flow decorators, the descriptive decorators and the execution decorators.
20:00
I do believe there's more. I think there's probably a hybrid or a mixed mode or a messed up decorator. Not sure that that needs to be enumerated. I have found some other decorators and listed them here just to put them out of the way.
20:22
And I think we have time for questions. Yeah, so let's thank Andy for his talk. I have some time for questions.
20:48
What is your opinion on decorators? I actually think that decorators are a member of a category of Python features that I've started to call shovels.
21:03
They are Python features that help you dig out of holes, but shouldn't be used prematurely because if you've dug yourself into the hole using the shovel, you'll be at the bottom of the hole and you might already have broken your shovel.
21:21
So they need to be used sparingly, carefully. They should not be used prematurely. So they should either be used as part of your initial design or after it's become very evident that this is exactly the right thing to do.
21:41
Okay, thank you. Here's another question. Hi, thanks for the nice talk. Quick question, how do you debug and test decorators when you write them? Yeah. So I tend to actually use a full-up debugger from PyCharm.
22:00
So if I believe that I have an issue in a decorator, I'll put the breakpoint in the decorator. Also, I will often write unit tests for the decorator and need my decorators to be pretty highly reliable codes so that issues in the decorator
22:23
appearing in other code is rare. Same thing needs to be done with context managers. Again, because they are sort of out of band code. Beyond that, it's not unreasonable to have them log
22:40
a debug level statement that still indicates what's happened. And if you're looking at a control flow decorator, then you need to be aware, and this is one of the places where I hope that either this taxonomy or this talk alone has helped you if you have a control flow decorator,
23:02
realize when you're debugging, your code may not even be running. And there's one big one for that right here, LRU cache in the standard library. Any more questions?
23:24
Thank you for the talk. How can we build on this foundation to think of better ways of writing decorators or things to avoid because it seems that there is indeed a lot of problems coming from each types of decorators.
23:43
Is there some you would definitely say avoid them at all costs, some that we could find more than unit tests, some ways to make them better, or what do you suggest? So my suggestion would be when you're writing a decorator,
24:00
especially when you're, well, anytime you're writing a decorator for any degree of publication, document the decorator. I would love to have you document it in these terms. This is a descriptive decorator. It registers to here. This is what it does.
24:21
If you look at descriptive decorator, I noted the problem with it is primarily that you don't have a good idea where it's registering, where it's putting its information. So this is a descriptive decorator. The net effect is that it's maintaining a list
24:42
in the module namespace of this module. And then anyone can look at that and go, oh, this is a descriptive decorator like this other descriptive decorator that I'm passingly familiar with, and I understand that.
25:02
This is a binding decorator, or this decorator implements the descriptor protocol, which the descriptor protocol is right in the Python docs, so we can use that terminology too. This decorator does exactly this.
25:21
That's what I think would really help is we sometimes just put our decorator in and we're like, it's a decorator. It does this and we describe it in purely functional terms that don't establish or maintain the commonality with other things.
26:00
Yeah. Let me see if I can even get it out as like an official Bloomberg white paper. That would make it easier to find than bury it on my own blog. Yeah, it would be at Tech at Bloomberg. I'll try to get it out. I mean, I have my own blog and Twitter,
26:21
and Twitter's linked from the conference stuff, and I guess I think we can get this up through the conference slides, but that's not very referable now, is it? Let me see what the most referable thing is. Okay, more questions?
26:40
Okay. So decorators have this funny situation where you can pass some arguments to the decorator or you can just leave off those brackets altogether. This means when you implement a decorator which supports both, you have to have a multifaceted approach because the first time you're called,
27:00
it may be with arguments or it may be when you're given the function when the decorator is applied. Do you design, do you implement decorators that do both or do you agree this is a water Python that we have to, as decorator, implement as cope with this? No, I probably would not bother in most cases
27:24
implementing the ability to do both. In some library cases, the people that are worried about that, I think they have legitimate concerns. If I ran into that, I think I would probably look at, I probably would adopt exactly the sorts of things
27:42
they're doing without too much hesitation but I don't personally generally worry about that. I would simply say like, you always need parentheses. You know, move along. And there's a decorator module that I think helps with this.
28:01
Yeah, that looks good and I'd probably just use it and move along with my life but like I said, the whole key to understanding decorators with arguments is that it's not doing anything. It's absolutely not a syntax.
28:23
It is simply calling that function and using whatever it returns from that function as a decorator. So I think if I actually went to make a kind of comprehensive solution for that,
28:42
I'd probably do a class, build out some sort of class-based solution and maybe pre-initialize it on module load as a singleton. I'm not sure quite how I'd go about it but if I had to resolve the problem, I'd start hunting down that path.
29:02
Okay, unfortunately we don't have any more time for questions so let's thank Andy once again. Thank you.