Add to Watchlist

Metaprogramming, from decorators to macros


Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Metaprogramming, from decorators to macros
Title of Series EuroPython 2014
Part Number 47
Number of Parts 120
Author Crotti, Andrea
License CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/19939
Publisher EuroPython
Release Date 2014
Language English
Production Place Berlin

Content Metadata

Subject Area Computer Science
Abstract Andrea Crotti - Metaprogramming, from decorators to macros Starting off with the meaning of metaprogramming we quickly dive into the different ways Python allows this. First we talk about class and functions decorators, when decorators are not enough anymore we'll explore the wonders of metaclasses. In the last part of the talk we'll talk about macros, first in Lisp and then using the amazing macropy library. ----- This talk is a journey in the wonderful world of metaprogramming. We start off with the meaning of metaprogramming and what it can be used for. Then we look at what can be done in Python, introducing function and class decorators. When decorators are not enough anymore we move to the black magic of metaclasses, showing how we can implemement a simple Django-like model with them. In the bonus track we'll talk about macros, as the ultimate metaprogramming weapon, showing briefly how Lisp macros work and introducing the amazing [macropy library].
Keywords EuroPython Conference
EP 2014
EuroPython 2014

Related Material

so have run and then this might with the count and this link here that you can see you can already download my older multilingual to show that would be
the the cold anger to show and this slide so so
you can have look and play out there when it comes to open vitamins and Inc interactively tried examples I'm going to show you because that's what I will do as well so i come from the thing about the live London and work for this company called the problem is
that the smaller the president's like that when I tell them and they're just the lead you that we
are hiding we're looking for an API developers and few it just have a look at the public on jobs using Django salary all nice thing small would be posters so right so what we're going to talk about like 1st of all I'm trying to define what is meant the programming
and why is usually and and then we're going to talk about least believe but not too much so that this can and then we're going to talk about Python about
vitamins how some techniques we can use for the premier by 2 so this is the definition of found Wikipedia momentum parameter elizabethan let's have a little this would make the parameter is the writing of computer programs that right or manipulate other programs or themselves as their data all that do part of their work at compile time that would be otherwise than so this this kind of still not very clear but I think we're gonna see now in in some examples how these will these really mean and manipulate the program itself manipulate themselves well enough as it's really that 1 that makes so why on earth would you do that that's the most important question because it sounds really magic and really strange as there are some scenarios where this really makes a big difference and normally it's use is not really used for the performance of reasoners
or for making a better product is maybe normally just used to be closer to the domain you're working so you write an abstraction in a more powerful way and in the end you read you managed to minimize this line of code the whole you managed to have make the old program the of the main the more easy to understand for the people in that domain so what's the cost of that well and do you know if you do it well nothing if you do wrong you're might screw yourself off it next slide this there was
very small Introduction to Lisbon Chloe was invented by McCarthy 1958 so there was a very long time ago as they get farther of languages in and what people think about it is that there are lots of time to this that's how people see normally and there is a reason however for all this point to this so then went back to show
you that while this this this this simple as possible well this simple total function and this has defined the is a special form of the 5 which is a way of defining functions which takes an argument it is the name of the function the arguments of the function and then there is another special for me which takes the arguments and then the conditional they that and the else about here and also that all these things are expressions as expressions process but I'm going to do a lot of work and of life according so many of time now for the 1st time to just show you how things actually works so we go in the next year because so and I define
this is there right look for
and as this year which is a common lisp interpreter and
then I hear the 5 fold I could compute and using words find interesting thing to
note is that even if I give a very high number it
doesn't blow up like Python will do even at this recursive
because it's is smarter about aggression divided so let's
continue so this is very small the very limited syntax and is another property which is quite an uncommon and not too many languages which is home or encoding seeking this a very difficult war that the uh what is to defined by the practical again means that the program itself is a value of representation of the data and so here a sample what we have is the computation of the products between the sign of 1 and the cosine of 2 points is the and what this
actually means is that he's like setting an expression to a list where the 1st element of the list is the symbol last time the 2nd element is not the least we design symbol 1 1 and the 3rd is not least the cosine into the and then you evaluate that expression so why is always come that what this because you can actually transformer and pass around a program and manipulated easily as he was data so to make it more maybe more clear I'd like to say how despite all more incoming but not my what how will be look like if it could be possible that might be something other than that makes sense that so suppose you can write suppose this is my and is a way to define a function you you create a list and then 1st release is death and then the function then you have argument comma past so this is a list and this is also how you define a function is not true because that that's how what it would look like if was to 1 December this thing is so important well 1st this is just there to show how the evaluation is done in Lisbon just to make clear as you know that every an operator the fixed and then everything is a cones the so you have here some of the some of the most efficient sign any have the 2 then you have another branch where you have a blast is 3 of 4 and he had nearly nearly because there is no other elements in the course so why is this property important because in least you can make make a programming to a very deep level you can make use macros so I assume and which may be the same but listen to set it to do an assignment of a viable you said you extend supper which to the symbol x the value that I suppose I want to do something like said that then to to valuables at the same time I guess I mean 2 things 2 1 value In 1 call Is that possible well moment is impossible but what if I write a function so I write this function defined set you have X Y Z and then I do use these this special form program which are concatenated tools multiple expressions but within an hour but that doesn't work so you can try it out the syllabus interesting was it's in the so this could the so now
sensitivity to a lot of of the assistance a failed this as the Bible ladies among so what i
have other bit but that's kind of understandable in a way because I'm trying to say to create a special
form with the function so this is not really possible so that however if I rewrite this thing with the micro we know where the only difference here is that micro and here I use this method called and these comments here from instead everything was fine In the final micro and then a Doucet could choose and because they MIT everything work
so what's the 1st the 1st in small but is quite well because here uh these micro and basically
creating at compile time in you and a new function that we get the templated when equality so this is like it takes the same body doesn't trying to evaluate it doesn't I developed a or B it just pass it around to these quarters thing which then gets evaluated runtime and these runs normally positions so by powerful technique can we do the same if I could mean almost well I mean for to do this in this kind of thing you would do something like that which is horrible and yeah almost works there so that elite you 2 things like a and B and in the past an expression you evaluate this direction and any update the global with this think that it is not even the same is but can work by and on the left so I am not the least of the prior interaction is always like would have passed something more interesting patterns so what can we do with Python for the programming where can never function the greatest classical meet the classes and then if you really want to be fancy you
can manipulate the is the US doesn't expect and to show them in order they in order of difficult English what is the
correct 1st because it is just the syntactic sugar for a function that modifies the behavior of another function or a class so this simply means below you will you do it at the creator of all of the function and the same thing as defining function and then redefining assigning it to the return results of itself and the the the 1st problem that we try to solve with the the correctness and assume I want to measure the time spent in the function and printed out to to the standard out and see how long everything takes this is said the correct of the desert so it takes the function he uses these nice drops the correct or defined in find tools and this is just to make sure that all day like underlying yeah the attributes like doll strongly gets passed correctly and they don't get lost along the way he would walk anyway it's always good to put this so yeah and then you define a function which we'll call the original 1 store the results somewhere they're stored the time before and after and then print out the time span and overturn original and then here you return to this In the function but you can show this how how that works for the when we good to show pointing the can
is sitting in the back because many you
can from components of that can
always cold conditions size of the friends
so now I created as a function of time in prison
was something kind to
anything useful and I call this and I get this
message function time
made and took they should have said something else I could change before this the total of course in them in state
yeah this is the again I'm
not so the 2nd class of
I have taken
1 the action of the yield of liquid In this work want on
no just orange or again as those that I showed that the next
assembly more interesting than that that that
might so we have another problem has been we timing but slightly different because I want to know when a when I'm invariant is something new like a new feature or
a change algorithm and I want to know the new 1 runs faster than the previous 1 but I just don't don't want distant only once I want to make sure this always of that so I want to have that running a test uh somewhat like integration tests and say my new fancy algorithm is still faster than they can field 1 after 6 months of hacking on it and adding things uh and so I mean theory with the previous approach we couldn't do that because we were just thinking about so you can just check all the time how things look like uh and but we another the correct which also change the signature of the original function then we can do that so what I have here is a test to check that the 1st implementation is fast and the 2nd 1 and the way it works is that the new this the content here will change the function return values and of the time spent in the function as a corner and as a 2nd out of money as I do this do this for the 2nd 1 and then I can check at the time of the 1st 1 is less than the time of the sort so no the see how actually implemented so it's still quite simple it actually is the same almost knowledge difference is that here we when we do that we do we also the time here so this is an interesting case and the difference is that you can't use it he import time way because if you cook the the if you decorate function in the model like that then he would mess opposing because in 1 you don't care about that time spent in production so you can only do that in this way but it's perfectly valid thing as it was stated to just think that the sugar to just that it plays a function of the runtime in this way and then calling so that's a new dataset of more likely this and so and they could close the what is it at the same
time times that works there see
again couple and the the 1st element is the result and the 2nd element is constant
in tool actually see how things work it's interesting also to look at the code here of timing the yes
and that rocks function the denies thing to do also make sure that the source code that the term is the original 1 not the rock not thereafter in into the content of so another example this is how you can the credit classes that suppose
I want to I always the same method to many classes
and this many I I don't want to go in every class and change it and I don't want to make a superclass to made to make this this is not a very simple way to do that you just this is the test where I have a class which had a sparse methods and then when I especially this class in the cold response I get the terminal 1 the money 1 so this is a simple way to do it very clear simple you just take a class here you were assigned to response an anonymous function which takes self have does this and then you the class itself yeah that's the petitioners so let's passed to make the classes nevertheless this is the next the other way of doing it means that programming in Python is more powerful than this more is also a bit more difficult to understand at the beginning but is really not that hard to assess how this it looks isn't that the trade so what's the the middle well metaclass in Python is this the same as the the metaclass status to a class like a class that's going to and so is the type of a class 1 if you
create a class and its way here is exactly the same thing as doing in this way With the tighter function when to try to show you know little
1st sample if I know the this I forgot to mention
the beginning that all the columns showing is set by 2 and so the only well the difference is not many but he has retired right
entitled to it with my finger on monopoly on OK so was the type of C study and then the 1st times that to create 2 types the same will this takes as arguments takes at all did domain and
the least of basis in the dictionary so for a
simple suppose I want to create and runtime class but be but I would hazard as basis from the origin and then as dictionary have like a class attribute the 10 must be a couple minutes mistake OK so this is now class and they can instantiated plants and I can access the so this a way of
creating classes of on them and this for in town in the back of works this is the simplest possible metaclass implementation
and how to use it so we
have a class simple tell which subclasses type and then to use it in our own
class we only need to create a class of form and subclassing object and then passing class equal to simple made here invite until we would have been underscore the class here equal to something but it's just the same thing and now discusses some of the more interesting examples of things every or more reactive jungle framework of some is very easy to define models that told that the based on data keep some information about the type of strong as before and this is kind of possible worlds syntax of issues which are just made up about question thing and it looks like about magic but it's not that hard to believe in the reality and this simple way to do it in this case if I want to accomplish something like that if I don't have met if I could just have a model for this particular case which takes the argument with X and Y and then initialize them and so well I mean if you you can do that if you have some very simple status but if you want to find many models if you will and use the variables because every time you have to declare everything and assigning there's a lot of border prize and also the most important thing the thing is that you have no information about the types and while vitamins ductile databases of nominal so you may still need to add information someone an so how humble how can we compilation this kind of very nice same but we
start by looking at how we can implement the field that I so which is this 1 so if inside that will be a base class for things like integers or strings and you can implemented as so this is the test that will this requirement for this kind of stuff type I want to have the same different things the and want to be able to and this is what the test again this idea that this so it's not they can make this in the next so this is this in the model I want to have and is obviously a requirement that I have for this model to behave like a war so I want to be able to create an object like this for sample Y and we'll passing x x as the full value why doesn't want and so I check that after I passed this otherwise it is set and X is also set to to the full body I want to be able to know know that if I pass something wrong like unknown hello he would give an exception and I pass something of the wrong time it will always given type this case so 1st so that we can look at how few son implemented and here again this requirement for the field so if I could mediator an integer where past 0 so as volume and the full 1 the value will take the president's if I tried to set at any time something which is the wrong type I get the type of and also very important in the fight in the beginning create the element correctly and then I change the type to something else and then else again type of but the implementation of field of which is not you down using metaclasses but this is not done in this way so they need them the most important thing is that we have the
property here which has also a certain and this property is they actually representing the value a whenever you try to set here the value which is also done at this point it always always check the the type the checks if the type is not an an instance of this basic tight which have a class we have define then is a type of let's see how useful simple implementation of the string integer so very simple here we have the
basic classes here
integer and here thing and so that's
all I wanted to do is to define the base type
and if there called your
past these are the basis of that then you type yeah so let's go back now to the model the model Smith a and this is the actual that the solution for all these which fits in
this life size must not be that difficult is that the solution for the origin of problems and the ways that is the days subclass type as that before in the previous example and then I and overriding and this comes with a new method which is used to in during the construction of the class itself instead of the initialization nationalization of the object like a unit and and then I get for arguments here I get their class I get the name I get the least of these classes and they have the same problem the class leaked which is a whole day uh this the contribution and methods of the class but which we can modify so the way the way I want to do this is basically just the replacing the of the new class that we've created now with another unit they will lose some extra checks and extra things for me ultimately with the 1st of all I looked over the active use of the class and they check at the beginning if they are to be that of the class he's an instance of field then this is such a field that I just thought dictionary here and then I'd defined any here which states some kW so what you can take x equals 1 and y you will was on fast and then it will 1st of all check if there are some arguments which have passing that are not known so if there is some of the the difference between the k they with passed in the fields that I found that looking inspecting the class these going 0 then given exceptions otherwise if not believe everything is fine then look all this and then generate set inactive due to this object cells we then that's a key they and then the name of the document and there's a value the value which is just now created by calling the class and passing the value so it sounds more complicated than this be of the show maybe now and behind how works and then after that that might be important step is that idea defined the Aeneid of the class the to that thing which already defined here and then I'll return and just this will called and try to show you just said is simple how this behaves in so I now yes now the only thing I need to do is to define a model in this way class
model yes and then I will simply find the final another model for and on top of it so we can
take 2 things apart from 1
0 and then identifying x equal to
models strange can now
suppose I want to create an instance of this but they pass wrong thing
complaints anonymous way by task this is a checks type and this another if I pass this everything would work correctly upset and the
interesting thing if if I look at the beginning of this it tells
me well actually it becomes and also from the condition
yeah but look at the beginning of the class simple to he tells me that it comes from here function models metamodel new
local in so that's where we find and analysis and another example which is a bit more simple is helpful for
example make sure that we always called will of that you need in the subclasses of a superclass and we define and I've seen is doing for sampling and the modules finding the time if you forget to call the it will blow up and complex and the physical that implementation is basically just this he has a initialize falls and in each in in it it was set initialized through and in every method that has to be uh working on initial object you need this this check so that's a bit annoying and that's a bit too long minutes so this is what I want is that I call or something and if I forgot to call this will in the it should I'm giving a session that and I think that's the implementation by simple I do that I store regions In somewhere and then I Call it here I I replace the it with another age which I'm doing I call it here just to make sure I don't lose anything and then I also and in the 2nd session looking for a flight valuable that is set up so this is kind of doing the same thing but is in a more generic way and so all you have to do in your class to use this is to just go here I define a class based sparse in the metaclass
chickens and all I have to do is to define a flight valuable
equally violence and is a valuable did I know that I
always said in that you need here here so if something tries to subclass base and forget to call in these here will be not set and then this we live in a given session to yeah and last thing the shovel
quickly is there just you can do even much worse than that if you want and this is a very simple function that underlies the AUC of a given function passing and powers in the source and then trying to find out a particular value a stream and replacing it with something else In petitioner show you the actual examples for you could this then this is this is the
function here which I want to the place it on time and what I want to replace is to remove the hello from this thing so and this is what it's doing I call it 5 XX these
and the term of this and then I call it again so what is going to happen
is yeah the 1st time calls
along the 2nd time only work when everything works fine so this thing is that of course this is very hard coded and it's not really a big good idea but this URI
is the is just 3 so you can traverse evening anywhere you want so
you can look for all the things that look like strings that discontinuity look for all that you can do anything you want and you can put them in the correct order all you can and in this this is the only way I think that you can reach something like as powerful as least as invited and that is a library which I'm going to be and the left time to show that the new was too much there is a library
called macro where you can actually do things like that and this is value of items even my syntax highlighted doesn't think it is but you will arise the would work and you can make a case class with a cat declare a class like that posting x and y and is automatically we've created unit where the X and Y and you can just expatiating data and everything will work correctly and that's how if implemented so if we want to look at at the school might provide so we have this information so any questions and then there's the conclusion sigh of what metaprogramming is fine you can make it in just from 1 feeding was like the call to create a very simple site on and then just be careful but it is the on this this
being on the ground will there will be at the heart of the new year yes I if I knew that the main thing might be its high-capacity lower the the user made the framing the yes leveled command dates maybe leave out my name or but change the story of the need to make sure it doesn't come to me yes no question on how to all I think the philosophy of Python is that you shouldn't do things that up to 1 of the the ball into problem is then so you you will kind of go against the the language that designed to do 1 more and that the lower the you could but there things you you can go crazy things like might provide us so it is possible already is just think is enough and haven't think you need
more than that it be
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation


  693 ms - page object


AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)