Dynamic Class Generation in Python

Video thumbnail (Frame 0) Video thumbnail (Frame 1108) Video thumbnail (Frame 5255) Video thumbnail (Frame 8159) Video thumbnail (Frame 8764) Video thumbnail (Frame 9361) Video thumbnail (Frame 9897) Video thumbnail (Frame 10653) Video thumbnail (Frame 11375) Video thumbnail (Frame 12139) Video thumbnail (Frame 13021) Video thumbnail (Frame 13685) Video thumbnail (Frame 16314) Video thumbnail (Frame 21106) Video thumbnail (Frame 25938) Video thumbnail (Frame 29737) Video thumbnail (Frame 30864) Video thumbnail (Frame 31968) Video thumbnail (Frame 32704) Video thumbnail (Frame 33584) Video thumbnail (Frame 34670) Video thumbnail (Frame 35391) Video thumbnail (Frame 35975) Video thumbnail (Frame 37341)
Video in TIB AV-Portal: Dynamic Class Generation in Python

Formal Metadata

Dynamic Class Generation in Python
Title of Series
Part Number
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Kyle Knapp - Dynamic Class Generation in Python This talk is about dynamic class generation in python: the practice of writing code that generates classes and their functionality at runtime. It will use boto3, the AWS SDK for Python, as a basis to dive into the basics, the benefits, and the drawbacks to dynamically generating classes. ----- This talk is about the concept of dynamic class generation in python. The whole idea is writing code that generates classes and their functionality at runtime. You now may be asking yourself, “That sounds like a neat trick. Why would I ever generate my classes at runtime?” Here are a few reasons why: - It can decrease the physical size of your code. - It can improve the workflow in adding new functionality. - It can improve reliability of your code. One example where the power of this concept has really been leveraged is in boto3, the AWS SDK for Python. Dynamic class generation has allowed boto3 to become heavily data driven such that most of its classes and methods are generated based off JSON models representing aspects of an AWS service’s API. For example, to add support for a new AWS service API in boto3, just plop in a JSON file into the library with no additional Python code required. Using lessons and techniques drawn from developing boto3, this talk will dive into the following topics related to dynamic class generation: - The basics of dynamic class generation such as how to effectively dynamically generate classes. - How to overcome some of the challenges of dynamic class generation. - The tradeoffs in dynamically generating classes and discussion on when it is appropriate.
Area Dynamical system Topological algebra Computer animation Software developer System administrator Symbolic dynamics Universe (mathematics) Video game Right angle
Functional (mathematics) Topological algebra Code Multiplication sign Source code Letterpress printing Frustration Insertion loss Parameter (computer programming) Topological algebra Menu (computing) Arm Computer programming Workload Latent heat Symbolic dynamics Endliche Modelltheorie Data type Inheritance (object-oriented programming) Run time (program lifecycle phase) Physicalism Instance (computer science) Cartesian coordinate system Variable (mathematics) Category of being Type theory Computer animation Symbolic dynamics Logic Personal digital assistant Pattern language Object (grammar) Resultant Tuple Row (database)
Functional (mathematics) Service (economics) Run time (program lifecycle phase) Code Standard Model Multiplication sign Demo (music) Source code Client (computing) Product (business) Term (mathematics) Symbolic dynamics Cuboid Energy level Endliche Modelltheorie Demo (music) Concentric Client (computing) Bit Cartesian coordinate system Software maintenance Word Computer animation Personal digital assistant Pattern language Energy level Electric generator Resultant Computer-assisted translation Library (computing)
Cognition Service (economics) Run time (program lifecycle phase) Standard Model Scaling (geometry) Interactive television Library catalog Data storage device Client (computing) Blog Object (grammar) Information Fiber (mathematics) Lambda calculus Data type Default (computer science) Point cloud Area Service (economics) Gateway (telecommunications) Trail Online help Interior (topology) Code Ext functor Client (computing) Event horizon Computer animation Synchronization Physical system Identity management Electric generator Computer-assisted translation Elasticity (physics)
Service (economics) Gateway (telecommunications) Service (economics) Trail Standard Model Scaling (geometry) Interior (topology) Code Library catalog Data storage device Event horizon Computer animation Blog Order (biology) Synchronization Set (mathematics) Identity management Computer-assisted translation Lambda calculus Point cloud Elasticity (physics)
Service (economics) Computer file Scaling (geometry) Interactive television Library catalog Open set Client (computing) Ordinary differential equation Subset Blog Object (grammar) Operator (mathematics) Set (mathematics) Information Configuration space Lambda calculus Point cloud Data type Default (computer science) Service (economics) Dialect Trail Online help Computer file Interior (topology) Standard Model Code Client (computing) Ext functor Event horizon Computer animation Symbolic dynamics Synchronization Identity management Physical system Computer-assisted translation Elasticity (physics)
Point (geometry) Dynamical system Service (economics) Online help Computer file Interactive television Source code Standard Model Ext functor Client (computing) Line (geometry) Canonical ensemble Content (media) System call Computer animation Symbolic dynamics Object (grammar) Information Configuration space Physical system Data type Default (computer science)
Dynamical system Server (computing) Standard Model Code Multiplication sign Source code Database Database Client (computing) Cartesian coordinate system Computer animation Personal digital assistant Order (biology) Library (computing)
Injektivität Inference Dependent and independent variables Computer animation Code Hypermedia Building Weight Interface (computing) Client (computing) Endliche Modelltheorie Cartesian coordinate system
Point (geometry) Addition Dependent and independent variables Validity (statistics) Mapping INTEGRAL Code Multiplication sign Sampling (statistics) Client (computing) Menu (computing) Parameter (computer programming) Cartesian coordinate system Element (mathematics) Revision control Type theory Latent heat Computer animation Personal digital assistant output Endliche Modelltheorie Resultant Computer worm Physical system
Presentation of a group Run time (program lifecycle phase) Computer file Standard Model View (database) Numbering scheme Parameter (computer programming) Client (computing) Function (mathematics) Mereology Element (mathematics) Web 2.0 Inference Medical imaging Latent heat Term (mathematics) Operator (mathematics) Touch typing Endliche Modelltheorie Proxy server Position operator Exception handling Dependent and independent variables Validity (statistics) Standard Model Electronic mailing list Client (computing) System call Type theory Computer animation Personal digital assistant Internet service provider Order (biology) Formal grammar output Right angle Resultant Spacetime
Point (geometry) Functional (mathematics) Standard Model Code Workstation <Musikinstrument> Online help Client (computing) Data dictionary Element (mathematics) Latent heat Different (Kate Ryan album) String (computer science) Operator (mathematics) Factory (trading post) Software testing Office suite Extension (kinesiology) Proxy server Address space Inheritance (object-oriented programming) Military base Line (geometry) Instance (computer science) Variable (mathematics) System call Category of being Type theory Latent heat Word Computer animation Personal digital assistant Factory (trading post) Order (biology) Right angle Figurate number
Point (geometry) Functional (mathematics) Server (computing) Standard Model State of matter Multiplication sign Letterpress printing Set (mathematics) Parameter (computer programming) Client (computing) Root Well-formed formula Computer configuration Semiconductor memory Operator (mathematics) String (computer science) Factory (trading post) Software testing Exception handling Physical system Inheritance (object-oriented programming) Mapping Military base Data storage device Sampling (statistics) Client (computing) Basis <Mathematik> Line (geometry) Cartesian coordinate system System call Electronic signature Type theory Word Computer animation Logic Personal digital assistant Order (biology) Resultant Tuple Spacetime
Server (computing) Code Electronic program guide Standard Model Electronic mailing list Medical imaging Computer animation Different (Kate Ryan album) Order (biology) Integer Summierbarkeit Endliche Modelltheorie Freeware Perimeter Resultant
Functional (mathematics) Demo (music) Computer file Online help Standard Model Demo (music) Interactive television Heat transfer Ext functor Client (computing) Content (media) Computer animation Symbolic dynamics Object (grammar) Information Endliche Modelltheorie Physical system Data type Default (computer science)
Online help Standard Model Code Interactive television Ext functor Client (computing) Attribute grammar Differenz <Mathematik> Computer programming Error message Computer animation Computer configuration Object (grammar) Information Endliche Modelltheorie Physical system Multiplication Data type Default (computer science)
Topological algebra Error message Computer animation Object (grammar) Mathematical singularity Sampling (statistics) Attribute grammar Differenz <Mathematik> Cartesian coordinate system Number
Functional (mathematics) Computer animation Standard Model Symbolic dynamics Source code Order (biology) Iteration Machine code Cartesian coordinate system Resultant
Computer animation Code Bus (computing) Client (computing) Communications protocol Software bug
Functional (mathematics) Word Topological algebra Computer animation Code Term (mathematics) Repository (publishing) Client (computing) Communications protocol Proxy server Software bug Product (business)
Type theory Functional (mathematics) Different (Kate Ryan album) Order (biology) Cuboid Self-organization Right angle
OK so we're moving on to the last speaker before lunch so I'm guessing everybody's hungry but please refrain from disclosing the goal is to me is before lunch but please palmapper go talk about dynamical as in recent Python right and also my name is Hanna and I am a software developer at US where I've ever really focused on developing the universe as a caper Python also known as 3 and the area so I will Python-based command to administrative resources and talk to you about dynamic class generation life so when me 1st go over what you should expect this talk person to talk
to you about the basics of dynamic generation mainly what it is and how you go about doing it then i'll talk about the benefits of doing it and when it is appropriate to use it and finally I'm going to build an application from the ground up that uses dynamic frustration in really shows you all the best thing to do so let me start by answering the question what is dynamic class generations what it is is it generates the patterns to generate classes that went time so all the functionality of the specific class so the definition specific classes exist in the source code and the result is it's generated as the program runs and because the cost of an existing source code departure now you need to be generated from some arousal in most cases it's some other data sources such as a schema API so let me show you a simple example of how you can dynamically generate a class here I have a whole world function that takes an object in prints hello world from the edges I also have a class they're called base class that takes and sustained as a parameter insisted as a property now I call type used usually you're familiar using type user takes only 1 argument tells you the type of object but here type can also accept 3 different arguments where the 1st argument is the name of my class the 2nd is a tuple representing inheritance and what this new class that and the 3rd parameter is the dictionary representing the additional properties I want to add to this class and once the call-type it will actually create menu classes such an instantiated as so now I oppose introspect some of these variables here if I look at the the name my class appropriately show my classes next if I try to explain the properties like and sustain go show up all those models and actors that property and also I can call hello world from this instance stated that additional functionality of pretty Hello world from this specific so you probably things like OK that's cool but why should I care about that in a question 1st of all it improve your improve your workload in the sense that you will have you will have to write a amount of Python code to actually add new functionality maybe not meet United have to write I think could at all which is cool 2nd it can improve the reliability record a lot of times how dynamic class generation works out is give a generalized that most logic was that generalized as result it becomes more heavily well used and tested than 30 can actually reduce the physical size of because specific
losses match exists source code and therefore doesn't take up against the size and finally there the
production of a pattern of going to application that uses the later in talk so now entitlements and the downsides of it 1st of all trace back to a little bit difficult the trip to follow mainly because a lot and using a generalized source that so spirit and as a result you have to look through the data the data source that your getting this functionality from to really identify words going wrong secondly for IT support doesn't come right out of box mainly because need the source code to be able to suggest and it simply isn't there so now we go into production level application in terms of but but 3 is that this is the case for Python so allows you to interact with all the different in the CPIs available and but 3 is dynamic and stage of its dynamic in the sense that all of its clients enormous resources are generated at runtime a data-driven in the sense that the functionality for these dynamically generated classes are called from Jason models representing and this it yes and with these 2 qualities allow me as the maintainer of this decay is an efficient work us because but they have rest it's constantly innovating constantly adding new features and services so being able to stay to date and provide all these new features the best case very important but it's very difficult when there's a lot of a guest worker with both 3 in the fact that dynamic in data-driven politics for me is to simply update adjacent model that's packaged in the library and I don't write any pattern code at all and all immediately be able to take up the functionality I can spend my time doing some of the work that that they can be really handle dynamic concentration so now when you do is show you a demo
of what this would look like the very cool so now I'm going to do is open up here and now let's say
areas have a new service called myself so if I try client has fiber my service and I will probably get the runtime you're saying
this is not available so let's actually got Texans so when we you do now
is that and we got a new API model for service and and open up um
this model so you wouldn't cannot looks like and if you look at the
various just this is Amazon E
C 2 using general it's nothing new here and now that I
wanna can use now is that it still alive which share the same data path as noted 3 In
order to add this model on the same data that share support provided for the service model that while they're just opened and then I provide a new CIA was renamed the the service into my
service but that just
copy the file into the correct that but research subset of open up IPython again I can
then use so provided 3 increase try to create new client inappropriately creates and now I can actually just college easy-to-use operations and is going to describe regions called and
appropriately was able to make the call as well so this is really this is really
cool because I did not write a single line that encode enabled to add new functionality so that being said you probably would now wondering when should I
actually considered dynamically generating these classes the 1 big point you probably want to look for is if there exists some canonical source of data where you can actually pull functionality from and better yet if there's more than 1
application possible using this source of data uh class examples where a b and a lot of
times what happens is you have to update the model anyways in order to generate some the server code so being able to use that for the client code you get the bigger the better propre which is also then you can get in the case of databases where you have a defined schema and libraries style exam and are able to use a schema to create a dynamically eyes to interact with this database so now we say
actually build an application from the ground up that uses dynamic class the application on a bill is
a lightweight but it could be it involves having a client interface where it's methods are one-to-one mappings to the various media and this applications we all model-driven there's not going to be any encode that specific to a specific it can happen is to be able valid inference based on models is to be able to 1st responses based on models and it's pronounce something support is not reciprocal so let me go over the
central point as well of were 1st I'm going to ship Shiite write a simple days on the supply and then I'm going to integrate PPI models to pick up input validation response passing then add case specific methods to this application are declining face stuff I make sure it's extensible 2nd inject menu classes and by land actually add more at the so for those of you not
familiar RPC on the briefly go over so do is obviously relies on using comes against a single year where there's not any additional half free-trade if worry about and mostly data is some bodies through the request response silicon request you will specify like a method or parameters you want right and acted in a responsible return result from that method call the now let me go over some sample code of how the system RPC find work out don't worry about looking through deeply right now all walk through with you so now if I was to try to this this is this time with an end point 0 0 so that you can import your all related when I try to make an API chemical before they make ought to specify a method and parameters such that the method gets maps of the method element and the J some payload of sent in the parameters will be mapped to the on only that was and that it was said I can just use requests to just send opposed that you're all with the jails on document and will return to me is the is document as well where you can see here all have the result of multiplying 1 by 2 by 3 together which as you the 6 so it's not that we need to be expanded on this this feels very general there's obviously you don't even need this got this biclustering now you could probably just call request directly you don't have any input validation you have no idea what methods you use what parameters prior the types LCA returns entire response seeking others in the responses of elements talking about with the idea of the request response was in the version adjacency being used
then talk I said to you which is agreement API models since this kind so currently
wherever we can precrime with an for your own generalize call by then this step will be always view is actually take adjacent model loaded and have the new client class consume this model such that I can make a model EPI call which will do is be able to know if its position or keyword arguments except and appropriately print out just the result I 1 and then similarly if I a bypass and inference parameter type sealing the strength to the multiply 1 by you that will runtime error saying this is a touch I expected that so in order to get this working let's talk about the piano here is a simple API scheme that we're going use this is a lot simpler than some other scheme you might see out there like in Web for users or something like swagger of abilistic in this presentation simple so here what do in a schema is you can't define what the end point you're all here is for years EPI and also you can provide operations a new 1 provider in our case when I multiply for now and for each operation you can specify an over over our working uh documentation for that method you can specify what inputs was look like and also with the outputs of specific like as well this but in terms of the amplitude and say what the type is in our case the stimulus and therefore we want to have the additional arguments and you can also specify what the document will tell you when described this as well and you can also do model what what each element in this list will be in this case it's images and you can do the same for the output which you get documentation and also specified well so now we're going to do is actually take this model and then trying to remedy the client of don't worry about trying to read through this right now all over his wall so now what we do is important they get adjacent file into model we now have this model multiplying class consume the model and it's an initializer well do of creates a grammar data in the response which will use the model to both validate in parts responses based off the provided parameters and the API response given that so now what I'll do is continue on and instantiate the inherited kind initializer and then I'll be able to use the make model in the article in which case well 1st do is try to validate the parameters provided using the validated and improve model once we know these are these parameter values will then use the make API call which is inherited from the client to order to actually make a request against the then we will be able to pass the response space of the model and the response given back to us as now about when he's expanded honest still feels to general mainly because they can still completely undocumented we have no idea what methods to invite you have no idea what parameters and without produced in B so let's go fix this by actually
adding address specific methods by dynamically creating them so
only what we have is we note that the model for each station applying the commodity chemicals but by then step will be able to do is now use a factory function to create a new client L actually have these new methods that we want on the client and similarly if we got help on a client we can also get documentation on how what will actually look like so let's go over and code and how this will work so once we will introduce a model and pretty quiet we'll answered initializing similar variables needed for the type of related to call later down the road in which case we just called the class name my client in that and bases we wanted to hear from in this case and also we will create an empty dictionary for word class properties everyone and in what the class so now what we do is actually open up operate the model and look for all the operations available to us and call this helper functions declined with the correct current method will do is actually defining unique at a new function called underscore EPI call in which will be used as used by the extension point so what we to return that defined functions we attach it to be added to the class properties that will provide 2 types which will actually create this new point for us and then with this new point class will stantiated instance of so set now if I call multiply 1 2 3 it's actually crossing out to this newly defined that defined PPI call here where itself is now referring to the client that a attached to the method of a test you and the make model PPI call which is what's called underneath of this function is inherited from the model class but there's 1 big issue the doctrine still not specific so if I try to call help on this firing now goes see exactly what I described which is multiplied is just a proxy activists 104 API call so let's figure out how to do that that's that's actually not too difficult you just add these 2 lines here which were sitting down the name and the or the method that we add to the class properties such that recall help now we will actually pick up the new the new documentation the correct name for the method so by setting the naked under name right here to 1 of the string operation and allows you to override that says the fact that it looks like a proxy people help and by 7 and then adopt you're actually able to set the documentation for that matter that get docstring method for is pretty much what it does is takes operation model and look straight the all the different documentation elements of you remember from before the EPI schema in order to generate the concatenated together a string that has the operations in the type of printer types all returned to 14 so let's talk about when he's extended officer model 1st of all it's not accessible it's an example sense that many of you will support looks something like custom class names for custom inheritance in the sense that we all 1 actually only have to rely on these model the TI
methods we want to be able and new functionality topic so let's talk about step forward in which we make quite sensible let's
start with this sample application where a critic cast flying in a sense that we know that a given as a set of a method and some parameters running at the same result every single time so this actually makes sense that the server every single time do why not just store the map the result memory and return that we needed so with this new point class all pretty new dictionary representing the cash and then I override the make model the API calls such that creating new the consisting of the method arguments and keyword arguments that provide instructors see it he is an operation test and if it is of this print out where traveler retrieving a from and also returned result from the catch than it is now possible will do is actually make a call to the server and the result in store for us so thoroughly worried that we have this logic but there's not really a great way to actually cocaine our test line causes there's no option for us right so but it formula stick this new model it actually CreateClient system can override what the name is of the class 1 use and it's inheritance such that now if I can't find that multiply you can see words receiving the result from and then called again FIL actually say that result in states material from history and then I can also look at what the name is as well and see that they were return my super-smart client as in defined so how to do this is actually pretty straightforward other than updating the signature for this function I just added this these 2 lines which sets of people that know class bases were about In order to do that Our it's not walk through this and I create a new point with the model and the customized the strain could name and the cost basis you can see that my super-smart point gets mapped to the class they type this fall in and class cases it met her request finds tuple gets mapped across spaces when tactical so now I wanna come multiply you can see that it's a verbal using the cash client and when I call that is gonna name ill appropriately use my SupersPort for the string the so as I'm going to be standard here but to be honest based root solanine glaring holes except for the big elephant in the room which is there's only 1 API methods so let's actually go fix them so in this final
step and API such that our
engineers of working server and were hard at work in the end new API and insecure so and in order to actually update this API they have to the models as well in order to generate some server code and as a result we got these new make API models for free and to look at some of these API models so you can see that for ad it's very similar to multiply in a sense it's a list of integers and will return to you is a sum of images of the a single subtract is also the perimeter and where the list of integers and they will return to you in the difference of those images as well as in the so now let's
actually do a demo with us for like so if I have right here but I'm going to do
is I'm going to import a couple helper functions person and for 1 that lets me get my models referred frequently without having to with open up the files and loading it and how important they require function so now if I open if I probably model what I'm going to do 1st is i'm going to open up the old model the 1 that only has the multiplying it so that when I predecline you can see that it only has 3 methods
as that generalize make API call it as a model 1 and also has a multiplier so I call multiply now if I print out uh the correct answer 6 but now patrilocal added something that I will be able to do it because the guy models not up to date here so now let me go fix that and use the new age of the API models I got free so I create this kind again and look at all the
different options that our method that don't the available to me now I can see that adds to do had 1 1 against each other it's going to get me to invite subtract on this instead the program show 0 so this is really cool because I want to have the right anymore Python code
once we once the application was built from the ground up and adding new features and feature is not very difficult at all than alanine gets to the kind of conclusion of this talk where then a class
generation I realize my sample application was super simple it was just adding subtracting multiplying numbers together there's I really even need to be aware that this
but what I want to get at is the fact that we start of an application can filter to the ground up such that in order to add new functionality all required was updating a new model for getting model hopefully you're running on the 1 hand have update this new data sources in order pick up function and that's really powerful especially when you have a positive another applications possibly using this similar model such that 1 updates the model can update and different applications that may be consuming dynamic last iterations produces really
robust coding because it's a generalized other could path and all your logit long after that down that as a result
you did very well tested heavily used code and 1 thing I really don't talk about was features and bug
fixes have a lower much wider spread impact in the sense that right now for the client application I wrote it only supports this RPC protocol if I was said something like rest Jason arrest an awkward protocol
what that opens it up for me is to use to be able to support a bunch different you guys and maybe only running on a protocol and in terms of politics is applied it's a bug in a certain date in a certain code path chances are even on was chartered for 1 functionality if another function is using the same code path politics about in there too so I really hope you guys got
a lot all ideas about how plastics Dana class generation words users where we use it in class we have some ideas and using it in your next that product or even for a product to some electoral thank you all for coming if you want to look at some of like the precision code here is again have repository for that here is about a free repository the 1 I can look at the more nitty-gritty stuff on resources incline generating dynamic generation and all that but of course that's were mostly client bank find generation happens but if we just an approach proxy that to that's if you want actually see how clients created there I recommend checking that out but otherwise I'll be here all week so if you guys have any questions about what 3 but of course the alive or about ATBs come find me all the happy talk about it but that's about it is a if you don't the thank you very much
could you contrast this technique that you showed move with the 2 other ways of doing this and the thing that is using that did not move that have to do organ and to using with the classes yeah so many classes that really something you can actually try to reason with and this is in reality type is just another class right so you can define your own metaclasses if you want some specific functionality is out of it as well so like there's a bunch of different other approaches in order to dynamically generating classes is just type is 1 of 1 that is kind of built that filled with for you right out of box anyone else what questions OK and that's wants to think about