Python Descriptors for Better Data Structures

Video in TIB AV-Portal: Python Descriptors for Better Data Structures

Formal Metadata

Python Descriptors for Better Data Structures
Title of Series
Part Number
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Adrian Dziubek - Python Descriptors for Better Data Structures Have you ever wondered how Django models work? I'll present a story of data structure transformation. I will talk about ideas from Django models that I used and how I rediscovered descriptor API. I will talk about printing, serializing, comparing data structures and some other examples, where descriptors excel at making declarative code easier to write. ----- I worked as a developer of a testing framework for a C++ server. The framework tested binary protocol implemented by the server. Most of the work involved testers preparing test cases. The data format was primitive structures -- hard to read and easy to break. Field order and all the data had to be entered manually. At the time, I have already seen the better world -- the models from Django. Have you ever wondered how those work? Step by step, I used the ideas from there to make the structures more friendly and on my way I rediscovered descriptors. I'll show in incremental steps, how: - used keyword arguments to lower signal to noise ratio, - order of definition for sorting the fields, - realized that ` call ` is used instead of assignment, - used ` setattribute ` as first step to extend primitive fields, - discovered that I'm actually reimplementing descriptors, and how it lead me to: - implement printing in a way that is friendly to regression testing, - use diff library for less code and better results, - implement more readable validation. I want to show how descriptors work in Python and how they enable declarative style of programming. By the end of the talk I want you to understand what is at the core of the magic behind field types used by object relational mappers like Django.
Statistical hypothesis testing Polar coordinate system Satellite Context awareness Presentation of a group Mapping Observational study Workstation <Musikinstrument> Data storage device Positional notation Network topology Electric field Energy level Data structure output Form (programming) Area Programming language Default (computer science) Workstation <Musikinstrument> Satellite Inheritance (object-oriented programming) Building Cellular automaton Projective plane Code Position operator Word Process (computing) Field (agriculture) Software Function (mathematics) Network topology Video game Summierbarkeit Game theory Data structure Physical system
Statistical hypothesis testing Functional (mathematics) Regulärer Ausdruck <Textverarbeitung> Freeware Dependent and independent variables Cellular automaton Letterpress printing Database Parameter (computer programming) Value-added network Programmer (hardware) Sic Network topology Electric field Energy level Data structure output Communications protocol Addressing mode Form (programming) Data type Module (mathematics) Workstation <Musikinstrument> Dependent and independent variables Namespace Binary code Projective plane Volume (thermodynamics) Binary file Statistical hypothesis testing Statistical hypothesis testing Field (agriculture) Function (mathematics) Revision control Social class Condition number Object (grammar)
Presentation of a group Set (mathematics) Price index Instance (computer science) Parameter (computer programming) Mereology Sign (mathematics) Different (Kate Ryan album) Pattern language Special functions Library (computing) Social class Potenz <Mathematik> Theory of relativity Interior (topology) Fitness function Attribute grammar Variable (mathematics) Sequence Element (mathematics) Field (agriculture) Configuration space Clef Data structure Server (computing) Data recovery Letterpress printing Student's t-test Bit Attribute grammar Element (mathematics) Revision control Chain Differenz <Mathematik> Field (agriculture) Manifold Electric field Configuration space Data structure Communications protocol Traffic reporting Data type Default (computer science) Server (computing) Gender Projective plane Physical law Line (geometry) Cartesian coordinate system System call Word Film editing Maize Function (mathematics) Revision control Social class Library (computing) Traffic reporting
Default (computer science) Functional (mathematics) Inheritance (object-oriented programming) Inheritance (object-oriented programming) Transformation (genetics) 1 (number) Set (mathematics) Usability Instance (computer science) Instance (computer science) Mereology Data dictionary Value-added network Attribute grammar Mathematics Personal digital assistant Chain Reduction of order Social class Clef Error message God Social class Default (computer science)
Statistical hypothesis testing State of matter Multiplication sign Range (statistics) Parameter (computer programming) Mereology Different (Kate Ryan album) Forest Personal digital assistant Pressure Proof theory Touchscreen Linear regression Attribute grammar Element (mathematics) Demoscene Statistical hypothesis testing Category of being Proof theory Arithmetic mean Field (agriculture) Pareto distribution Right angle Reading (process) Functional (mathematics) Bit Statistical hypothesis testing Element (mathematics) Number Revision control Field (agriculture) Electric field Configuration space Data structure Communications protocol Data type Matching (graph theory) Server (computing) Projective plane Physical law Volume (thermodynamics) SCSI Performance appraisal Scalar field Personal digital assistant Logic Function (mathematics) Revision control Social class Traffic reporting
Reading (process) Group action Moment (mathematics) Set (mathematics) Attribute grammar Field (agriculture) Field (agriculture) Personal digital assistant Revision control Social class Intercept theorem Clef Communications protocol Data type
Point (geometry) Reading (process) Inheritance (object-oriented programming) Insertion loss Instance (computer science) Mereology Dimensional analysis Number Electric field Integer Data structure Social class Pairwise comparison Noise (electronics) Default (computer science) Touchscreen Weight Electronic mailing list Line (geometry) Element (mathematics) Statistical hypothesis testing Wind tunnel Arithmetic mean Network topology Social class Clef Spacetime
State of matter Interior (topology) Line (geometry) Element (mathematics) Software development kit Finite element method Field (agriculture) Process (computing) Different (Kate Ryan album) Web-Designer Network topology Revision control Social class Clef Library (computing) Data type
organizing things and privately unsightly study make like society with their bikes and ride bikes I've make up a graphic and also an ultimate frisbee player if you don't know the game is this something like vitamin among the programming languages the same thing is both of the main reason being words so check it out it's a great thing to do so the that is going to be about how I rediscovered descriptors and so what I did when I was working on tree structures the the tree structure that's basically where you have parent nodes you have some fields inside so many other way so if you have many notes 1 can be putting inside another and we call this knowledge field of for the final and so you can also put value inside the field so the basic data structures like that in general you know the most general the general so so if using this is 1 level tree and so what I discovered when I looked at when I was watching the presentation by friends is that I think many people have seen the descriptors even heard about learning areas but they didn't see the obvious way to use it in other places or to arrive the house default behavior so not anything about descriptors yeah and so have you ever overridden the default behavior that's what it was and so on and so on I understand that it is in the projects and some context and then only about 20 mili gets a cold and how why when I got the opportunity I have written the tree structure that was just over there and so what I learned from that so there's going to be illegal schools have the 1st project project with bullet positioning system it means we have a mobile phone which sends the a GPS position along with what's there it's sees in the radio so the what the base stations of the cell network PCs and we process does become positions of the radio stations we got it is and then when the column away from the 1 down sees only in the sum of workstations we can use it's an approximate position and this is using an assisted GPS when you because otherwise the the form would take like 10 minutes to look into the satellite because he doesn't know anything about this place and the here achieve insight so my life
forms and so that was discussed was project and we had a that's further coincided with written by a specialist programmer and this project basically replaced the mobile phones are also had access to base so we were sending binary of that we're comparing the responses against what we want to see and so we resort checking with other ways that the positions are updated in the right way so there's this heat
and the so there were few problems with the goal and we had started imports because this is just a parameter no another way to deal with the modules he wanted everything basically in the same namespace so every package was in a of itself and was important everything from the submodules themselves the other the thing we did in the with because the project which is another and also and so we had a few things that I think stand out of it not having introspection into but that in such an extent as so we have a repeated names as you can see here there are slots that protects us from assigning go to the wrong name and we have initialization where we call the base alignment and we sell OK so under these names we want such field and we also give those names of the fields again to the fields themselves so that's really hot then again in the top level structure we have all the functions defined in such way that they are more money will only looking at the Trevi log and they're doing the printing this realization the comparable the comparing to other objects and so that doesn't have to be like this it can be automated because we can know everything about this is that the structure but there's also something very peculiar and which I would like to direct your attention to this is the more of a common guy over here to show you this is the called function the called function was used to start the value inside of so instead of assigning where you would lose the field and instead and it had a different you would for the this simple object inside on this actually we have we're calling the field and this way we put the volume inside a field and the field is the user with always methods
so what happens when you have structure like this so you have to be very careful with references and so having variables to collect some parts of data and then constructing this is rather risky so although definitions were not in a long chain of attribute axis where the lines were covered by students from various users and also we had another interesting trusted from the was last where we used and the static methods within in the class so we have recently making a single atom in our molecule which already this is this is on the thing find here we have some special functions of the with some of the data like this version we have those calls as you can see where the cut is the 2nd kind uh where that's actually send the data and so on and that's about it so when you think about substructure
and what is your goal when you're doing such this project I thought my goal was actually to provide for the best so there are the guys that are the learning my work and so what I wanted to do it is given easy-to-use trees so that they can easily define the data and that's how I imagined it so so I would not have this repetition I would have liked to use structure i would know which element goes where and when and when you look at it because those 2 structures were used everywhere there were used in a server configuration that were used in different reporting they were used in fitting the other were using fitting of base so so when you look at it this definition looks very much like an assignment and so on so I thought this is the right way to go and have for a while I like to have something like jungle where you have those fields where again assign values for these can still use them and commented on them so I try to make it like that in the 1st step was to use the keyword arguments and so on so I added those Q with arguments into the Bayes alignments and I use them everywhere in the curative laws we have these ABC element which inherits from maize lemons and I have a few words there and so the 2nd thing I did is I actually use the printing to give the differences so use the diff library to give the reporting to the that's there and this way you didn't have to auditory for 4 different parts but so again this is more of a common guy shows you the place where there's something interesting happening so in the Aeneid we call the get out on the item and then we call it with the value the set of values and so we have this cooperation between calling apparent which is the need 1 and the goal in the child which is the call 1 which intersects the value and put it inside and so on so that was the 1st thing and pretty soon the another probable came and and and uh have to deal with the rest the especially because of the also using it not only my team so what I left there was starving ports and operate the repetition when defining the data I left there is relation because it was using some see what and so on it's because the other presents were using it uh yeah so the next thing I had a chance I took the opportunity to actually implement those other structures and you because we have a new practical to implement and so and I guess I was looking at gender manifold was maybe not only the initialization can be dealt with but also like assigning the values of the attributes so when I assigned I want to be like you to get into the field and I wanted you to be usable still and so on the 1st straightforward way outside is seen was to use the set of which which have in which I have already the default behavior and I redirected the assignment to the assignment of and so in this place I use that so the set of in the when you answer the attributes and then this assignment is redirected to the child and this child is dealing with putting this as you read in the right place in the field and also putting the other fields that can be in the subtree so we are actually had this into separate classes but I couldn't fit in here so this class is basically both the and and the null so each node has this ability to intercept the assignment and each node has this ability to take care of it when it's a child and so we are taking care of the value and we're taking care of the sub fields that are in the and indeed some in the chapter the another thing that from January exponents and it was the creation which let me get rid of the repeating the names and the slopes so we have a creation of the recovery from the last into stands and this way we can easily so the stuff because as you go when you define when you call those glasses interest instantiate those fields they each get uh creation to in the sequence like playing the lines doing the fight and so it was almost there really right it happened to be that's actually I invented something like that of descriptors and but that the just go in a different way so I also have already written in this set of attributes magic methods and I call sign and what the descriptors that because do they actually we have this the get attributes and they use for setting the value of the so those of those 2 being set
things and signed change the
functionality and I might put them instead there's a get of reduce goal or set and so the set is in place of our science so so this is the transformation of happened and basic using the and good and what I think this doesn't seem scary I think people get get scared because they see this call and think 0 my god this must be complicated and why is it here and this is complicated because this is actually a few things makes to get there is inheritance here there's descriptors and there is no other descriptors so the ones in red and again I have to say about those too much 1st marketing being so there is a power and goal and there is a chart of gold and so the red ones are so we have basically checking within the class because we have to do it before we access the in the uh feeling of the instances and if there is a gap sets we call and the set is doing a lot and so now the green part is basically an inheritance chain so we take care that uh we take stuff from a misunderstanding case that is within instance with occurred to take it out of the class and if we take it from there we check again if there is a get matters so that we can override behavior and at the end of the white 1 is a full but which you can use and which is also quite fine too define something like which the full dictionary without attributes when there is an attribute missing you just you can override the behavior of 1 what happens when when you get no such attribute and in the in the default is not up to the error that you get and so so
after years now this and then you make an structure all the inheriting from this base elements you can do things like put some fields in there to give you the ranges for evaluation of some fields to give you default values and all this is safe from right and when you are when you have to find other so this is about definition I was I had actually 2 kinds of the the 1st thing was between the structure and the items so the structure was dealing with combined the fields the was dealing with combined value within the field so so the structure was basically like a subtree the the was like like a single alignment wearing the structure so at least and that is simply the final data say something about the among value and when you have you in for example you from means and the mean value the euro and so on and so on and this becomes more readable but the most important thing is that you actually get this kind of use that let's look at the number of the law of art and so you have this ABC elements which are the version and this has a mind of the so we want to change minor field and we don't have to call it so it doesn't exchanged but Europe when you are assigned to you still can validated you can serialize it's and this way you move all the logic down with this structure so you don't care about what this particular field is doing where in respect of serializing and then in the field of all of us is doing is calling this field to get the serialized states are and then construct the bigger parts so this way you can express the functionality and actually in most cases you don't have to write it again uh in in the fields in those small items of and so the descriptors are basically that the overriding the access to the and this gives us the possibility to To put it somewhere else would things that we're going with like assignment somewhere else and and still use the match field that has some properties and and methods to and so possible almost when this project it gave in the structures that looking so easy and that's I was actually able to redefine the printing in such a way that I was able to print those structures onto the screen so from the previous situation on the right where we had a different way of thinking and different ways of defining that we came to the situation the thinking and defining was same and so we can do regression testing within the project and the test got reading the performance was not because they were able to copying the structure from the screen and just put in place to modify some things and and this regression registered and so and that's about it so I think the moral of the story is that you can prove anything with other countries for example and I like how it's uh Honduras calls and I also this scene it in the morning and some other stuff that there is this forest of principle and I wanted to say that that you should actually when you're doing something you should choose the most bang for the buck and try to not going to get into writing style until you get to new features because it doesn't bring business right and so on and the possibility that I did the new solutions and new features so it's broken don't fix it so do you have any questions I think restaurants and the thing is that the time so so so the story is 1st I had to cover all the low-hanging fruit and I would do keyword arguments in there to initialize the volume and then I realized that when you look at the data is it's pretty much like
an assignment and I wanted this assignment and not only be used in In initialization but also to be able to assign again to that's the the structure so when constructing the data structure I could have some definitions but then I could cooperative that structure and then assign again and qualified for some other still have some basic complaints and modified modified later and this
led me to try to override the assignment which is basically attribute axis blessed be assignments and so I tried to do that we the set of and science and this but then I realized that actually those things are already there so how I think about it and I think this dude descriptor or API that before is mostly about this kind of thing where you try to take over at the moment where somebody's listening to something then take this data foot inside the field or knowledge and then still have this magic now that does all the work for for you but you have the ability to write cold like like
that where you find something and then call the stuff that needed for for you and so and thing 11 into thinking that this group there's another very much
the big called rather over here but this is only due the red part which checks the dimensions are there and calls them the so this is the default behavior together attribute stuff going to invited and for the lecture and noise and I just called those subtrees which contains on the other fields which could be also subtrees uh structure so build a class here so I didn't have to bring them and the stuff that would become 1 for for example the integers 2 numbers to Williams and so this is the idea tunnels and for structures by weighted by government lists trees and so here I'm actually showing something that is the semi-solid 2 because I'm over here on combined value that's what at a loss and combined fields because in in have space and then they want to complicated so I want to say that there is an all you can put a value on it but you can also have substituents and those subjects can also again be an old you can put it right there you can be put some efforts and this way to tree from that in the as I said I think it had a big impact on the performance of the team by means of this 1 where they could actually copy based on the data from on the screen and I think that was the biggest thing from the other stuff was also would change the comparator so we had to structures normally it was walking down the tree and trying to compare each of them and say OK this is different from that 1 but maybe even didn't give a value so I changes to such way that actually use the printer I printable structures and then put the from that of also non differing lines from the data which eventually actually and so this
is this is the point from the from the
cold so some of there are 2 biggest improvement the other improvement was about the configuration and but this was the kind of different tree so so I think it's not related while I no longer work the company so right now I'm doing web development of the states and this is a much better a line of work and so I think we have refined processes with a lot of we have almost 100 per cent coverage we have the the and the we have working scholar and things and not going back over there to fix that anything more for years thank you what is this is because of this


  168 ms - page object


AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)