Write more decorators (and fewer classes)
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33685 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Social classSoftwareIntelExecution unitGame controllerInheritance (object-oriented programming)Process (computing)Metric systemSoftware engineeringState observerReal-time operating systemMultiplication signDemosceneConnected spaceLocal ringWebsiteInheritance (object-oriented programming)Presentation of a groupBlogPattern languageGame controllerWeb applicationOffice suiteProduct (business)Social classSoftware testingCodeMobile WebUniqueness quantificationMoment (mathematics)MetreObject-oriented programmingSelectivity (electronic)Computer animation
02:07
Message passingFunctional programmingScheduling (computing)Web 2.0Order (biology)View (database)RootPattern languageWeb-DesignerWebsiteSynchronizationTask (computing)Process (computing)Queue (abstract data type)Configuration spaceMobile appParameter (computer programming)Object (grammar)CASE <Informatik>Uniform resource locatorLogicSoftware frameworkSocial classServer (computing)Real-time operating systemAreaGame theoryBus (computing)Computer animation
03:53
Social classSocial classInsertion lossUnit testingObject (grammar)Order (biology)Software testingCASE <Informatik>Functional programmingInterface (computing)LogicRight angleSubsetDistribution (mathematics)Software frameworkGUI widgetNoise (electronics)Electronic signatureQuicksortNetwork topologySystem callSubstitute goodCodeMetre
05:42
Order (biology)BuildingCodeSocial classWordInterface (computing)CASE <Informatik>ImplementationProjective planeLogicResultantSoftware frameworkBitRight angle
06:49
Functional programmingParameter (computer programming)ResultantCASE <Informatik>WeightLevel (video gaming)Lecture/Conference
07:58
MereologyFunctional programmingFunction (mathematics)Multiplication signInterpreter (computing)Neighbourhood (graph theory)
08:46
CASE <Informatik>Game theoryParameter (computer programming)Functional programmingMereologyMultiplication signInterpreter (computing)Speech synthesisProper mapGradientWordCodeFunctional (mathematics)Cache (computing)Ideal (ethics)ResultantHardy spaceWrapper (data mining)Computer animationJSON
10:45
Multiplication signSoftwareCASE <Informatik>Software frameworkBitFreewareAssociative propertySocial classPoint (geometry)Degree (graph theory)Parameter (computer programming)Computer configurationFunctional programmingVideo gameHypermediaObject (grammar)Type theoryMobile appRootRule of inferenceSinc functionHardy spaceComputer animation
12:02
StatisticsMonster groupMetric systemBlogSoftware frameworkCodeMetric systemInterface (computing)Group actionoutputMultiplication signCodeScalabilitySoftware frameworkStability theorySummierbarkeitOrder (biology)Service (economics)3 (number)Semantics (computer science)Computing platformWordContext awarenessEvent horizonBlogMatrix (mathematics)Social classDimensional analysisAttribute grammarTimestampLoginFunctional programmingResultantLogicInformationPresentation of a groupLine (geometry)DatabaseMonster groupFunction (mathematics)Data dictionaryReal-time operating systemStreaming mediaTime seriesComputer animation
13:55
Multiplication signTimestampTask (computing)Dimensional analysisOrder (biology)Forcing (mathematics)State of matterType theoryMetric systemFunctional (mathematics)outputFreezingEvent horizonData dictionaryPartial derivativeLoginTelephone number mappingTupleCounting
14:37
Monster groupMetric systemInterface (computing)Field (computer science)Social classCASE <Informatik>Attribute grammarFunctional programmingOrder (biology)Line (geometry)CodeDefault (computer science)Event horizonMonster groupLoginInheritance (object-oriented programming)Dynamical systemMoment (mathematics)Run time (program lifecycle phase)Software frameworkFunctional (mathematics)Scaling (geometry)WeightForcing (mathematics)Source codeComputer animation
16:10
BlogMetric systemMonster groupLogicCharge carrierSoftware testingSpacetimeSocial classLoginMetric systemLogicMonster groupInterface (computing)MetreFunctional programmingMultiplication signLine (geometry)Process (computing)Content (media)TimestampException handlingEmailCASE <Informatik>Electronic mailing listString (computer science)InformationEvent horizonBitLoop (music)TupleField (computer science)Figurate numberIterationCodebuchLogarithmCost curveScaling (geometry)Matrix (mathematics)Computer animation
18:38
Graphical user interfaceMereologySet (mathematics)Monster groupLoginSocial classCASE <Informatik>Line (geometry)BitComputer animation
19:31
Inheritance (object-oriented programming)Endliche ModelltheorieModule (mathematics)Login
20:02
Inheritance (object-oriented programming)Social classInheritance (object-oriented programming)AdditionDimensional analysisAttribute grammarMetric systemMatrix (mathematics)Module (mathematics)CASE <Informatik>Functional programmingLine (geometry)LogicPattern languageField (computer science)Computer animation
21:12
BitModule (mathematics)Monster groupDimensional analysisOrder (biology)Metric systemAttribute grammarAdditionSemantics (computer science)Point (geometry)Symmetric matrix
21:45
Social classInheritance (object-oriented programming)Social classProcess (computing)Multiplication signInheritance (object-oriented programming)LogicElectronic mailing listGaussian eliminationModule (mathematics)Parameter (computer programming)Attribute grammarCodeComputer animationJSON
22:56
Software testingSocial classMultiplication signNetwork topologyMetric systemMetreResultantComputer animation
23:33
Different (Kate Ryan album)Social classFunctional programmingType theoryService (economics)Run time (program lifecycle phase)Figurate numberAttribute grammarRight angleMultiplication signData miningInternet service providerGoodness of fitComputer animation
24:53
BlogMonster groupMetric systemInheritance (object-oriented programming)Power (physics)Interface (computing)Module (mathematics)outputFunction (mathematics)Social classEndliche ModelltheorieAbstractionInheritance (object-oriented programming)Metric systemLoginLogicInvariant (mathematics)Computer animation
26:00
BlogMonster groupMetric systemInterface (computing)LoginBitMathematicsFunctional (mathematics)Social classParameter (computer programming)CodeDifferent (Kate Ryan album)Set (mathematics)Equivalence relationMonster groupStatisticsComputer fileMultiplication signCASE <Informatik>Functional programmingState of matterPhysical lawMetreComputer animationJSON
27:14
Metric systemBlogMonster groupInterface (computing)Process (computing)Functional programmingCASE <Informatik>MultiplicationMultiplication signSuite (music)Parameter (computer programming)Social classOrder (biology)Block (periodic table)Object (grammar)LoginoutputGame theoryNetwork topologyMetric systemLogicAlgebraic closureFunctional (mathematics)Hardy spaceMetadataString (computer science)Electronic mailing listMetrePatch (Unix)CodeInterpreter (computing)Online helpComputer animation
30:31
Inheritance (object-oriented programming)Physical lawSocial classCodeFunctional (mathematics)Phase transitionResultantPartial derivativeLoginFigurate numberFunctional programmingInheritance (object-oriented programming)Computer animationJSON
31:03
Inheritance (object-oriented programming)Functional programmingLogicMessage passingMetric systemField (computer science)Interpreter (computing)Social classAttribute grammarParameter (computer programming)CASE <Informatik>Dimensional analysisAdditionJSON
31:48
Social classInheritance (object-oriented programming)Social classBit rateFunctional (mathematics)CodeTask (computing)Inheritance (object-oriented programming)Computer animation
32:19
Software testingSocial classWikiFunctional programmingDescriptive statisticsMilitary baseRandomizationString (computer science)Parameter (computer programming)Type theoryInheritance (object-oriented programming)MetreComputer animation
33:04
Function (mathematics)Functional (mathematics)Software testingInterface (computing)LoginRun time (program lifecycle phase)Monster groupStatisticsFunctional programmingProduct (business)Bit
33:44
Physical systemTorusBitForm (programming)outputCodePhysical systemSocial classInterface (computing)MereologyMonster groupStatisticsFitness functionVideo gameRevision controlSlide ruleComputer animationJSON
34:30
Inheritance (object-oriented programming)Social classTheoryComputer clusterSoftware frameworkSocial classInheritance (object-oriented programming)Closed setCASE <Informatik>Functional (mathematics)State of matterPhysical systemComputer animation
35:47
MaizeOffice suiteReal-time operating systemProcess (computing)Streaming mediaFacebookTwitterProof theoryNumberNetwork topologyBlogPlastikkarteComputer animationXML
36:22
Presentation of a groupMultiplication signLecture/Conference
36:53
Goodness of fitFunctional programmingTouch typingOffice suiteSoftware frameworkCodeProduct (business)PlanningMathematicsRepresentation (politics)Network topologyType theoryTracing (software)Universal product codeOpen sourceLecture/Conference
Transcript: English(auto-generated)
00:05
Hello, thank you very much for coming My name is Antonio. I'm a software engineer at Yelp. I work on the metrics team So what I currently do is working on observability and real-time metrics and real-time processing
00:21
Yelp mission Okay, the whole mission is to connect people with great local businesses. We have more than 127 million reviews on our website and On average every month we do the 26 million unique mobile devices accesses Via the Yelp web app and 84 million unique visitors via the desktop
00:43
With a desktop website we have offices all over the world And we are more than 400 engineers working both on the product and the infrastructure So a new introduction before to start This talk has been inspired by the talk stop writing classes by Jack the other rich
01:02
I highly recommend you to watch the talk is great and By the blog post the controller pattern is awful and other objective-oriented heresy by Evie again great blog post highly recommended All the code that you're gonna see in this presentation and trust me is going to be quite a lot Is available on github at that URL so all the code runs and as test filter Crony to play with it. It's for you
01:29
Now I'd like to start from the very end of the talk with the main two takeaways so They might make little sense to you at the moment, but keep them in mind that maybe at the end
01:41
There will be like, you know Cool ideas that you can use at your day job So the first one is let users utilize all Python features instead of just inheritance and The second one is go for decorators when your classes have only one method and are instantiated only once
02:01
alright So what's this all about why we're here? Well, this is the reason How many of you have ever used celery? a Few nice nice. Well celery is in a synchronous job queue based on distributed message passing It is focused on real-time operation, but also scheduling this kind of stuff. So the way
02:25
you define a task in celery is Defining an app using the server API Then calling the decorator task on the app object that you just created in order to decorate your task function So this is the first example on their website and this is the pattern then I'm gonna show you today
02:47
This button is very very common actually and is used by a lot of cool frameworks. For example, you know flask, right and flask is a micro framework for web development and The way you define logic for your routine flask is exactly the same as you define task in celery
03:05
you define a flask app and Then you call the root decorator or the app object you just created with some parameters in this case The URL you want that function to be executed for and then you just give it the function that is going to encapsulate the logic
03:22
You want to execute? This is a very common pattern for web frameworks In fact, also the pyramid frameworks use it in order to define views Which are basically the same concept of roots in flask as you can see you just call the at view config decorator
03:40
You pass you pass in some arguments and then the function you decorate it It's gonna be the logic that is going to be executed This is again the first example on their website Now, what about classes as you notice the name of the talk is bright more decorators and fewer classes
04:00
So if you ever try to write a test in Python It's highly likely that you have read about unit test unit test is now in the summer living Python 3 so I guess it's the recommended way of writing tests in pythons and The way you do it is you define a class inheriting from unit test or test case and
04:22
You define some test methods that are gonna have your assertions in order to understand if your code is doing the right thing Because what at least what you expect and in case you want to execute some setup or teardown methods what you do is that you override a meter with a specific signature in this case set up and
04:41
Here you may do all your setup logic in this case. For example, what the example in the Python documentation does is instantiated a widget object and attaching it on the fly to test class now Pytest, which is an alternative framework to write tests avoid the usage of classes
05:05
Substitute in them with the decorator. So if you use the pytest dot fixture decorator in order to decorate your setup logic Then you don't need a class anymore. And in order to access the object that is returned by your setup
05:21
Setup function what you do is that you pass the name of the function to your test Functions and it works magically. So as you can see the interface is much more leaner You can use all the function all the concepts from functional programming and everything and well at the end of days just shorter
05:41
so That's cool. At least that's what I think so. I wanted to understand how you implement this stuff Like how do you write these decorators in order to avoid using classes or to build clean interfaces? Well, what do you do when you want to understand how some code works? You go on GitHub and read the source, right?
06:01
So I went to the pytest project and this was the result which is Kind of complicated. So I said to myself well pytest does a lot of magic Maybe pyramide is gonna be easier Nope It was you know a bit worse maybe but yeah again pyramide is a mature framework and there's a lot of business logic and
06:23
And there's a lot of you know corner cases So celery, celery was simple, right? Nope, celery was the worst of all like as you can see it's a wall of code I have no idea why they put all this code probably because they wanted to handle a lot of corner cases
06:40
So my reaction after seeing the celery implementation was this one And I say to myself decorators are hard. I'm not gonna use them. So we believe this sentence is true No one, that's cool. That's cool. Well, maybe someone there Well, it's actually not entirely correct a few months later
07:05
I looked more into how decorators works in Python and Basically the way they work is that you define a function Which receives as an argument the function you want to decorate and the result of the decorator function is
07:21
Assigned to the function name That's pretty easy, right and In case you want to pass arguments to your decorator the way it works Is that the decorator function is going to be called with the arguments before and then the result is going to use It's going to be used as a decorator as we have just seen and in case you have more than one decorator
07:44
The way they get executed is that the closest one to the function that you want to decorate Gets executed first and the result gets passed to the decorator this upper level So actually I changed my stance after that And so I decided decorators are pretty easy to use but maybe I just are too bright
08:05
Well, this is not entirely true as well So this is the simplest decorator ever made I guess As you can see what it does is just when decorating the function it prints
08:21
decorating function Pretty useless, but it's a decorator and it works in fact if we execute it in the Python interpreter That's what we get the first time we import the function the decorator function is going to be executed and decorating function is going to be Is going to be on the output and then every time we execute the function nothing happens to just execute the regular function
08:45
So after this, maybe you can think that decorators are easy to write Again, that is not entirely true So in case you want to implement a decorator with arguments as we saw before
09:01
the way you do it is defining an inner function in your decorator function and The inner function is going to be the real decorator the one that we saw before so then we're receiving the function as first argument while instead you can use the arguments that you pass to your
09:20
Ideal decorator in any other part of the function itself So the way it works For this decorator implemented over here is that when the function gets imported It prints decorating function with bar and buzz which are the two arguments that we pass to the decorator And then when the function runs the function runs without any modification at all
09:46
Now if we want to do something more useful with our decorators like for example Changing the way the function is called or saving the result into a cache and the kind of stuff We need a proper function as well. There's going to be in defined inside your decorator function
10:03
And these wrapper functions receive an argument for arguments the arguments of the function you want to decorate so piece of advice If you want to write code that is reusable just use args and args So you need to care about what the function you're going to decorate is taking as parameter and the way
10:21
This decorator function works. Is that again when executed in the interpreter? It decorates the function and at import time it prints decorating function and now Every time we call the function The code that we've written in the wrapper function inside the decorator function is going to be executed before
10:40
the function get executed for real Now with that in mind My stance on decorator is that they're neither easy or hard They're just a bit tedious to write you see quite a few corner cases to remember what once you know how to write them Once you have this free example in front of you, but it's kind of okay
11:01
I mean doesn't take much time and in fact if we look at a flask framework Which is a great piece of software in my opinion. This is the way they implement their decorator Which is not much different from what we saw before and it's much much simpler than all the others decorator we've seen and so far So basically in the flask
11:22
Class which is the one we use to instantiate the app object We define a root method which takes some parameters and since it's a decorator that takes some parameters We need an inner function and is ignorant function is the aqua decorator and in the actual decorator What flask does is?
11:41
defining an endpoint Using the options parameter that we provided and then adding a rule for the end point to the app returns the function and returns the inner function So My talk was supposed to stop here to finish here But then I noticed I booked a 45 minute slot instead of a 15 minute slots
12:01
So I decided to go for a real-life example So basically we're not gonna see decorators for a while what we're gonna do is looking at the class based interface that these example implements and Then we're gonna look and how we can rewrite it with decorators. So the example I want to take is that monster
12:22
Start monster is a real internal framework that we use at Yelp to extract real-time metrics out of logs where logs are just stream of events and This is not really important for the presentation. We just give you much more context So the way that monster works is that it consumes log from Apache Kafka
12:41
Which is a distributed streaming platform usually used to you know Put log like carry log around your infrastructure and then it does some custom logic which is implemented by the users via an API and Emits some metrics to signal effects, which is a third-party services
13:01
They will use in order to visualize and analyze our metrics and Kairos DB Which is a fast distributed scalable time series database on top of Cassandra that we maintain internally So To reiterate the input for that monster are logs and the output are actually metrics
13:21
This is an example of what some monster does So in this example, we see that the input is a log line, which is JSON formatted with some information Then some user code gets executed inside set monster a step motor step monster emits a timing metrics So basically just you know timing the result of a function or something similar
13:42
which has a name a timestamp a value and the dictionary of attributes that we call dimensions in the metrics word Now, let's look at the metrics interface the metrics interface. It's composed by three little pieces The first one is an enum defining the two kind of metrics that you can
14:05
Emit which are basically just counters in order to count events and timers in order to time them Then we have the metric definition We use the name tuple here and a metric as we say as always a name a timestamp a value
14:20
dictionary of tags which we call dimensions and the type and at the very end we define two commodity functions in order to Give the user the possibility to instantiate a counter and the timer without providing the type Using partial. Okay. So let's look at the input interfaces that so the logs
14:42
So this is where this is where the class is already come in play basically, the way a lot class is defined is Defining a decoder function which is going to used in order to decode your log line in this case, we provide the JSON decoding as default, but you can just override this default just setting your
15:06
custom decoding function as a class attribute and Then the base class which you are seeing at the moment just define the code class method which is going to be used to decode log line and The second thing is the name of the log you want to tail. So start monster is a dynamic system
15:25
So the way it works Let's just inspect at runtime all the classes defined in the framework and it checks for this name field and It just tails from Kafka all the logs named this way
15:40
So if we want to implement a log to say to start monster tail this log for me That's the way we do it. It's pretty easy. It's reasonable So we just subclass the log base class and we just provide the name in this case events Now in case you want to provide your custom decoding function the way you do it
16:00
it's just that you define your function at the very beginning and then you Set up the decoding function as a class attribute of the log Now, let's look at the start monster interface. So the real code is the start monster interface is based on the concept of trigger a trigger is a class which
16:21
encapsulate the logic to extract metrics from logs And this is the base trigger class. It does very few things. So the first one is defining owners for every trigger, so we always know which team is responsible which logic and What it does is asserting the owner
16:42
Then this field is going to be a list of strings. There is no way for us to encode this information in Python And so we decided to start writing a little bit of a tutorial so people could read it and you know understand what they need to pass to the function to the to the class and then Every trigger needs to implement a digest meter the digest meter is what is used
17:05
To transform the decoded log line into metrics Actually in the it's supposed to be a generator and it's written in the tutorial as well So this is the chorus at most is all The logic that really matters and that's the process function that receive the log
17:24
We are consuming from the line We just consumed and then it iterates over all the triggers and it tries to yield from The digest meter called on the entry so the decoded log line and in case of any exception It sends an email to the owners
17:44
Now this is the way a user basically Instruments that's monster to consume a log and then to emit some metrics out of it So as you can see you first define the log as we saw before then you define the owners for your trigger in this case just me and
18:02
Then you define your digest function your digest meter What it does at least for this trigger is just yielding a counter saying we saw an event with this time stamp And we just count one Here it is another example you define the resource usage log
18:20
You define your trigger with your list of owners and then in the digest meter here. We loop over Just a custom tuple as time and you time and we emit two timing metrics based on the content of the of the log line So this all worked fine We published a tutorial and we started to have some users
18:42
But then we realized that our users are not just regular users They are engineers, and you know what the problem with engineers is right you give them a Lego set And then they build a flying dev stars out of it, so The first question we got as soon as we released a tutorial and start monster was how do I narrate from a base log class?
19:05
Well, I didn't think about it so I Answered well Let's let's do so so you define a module starting with underscore and Here you define your base log class in this case what the base log class is doing is basically providing a decoder for Apache
19:27
log lines and now With a bit of magic we tell start monster not to look for these Underscore prefix it modules, so we don't start consuming logs that don't exist
19:43
Okay, that's cool. So that's the way you use this kind of base classes in your regular module You just import them and you set the name of the log you want to consume pretty easy You update the tutorial and we thought we were okay well It was wrong because as soon as we updated the tutorial
20:02
We got this other question How do I narrate from a base trigger class because now I saw you I can inherit log class So I want to inherit trigger classes as well Alright Didn't find they didn't think about it either So what we decided to do Was basically reusing the same pattern so in these same underscore
20:24
Module you define your base trigger with some logic in this case. For example, you Ask the user to set the metric name and the endpoints that are written in the log line as class attributes and then you also ask him to implement your
20:44
Get additional dimensions metric method for example Where you can actually customize the matrix we are going to meet and then you provide a digest function Who does some logic and calls the get additional dimension method and
21:01
Uses the end point and the metric name that you define as class attributes and at the end yield the counter So with a bit of magic we say to start monster again just don't look for these underscore modules for triggers as well and
21:22
That's the way you use them So you just import them and then you just set those owners endpoints and metric name attributes for the Subclass and then you implement the get additional dimensions in order to customize your metrics and that's it. It works So again, we updated the tutorial. It started to be a bit long but users were happy for like 24 hours
21:47
Because then I got this question How do we narrate from two trigger classes at the same time? well, I Didn't think about it either but you know Python supports multiple inheritance. So we started to occur around it and
22:05
our answer was You just don't So let's keep things simple you implement your base class with some logic and then you In air it from me in your other base class and you define some other logic and then
22:25
What you do is that define another base class? Well, you still inherit from it and you define some other logic and then in your real module you just Subclass from both of them and provide to them the same parameters as class attributes
22:42
So I get it this is not ideal but as long as it's documented it's gonna be okay, right? then after more or less one week, we got the first code review in for adding a new trigger and This is the first question we got but how do I test my trigger class because you want to write tests, right?
23:03
Well, guess what this time I was prepared You remember pytest, right? So the way you can test your trigger It's just defining a fixture in pytest when you return the instantiation of the trigger you want to test and then what you do You call the digest method on the trigger and you test as a certain metrics
23:26
Is in it's in the result of the digest meter easy One minute after he asked me But how do I test my base trigger class? Right, what's the difference I don't understand well if you look at this class
23:45
It inherits from service based trigger and service based trigger in its in it Assers that the derived class define a metric name and the same time It in here is from trigger which in this in it
24:01
Assers that it defines owners as well so You cannot really instantiate it because the assertion is gonna fail I was very puzzled by this like I stayed one day thinking about this even under the shower I didn't find any good solution, but then a colleague of mine Came up with a very good idea
24:22
Let's use in the pytest fixture the type function that pythons provide so with a type function, you can instantiate a subclass of a certain class at runtime and Also, you can provide class attribute as a dictionary
24:42
that worked and We updated tutorial it started to be long enough that people didn't want to read it. And yes, I guess it's okay but Let's try to make things better, right? So I'm going to demonstrate you that my face before using decorator was this one and after switching to a decorator based interface
25:04
You change it up to this one so Looking back and the submaster module. We got logs as input and metrics as output. So what's missing here? What is missing is the trigger? What is the trigger? So we say the trigger
25:23
It's a class that encapsulates the logic to transform logs to metrics Well, just get rid of it it doesn't have any actual Reference to the real model. There's just an abstraction that we constructed ourselves. Why do we need it in the first place?
25:42
no idea, so let's remove it and Let's not force people to utilize just inheritance for the interface Let's give them the power to use all the features that Python provides So in the new interface the metrics interface stays exactly the same no modifications
26:04
The logs interface started already to change a bit So instead of using classes only via subclassings we decided to give the users the ability to just use classes either were as they were meant to be used by Python so you can instantiate your class this time and you pass the name of the log and the decoder function as
26:26
parameters of the init method and No more session over here and the decode class method now became simple method the only real difference from before is that
26:40
Now every log class defines an empty set of functions those functions are what are going at are basically the equivalence of triggers in the new interface Now this is the way you define a log in the new stat monster Just use the class. That's it. You don't even need the file for each log. I just you instantiate it with the name and it works
27:04
And in case you want to pass in your own decode function you just call the class and you pass the parameter That's it. It's already much cleaner But now let's look at the stat monster interface so first of all the process function stayed exactly the same but became a method of the log class and
27:24
Again, we the only thing we did was substituting the trigger with the functions set that is attached to the log class Now this is the way we wanted the user to utilize the new interface The way you define some logic in order to transform your logs to metrics is now importing your log
27:46
object that you created somewhere else and then Calling a decorator method on top of the object also using another decorator to pass in the owner's metadata to the function and just
28:02
Define the function the function is exactly the digest meter of the first trigger we saw without any modification you just remove all the class thingy and Basically The way the decorator is implemented. It's the simplest decorator ever
28:22
what it does it just receive the function as an argument and Just adds it to the set That is already attached to the log class That's it Now this is even simpler than flask and in case we want to provide some more helpful
28:46
Decorators like for example this one which makes you able to register the same function for multiple logs at the same time what we can do is implementing a decorator with arguments and the arguments is basically the list of logs and In this case what we need is just an inner function where we iterate over the log
29:05
And we call the register method once again because remember it's a decorator, but at the same time it's emitted and In case we want to implement this owner's decorator to give some metadata to the function What we do is exactly the same as before
29:21
we pass it to the handlers as parameters of the function and We just attach on the fly the handlers to the function because every function in Python is also an object So I know that some people are religiously opposed to these so in case you don't want to monkey patch your function You can always define a dictionary as a global and then use it in the closure of the upper function it works exactly the same
29:48
Now this is the way these registered decorator is used as you can see we import two logs and we pass two logs to the decorator and Actually, I also want to point out that the owner's decorator removes these ambiguity about oh
30:05
Is the owner suggest a string or is the owner's a list of strings like if you want to pass in more owners you just pass in more arguments and the Python interpreter is going to check that for us as We don't need more assertion because the Python interpreter at import time is going to check that we created the two logs
30:22
We want to register the function for The code of the function once again is just a digest speed of one decorator we saw before Now let's go back to the questions We got so how do I inherit from a base log class? smiley face If you want to subclass your log class you can
30:43
But if your idea is just passing at the coding functions and then avoiding to pass it over and over and over What you do is just to use partial and then you instantiate the new logs using the result of partial Next question, how do I inherit from a base trigger class?
31:02
Well, no more base classes. No more assertions. What we do is just defining a simple function we define a function yielding some metrics and having some attribute as the attributes of the class before as parameters in this case the endpoints and
31:21
Even the additional dimensions method is now just a parameter for the function And then we call it That's it. So we just yield from the function within our You know logic and that's what we do. We don't even need methods. We don't need some classes. We don't need to remember to Assert for class attributes. It's all done by the interpreter for us for free
31:48
But I'll do any rate from through trigger classes double-ep about is one What you do is just you define two functions. This is The digest method or the first trigger that we saw when we saw the example of the multiple inheritance
32:06
This is the exactly the same code of the second trigger that we saw before You just call them both You don't even need to ask me just call functions Now let's go back to tests
32:21
How do I test my trigger class now? Easy as before even easier. You don't even need the Python features anymore. It's just a function you call it. That's it and How do I test my base trigger class That's the thing. I'm the most happy about
32:42
It's basically another function. You just call it again and the way you mock Yourself it's just passing some random strings like for example test as parameters. You don't need to use the type Function that Python provides you don't need to remember about inheritance and everything. We're just passing a string
33:04
So but now that we removed all the assertions, how do I make sure that all functions have owners? Well, we write the test What the test does is calling the collect function Which is the one used by stat monster to collect runtime all the logs
33:22
We iterate over the logs and we just check that they have the owner sub tribute sets So well That's even better than before because we transformed a runtime error To a test that could fail before we even push some function without owners in production
33:42
So it's everything we indeed saw implementing these decorators and changing a bit interface only for users sake no, actually for The most part it was the same fishy interest Because you remember they talked a lot about the import system of stat monster the thing that makes that monster understand
34:03
Which logs it needs to consume with classes it needs to execute and the kind of stuff Well, you don't need to understand the code, but this is the import system in the new stat monster Just look at it and compare it with the old one It doesn't even fit in a slide and actually this is a version that I wrote in Python 3.6
34:24
so it's much more compacted the Python 2.7 version that we used and That makes me very very happy now Closing up The main takeaways of the talk were let user utilize all Python feature instead of just inheritance as we saw and
34:45
Go for decorators when your classes have only one method for example, the trigger class has only the digest method and They are instantiated only once Actually, we never haven't even
35:00
instantiated trigger class ourselves the framework was doing that for us and Now instead what you do is you instantiate your log class whenever you want you even want to subclass it You are free to do it. You don't want to don't do it and All the triggers disappeared. They're just functions So just to clarify I'm not saying that we should move entirely out of classes from it
35:25
Just use classes when they're useful in this case The log class is still a class because there's some states and it has some behavior And it's used as class are supposed to be used in Python not only for inheritance
35:40
so when it makes sense go for the greater when it makes sense go for classes and That's the end of the coke Remember that we are hiring especially for our office in London and Hamburg So if you want to work with Python and on big real-time processing streams and kind of stuff just come to talk with us at our booth just in the big goal and
36:03
Don't forget to follow us on every social network possible Facebook Twitter and GitHub. Yes. Nowadays. GitHub is a social network and Please read our engineering blog where a lot of smart engineers. Why better than me? Speak about all the cool stuff that we build every day
36:23
questions Thanks, Antonio Presentation we have time for few questions No questions, please
36:46
Did you run into any specific situations where switching to using decorators made it hard in practice To debug certain problems. So where the decorator obscured the actual problem that you were trying to get to that
37:06
That's a very good question actually, so you may have used decorator they're very complicated like to do a lot of stuff and They are then very difficult to debug because they change the way the function is executed And then you got this weird stack traces and you really understand what's going on. Well, my true sense of this
37:25
Is that as in any? while always writing code You want to keep it simple. So my true sense is Decorators should always return the function and touch It's like the function should be able to be called
37:41
Without using the decorator and the decorator should stay really small So the frameworks that I showed you before a very complex decorators, but they are very much your framework with a you know Good open source community and a lot of features just came into it So my true sense especially for your own Production code is keep your decorator simple and avoid to do a lot of stuff into them. I hope it answers the question
38:10
more questions to Antonio No We thank you and Tony for your presentation. Thank you for coming