Effortless Logging - Let the loggers work for you

Video in TIB AV-Portal: Effortless Logging - Let the loggers work for you

Formal Metadata

Effortless Logging - Let the loggers work for you
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Effortless Logging - Let the loggers work for you [EuroPython 2017 - Talk - 2017-07-10 - PythonAnywhere Room] [Rimini, Italy] Logs are your best friends on those late nights when you try to troubleshoot that problem in production that was written by a friend of you who is on holidays at the moment. Logs are the main way to know what is happening at runtime with an running application but as we don’t realize how important they are until we actually need them it is usually an under appreciated part of our development process. This talks overviews the logging module of the standard library and demonstrates some basic best practices and techniques make the most out of our logging when we need it. After this talk you will be fully familiar not only on how to use the logging library but also how it is actually designed, how it works, how to extended further than the basic usage and some sample recipes and configurations for complex applications
Human migration Sample (statistics) Direction (geometry) Software developer Connectivity (graph theory) Operator (mathematics) Personal digital assistant Core dump Hill differential equation Login Configuration space Grand Unified Theory
Sample (statistics) Information Personal digital assistant Software developer Personal digital assistant Sampling (statistics) Login Grand Unified Theory Configuration space Cartesian coordinate system
Multiplication Integrated development environment Bit rate Block (periodic table) Code String (computer science) Software developer Letterpress printing Endliche Modelltheorie Pole (complex analysis)
Category of being String (computer science) Right angle Endliche Modelltheorie Object (grammar) Video game console Line (geometry) System call Library (computing)
Email Computer file Network socket Physical law Letterpress printing Row (database) Object (grammar) Endliche Modelltheorie Instance (computer science) Login Row (database)
Degree (graph theory) Information Computer file Code Structural load Row (database) Information Object (grammar) Event horizon Social class Row (database)
Category of being Word Functional (mathematics) Information Computer file String (computer science) Right angle Object (grammar) Endliche Modelltheorie Row (database)
Filter <Stochastik> Default (computer science) Digital filter Sigma-algebra 1 (number) Insertion loss Sinc function 2 (number) Condition number
Filter <Stochastik> Digital filter Functional (mathematics) Information File format Code Mereology Field (computer science) Hierarchy String (computer science) Factory (trading post) Right angle Row (database)
Filter <Stochastik> Hierarchy Digital filter Inheritance (object-oriented programming) Inheritance (object-oriented programming) Physical law File format Sheaf (mathematics) Row (database) Mereology Row (database)
Hierarchy Digital filter Inheritance (object-oriented programming) Inheritance (object-oriented programming) Energy level Row (database) Endliche Modelltheorie Mereology Abelian category
Category of being Inheritance (object-oriented programming) Code
Computer file Projective plane Information Endliche Modelltheorie Exception handling Physical system Product (business)
Default (computer science) Server (computing) Information Key (cryptography) Software developer Physical law Electronic mailing list Parameter (computer programming) Category of being Software testing Information Endliche Modelltheorie Exception handling Error message God Exception handling
Type theory Arithmetic mean String (computer science) File format
Error message Key (cryptography) String (computer science) Multiplication sign File format Representation (politics) Information Exception handling Error message Exception handling
4 (number) Error message Set (mathematics) Login Information Configuration space Limit (category theory) Login Exception handling 2 (number)
Point (geometry) Default (computer science) Asynchronous Transfer Mode Computer file Linear regression File format Diffuser (automotive) Streaming media Parameter (computer programming) Revision control Protein folding Operator (mathematics) Configuration space Configuration space Object (grammar)
Computer configuration Network topology Revision control File format Configuration space Video game console Configuration space Information
Hoax Code Demo (music) Binary code Login Line (geometry) Newton's law of universal gravitation
Logarithm Blog Source code Energy level Software testing Right angle Row (database)
Computer virus Inheritance (object-oriented programming) Information String (computer science) Polygon Revision control IRIS-T Software framework Information Parameter (computer programming) Row (database)
Filter <Stochastik> Digital filter Functional (mathematics) Inheritance (object-oriented programming) Multiplication sign Demo (music) Revision control Video game Information Social class
Web 2.0 Multiplication Email Sample (statistics) Computer file Personal digital assistant Personal digital assistant Error message Hydraulic jump
Module (mathematics) Strategy game File format Multiplication sign Personal digital assistant Right angle Process (computing) Cartesian coordinate system Thread (computing) Form (programming)
Area Filter <Stochastik> Context awareness Digital filter Context awareness Information Field (computer science) Error message Personal digital assistant Configuration space Right angle Family Gradient descent
Inheritance (object-oriented programming) Code Multiplication sign Login Mereology System call Mathematics Personal digital assistant Buffer solution Personal digital assistant Buffer solution Right angle Freeware Error message Newton's law of universal gravitation
Functional (mathematics) Performance appraisal Operator (mathematics) Multiplication sign Personal digital assistant Order (biology) Login Pattern language Object (grammar) Resultant
Context awareness Weight Sound effect Endliche Modelltheorie Game theory Mereology Host Identity Protocol Physical system Number Library (computing)
Email Loop (music) Computer file Divisor Information Basis <Mathematik>
Functional (mathematics) Process (computing) Voting Information Computer file Personal digital assistant Normal (geometry) Right angle Parameter (computer programming) Product (business)
Scripting language Graph (mathematics) Computer file Spline (mathematics) Block (periodic table) Software developer Shared memory Parameter (computer programming) Variable (mathematics) Sign (mathematics) Process (computing) Integrated development environment Bridging (networking) Different (Kate Ryan album) Query language Right angle Data logger Data structure Quicksort Flux God
Expected value Message passing Process (computing) Computer file Inheritance (object-oriented programming) Multiplication sign Image resolution Order (biology) Energy level Object (grammar)
however 1 as well 1st of all I I think you're much here by components to me and I think this is 1 of the hardest the at conferences because you don't only have to compete with the rest of this because this is where we the beach I'm surprised to see have that many people here as the money's migratory and I work at Bloomberg on a Python developers Our can use operation you're to speak about my company or Spain some spanning so that I can Spain or anything else of qualifying around afterwards so they women's beguiled looking at will have elaborating direction on why why should you look at how looking works how can you use it how do you can figure it
out will do was right to do a little and called them all of their so I'm going to be jumping your for and then some
sample use cases that you can implement on the floor and in the hot sometimes what with high to have something that you so
1st of all welcome to a full talk while matters that embrace yeah I see you I see the connotation as the information you give to the rest of the developers when they're coding and I see log as the information that give to the developers and this about the and when you're application is running this this is 1 of the things that we usually don't don't care that much about right we With all kind of logging trashed and then when we have an issue we we really wish that we had the proper lots so uh why
should you use log instead of a simple just bring the bubble block around uh all your code at 1st of all well some people likely say that you have to use the other buyer that's Ismail that you're not using loading properly add add if you want to go that string that uh using log instead of printing it you see a lot of benefits is most versatile and comfort all these really beautiful how it splits and that's how you look from the what you and you have a multi environment uh it will work you comparative print which have some users to rate trying to come with all the nodes have been my lights serve bear with me and there more thinks that promise that you can really take the flights afterwards and so hello look how loading works in Python haiku copies there any Jabba the pole developer in the room it in the hands wicked does not see the spot France's written that we are not going teach you any we we respect people this is by the current i gets easier for me we fell off Apache 48 that Python was built the title multimodal was built with that in mind following that stuff so we're gonna see all
different elements on on the log model so the 1st of
all is on the log library we have uh lawyers lawyers are your main uh we found to be able to log anything and they
is all you to pass a string and where model is you have with a category you know that this is gonna call and the letter objects and some log
line is gonna go to uh Sastry like to the the console or of horrified right we really
quickly this animal was next and self yeah but he's not just like that so the learning model basically creating a larger record which is an object that is gonna shuffle everything in and that object basically was gonna be locked and you might be wondering about OK that's an object how is the object likely going to the council to the 5 so that's where we
use handlers OK we have seen lower subjects uh like log log log records and other objects and knowing your and the the our the instances at the law and the objects that allow you to uh print things to a file or to um the Council of India out there many lawyers that there are many handlers already in bed built in learning model for example you can to file you can look to go so developed by uh by b because some things to a socket you can send them by e-mail an enormous in 1 it's in the notes stick this lights and so now
we say we would electrical other objects
virilocal or whatever degree 1 load with bustle information those chemicals that they're the code at least in the larger class which is the cradle of our is gonna possible the there which is gonna do its magic spinning at it uh with his coat and he's gonna transform it into
uh as you know it's gonna suffer is not put it into a file into his I and you might wonder how does the handler knows how to put all these beautiful log record information into a file so that's what we have for matters for mothers event makes all the
information in that you have in the lover uh in the log record and they're gonna give you back a strict OK so
we say we have all other objects we call at you for 1 of the category in it with all the information that's the model to the local the local is going to create a larger record below the occurs in the hand the hand the WikiLeaks for matter which will give you back string and the can pass it to a file or a function 1 right so we know how long the word how model
works and above all this we have filters read on and this is just an effective 30 seconds of them uh filters are way to
be able to is then that and I really flexible to to be able to feel the love since somebody find conditions
by default I mean the followers note that might useful but you can create your own 120 and dollars you to attach those ones to add to your loss on the handlers to and filter out some some lovers sigma of 1 to look so we said we cradle we buy the
information it goes to the code of the longer it takes a field so if all the filters down true cold we and we in middle record that we just created it goes to the 100 handle have some filters if that passes then we format below record we get this string and we send it up whenever we decide the right the we also have a larger
OK so what is the cure couldn't so we haven't seen so far have we actually parade lover all the letters are defined that factory function which is called a get loaded ideally should the busses string which is set out both separated that name information so for example here we can see with Britain part and a child and what that means is that and here we are defining the title over which is a child of the bond lover and and so on so we if you were too great a longer 1 OK so now we have the
final might so we we know how to do and now we will at something it goes the longer and it goes through the filters if it's sort a fissile good then we go to the hand section which is we need the log records those the 100 law but then
also eased a lover has a planet with Culpepper did you will go to its parents and any day we don't require following again the same so with that's execute the feel the cold no this is a great people that have fallen in fact so it will just execute this part of the parents so what went and the what the hearer k means that
once you have a went where you lot when you call your handle you gonna call the handles of your parents is not that you're gonna call you parent is if you're gonna called upon the free part also in a way that they purity is that you have set the level on the other we will use the parents so we we know a lot of outright well it's the is to more to
model this is appro as this is the final OK so
the love that lowers can be enabled or disabled and you have the category which is also in hand the that this is more or less the whole workflow hidden from the inductive accommodation
which have this this other 1 which is this compensation OK so we know that we have a
lotta works right it's let's see how how we use this this huge the implantation of this huge based code so here we can see how can
we lots of stuff this is how you asset we just look things this is the what you what not to how you would that's I think this is and I
this way of separating the concern of how you want things from what you think yeah usually find separate them so we just login as we get a the we use name so you can use the name of the model which it's it gets so name is uh it will give you the full of many here that your system for simplicity have project 1 and a folder called uh folder 1 ending a file phi 1 is will be product 1 and if
there I models for the 1 and 3 but it 1 for the 1 I want so it keeps you by default a really nice indeed if you have a key that follows you file character we do we not abide so do we have different categories the biking for error and critical will we have added by the law that we is usually want to see maybe north tests or maybe not development server we directly some we gets an exception we do love exception is not a new category diseases error but passing x and exceed 4 is gonna love all the information it has uh like all this phrase like undue all information it has about the exceptions so we do this we can do the same with other categories by just enforced forcing it to 400 and you're wondering 0 my god I love this phrase like thing I want to see this well in a lot from Python 3 . 2 Italy's don't quote me on that and you have this thank you for a large that you can pass we'll of which will also be the list the article
now some some things I wish they knew when they start the things they should but the with log in
this if you do this even if is the body's gonna form of and it's gonna use this has a competitive competition because for might industry needed you don't use it so just as the them plate and then
uh posterior mean that you want to put in the place holders if you don't like this kind of string uh quotation and from others have different types have is a have don't them in more wow all so really quickly
and you're doing this this is a problem this what think think so he didn't this you probably will see
errors like at the end of a terrible her has happened data and know how many times have you have seen this in In problems so quite quite you've vital funding in your attention exception and log that exception but what is happening is is looking this string representation of exceptions which 40 key errors it is the key that is missing this extremely forceful faded
so what you probably want to do is as example which we saw before 1 you can see a 4th i'm gonna
generally quickly uh we spoke limit of this and now we know the
set of fear he said before our you
5 seconds this is how you can configure the whole login and this is in indirect so this is the basic uh this is 1 of the ways you think of it the
have those that operators political is going up into the to go so but if you pass of islands in the be the summer point to a file you can go over all the parameters with basically this diffuse defaults if you like the folds et
. 2 other ways to configure the linear model uh what that 3 you can configure called like just reading all the objects Monteleone uh wire and then you can also configure it by a a configuration file which I don't like so I want so you and the 3rd 1 is you can configure reducing that dates conflict so the county
elicit opacity tree with all the options but you can get even higher than this of the ability of saying that the logging configuration Django basically what you're doing and that's the reason that the comfort and I usually use I I personally prefer uh the comfort and I'll put this config in a gamified although the unify landing at a the in FIL so uh here we can see uh where we you can define on the complete for matters and
lost Our lovers down more indentation the so as I
want to show you the that's all I said is true and them so yes because there are not enough technical difficulties right so and after they would want so we're going to just going to see how how our log line is gonna go through all the code we weeks binaries so we're just going for logging we do basically basic on fake we get a lotta I just printed and this is my cheat so we use Python I found 2 . 4 and don't care in 5 and 3 minus and the
and Our you see that it really wants you they are 0
and OK but so was the 1st
thing we see where the check if Dolores enable for that level right the test because we we have a on or informed and then we go to the innovation on this blog which has all the interesting stuff is getting a striking for source file above it's the coloring from and now we're creating these love record that we spoke about right so next next you can print with parenthesis because he's 3 the the current so here you can see how this logarithm looks like
it so it has all the information that can capture around and also there are the string and the parameters of the past we go next I
Ch polygons that story uh start again don't do them with a single mom and about about next next known don't I told you so we go inside next next next next next week greater longer they literature record here goal and now we going to handle which is not
handle class but the handle method within the function and what do we check if the always enable not
and we take all the filters because they return true
then we're gonna call all the handlers I'm gonna I'm running out of time this is really interesting I
recommend you do it after was in this life but there is still more content of of and we have 5 minutes and the
I promise you will through always spoke but there is no and that clicker watch OK so I see
some use cases of his might so I'm going to
jump over this 1 and basically defined as multiple handler something a I have my web is a center my to myself for the for the critical there's my e-mail I have a single file with all their error looks because I may want to see in 3 months likely happen with it's why for the 4 underbite only available in that I will just rotate them every 4 dates for example so I keep only for the of what's
you can be Jason where the where would we do the Jason formatting in the handler now in the form of the right so this is how a you could you could have your application through obligation because you know young height and you know that last should not be human readable I you prefer to
send it to some kind of postprocessing lots that will do have my strategy of the data Jason there's cult of tried to upload it and I don't have time to the and is until
there's the context you are over Python 3 . 2 there now method you can uh configure to capture all the calls for undeveloped
rigorous created and you can some extra context information if you are before 320 some confirmation is you can use the field you can over use the filters but information so why is this and this is using ionizing scary stuff there is gonna be passed through like all the lots of in the past 3 years because we show how the filters can be added to both lover some kind and you can enrich it with something and they put it in your from other why would you put it in the locker or enhance if you want to do it for all the lots not everyone in the same family descent on understand what he said In handle area right because you
did in their longer it will only happen for the full that part and 1 right because we so that as with any archaeological gonna call all their handlers and the hand of your parents but you're not going execute the code of your part the cake and more cool stuff about buffering how many times we have had an error you have a lot in their I you wish you had only change that had the bad luck to you all right so this is what this smart buffer and the reason they do is is is gonna
have about I is gonna buffering and that the the prettiest lock in this case what I wanna be detects an error is gonna not only look bad error but also the previous you've so you can do for example whenever there's an error free in the last 20 and the magnets
have time for order so that doesn't to get operations have you want to do if you want to let's say that you want to get all mean that some if some you 1 start you want to log their result of a functional
mean that other you would good because you might have for temple the at school I hasten but which is that about that we have is we have blocks in about you can use this kind of uh pattern which is that you create an object that will call the function only when you call this street and that
said take away stop illegal logging the number so there was the talk about and not really like a lot of model is SIR uh it's amazing you can you can build on top of it as much as you want and is really I lost the weight what and how and and how is this a separated is it always all the parts of the system to collaborate on log so for example use you saw how we did before this thing of rich with there and the context of the letter so something you akin do it you that is not only that you log in you know lots are gonna love that global scary stuff that also the library that you call analogous so yeah really yeah is a frightened there what and how I and that so we have
2 minutes for questions that in effect few hasn't president a reaI you deplores were my game you go over and in the middle of everything so we have a question here it seemed that the logging the model does
a lot of you what what is the performance in fact city so likely something really scary is that if for example this is why so that more or less you can split there the what and how if you have 1st sending an e-mail this is the best on hand the microphone is everything is have happened synchronously so it's really dumb impact your your performance if you if you for example enable at divide of and your dividing something in a for loop is really gonna go to the buffer and into the file onerous information so light don't don't just ignore it be careful with let's read that have high buying but have measure in normal the normal the performance forget that the but the body yeah so it it depends what you are right so you can see it depends on how much we lot but he he does have an impact on how many milliseconds is gonna take the depends so for some of you are doing this sending an e-mail is gonna take longer if you looked at if you look of of how long it takes to a provided basis for you this so there is another question and some what's your opinion about to
use decorators for for logging
is by this thing where you put the greatest or uh on a function will log when he gets in and when it gets out with all the because I heard that it's and great use case for the parameters set of here yeah of saying that it might be coming depends on your use case I'm I'm worried you know I really high I personally prefer to write my own once because the greatest it feels more like I'm dumping this here's something I wanna get information 1 and it whilst I really like to think of my love my lots I really want the single you know I I wonder lots to tell me a story right under what I log files and see I am doing this I want to see value because when you do the crater traits for today but in 6 months we have a problem production and you see this function name which is do stuff um what was this thing going right because you go to the polls but you can say that that person this is not this is having which might not have any deal with finite you can say that step and that's really what thank you the and did
you solve the problem with the multiprocessing voting because it's always on the issue yeah you can you can do it that's at the 2nd half of behind it and he was you will have to do like uh you cannot use the the normal you cannot use something that is long file like it you cannot use that a norm handler that will look to a file from 2 different process because they will
share there the file a script to write this book for a studio about for example the problem is that oceans that might actually the graph that I mean what I thought that they should be for a lot of donors I would love I TCP to another process which is likely gone up a little too and the to the story I think excuse me yeah so a lot of the poorest bridges everything into a logging for right of so just done the different variables and parameters so what of approaches to solve this problem is to use struck blocks how did it if you know the structure of the sharing struck flux so so sort of reality and get so high so you view on solving this problem so the developer just put everything in the I need to use structure lying God and so so this fragment where I'm looking to see
Beckley comes from the uh cost shoveling it and then you can Petrossian trusted afterward and choose what you want to see how the once said thing is I usually I'm still very old-school using log files another they like it but the my environment is but we have and at home at work for some queries things like splines Splines them more convenient thing so you have the infrastructure to do like strike the logging where you can love Jason and them faster than only to them but you want go for it you're going do 1 up at home that they're gonna log files or even your company does have that you and your speed breaking for log files you want a sign but I think this is cool I think if you know what
we do today as well we look into a file and then we we pass it if you can you can say that it makes both the and to comment on me if your comment on they don't don't have a a quadrature says do stuff so 1 best practice I heard was always have at least of hierarchical naming that you can see what I just saw at up to your comment that don't just have a lobbying message that says do stuff at best practice I heard is preserve at least enough levels of hierarchical naming on the object that you can on the biggest single 1 and I had a question for you and what is the guarantee you conformant you can form a millisecond resolution timing was a actual guarantee that the order and the accuracy uh apart from level accuracy of the timing and mocking messages especially the dust ripple of to multiple parents so did did you you as the mysteries did you ask if it guarantees that things will go to the filing the same or the the log for the the pressure in in supposedly it's guarantee that in a single threaded process you stuff is in order but the millisecond timings may still be inaccurate so what's your level of expectation that the actual accuracy of the millisecond should you what can you trust is it good to the nearest 10 millisecond 100 and amateur a similar to the US the maybe the might knowledge of yet even if you don't mind days can learn so we came later OK and thank you so much for everywhere and that outperform idea