Type Annotations in Python 3: Whats, whys & wows!
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33675 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2017147 / 160
10
14
17
19
21
32
37
39
40
41
43
46
54
57
70
73
85
89
92
95
98
99
102
103
108
113
114
115
119
121
122
130
135
136
141
142
143
146
149
153
157
158
00:00
Computer fontCodeSoftwarePosition operatorCycle (graph theory)Video gameProof theoryFormal grammarError messageHacker (term)Software testingMathematical analysisProjective planeSoftware bugComputer programmingFreewareGoodness of fitPhysical systemFunktionalanalysisUnit testingCuboidObservational studyINTEGRALReading (process)CodeCASE <Informatik>Formal languageFluid staticsComputer clusterStrategy gameMathematicsStatement (computer science)Point (geometry)outputPhase transitionLatent heatCompilerSoftwareLine (geometry)Different (Kate Ryan album)IdentifiabilityOrder (biology)Revision controlInterpreter (computing)Network topologyBitVariable (mathematics)ConsistencySoftware testingArithmetic meanMultiplication signVector spaceComputer iconMathematical analysisWebsiteMereologyRight angleBlock (periodic table)NeuroinformatikForm (programming)Fitness functionLetterpress printingFormal grammarPattern language
07:14
InformationFormal grammarPauli exclusion principleTheoryFunction (mathematics)Parameter (computer programming)Variable (mathematics)ArchitectureSound effectRun time (program lifecycle phase)Interpreter (computing)Electronic mailing listCodePersonal digital assistantPauli exclusion principleFunktionalanalysisPhysical systemCodeVariable (mathematics)Core dumpParameter (computer programming)Interpreter (computing)Electronic mailing listBoolean algebraIntegerRun time (program lifecycle phase)MultilaterationMathematicsState of matterOrder (biology)Goodness of fitRevision controlTheoryElement (mathematics)Arrow of timeComputer programmingFlow separationBitString (computer science)Data dictionaryMappingPiMereologyFormal languageError messageIntegrated development environmentDecision theorySound effectModule (mathematics)Arithmetic meanSoftware developerAdditionUniform resource locatorNumberMultiplication signDifferent (Kate Ryan album)Data structureoutputRight angleCommunications protocolAreaGradientSocial classPattern languagePie chartPoint (geometry)Network topologyEndliche ModelltheorieTypprüfung
14:28
Installation artDrop (liquid)CodePiDomain nameCodeMalwareNeuroinformatikSurjective functionHypothesisSoftware developerGame controllerConfidence intervalLibrary (computing)Revision controlLocal ringDomain nameMultiplication signComputer fileVirtual machineProjective planeNumberCybersexOpen sourceData structurePrice indexFilter <Stochastik>Self-organizationInformation securityWordTranslation (relic)outputSystem callView (database)Drop (liquid)WebsiteDrum memoryXML
16:56
CodeSoftware testingLine (geometry)CodeBuildingProjective planeData dictionaryLine (geometry)Electronic mailing listOrder (biology)Interface (computing)Product (business)TheoryLibrary (computing)Pattern languageNetwork topologyLogicSoftware testingCASE <Informatik>Source code
18:06
Digital filterScripting languageSoftware testingFingerprintScripting languageElectronic mailing listString (computer science)Run time (program lifecycle phase)Library (computing)IntegerNumberInterior (topology)FunktionalanalysisElement (mathematics)Software testingDampingPiInstance (computer science)Order (biology)Error messageBit rateUniqueness quantificationFitness functionCodeSocial classJSON
19:22
Function (mathematics)CodeModule (mathematics)Parameter (computer programming)CodeModule (mathematics)Variable (mathematics)Data storage deviceFunktionalanalysisMereologyPiOrder (biology)Endliche ModelltheorieXML
19:59
CodeModule (mathematics)Function (mathematics)Revision controlError messageError messageFunktionalanalysisSequenceCodeLipschitz-StetigkeitComputer filePiVirtual machineIntegrated development environmentDampingInformationRevision controlLibrary (computing)IntegerResultantSocial classEndliche ModelltheorieSoftware testingObject (grammar)Reading (process)Product (business)Quicksort1 (number)Weight functionGroup actionArea
22:01
Social classAttribute grammarInstance (computer science)Menu (computing)Execution unitParameter (computer programming)Glass floatDigital filterCodeModule (mathematics)MereologyString (computer science)AdditionPhysical systemRoutingState of matterScripting languageMultiplication signProjective planeVariable (mathematics)Object (grammar)Parameter (computer programming)Electronic mailing listBlock (periodic table)CodeDampingDependent and independent variablesFitness functionResultantIntegerLibrary (computing)Social classCrash (computing)Conservation lawBitError messagePoint (geometry)Software bugNetwork topologyPiMessage passingArrow of timeFeedbackTraffic reporting
25:22
CodeModule (mathematics)FunktionalanalysisElectronic signatureEndliche ModelltheorieResultantIntegerMultiplication signModule (mathematics)PiDampingWordJSON
26:08
CodeComputer fileDirectory serviceElectric currentBuildingMereologyCodeInformationAdditionFunktionalanalysisEmailComputer fileRepository (publishing)Directory serviceVariable (mathematics)ResultantInterface (computing)Library (computing)Message passingRight angleSystem callClient (computing)Integrated development environmentAreaPulse (signal processing)
28:17
Computer fileBuildingCodeMereologyLibrary (computing)InformationCodeTouch typingFunktionalanalysisSocial classResultantDot product
28:58
Pairwise comparisonBuildingExecution unitCodeCanonical ensembleVariable (mathematics)Module (mathematics)Revision controlComputer fileSource codeSoftware maintenancePauli exclusion principleTupleParameter (computer programming)Function (mathematics)Standard deviationLibrary (computing)Core dumpSoftware developerComputer virusGeneric programmingSocial classElectric generatorRepository (publishing)CodeSoftware maintenanceModule (mathematics)Computer fileSource codeGeneric programmingSocial classCASE <Informatik>Electronic mailing listDifferent (Kate Ryan album)INTEGRALResultantRevision controlElectric generatorVariable (mathematics)Branch (computer science)Intelligent NetworkRepository (publishing)Element (mathematics)Interpreter (computing)Software testingLink (knot theory)BuildingInformationSynchronizationExterior algebraMathematical analysisHierarchyData structureSystem callEndliche ModelltheoriePairwise comparisonMathematicsNetwork topologyPoint (geometry)Group actionLine (geometry)Similarity (geometry)Data conversionDiffuser (automotive)WeightPattern languageBitXML
33:39
Computer fileOrder (biology)Software testingNumberScaling (geometry)DiagramProjective planeCASE <Informatik>Medical imagingCodeComputer fileToken ringQuicksortProduct (business)Network topologyComputer animation
35:07
Repository (publishing)QuadrilateralBootingVariable (mathematics)Function (mathematics)CodeSlide ruleAmerican Vacuum SocietyCodeMilitary baseInformationProjective planeMereologyGoodness of fitConnected spacePhysical systemNumberDefault (computer science)Range (statistics)Parameter (computer programming)Position operatorIntegerRun time (program lifecycle phase)Interpreter (computing)Demo (music)EmailFunktionalanalysisHacker (term)Theory of relativityDesign by contractProxy serverBuildingRepository (publishing)Arrow of timeControl flowOnlinecommunityDampingSlide ruleLetterpress printingLink (knot theory)State of matterAbsolute valueDataflowTypprüfungParticle systemTwitterBasis <Mathematik>Moment (mathematics)System callDrop (liquid)Physical lawAffine spaceCuboidCore dump
39:59
CompilerBytecodeDirection (geometry)PredictabilityPlanningVirtual machineMathematical optimizationFunktionalanalysisSource codeDifferent (Kate Ryan album)WebsiteOrder (biology)Product (business)InformationCodeProjective planeLecture/ConferenceMeeting/Interview
41:08
Order (biology)Theory of relativityRight angleSystem callCodeSoftware testingInformationLibrary (computing)Execution unitScripting languageLecture/Conference
42:08
Form (programming)Positional notationElectronic mailing listQuicksortAbstract syntax treeDirection (geometry)IntegerCodeGoodness of fitExtension (kinesiology)Endliche ModelltheorieProjective planeMathematical analysisPhysical systemMultiplication signInformationGame controllerComputer programmingFluid staticsFrequencyData modelFunktionalanalysisSystem callConstructor (object-oriented programming)Interior (topology)TypinferenzMultiplicationControl flow graphPiSingle-precision floating-point formatModule (mathematics)Graph (mathematics)DataflowDampingLecture/Conference
44:27
Figurate numberIntegerCASE <Informatik>Product (business)PiNumberLecture/Conference
Transcript: English(auto-generated)
00:04
So good morning everyone. I'm pretty happy to be here, and I'm also happy that there's such a large interest it seems in type annotations Probably some of you already heard of type annotations in Python or maybe even use them so that's a good thing because type annotations are actually a pretty old feature already and
00:25
I wanted to give this talk because right now. There's a lot of activity in this field and As I personally have worked a lot of static analysis, and I'm interested in everything that is connected to code quality I found is an interesting and very good opportunity to actually have a look at
00:41
Type annotations again and see what happened in these ecosystems in the last three and four years and spoiler alerts And it's quite a lot that happened So I have three learning goals For you and this talk first I want to explain to you why actually we need type annotations or we might need them and
01:04
How they made their way into the Python language secondly I want to show you how you can actually use those type annotations and type hints in your own code base and For that we're gonna Have like an integration project where we take a small code base and actually try to put type hints in there, so
01:21
then I want to Show you also not only how we can use type hints, but how other people are using them in the wild so in public projects and To see if actually somebody else is even it's doing something with them So finally then I want to see I want to show you what else you can do with type annotations if you're not
01:40
interested in type hints okay, so The first question you ask yourself is probably Why do we need type annotations at all? I mean after all our code is just running fine without them right so to answer that we
02:00
Will just have a look at How and when we find bugs when developing software, so if we ask ourselves What's the first opportunity to find a bug when we have a new software project, and it's actually not It starts even before we write the code and the design phase you know because we could just in a perfect world
02:20
Think about what we want to build Like think about all the edge cases and all the input variables that we might have in our problem and then with that perfect Specification just write the code and have something that has no box in it and of course this would be nice But you know this almost never happens Actually there are some system that can help you to write systems that write software that is almost bug free
02:43
but most of these do not apply to many of the daily projects that we are working on so Unfortunately we have to do something else The next opportunity to find a bug in our code is after we have written the soft the code and for example you could
03:03
Have somebody else look at your code and try to find box in there so-called code review and actually this is a pretty good Way of identifying box if you believe the empirical studies that exists, so if you're not doing this already You should really start doing it and as an alternative Instead of having a human read your code you could also feed it to a computer program so-called static analyzer and
03:26
Have that static analyzer try to find bugs in your program and for Python also several solutions available for this that you should use and they can help you to identify some of the problems that can occur in your code and Okay, the last point actually doesn't apply to Python
03:40
But it applies to other languages for example if you have C++ or rust you have a compiler That takes your code and puts it into an actual executive executive of the form and while doing so it also performs a lot of so-called type checking and Using this type checking it can tell you before You run the program if there's actually some problem in there because there are some types that somehow don't fit together
04:04
And as you know from your Python experience The Python interpreter in that sense is very lenient because it allows you to almost Run any code at all any code you want as long as it Corresponds to the Python grammar, and I mean you probably all know the situation where you just make a small change in your code
04:21
and then it just You run the program and it works fine in the beginning But at some point it crashes because you might have forgotten to include include like an import statement or some of your variables that you're using Deep inside your program undefined and so this is a situation where like a compiler would really come in very handy because it would catch those bugs already before
04:41
Okay, so if you can't rely on that what can we do to find other bugs in our program? well, we can build a little exercise yard for our code and Tested and that's what we're doing a lot of your unit testing functional testing and integration testing with the goal of actually running As much as of our code as possible and running it through as many different situations possible in order to
05:04
Identify possible edge cases or things where the code might break and this is also best practice So if you're not doing this already, I'm happily recommend you to start with that because it can help you to find many bugs in the program and So the last method that often is not discussed, but which is quite useful to find bugs is of course
05:25
Not find them, but have the customer do that And that's what I call it the code I'm so called the banana strategy because if you know if you like shipping bananas you actually Take them off the tree when they're still green and unripe and then you let them ripen at the customer side
05:40
So to say so with software you can do the same you just ship unready code and then your customer kind of discovers the bugs For you and then you go ahead and fix them. And again, I'm not saying this is a bad practice or something It happens a lot and but it's not always what we really want because we ideally we want to deliver software to the customer That is working as expected and not have him or her discover the bugs for us and the Python community
06:02
Also thought that and that's why They decided to do something which is a bit like implementing a compiler for Python By having a system that makes it possible to actually check our programs for internal consistency Before we run them so a type checker for Python if you want and that's what type hinting is about
06:29
Now the problem is of course There are already like billions of lines of Python code out in the wild and you know from the situation we had when we tried To convince people to switch from Python 2 to Python 3 that it's not so easy
06:42
To get people to rewrite their whole code in a new system So if you would just introduce like a new version of Python that would require everyone to like Statically type all of their code This would be a very good way good way to drive people away from Python or have a fork in the Python interpreter Maybe so that's not the strategy that the Python community pursued and instead they did the so-called gradual typing
07:05
Which is also pursued by other languages like typescript for example, and which lets you instead of having to Statically annotate your whole code base just lets you add Type annotations or type hints to like small parts of your code where you think it might be worth having them
07:20
So if you see here, we have some external code for example in gray would be unannotated code in green It would be annotated code and then you can write your own code basically and you can also decide For which parts of your code base it actually makes sense to have these type annotations because maybe the code is really critical or maybe you really really want to be sure that there are no mistakes or no errors in that code and
07:42
So this is nice but of course now you need a system or way to also interface your own code with the external code that might or Might not have type annotations and later. We will see how exactly we can do that in Python. All right, so Type
08:01
annotations by themselves are actually not a new feature and So I want to give you a short history of that The first change that led to this current state was the introduction of so-called function annotations Which was done in 2006 so more than 10 years ago And it was formalized in this so-called PEP
08:22
Which is a Python enhancement proposal and that's a formal way so to say to get new things into the Python interpreter Or the language in general and this PEP 3107 introduced a new syntax that would allow you to Write these type annotations in your Python code without though. I'm giving any meaning to those annotations
08:41
so basically what you could put in you could put in there whatever you wanted and the Python interpreter would just ignore that and This feature landed in the Python 3.0 version in 2008 After that not much happened for a while, you know The community just wanted the users to decide what they wanted to do with that feature I think honestly users didn't do much with it if at all anything
09:04
So in 2014, that's why the community again decided to Introduce some new Feature so called type hints. This is PEP 484 and this is actually the PEP that I gave meaning to the annotations that were introduced in 2006 And there are also some other PEPs here 482 and 483 which
09:24
Explained the theory and the ideas behind the type hints So if you're interested in that you can check that out by just replacing The number and inserting that into into the URL that you find below so this new feature set landed in Python version 3.5 and
09:41
Allowed us for the first time to really have a formalized system of using type hints in the in the language Now you could think that might be the end of the story But actually no there are still some open and active developments going on here. For example in 2016 there was another addition to the syntax of Python via PEP 5.2.6 which introduced a
10:03
Small change but a new possibility to also annotate a different type of thing and namely a variable because that's something that The writers of the original PEP in 2006 didn't think about but which turned out to be quite useful If you want to do good type checking So that's something that they implemented just like last year and which landed in the new version of Python 3.6
10:24
And there's still some ongoing changes here. For example PEP 5.4.4 which I find really exciting Which is about so-called structural subtyping via protocols But which we are unfortunately not going to talk about it about in this talk because this goes a bit too far so as you can see, it's still a very active area where new things are created and
10:46
It's at a state right now where you can actually start using it So now we probably say okay stop with the theory. Let's show me the syntax already so here's an example in all its glory and you might already spot some of the
11:02
Unusual things here or some of the annotations in the code base so you can see here. For example, we have a so-called argument annotation Which annotates the input value to function and If you annotate the input value, of course, you also want to annotate what comes out of the function So the return value and this is done via so-called return type annotation
11:23
Which you see here on the right and this is done by this arrow syntax and the final element is this New so-called variable annotation which was introduced in Python 3.6 Which also allows us to annotate a variable inside a function body and you might ask yourself. Why do we need that?
11:41
Well, it's actually useful because here What we have is that we have we initialize a result variable with an empty list and for a type checker And this would be a problem because it would only know okay This is a list but it wouldn't know what you would actually want to put into that list And so this we can tell to it by saying, okay We want to have a list of integer values and that's the main motivation where this feature was added
12:04
Alright, so if you take a step back now and actually think about what those type hints or type annotations are doing Well, if you just would run that code with Python 3.6 it actually would do not much nothing almost, you know Python the Python interpreter would just take the annotations put them in a magic variable
12:24
Which is called underscore underscore annotations and then just go happily running your program without any runtime effects at all So by putting these annotations you do not change the behavior of your Python program per see Now you also saw maybe that we are importing this weird
12:43
module there which is called typing and The motivation behind this typing module is that if we have a type checker if you want to implement one in Python we of course need a way to specify what types you actually have and We have some built-in types like integers and floats where you could do that Directly in the code base, but often we have types which are a bit more complicated
13:04
So for example dictionaries that map Strings to integers or lists of given other types, you know and the typing module helps us to specify these types by providing us several classes and several ways to generalize these classes and adapt them to our own need when we are typing type annotating our
13:22
Program you can see here two examples We have a mapping of a string to an end and the third parameter and we have this list of ends inside the function body So, okay now you're probably asking yourself, okay if the Python interpreter doesn't do anything with that annotations then how are they actually getting used and the answer is and they're getting used by external tools
13:44
So there are several of them available. For example, the most popular one right now is probably my pie But there are also some IDEs like pie charm that can use these type hints and these External tools actually do the heavy lifting and type checking so they go through the Python code Look for the type annotations and then try to make sense of them and try to find problems in the code
14:05
And this decision of having that as an external tool and not as part of the core language was a deliberate one And I think it's a good compromise in order to have a system that is easy to evolve But I think eventually a lot of that functionality might also land in the core
14:21
Python interpreter or at least as a tool that is shipped with Python but right now you have to install an external tool for it and Now we're going to look at one of these tools and that's So called my pie and my pie was written by Yuka late to Salo I've probably horribly mispronounced his name right now
14:40
Who's as far as I know working at Dropbox now and developed many of the ideas for my pie during his PhD thesis and The good news is that my pie is strongly pushed by team at Dropbox under the lead of keto van Rossum which should give you some confidence in the Stability or like at least the active development of the tool that's pretty easy to install you can download via pip
15:03
And then just run it on the files which we were going to do in a minute and here on the right side I also included their website, which you can see it's quite active and they have like several releases almost one release per month and As you see from the version number the tool is still evolving but it's already functional enough so that you can actually use it
15:25
Okay, so If we want to use it we need of course in example code base And I choose a project I did for the an open source project for the German cyber security organization It's called floor floor is a German translation of the word bloom and you might have heard of bloom filters
15:45
so what the libraries is doing it provides such a bloom filter and What does a bloom filter do? Well, it's a it's a so-called probabilistic data structure to which you can put in some values And then you can later ask the bloom filter. Have you seen this value or not? And
16:01
If you have put in the value the bloom filter will always remember it and always say yes I have seen that value but what can happen is also that it misremembers something so you it might answer. Yes I've seen it with a value that you actually didn't put in there and this might not seem terribly useful but it's actually quite neat because we Did it says oh we have lots of indicators for example domain names
16:24
Corresponding to the mains that like control malware or download malware onto computers and we want to check those domains On a local machine for example So what we can do is we put all those values like a hundred million into a bloom filter Which is then about depending on the probability that we choose of this
16:41
Misremembering about like a hundred megabyte large and we can just copy that to the machine and ask the filter every time Have you seen this domain or not? So pretty useful Library very small code base and we're gonna try now to add annotations to that and see how that goes So I choose a code base and as I said because it's really compact It's less than 200 lines of code, but it actually does something useful
17:03
So which I always find better than using like a toy example just to show how the stuff works So it's actually something that people use in production and the nice thing about it is also that it doesn't have any external Dependencies beyond the standard library, so we don't have to type annotate or interface with any external code And again, this is good because it makes it easier
17:22
I don't want to give you the impression though that it's always that easy to annotate code So, of course if you have large projects, you will need to put in some more effort to do type annotations But in order to get started, I think it's a good project to just see how it works in theory But you can also actually do that with a larger code base, of course and it doesn't also have many exotic types
17:43
So most of the types that we're using in there are built-ins or simple compositions of buildings like lists or dictionaries So it's not too hard to understand what's going on I hope and Last thing is that we keep that code base compatible with Python 2 and Python 3 because some of our customers they can't Use Python 3 yet
18:00
so it's also an interesting case to see how type annotations can actually work for a type of Python 2 code base and In order to test that we need we need a script, of course So we have like an example script here which imports our library Imports it creates an instance of this bloom filter class, which we initialize with two
18:21
values so normally it expects a value n which gives the number of the elements that are in the filter and Probability P which gives the false probability rate so the the probability that the filter will miss remember a value that we didn't actually put in there and The number of elements should be an integer, of course The probability should be a float and you can see here. We instead put in a float and an int in the wrong place
18:44
So this is something that would blow up so to say in runtime and then secondly We want to check or add a value to the filter again We have a problem here because our filter expects a array of bytes and what we put in here is a Unicode string so this again would be an error and
19:01
lastly we want to do something with the fingerprint of a given value that the filter returns and What this function actually returns is a list of integers But here our user kind of assumed that it would be a string and he tries to or she tries to add another string to That which of course again is a problem. And now let's see if we can get my pie to find these problems
19:22
So in order to add type in store code base We would go through the code function by function and then try to add annotations to the arguments and to the return types, so If possible or if needed we also might need might add annotations to other variables inside the function bodies or inside the module
19:44
to help my pie actually make sense of them if they are not used as a part of a function argument or return value and to do that we need to import of course the necessary types from the typing module and then If we have done that we can just run my pie and try to make it happy so to say All right, and the result of that is shown here
20:03
So you can see that we added some type annotations to our init function so we specified that n should be an integer that P should be a float value and We have a function also to read a bloom filter from a file Which of course expect something that you can read from and there's also a way to express that using the typing model
20:21
Which is this IO class here And so what we're doing it to be telling the read function that we are expecting an object Which is readable and which returns by it that the function can then use to read into the in the bloom filter values Finally this contains function which checks if a value is in the filter expects Of course a byte sequence so byte array and returns a boolean value, which we also annotate
20:45
Okay, and after doing that we can just run my pie This is shown here. So you can see on the left. I specify this environment variable, which is called my pie path Which we need here because my pie actually has its own search path So if we want to check some external code
21:01
We need to tell my pie where it can find that code and in my case I have my Python libraries here in the lips folder and then I run my pie I also tell it which Python version to use in that case We want to use Python 3.4 because we don't use any variable annotation so we don't need 3 6 and it's not running anywhere on my Ubuntu machine and
21:20
then we tell it the file that we want to check in this case floor test of py and Hopefully we would get them out all the information about the mistakes that are in that file, but To my surprise what I got out of there instead was this like really weird cryptic error message Which really made me doubt if I actually understand anything about type hints at all
21:43
So it was really like when I do this I said, okay Andreas, this is easy. I got this I just gonna annotate that stuff and I said what? Why by it but this is this should work, you know and so I was like digging in the documentation and I tried to do some debugging and finally, I found the reason why this is not working and the reasons actually that I broke my pie with that and
22:05
The problem here was and that I had in the bloom filter class a variable which is called bytes and which my pie Somehow mistakes for the built-in by its type So when I'm in it try to initialize that class here my passings, okay bytes This is a variable, but this is not a valid type. So I'm gonna throw an error and then just crashes
22:25
So as you can see, it's still not perfect But on the other hand as they always like the debugging of your system is only as good as your most stupid user and Probably I kind of lowered that bar from my pie by naming my variable as a just like a built-in type
22:42
But the good news is the guys who are developing my play They're really awesome. So they I posted this issue here on Monday I got in response to that in like 30 minutes and last Yesterday morning. It was already fixed and merged into the master So as you can see, they're really like very are very active and they really take feedback and bug reports very serious
23:04
So if you have a bug that you encounter when running my pie I encourage you to just go to the github issue tracker and post it there and they guy and they will definitely Help you or resolve that in as fast as possible So having that fixed a quick workaround by the way was to just rename my variable with an underscore
23:22
And then it didn't complain anymore. So the results of that we can run it on our script again and Amazingly as expected it finds all the tree problems that we put in here So you can see when we initialize the bloom filter. It says oh, this is an incompatible type I'm expecting a float here. Sorry an end here, but I get a float and there's actually also an arrow for the second message
23:45
Which I don't show Then we try to add a value to the filter with the wrong type and there my pie also complaints at all He should and should put in a byte objects, but I actually got a string and finally it says oh here You try to add a string to a list of integers and this is also not possible
24:02
So this is pretty pretty cool. I find I mean it might not seem like a big thing But you have to realize that just by adding like a few snippets of additional code to our project We made it really much much harder for a user for users to use our code the wrong way So make we make it much easier to like have a good state in the system and to avoid problems when?
24:26
Using that library as a user, but also of course make the code better for ourselves, you know So normally this could be the end of the talk and we could go to our home But if we just push that to master It wouldn't take long for customer complaints to roll in because as I said some of our customers still
24:46
Rely on Python 2 and since type annotations are not supported by Python 2 and adding them to the code base would just break it So that's bad news huh, but can we do something about that? yeah, actually we can and because my pie or type hints have a
25:02
Second approach for adding these type annotations to code base which are called type comments So type comment is similar to a type annotation in the sense that we specify Function arguments and return types as well as variables, but instead of doing that in the code We do it as a comment
25:21
So if you would again do the same annotation as before of our code I'm only this time using comments the result would look like this so you can maybe I hope you can see them So we have one comment after the init function which starts with this magic word type in the column Which tells my pie that this is a type comment and then afterwards it contains the signature of the function
25:42
so it says okay, we have an integer value a float value and we return nothing and You can do that for all of your functions again as before and the nice thing about is you don't have to use The new type annotation syntax what you still have to do though is as you can see above Import the relevant types from the typing module Which is possible though because the typing module has been back ported to Python 2 7 so we can just install that
26:05
Via pip for example Okay, now we're fine, right? So actually no because there's also Often code that we just can't change, you know, maybe because of the code is upstream somewhere We don't have like the right rights or it's an external library that we want to use
26:23
So they are really out of luck both of code Comment with type comments and with inline annotations But luckily again, there's a third approach that we can use for scenarios like this, which is called a stub file And I've shown you here an example of such stub files. So here we have our
26:41
Directory with the code file. So we have our init.py the filter.py which contains the code that we saw earlier and Some helper functions and what you can see here that we also put some new files in there which are ending with pyi and these pyi are of course the stub files that we're talking about and
27:01
Mypy when going through a code base will also try to find such stub files and it has its own search pass for that so it will look for example in the current directory in The search pass that you give it via this mypy variable environment variable and also in another place Which is called a type shed Which is a github repository that contains many type informations for for example built-in Python functions and also third-party libraries
27:26
So and the thing is if mypy finds a pyi file It will only load the information from there and will completely ignore the corresponding py file And this means we can instead of like changing our code Just add this py file pyi file and add the type annotations there
27:44
So again, it's the same The same operation as before we just start with our code and then we like go through it But now we in addition to adding the annotations. We also remove any actual code that is doing stuff, you know, so
28:00
You could I've probably think about this as like writing a header file for Python I don't really know if I like that concept or the idea but it's very similar to have a C or C++ header file where we specify the type information and have an actual Python file that implements that Interface so to say and the result is looking like this So you have here again our class with the different functions and you can see that we added our type annotations
28:27
but you can also see that Instead of having the function bodies written out. We just put this ellipsis there So there's three little dots which tells mypy that this okay There's a function body coming afterwards, but it's not important. So we don't put it there
28:41
And so this is a very compact very nice way to specify the typing information for code that we can't actually touch and We can as a set store that in various places and have then mypy look for that information when we try to change To check code that actually uses our library for example So these are three approaches that I showed you and now you might wonder
29:02
okay, which one is actually the best or how they compare in in practice and To get an idea of that. We push these three new branches to get up and we have a Travis CI integration there which automatically automatically tests our code against the different Python versions and as you can see The results are here
29:20
So for the if you look at the last elements where we have our stub files in there Our code is still compiling or running fine, which is not surprising because after all we didn't change any of the Python code We just added some new files that the Python interpreter doesn't care about. So for for the interpreter deals files don't even exist But for the other two cases where we use inline type ins or where we use commented type ins
29:44
We actually get an error and if we go Look at the details of that. You can see here the results for the inline and for the commented Build for different Python versions so we can see that if you look at the left column the inline build for Python 7 of course fails because there are no type annotations in Python 7 and
30:04
This is 2.7 and the same of course for pi pi and pi pi 5 3 1 here this Failure for Python 3.2 and for pi pi 3 is more surprising Because you see an exclamation mark there instead of an X which means that something
30:20
Went wrong during the setup of the test and not during the actual running and what went wrong there Is that we try to import the typing module, but unfortunately the typing module requires Python 3.3 so Python 3.2 can't import that and neither can pi pi and that's why those two tests are failing and The same is basically the reason why the commented type ins are failing here because also
30:43
We still have we only have comments in our code But if you remember we still need to import the typing module, so we still have the dependency on that So this is why the code is not running on Python 3.2 or pi pi and actually this is not really true Because I found a workaround for this which is maybe a bit ugly But it works so you can if you follow this link here actually also make the commented type ins
31:06
work in Python 3.2 or pi pi 3 alright, so So which approach should you use um If you look at the inline approach, that's kind of of course the right way to use type in it's a way type hints or type annotations are intended to be used and
31:23
The advantage is that it's easy to read and that the code and the hints are kept in one place The drawback is of course that your code will only be compatible with Python 3.3 or if you use variable annotations Python 3.6 even The alternative here using type comments is good because it keeps the code compatible with Python 2.7
31:43
But and it also uses allows to use variable annotations But it might be a bit ugly and it also still requires importing the typing module Which there's a workaround for though. So the final approach using the stub files is nice because doesn't modify any of the original source at all and it allows you to use always the latest features that are available in
32:05
my pi so The advantage is of course that you can that you have to write now instead of one file To so you will kind of duplicate your maintenance efforts and you will always have to take care to keep those two files in sync with Each other which is not always easy
32:21
so All right, of course there's much more to the To type in send this and we just like had a look at a very simple example If you want to learn more about that I recommend you to check out the documentation of the typing module which contains information about how you For example can create types for your classes or class hierarchies how you can use so-called generics
32:44
So think about a list that contains a different Element of a given type or like a new structure that you invent and that you use in your code So the typing module provides support for all of these use cases and many more like generators as increment things so it's really very well documented and
33:01
can Help you to type most of the variables that you will find in your everyday Python code, I would say So probably now we say okay. Well, this is nice. But is anybody actually using this already? I didn't know that so I checked and what I did for that is to download on the one the top
33:22
1000 repositories Python repositories from github, which you can easily do through the API and then Download the code of that and check it for inline annotation type comments and stops So the code and the analysis can be found on github. I just want to show you the results here
33:40
so what you see here on the left is a An image showing you these 1000 projects that are analyzed So those are really the most popular Python projects on github and the color code Represents the number of type hints or type annotations that are found in each project and you can see here on the bottom
34:01
There's a scale so it goes from zero type hints obviously to about like 10,000 and If you look at the number of projects that actually use Type hints you can see that from these 1000 projects There are 103 projects which have at least one type annotation in their code. And so you might say okay
34:20
This is like a homeopathic dose of type hints So maybe it would be more useful to see the project that actually use many of them And if you go on for example If you check the number of projects that have at least 10 type hints in the code base Then it's going down to 53 already and if you check for projects that have 100 in them It goes down again to about 24 so you can see that
34:42
Popular projects on github. They're already starting to use type annotations some of them are really embracing them, but others are just like maybe experimenting with them or like using them in some test cases and In order to see which kind of type hints they're using you can also have a look at the right Diagrams here which shows you the usage of inline type hints type comments and py files and
35:08
if you're looking for some great examples of Type annotations or type hints and Python code bases. You can have a look at these repositories. So The first one is no surprise, of course because it's my py project So I would expect them to use type hints if they're like writing that okay
35:24
The second one I don't know You might know that if you were have like a connection to Hacker school or recourse center because I think they use it as their main discussion system It's called Zulip and it's an online community system, which was also acquired by Dropbox Which might explain why they're using type hints a lot. So that's the second project
35:42
I mean these two you could maybe discount because because you could say okay They're working then part of Dropbox and maybe they are forced to use type hints Maybe don't like that even at all But the third project is something that you probably use a lot Which is the swings documentation generator and I was actually surprised that they are using type hints But they have a lot of them in their code base about 4,000 and I find they really made very well use of them
36:05
So if you're looking for a code base with good examples about how to use type hints in a really large project you should have a look at that and Afterwards just still you can see the number of type hints in the project is going down rapidly There are some other things here, which is I think this your age you are age project is a radio hacking tool
36:24
You have some home automation tools here a proxy and some AVS AWS related stuff So overall you can see that like of these 1000 Python repositories that we looked at There are at least 10 that I would say use type hints or type annotations in a serious way
36:41
All right, so I'm almost true the last thing I wanted to show you is That you can actually use type annotations for other things than type hinting because you if you remember I said The type annotation syntax was introduced in 2006 and it wasn't tied to any actual use case So it was deliberately left open so that users could just use that
37:04
Additional syntax to do stuff with it and Now if you look again at a simple example of a function with type annotations or like annotations in this case You can ask yourself what is happening actually when we run that and as I said before what the Python interpreter does is That it like creates a special variable called underscore underscore annotations where it puts everything that you put after
37:26
The colon here or after the the arrow so to say so that makes it really easy to access that information at runtime And it allows you to then build stuff on top of that and as an example
37:41
Again, I don't show you much code here. If you want to see how it's implemented You can look at the github gist here, which is linked below So what I did as an as a small demo is to build a runtime type checker using this annotations and it works like this you have a function here which in this case takes three arguments and We want to make sure that the first argument is an integer which is in the range between 0 and 100
38:04
The second one is a float which is between 20 and 40 and the third one It is again an in between 17 80 which also has a default value and we also say that okay the return value of that Function should be a positive float value and this is a so-called contract because you're specifying not in the body of your function
38:22
What the value should look like but already in the definitions of the arguments and then you have like a system that enforces that for you So if you call that function with an argument that does not correspond to the contract that you have the Decorator here, which is called check would complain about it and
38:40
This is actually quite easy to build with this type annotation Syntax in Python and it took less than 100 hours to implement it I didn't know if I should put it here because I'm not sure if it's a good idea Because the thing is if you use it in this way, you will of course break the type hinting with it, you know because
39:01
Tools like my pi they expect The type annotations to make sense. So if you put something like this here, which my pi doesn't understand it will be a problem. So That's why I'm saying if you can use the stuff in that way, but you shouldn't you should think about it Well, okay to summarize I hope I could show you that type and type hinting in Python actually works and makes your code more robust
39:24
For yourself and for your other people that you can already use it now Even if you have Python to code bases and the annotations can do more than type hinting if you want them to All right, so that's basically it you can get the slides on slideshare You can find me on Twitter or contact me via email and now I'm happy to take your question. Thank you
39:44
Thank you for the talk
40:01
Are there any plans in predictable future you are aware of of using? type annotations for Optimizing bytecode generated by compiler Yeah, I think there was definitely there definitely ideas that are going in this direction I think the piper project they considered this but they said they had like other sources of information that they can use to do that
40:27
Optimizations, but of course like having these type annotations in the code makes it much easier to do like Specializations of your functions so you could use that definitely to make Python code run in an optimized way or even have like a virtual machine
40:41
That is running that is compiling it to something else I mean Satan is basically doing this by using a kind of different annotation syntax. So yeah, it's definitely possible more questions
41:10
So they suggest also write unit as the test for the correct typing If you remember you you showed that the test script First failed and then correctly correctly felt. Mm-hmm
41:25
Also So the idea of the The type checker is of course that you don't need to write any additional code to to check your code base So to say you can of course do that, but this was mostly an example
41:40
Where I wanted to show how you can use the type checking how how users of your library could use the type checking in order to make sure that their code is using your code correctly and You can of course like write your own test, but it's not really necessary in order to benefit from the type checking information Yeah, but it's possible. Yeah
42:05
Hi, and you saw how to access the information of the annotation and the Function and is it possible to access the information of annotated variables? Yeah, it's possible. I should I don't remember where exactly it stores. I think it's also inside
42:23
Like an annotation subject, but I really have to look that up here But yeah, and you can also of course get those the annotations in the AST Which is how I found them in the Python projects on github. So I just didn't run the code I just compiled it to an AST using the AST module and then look for the annotations in the syntax tree
42:42
So both ways are possible now Hi, so I've noticed well so the type checking Is there support for multiple types so say X's int or float Yeah, there's I didn't show that but the typing module actually has support for many of the things that you would need in a
43:05
typing system Including so called union types. So a union type would basically allow you to say, okay this variable It could be either a list of integers or a list of floats or maybe something else so you can do that I didn't show that but it's possible
43:22
Hi there nice talk there is also maybe a reason why Sphinx is using So many type annotations. You can have a direct benefit also by using an extension for Sphinx that will put your your comments like
43:41
If you use type comments, then it will search for them and put them in the documentation Okay, yeah, of course that's a good reason now, thanks for pointing it out. Mm-hmm Does and my pie also do some sort of type inference So, I don't know how my pie works internally
44:02
I've written some static analysis tools myself So normally what you do is you try to control a construct like the call flow graph control flow graph In or like convert the code to like single SSA form and then do something with that So I'm not actually sure how they do that. But yeah, there's definitely some type inference involved
44:22
So what I didn't show you in the code examples was that I can just go back here you see that we have like these definitions here of the PN and the big N and There we just put like a literal in there and my pie can of course then figure out That the value in there has a type of integer for example
44:41
So it can do some of that stuff. It doesn't do I think too much of it But for simple use cases, it can actually infer the types. Yeah Okay, maybe finally a question for the audience who of you is already using type hints. Oh Awesome in production Whoo nice Next year and we hope this number of hands goes up. So thanks again. I'm doing us for this very nice talk