Merken

Import Deep Dive

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
a letter believe that there is this famous interview
questions and answers you take python . org into the web browser press what happens this talk is a bit similar it's about what happens when you try to import some random put lots of stuff happens that a
little bit after I submitted the talk I learned that even these leading a 3 hour tutorial on this on the this year's icon so I'll try to look at it from a different angle but that if the stock is not enough for you then there's lots middle them lots more material you can you can use to learn more than
that the dive it'll be like of guided tour through what happens when you can point something but hopefully when we're finished with the talking can take deep dives through the source code yourself so what happens when
you have exactly this under the covers there's a global gender import functions that gets called and the results from that is assigned to a that's pretty much all that happens now the import statement is a little more more powerful than that kind of evolved over the years so you can do some package imports with these dots you can import stuff from modules in the mapping from this student under import function is not always trivial out but it's a documented pretty well in the box so if if you want to do that then right hand and read it but all the donor import function is is an interface can be imported machinery which nowadays has its origin in Python itself in the important module so if you don't want to do this when we import something programmatically there's also a continuous function called import module that's much better to use so if you have a string with the module name just is found it's also just an interface to the import machinery and the other thing you can do with the engendering port is replace it with your own function but that is not very useful because then you have to to reimplement most of the machinery yourself so you're not it's not useful to call them the importance not useful to replace its so it's probably better if you just forget about it the import statement called the important
issue here and I will talk about what the import machinery down as I'll skip
all the locking caching error handling and older Python's all the stuff that takes most of the library but it's not really necessary for you to know what what's going on so the
basic algorithm for and what happens when you import something is actually pretty simple looks like this and the 1st thing you get is this cash this this is modules dictionary so if you important a module that already has been imported it's stored in the cache so when you read importer module you get exactly the same object that there's a catch to this when you delete something from the dictionary and then re-import the same module it's have gone from the cis modules so the import machinery things it's never been important and it import the the module again you get a brand new IT module objects in every function in every class and there will be brand new which most of the time is not something you want because stuff doesn't expect this but you can do this the other thing you can do is poison the cash you can just assign anything just as molecules so you can put a string in there and when you then imported so you get a string as a model and you can use all the string operations on it and some modules actually use this to make modules that are callable loss of scriptable or have arbitrary at its there is some limited use to this that maybe you shouldn't do it and production so that's the 1st statement the 2nd there is this finds that function that takes the name of the module and the pack in most cases the fast will be system which is just a list of all locations that have modules can be imported from on my system it's usually much longer than this for details on how it's constructed CD easily solved he talks about it all at once so with these 2 and I call the finds that and that is the case that object the module spec object and that is just a description of how the module will be loaded and where it will be loaded from you know that there's actually utility function that you can call to get the without importing the module like this so the modules that gives you the name of the loaded which is kind the strategy how to be loaded and the origin which is where the module will be loaded from so you can do that without importing the object which might be useful and sometimes also the Michael spectrum becomes
a permanent record of how the margin was loaded so you can with any module we can look at the dentist that contributes and see where it's like uploaded from the
next step is that the actual loading and I will look at it it's a little bit more detail later but what happens here is an empty module object is good answers modules and after that it's initialized it's important that it's done in these this exact order 1st is put into these modules and after that it's initializing all the functions and classes get assigned to it and after that the machinery expenses modules and returns whatever it filed that and this this a
simplification of course but you can already use it to solve real world problems for example import cycles everybody's favorite thing when it comes to the importance of the line so we have 2 modules here anyone imports the other and the other imports the first one again this is a very bad thing to do batch and yeah it it it's usually results in areas that are not so nice but if you know this algorithm and reason your way to what is happening so if I import through it shakes his modules doesn't find food there so it finds the source code for a few and starts loading it starts going through it uh well 1st of ancestors modules and then starts going through the source 1 line by line the 1st thing it finds its important bias so it has to import buyer doesn't find instances modules so puts an empty model object modules and starts going through that 1st thing it finds is another important so it tries to import food let census modules and finds through in there because it already put it there but it's not gone through all the initializations so we get half initialized for object here and then we try to we call this function which by accountancy and this initialization so this falls within attributed and the whole thing goes out to get an importer and you start looking where the idea is it's not so obvious that there are some tools that can detect these imports cycles and and warn you which you should use and the best way to to solve this problem is probably due data functions that both modules we need and move them to a different model and put them but if you ever run into this situation you already know how to solve OK so here we go
again see I left some space here because there's obviously some more and the something more has to do with some molecules and packages
so let's give can go through a little bit of vocabulary are and module was a top level module you can import it directly so is you are element for example but you are also has some other marginal behalf below it so yeah as a package it's apparent of Europe piracy neural of requesting island response and are submodules this you are very clear on hope you knew that already so what happens when I tried to import some but this 1st the path is different for submodules the factors not consist fast but the
Baptist taken from the parent so the parent has this path attribute that says where all the submodules are loaded from the 2nd thing as that's different is these 2 parts so for some modules the parent is always loaded 1st there's no way to load your EPA's without loading it and so it's always done 1st and a floating the somehow causes only bias to also be loaded and this point it's just returned wouldn't otherwise it's important normally and at the end after everything's done the submodular set has an attribute on the parents so if you import your own of parts and the object you actually get is your related but it hasn't actually pass on that you can get by the dark because it's such as the attribute at the very end of import so there is a more complete algorithm which you can use to solve for reason about more complex from situations that involve some submodular so for example if I have this simple package in R . 5 with 2 imports are some constant value and some code that uses it and I tried to import that so what happens if the parent model is always loaded 1st no matter which 1 of these you import so 1st of let's census modules for food doesn't find that goes to find goes to all of the source and executed by on line by line we go goes to the import function the input statement which live involves the machinery again looks consist modules for food or main doesn't find that it's it's of submodular goes to load through which was already loaded it's already in modules so it returns fairly and goes to executing the code got to get to this import statement and it tries to import food again sensors love modules someone who there so it literature stuff and then we tried to use at which point we use the full module but it doesn't have the constant yet because that gets initialized at the end of this important that we didn't get to together so once again you get on that and this is kind of complex and you have to understand that this
algorithm which arguably is not that hard but if you have bigger modules and gets a bit complicated so i've prepared a set of little
rules that he he should follow to be OK 1st of all European it should be kind of of public interface to the package so it should just import stuff from submodules maybe said under all and do nothing else and then your submodules should not use the public interface they should import directly from the models that you want because you all probably know about the internal structure of package and obviously shouldn't have input cycles in the modules in the submodules themselves pretty follow these rules you should be OK otherwise understand
this algorithm and you can read in your way through they have so that's for this and maybe you're wondering what exactly this finds that does let's look at that and let's look
at 1st result where do you actually loader module from so if I can put my
random module can print out and I see it's loaded from some location on my system I can look at the end of the file attributes and get the same thing at the back constraints but if I can port another modules research print out and I see it's built into see doesn't have identified much attribute does anybody know where this is module is actually located on your system
now says actually built into the executable itself so in my case it's underuse of in-vivo it's built into the the actual program yeah but all the other modules that and this place so these are 2 different types of modules we have a look at the
aquarium of module types we can see we have the built-in models which are written in C and compiled into Python itself we have some source modules which are written in Python and loaded from files and we have some other types of modules as well we can have extension modules which are written in C or some other compiled language and looted from filed a shared library on my system that's mass uh for example some number like core modules can be extension modules as well and the 4th type is frozen models which are written in Python and compiled into the executable itself 1 example that everybody uses this frozen in Portland which is a copy of the import machinery that's built into Python for loading the real import machinery because you have to use the import machinery to read stuff from files and stuff like that has a flight to by to add by 2 eggs are actually compiles Python modules into the resulting in the people to make 1 final executables let's use case for that to happen you load all these different kinds of modules
almost this list of strategies to use insist on metadata and the algorithm is quite simple we just ask each of these finders turn if they can move loader module so if I'm loading the SIS model has to build an importer Haiti have assessed model and the build an important except the list of building modules and says yeah here's here's the information In gives me a spec for if I define importing random I look I asked the building important and doesn't find a random module and building modules as has frozen important doesn't find random and those afroasian models so I asked the pathfinder the pathfinder is a bit more complicated this is the thing that looks at that stuff that and goes through every entry insist that in order and for every path that term it has what's called a path wanted to lose looks like this so it'll go to the current directory and construct a profit for that so since the important can't about 3 so scared when there's a file finder which can handle the trees so that 1 is used for the current directory and there we look for these files and we probably don't find any of those so we'll give you the next entry which is a set file yes because it's the important for the this is the file if it can find any of these can have so we go to the next century master file finder if it can find any of these and there and again it's that kind of that applies actually in this directory and since the finals there the pattern and this is the trend now at this point we have a suspect and when we actually have a spike we don't look any further so when the file exists it's just that is written for it and the machinery doesn't look any any further in the past so the 1st much wins and what's in the module spec again we have the name we have the origin which is the source code to load we have a location for the cache file which may or may not exist we have the loader which is the strategy to use to load the sources and some others hello to specific information out you can read all about this and the that that I will link breaking just that that's how you get the but and we have a bit more time left so I can talk about how to actually load module so
once we have this thank the loading is kind of simple 1st we create a module object model object is nothing special just object that has a name attribute and either the loaded loader can create 1 or or if it doesn't want to then we create a default 1 after that we we set the initial model attributes of which are actually just copied from the specs so the spec gets copied to that expected the name that's competent and name so now we have 2 places for each of these bits of information which is kind of redundant and you can change each 1 of these individually so it's a bit of a mess as but 1 of these is always used later after that we put the module is the modules and execute whatever source code we find the global rivals variables so are actually just attributes on the module objects which is kind of fun to play with a few important the main module you can assign a global variable get it back as an attribute or vice versa and this is I and this is also where gender name comes from its assigned very early in the loading phase so by the time you get to executing your code solidarity kitchen what it so that's that's the killing the module and 1 more thing I have uh 0 1 more thing I have is how to actually get this so this code for source module 2 in the module spec we have put the origin and the PY file and the cache location and if the catch location exists and it was compiled from a matching PY file which has the same size and same modification time than byte code is read from the cache file and execute and of it and then executed if it doesn't match then it's straight from the and the origin filed and potentially stored in the cache if you're familiar with how Python to do this at the origin and cash we're in the same directory which had the problem that if you deleted and you see that gets you can uh if you deleted PY filed in the QIC got executed some of this is something that for some reason was there and the same thing as a deleted file which use to throw off of a flood of beginners not only them and by countries we have had the bike as territory which no longer has this problem 2 in the PPI cash we have the PIC but if p y is is not there the caches and even looked at what you can do if you really want to load things from PY sees as copy the p y C over to the old location and delete the wine cellar actually work and this
is all the code it's just a screen full of what you have to understand and if you want any more details the import is installed on your computer so if you just look at it now and see what's going on In the for it to be having questions and thank you
for a book I would like to know it what was the use case only been able to most them so it's called home justified because excuse me is where the users on loading golden survive from UIC files lawsuits the 0 C files video files OK when I and when I showed you 2 different kinds of modules I wasn't really complete at that looks like this so you can
look at the road uh load from native code for example redundancy in let's from Python code or by code as you have built in frozen extinction sources and Source less surface area the PY and you can also load source search list files from the and this is done for you to the strategy for example some Windows users don't lie deep directory structures you have lots of files and directories suggested that there was a lot into and you now it is it it usually has to the y z extension and you can import directly from that you can if you you can actually run those the UIC as assigned to 2 pi fun and if you have main module and the election Bremen also on Linux is that that has the shebang it can run those so
it's just said and easier to write an easier way to package things just some of 1 this file and everything is in that requested there are no questions I can find something else talk about the do we have time the parameters so 1 thing I forgot to say this create module inexact module so this is for Python modules for C modules like like the extension or built in everything happens in create there's a fight and it's hook function that creates the module and also initializes at that 1 step and then this exact is just nothing past so that was that is the current situation with Python 3 . 4 for Python 3 . 5 there's Newman a new mechanism that of does something similar to the Python modules so that create creates an empty object and then there's a separate exact and that you can do your work and which is better because of the times exact is run the module object is already insist modules for example what could happen before is if you ran some music over the ransom Python code and it tried to import your marginal again you would get into an infinite loop because it's not in the cache so it will try to re import your module again and there is after margin the loading is a bit more declarative knowledge in that time 4 8 9 and you can go read that if you're interested I so there's work going on in this area still and what I hope the top will be obsolete in a few years share and of wondering what would happen if you ladies model with class and once again any let me later model and it's good class in grade instantiate that class and then you do that take the section that stuff where you really initialize the model relate yes so so so what the reinitialization just reload does this that creates creates a new module that creates a class object but every instance of an existing class has a reference to the original classes so all the old instances would use the old class and all the new instances would use the new classes which so create some problems for example if you try if you use now you try to check for equality and it's implemented by looking at the class and the class is obviously don't match and you have a problem because you think there's the statement represents string representation the same but the class actually different and who looks at the class idea so there are some use cases for this but it's usually better to stay well away from it OK thanks the
Bit
Browser
Randomisierung
Vorlesung/Konferenz
Ähnlichkeitsgeometrie
Computeranimation
Videokonferenz
Bit
Modul <Datentyp>
Quellcode
Bildschirmsymbol
Computeranimation
Resultante
Lineares Funktional
Befehl <Informatik>
Quader
t-Test
Stetige Abbildung
Computeranimation
Überlagerung <Mathematik>
Mapping <Computergraphik>
Skalarprodukt
Lemma <Logik>
Zustandsdichte
Geschlecht <Mathematik>
Vorlesung/Konferenz
Zeichenkette
Schnittstelle
Nichtlinearer Operator
Lineares Funktional
Befehl <Informatik>
Klasse <Mathematik>
Stichprobe
Softwarewerkzeug
Ausnahmebehandlung
Mailing-Liste
Physikalisches System
Biprodukt
Punktspektrum
Computeranimation
Objekt <Kategorie>
Deskriptive Statistik
Informationsmodellierung
Caching
ATM
Strategisches Spiel
Programmbibliothek
Vorlesung/Konferenz
Restklasse
URL
Zeichenkette
Objekt <Kategorie>
Randverteilung
Lineares Funktional
Bit
Datensatz
Modul <Datentyp>
ATM
Klasse <Mathematik>
Güte der Anpassung
Ordnung <Mathematik>
Computeranimation
Resultante
Lineares Funktional
Quellcode
Raum-Zeit
Computeranimation
Objekt <Kategorie>
Informationsmodellierung
Algorithmus
Modul <Datentyp>
Flächeninhalt
Reelle Zahl
ATM
Dreiecksfreier Graph
Stapelverarbeitung
Gerade
Instantiierung
Bildauflösung
Bit
Punkt
Parser
Element <Mathematik>
Komplex <Algebra>
Code
Computeranimation
Übergang
Last
Informationsmodellierung
Algorithmus
Modul <Datentyp>
Endogene Variable
Wirtschaftsinformatik
Vererbungshierarchie
Gerade
Attributierte Grammatik
Lineares Funktional
Befehl <Informatik>
Softwarepiraterie
Quellcode
Ein-Ausgabe
Teilbarkeit
Endogene Variable
Objekt <Kategorie>
Bimodul
Menge
ATM
Mereologie
Attributierte Grammatik
Bit
Schlussregel
Ein-Ausgabe
Computeranimation
Informationsmodellierung
Last
Algorithmus
Bimodul
Modul <Datentyp>
Dreiecksfreier Graph
ATM
Datenstruktur
Schnittstelle
Resultante
Elektronische Publikation
Hochdruck
Physikalisches System
Elektronische Publikation
Systemaufruf
Gerade
Menge
Computeranimation
Last
Algorithmus
Modul <Datentyp>
ATM
Attributierte Grammatik
URL
Attributierte Grammatik
Subtraktion
Elektronische Publikation
Oval
Formale Sprache
Zahlenbereich
Ruhmasse
Maßerweiterung
Physikalisches System
Quellcode
Elektronische Publikation
Gerade
Computeranimation
Informationsmodellierung
Reelle Zahl
Datentyp
Programmbibliothek
Mailbox
Attributierte Grammatik
Speicherabzug
Compiler
Maßerweiterung
Optimierung
Lesen <Datenverarbeitung>
Bit
Punkt
Relationentheorie
Maßerweiterung
Code
Computeranimation
Metropolitan area network
Metadaten
Last
Variable
Informationsmodellierung
Verzeichnisdienst
Algorithmus
Typentheorie
Byte-Code
Mustersprache
Randomisierung
Default
Phasenumwandlung
Attributierte Grammatik
Caching
DoS-Attacke
Booten
Matching <Graphentheorie>
Objektmodell
Gebäude <Mathematik>
Mailing-Liste
Quellcode
Elektronische Publikation
Matching
Ausgleichsrechnung
Objekt <Kategorie>
Twitter <Softwareplattform>
Menge
Geschlecht <Mathematik>
Last
Caching
Strategisches Spiel
Attributierte Grammatik
Restklasse
Information
URL
Verzeichnisdienst
Subtraktion
Verzeichnisdienst
Typentheorie
ATM
Attributierte Grammatik
Vorlesung/Konferenz
Restklasse
Elektronische Publikation
Systemaufruf
Code
Computeranimation
Videokonferenz
Randverteilung
Bit
Gemeinsamer Speicher
Relationentheorie
Klasse <Mathematik>
Selbstrepräsentation
Maßerweiterung
Maschinensprache
Code
Computeranimation
Gradient
Loop
Informationsmodellierung
Flächentheorie
Gruppe <Mathematik>
Hook <Programmierung>
Bildschirmfenster
Vorlesung/Konferenz
Datenstruktur
Maßerweiterung
Parametersystem
Befehl <Informatik>
Elektronische Publikation
Matching <Graphentheorie>
Güte der Anpassung
Strömungsrichtung
Mailing-Liste
Quellcode
Elektronische Publikation
Unendlichkeit
Objekt <Kategorie>
Differenzkern
Flächeninhalt
Last
Caching
Strategisches Spiel
Garbentheorie
Verzeichnisdienst
Zeichenkette
Instantiierung

Metadaten

Formale Metadaten

Titel Import Deep Dive
Serientitel EuroPython 2015
Teil 54
Anzahl der Teile 173
Autor Viktorin, Petr
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20184
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Technische Metadaten

Dauer 27:43

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Petr Viktorin - Import Deep Dive Whatever you need to do with Python, you can probably import a library for it. But what exactly happens when you use that import statement? How does a source file that you've installed or written become a Python module object, providing functions or classes for you to play with? While the import mechanism is relatively well-documented in the reference and dozens of PEPs, sometimes even Python veterans are caught by surprise. And some details are little-known: did you know you can import from zip archives? Write CPython modules in C, or even a dialect of Lisp? Or import from URLs (which might not be a good idea)? This talk explains exactly what can happen when you use the import statement – from the mundane machinery of searching PYTHONPATH through subtle details of packages and import loops, to deep internals of custom importers and C extension loading.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...