Merken

Everything You Always Wanted to Know About Memory in Python But Were Afraid to Ask

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
now
we have a talk from get appear removes the difficult names sorry so good and everything you always want to know about Python know about memory implies and so and yeah but were afraid to ask things have fun and I get float to yeah and remember which thank you very much and
things of that nature so that's about it for your own cover all the topics of item memory because the subject is to to complex and I had to to something but I will try to do my best and probably I would run out of time and so if you have any questions please especially during the lunch and after lunch going back home OK so few words about me that I'm still the students and I work on some of receptor
system that's the glass Copernicus University my main interests and scientific interest that the basis and gpgpu you computing a try combine 2 and also I did some some of the stuff we see that the mining I had of these 8 years of fighting experience and I did some with the bible I these here 3 of them as so I was working with that I was responsible for preparing parts of a trading platform for and of the management of the company because most attractive the contribute covariance sometimes also responsible for preparing a mussel biomonitoring analysis and that the mining software for a laboratory and now we are thinking about commercializing and for my PhD thesis I prepared to assimilate at the reduced processing environment for evaluation of the bicycle is at the end of the world and I mention this products you and Burgulence because all of them had something in common they are what they are wearing a memory-intensive they where 1 training and during this computations they tend to grow it morning and at some point out that I decided that I have to know something more about how the title manages the memory and what are the size of the of the of different types and what are the strategies for locating different containers and so this will be mentioned in the fused to sections and later on try to say a few things about the Marmara providing tools so let's stop this in basic stuff I have found the simplest possible ground and I also teach the percentage of students and after I don't know if demands there really no what are the sizes of different types in the and the forces of class and in spite of this knowledge isn't required for you so actually you know you believe you can you don't even have to care about what what is the size until some point and your application is large enough and it's OK so lot of Mory's then you have to start to think about it what is the what are the size of different by fact so I won't go in we did this with this stable led to some some interesting stuff OK you have 1 of the reasons that that's a long in by tool and the Indian by country are actually limited by your memory so as far as having and enough moral can you can at a lot of number and you have to to also know that the sets are pretty large you compare them to do you see also just that and about because there is other there's some over head of government collector and other things and it's also want to note and how those things and unique answers are represented in memory so we have lot do not have the and we also have to pay to the bytes for each element in your the community called the same goes for typos where you actually the problem but even larger here at the and you have to to pay 4 and 8 bytes for each other shows you can do it yourself
and from by Python to speak to have to get size
of and you can look at it and can and through that site by which restrictions so all the built in into the objects will return you correct results of views on the Jewish party libraries it might be eat Maritornes some crazy stuff so be aware of that and actually calls the size of metal 2 and the 2 of the additional garbage collector overheads that don't out
the garbage collector so let's do
something more interesting to
he was a fun example because creating this is from and here we creates a to lists with the same size and here we we fix the number associated with this form and here is exactly the same number of plaster something more noting that there will be any difference between allocate memory allocated for being between was think 1 interesting tho so this is a fun example so yes there should be something different and as you
can see an the size of of the 1st surrealist is actually less than half of the of the 2nd so this is because of object entirely so what that so
OK and by the way there's a general rule form
critical objects so when we create object and assigned to the 2 of our variables so this object is created and the and the signs so we divide variables don't just want this objects they don't hold them more and you don't of all that is our exception this it's mainly Due to performance optimizations slow down in in the and it's highly implementation-specific all the examples from this presentation are from the fighting and actually it might change over time and there was this 1 change in the Python plementation about object in their and uh during the time so the what is the the internal couples so often used objects are preallocated and instead of creating every time that we I don't know if the side a uh equals 0 so that 0 woman created all the time will be it will be shared among all their responses so here we have been called that through and that's the that so here we go assign 0 to a and B and if you write a is b will get strong and of course values also through and here in this example you can see that we assign meaning of a large number of and a is the will of to false and the values are of course the same actually I know someone special to me an amazing tests from 2 days ago and there was similar question there but both there's still this is highly Python implementation specific so let's
talk about was something more about the object and the now use of and this is I will say this once more this is despite implementation-dependent this may change in future and probably will this is not the comment that the bike on the commendation for programmers and if you want to reference for those of us who values here you have to console the OK so and see I 2 . 7 to before we got an inter object intended for integers from this uh we also have some of the candidate for strings and unique coating by 2 and by country and unique open string seen by John uh 2 and pattern 3 answers turn will be 4 empty strings and all strings that are like that of the length of them is equal 1 and with the restriction for the unique called for only the matching 1 symbols and also to an empty tuple is another example of object but through shared among OK know
something but the different still internal it's drinking during so we start with simple example we'll create constraints almost here we will not the missing collateral today and try the same and to get also of course but if we use in of this is for Python so at and try this 1 and 2 that that's true so and let's try to use it for or something people who creates a lot life wait both things and students 57 megabytes of memory resident memorials and if we do the same with the internal here we actually reveals that memory usage but what's actually happening when we use terms so stinking during
almost became but that the fusion some semantic for storing only 1 copy of the 2 distinct strings what we have to remember that they should be in what tables so for but by entitlements we would have functional interim and you by country it was relocated to this this model so we have this environment and if you use this function you actually and the Austrian into the table of internal things get that references to data and that's the way it might matter actually the same string about was already turned or it might be a copy of the string and when can we use it so we can get this also from the commendation convexity of little performance on dictionary lookups and and some of your vitamins we'll be ultimately the government so for in programs and actually dictionaries of cold model class in the sense of articles have Internet he's the end of the previous examples
we can also reduce their ultimate reduce the space used when we we have a lot of the same using our good OK
so let's say something more about the table containers there are different in comparison by only dictionaries and actually behind the signs is strategy for locating uh contained so uh and it's just that you will try to the for growth of shrinkage so to better from gold was slightly over all take memorial each time we oftentimes on the alignment to always we won't have to relocate the memory uh and you know in our system and so we leave the room for for growth and then we also have to remember that sometimes we have to feed their allocated more for for the table contain and so this will reduce the number of expensive function class like my copy and so on on and of course we will try to use an optimal whilst for performance reasons so
respond to it's very very simple example and this is always 1st I mean put on elements for and into the latest we'll get an allocation but not for 1 element of 4 core elements and after that if we append something returned some thinking that we will we will have free up and so it's memorial operations free so we have compared to we can put another element another and then another when we put the elements we have to we have to so although by whom we will relocate the are for a 4 hour lists and we'll prepare for more than so how does it exactly
where so listen by the sovereign as a representative fixed of flanked are pointers so we just point to objects and thereby design view of the overall located the police or the beginning it will be something that looks for a lot of these it will be less than this percentage so OK some consideration about performance due to memorial actions involved when you think please we actually uh when we put some kind and of the of the least this is this a version of and what if we put something in the middle of or in the beginning we will have to copy the memorial should the memorial tool the performance operations and it is also a watermelon that's for 1 2 and 5 them these we look we waste a lot of space so if you have large number of Somalis have to allocate
over allocated for more than
OK and when this here's the overhead of broken chorus and you have to pay this price for each element for the need for a different architectures and the shrinkage of the this will happen when I look at the number of lemmas that we use we go below the half of the look at space OK and that's to talk
about locating for dictionaries and that's it breaking similar and books you will although we we we will all were allocated when we reach that years of of and dictionary or sets actually for small dictionaries and small small sets will quadruple the capacity in 1 day 2nd order it is big enough for you double the capacity to extent the more and then we'll have to calculate actually the use size 4 for this object and I look at the memorial and the shrinkage of the dictionary offset to happen when we the light a lot of the did you removed the last summer of OK from on basic now another
example of what we can represent data environment pyruvate represented in different ways so we can use all type of things like glass that you can use lots can use named tuples tuples these and dictionary and I repudiate on example
from the book to by Koehn 2004 for apprentices and but other it's more objects and covered some more fields uh and actually can see how did they differ for storing the same data that doesn't by defining different types so as you can see which are not some restrictions because you when you put as slots into your classes but a lot of restrictions for this task but you can gain uh memory minimization goes for leave you get on that's memorial use for a for those those classes OK so
notes on the garbage collector and reference counts by so actually as it probably all uh but has a garbage collector and do collects objects when the authors found goes to 0 there are some information on some of the operations that increment reference counter some there are some of the operation of the detriment of the reference counter but there is no warning candidates putting that into the office of the communication that if you actually overruled down and you can have problems because if you have because I don't garbage collector graph can deal with cyclist uh in uh in object references boats when you use that the it's not possible for about 5 months to get to the correct order of the using development as the objects in some on the cycle so actually uh this cycle won't be the allocated from from remember
OK I have some more time so I will talk about this and use me and but you can use for a White and remember providing analysts star would stop using those pretty simple it's actually trust that system for roman move for up API for a system utilities and so actually to get some information about current process Memorial can just use the price of the process and get your get off your process and transform this information done dictionary and then you can return the simple information the for most of the examples I just the could the cold but because it's most relevant for corpus for this that the proposed and to another to reduce memory profile and so it recommends to to use P a feudal system so this is good to have that you laughter dependencies into a cluster and there number of final my work in 3 different modes so you can get a line by line so you can get memorial that moment monitor and become to the use of some of them got to go so let's start with the alignment provided you have to put in your
coats profile of the crater on the function we want to provide profile and then you can run start with something like that of course you should be the name of the of of the code and then we will get such results that you get line-by-line Memorial that's and the increments from the from the memorial which for each and here we see that the the for loop you the the main uh minimum memory contributor and the 2nd way that we can use the memory requirement is by uh using the it's also memory usage monitoring time so you we will just when you turn them brought system where to do that and actually you can use it for any type of process not only for uh not only for the title but if you want to use with Python should both to provide the correct for functions that you want to track and surrounded with the option Python and here I run simulation and here and here is the result I got a plot and here are some things that you would like to add that the convex function markets us as the the 1 that those those of my operations here so I see that probably uh connect is responsible for the growth from here and there and functioning market here at frequency it doesn't change out from from our memory and that the adoption
forms from memory providers to use it as a double figure and so we can set up a stressful of MIMO use memory and run our process and then we will get and I will step into the the mother when we reach that memory of the set of the direction and another
1 another tool use Object athletes cool tool for visualizing the object references in by them and so actually for small projects is pretty cool because it gets the such plots like this and it's a good tool for
finding the reference cycles in your in your code if your product is laden half the people but will we generated will be pretty large and be have hard to check something there but we tasks some cold and manipulation and there's sold all this to our using in in the tutorial for enough you can actually track down the object reference to that site pretty pretty even this
OK the next 2 tools probably there will be covering more
in there to talk from the session chiefly and malaria and there are a few is the tool there are are treated the same as which some differences but the pretty good distracting in on the on the part of this project and so let's
see what we can do with that room so we actually can run codes and do our hips snapshot here and and some more Memorial extensively in intensive operations and on future of not a huge snapshot and that we actually can do some medics on on those
hits and get such a result so we see that we allocated a lot of integers entities with this 1 operation and another
and is a combination of many I answer on the ground so actually you can use this tool to about all the objects in your coat and then use the ranks snake uh with that of memory that's about it and get such interactive plot so we can this means and how to see how the of how the memorial located for different different objects OK
and this is almost the
end of my talk so and you can also use different molecule plementations with it's pretty easy and and you will find probably many block and entries about using different the memorial craters and so it's got some problems those so that you can actually game we which saw very little endurance neuronal called some additional at better but memorial process to system memory retrieval and that's the that's also gone so actually my works against you so it depends your application back so if you want to use a different uh molecule plementation you have to be so of course the from the different libraries and then you can run the Python which of the field and with the part the uh library want to use and you can get different results I actually prefer some small does so my called got some service that center used small Jermoluk and the model with the same code and you can see how the more it changes in different places so as you can see from our work you actually melodies actually pretty good now on Python there was something of a few years ago there were some problems the number now it works pretty pretty good what's the answer conceived and look you can get pretty the same results and trouble for a different applications uh you can gain something but we fishing model for discovered this example actually if you end up with a little bit more memory allocated not return to the system but again this depends on the application side that you will of of application back some other
useful tools you can always say you
can always built by combining that multi can you say about language Python pretty good cooperate with it you can use the experimental extension for gdb and probably for most of the Web developers you can use 1 of those of those er at those probably more convenient because outweighs the middleware version of terrifying memories of this 1 and you can just put it in your with the and that gets on memorial profiling as
a summary and try to understand better the underlying Memorial don't pay attention to what sports user profiling tools c can destroy this is actually the hardest part so try to find the root causes and think the memorial so probably the next the next talk will be about this and is thus also also we can do this solutions sometimes the t so we can delegate Memorial uh into under the proof of that monitoring this operation on the time of the process processes and collected the results and then to the process of stop the process and you can actually rest of the process if you just get them as much memory overhead and also can always go for a hanging fruits like loss or try different moral patients and just some great differences that they use Europe
when I was preparing this presentation so give it a try some of them are all the data like this 1 because they need insights about quite a lot of memory inside so thank you very much which focusing included
sometimes on a question so please come to the microphone over the rotating give some Christians I've experienced it sometimes that I've had to create many objects in Python and then that I removed all the references to them and actually force the garbage collector on yeah but this is the memories that wasn't 3 so and this just time widest Python sometimes not Freeman it's actually made the plants from the version of Europe by the interpreter that this is 1 and 2nd 1 it that sometimes the moment the moral code we have problems with returning the memorial it's a little bit more complicated but we can try to try the different them all allocations libraries that irrational and try to see that it is it will help you with your problem yes of course do you have any hands you should this be be and so on and you already have have himself to the work of the memory problems so what I experienced sometimes using PsychoPy due to play the whole process and was using like for gigabytes of memory but he the only showed that very few the memory of stuff so I guess it was related to his something yeah solid western is the question is do we have any himself to the a d-block like those of the the memory problems and so you can either going to try the devil version of Python some component with the devil version and then you can see that objects that were and be allocated by the garbage collector and the plants use walgreens if you want to go low level so we can talk about the normal OK thank you're much
Berline
Mehrrechnersystem
Gibbs-Verteilung
Computeranimation
Retrievalsprache
Kovarianzfunktion
Punkt
Prozess <Physik>
Mereologie
Natürliche Zahl
Leistungsbewertung
t-Test
Kartesische Koordinaten
Computerunterstütztes Verfahren
Element <Mathematik>
Statistische Hypothese
Analysis
Datenmanagement
Softwaretest
Typentheorie
t-Test
Vorlesung/Konferenz
Datentyp
Prozess <Informatik>
Programmierumgebung
Biprodukt
Tupel
Software
Menge
Forcing
Scheduling
Strategisches Spiel
Garbentheorie
Programmierumgebung
Simulation
Notepad-Computer
Objekt <Kategorie>
Subtraktion
Klasse <Mathematik>
Zahlenbereich
Systemplattform
Überlagerung <Mathematik>
Data Mining
Datenhaltung
Software
Datentyp
Leistungsbewertung
Analysis
Schreib-Lese-Kopf
Algorithmus
Eindeutigkeit
Systemplattform
Physikalisches System
Komplex <Algebra>
Basisvektor
Mereologie
Wort <Informatik>
Data Mining
Resultante
Objekt <Kategorie>
Addition
Web Site
Sichtenkonzept
Maßerweiterung
Systemaufruf
Computeranimation
Objekt <Kategorie>
Umwandlungsenthalpie
Programmbibliothek
Overhead <Kommunikationstechnik>
Overhead <Kommunikationstechnik>
Speicherbereinigung
Implementierung
Betriebsmittelverwaltung
Objekt <Kategorie>
Subtraktion
Zahlenbereich
Mailing-Liste
Maßerweiterung
Systemaufruf
Computeranimation
Mailing-Liste
Bildschirmmaske
Ganze Zahl
Umwandlungsenthalpie
Overhead <Kommunikationstechnik>
Speicherbereinigung
Betriebsmittelverwaltung
Implementierung
Objekt <Kategorie>
Ausnahmebehandlung
Minimierung
Mathematisierung
Implementierung
Kombinatorische Gruppentheorie
Computeranimation
Variable
Bildschirmmaske
Mailing-Liste
Ganze Zahl
Umwandlungsenthalpie
Vorzeichen <Mathematik>
Endogene Variable
Virtuelle Realität
Betriebsmittelverwaltung
Implementierung
Umwandlungsenthalpie
Softwaretest
Globale Optimierung
Ausnahmebehandlung
Schlussregel
Variable
Schlussregel
Arithmetisches Mittel
Objekt <Kategorie>
Objekt <Kategorie>
Videospiel
Nebenbedingung
Tupel
Dicke
Programmiergerät
Eindeutigkeit
Unicode
t-Test
Symboltabelle
Term
Computeranimation
Zeichenkette
Spannweite <Stochastik>
Objekt <Kategorie>
Quellcode
Tupel
Offene Menge
Ganze Zahl
Mustersprache
Demoszene <Programmierung>
Mixed Reality
Lateinisches Quadrat
Implementierung
Zeichenkette
Programm
Lineares Funktional
Schlüsselverwaltung
Klasse <Mathematik>
Konvexer Körper
Paarvergleich
Zeiger <Informatik>
Instantiierung
Computeranimation
Data Dictionary
Internetworking
Zeichenkette
Informationsmodellierung
Attributierte Grammatik
Data Dictionary
Optimierung
Tabelle <Informatik>
Zeichenkette
Lineares Funktional
Minimierung
Klasse <Mathematik>
Zahlenbereich
Paarvergleich
Menge
Systemaufruf
Raum-Zeit
Computeranimation
Strategisches Spiel
Data Dictionary
Zeichenkette
Mailing-Liste
Funktion <Mathematik>
Vorzeichen <Mathematik>
Strategisches Spiel
Schwellwertverfahren
Ordnungsreduktion
Betriebsmittelverwaltung
Tabelle <Informatik>
Betriebsmittelverwaltung
Punkt
Freeware
Selbstrepräsentation
Gruppenoperation
Zahlenbereich
Element <Mathematik>
Raum-Zeit
Computeranimation
Strategisches Spiel
Mailing-Liste
Endogene Variable
Zeiger <Informatik>
Operations Research
Betriebsmittelverwaltung
Nichtlinearer Operator
Verschiebungsoperator
Sichtenkonzept
Raum-Zeit
Element <Gruppentheorie>
Mailing-Liste
Zeiger <Informatik>
Gruppenoperation
Objekt <Kategorie>
Speicherabzug
Verschiebungsoperator
Raum-Zeit
Element <Gruppentheorie>
Zahlenbereich
Element <Mathematik>
Zeiger <Informatik>
Raum-Zeit
Computeranimation
Gruppenoperation
Strategisches Spiel
Mailing-Liste
Lemma <Logik>
Computerarchitektur
Overhead <Kommunikationstechnik>
Operations Research
Betriebsmittelverwaltung
Tabelle <Informatik>
Hash-Algorithmus
Kanalkapazität
Schlüsselverwaltung
Selbstrepräsentation
Element <Gruppentheorie>
n-Tupel
Kanalkapazität
Menge
Computeranimation
Data Dictionary
Objekt <Kategorie>
Menge
Code
Datentyp
Computersicherheit
Selbstrepräsentation
Data Dictionary
Maßerweiterung
Ordnung <Mathematik>
Programmierumgebung
Objekt <Kategorie>
Stellenring
Klasse <Mathematik>
Zählen
Computeranimation
Task
Datentyp
Zählen
Softwareentwickler
Autorisierung
Nichtlinearer Operator
Extremwert
Graph
Objektklasse
Aliasing
Office-Paket
Objekt <Kategorie>
Datenfeld
Körper <Physik>
Funktion <Mathematik>
Dreiecksfreier Graph
Selbstrepräsentation
Information
Ordnung <Mathematik>
Speicherbereinigung
Resultante
Prozess <Physik>
Momentenproblem
Extrempunkt
Weg <Topologie>
Konkave Funktion
Code
Computeranimation
Systemsoftware
Physikalisches System
Loop
Graph
Datentyp
Plot <Graphische Darstellung>
Gerade
Modul
ATM
Lineares Funktional
Nichtlinearer Operator
Prozess <Informatik>
Systemplattform
Profil <Aerodynamik>
Debugging
Strömungsrichtung
Plot <Graphische Darstellung>
Physikalisches System
Frequenz
Ordnungsreduktion
Konfiguration <Informatik>
Benutzerprofil
Funktion <Mathematik>
Selbstrepräsentation
Simulation
Information
Speicherverwaltung
Prozess <Physik>
Plot <Graphische Darstellung>
Debugging
Service provider
Computeranimation
Richtung
Objekt <Kategorie>
Graph
Bildschirmmaske
Menge
Endlicher Graph
Projektive Ebene
Strom <Mathematik>
Objekt <Kategorie>
Web Site
Ikosaeder
Programmierumgebung
Biprodukt
Information
Analysis
Code
Ähnlichkeitsgeometrie
Computeranimation
Objekt <Kategorie>
Task
Graph
Endlicher Graph
Dreiecksfreier Graph
Benutzerführung
Speicherverwaltung
Objekt <Kategorie>
Partitionsfunktion
Nichtlinearer Operator
Subtraktion
HIP <Kommunikationsprotokoll>
Vorzeichen <Mathematik>
Programmierumgebung
Information
Ähnlichkeitsgeometrie
Analysis
Computeranimation
Total <Mathematik>
Mereologie
Zählen
Projektive Ebene
Benutzerführung
Speicherverwaltung
Resultante
Nichtlinearer Operator
Partitionsfunktion
Snake <Bildverarbeitung>
Plot <Graphische Darstellung>
Computeranimation
Objekt <Kategorie>
Rangstatistik
Ganze Zahl
Total <Mathematik>
Zählen
Speicherabzug
Speicherverwaltung
Information Retrieval
Resultante
Subtraktion
Bit
Prozess <Physik>
Zahlenbereich
Kartesische Koordinaten
Physikalisches System
p-Block
Code
Computeranimation
Physikalisches System
Dienst <Informatik>
Informationsmodellierung
Spieltheorie
Mereologie
Programmbibliothek
Betriebsmittelverwaltung
Implementierung
Objekt <Kategorie>
Leck
Prozess <Informatik>
Formale Sprache
Versionsverwaltung
Profil <Aerodynamik>
Debugging
Gebäude <Mathematik>
Befehl <Informatik>
Maßerweiterung
Optimierung
Gerade
Computeranimation
Middleware
Middleware
Multiplikation
Interaktives Fernsehen
Total <Mathematik>
ATM
Speicherabzug
Web-Designer
Maßerweiterung
Versionsverwaltung
Betriebsmittelverwaltung
Leck
Resultante
Nichtlinearer Operator
Hauptideal
Subtraktion
Einfügungsdämpfung
Prozess <Physik>
Prozess <Informatik>
Speichermodell
Dualitätssatz
Kombinatorische Gruppentheorie
Computeranimation
Quellcode
EDV-Beratung
Wurzel <Mathematik>
Beweistheorie
Mereologie
Wurzel <Mathematik>
Overhead <Kommunikationstechnik>
Benutzerprofil
Betriebsmittelverwaltung
Betriebsmittelverwaltung
Interpretierer
Stereometrie
Bit
Prozess <Physik>
Momentenproblem
Versionsverwaltung
Code
Übergang
Objekt <Kategorie>
Programmbibliothek
Vorlesung/Konferenz
Zusammenhängender Graph
Speicherbereinigung

Metadaten

Formale Metadaten

Titel Everything You Always Wanted to Know About Memory in Python But Were Afraid to Ask
Serientitel EuroPython 2014
Teil 95
Anzahl der Teile 120
Autor Przymus, Piotr
Lizenz CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/20018
Herausgeber EuroPython
Erscheinungsjahr 2014
Sprache Englisch
Produktionsort Berlin

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Piotr Przymus - Everything You Always Wanted to Know About Memory in Python But Were Afraid to Ask Have you ever wondered what happens to all the precious RAM after running your 'simple' CPython code? Prepare yourself for a short introduction to CPython memory management! This presentation will try to answer some memory related questions you always wondered about. It will also discuss basic memory profiling tools and techniques. ----- This talk will cover basics of CPython memory usage. It will start with basics like objects and data structures representation. Then advanced memory management aspects, such as sharing, segmentation, preallocation or caching, will be discussed. Finally, memory profiling tools will be presented.
Schlagwörter EuroPython Conference
EP 2014
EuroPython 2014

Ähnliche Filme

Loading...