Merken

Lessons learned with asyncio ("Look ma, I wrote a distributed hash table!")

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
be and your thank you so the money and because a freelance Python development from the UK columns and this is an introductory talk really about facing higher the media distributed hash tables answer basic ideas on sure you all know the right thing my feet 4 on and online expectation of that you know that with the prices of so what we'll do is explore so the core concepts of space and time and I'll tell you the story of how I used a society creates a fundamental personal project was to build a distributed hash table which are called the area use 3 reference in his keynote this morning you have 1 is rather Austrian history so this isn't an exhaustive of exposition facing there's an awful lot have to leave out it simplifies a lot as well my main is on you with just enough information so you can continue exploring the module in the concept found therein afterward and it's also a personal pedagogical exercise as if I can explain facing in a simple total weight it's it demonstrates my clarity of thought about facing kind an understanding of what I think of as you can hear I'm talking very quickly because I have brought a lot of information that they give to you so yes perhaps like American
Ukraine will find this talk so the 1st thing that needs to be answered is what spacing Heidi well the patent documents are very clear about this they they state that this module provides infrastructure for writing single-threaded concurrent code using carry genes would like to I access over sockets and the result is running NetWare clients and servers and other related primitive article that I want to go up and down and you have from memory but no I understand all the more to it it from the documentation it doesn't give me a 2nd or a feel of how might you use small
arms and such documentation can make model of their little bit intimidating and turns of the realm of esoteric lead to behave like treaties so he talks Korea fixed with some of and the out
talks in any way what those facing higher do well we can do a lot better than you can keep the answer simple to what they think I do as treaty says it lets you write code that concurrently handles asynchronous network based on the answer must be clear about what I mean by those terms concurrency is some things in here it happens not 1 on asynchronous so literally means not synchronize there's no way to tell when something may happen in the network of course as you know immediately communicating with with another device you survive the Internet and find it so it is actually 1 of the input output when program communicates with the
outside world so the problem clearly stated that messages arrive and depart from that would be predictable time and Eysenck I'll let you do with such interactions simultaneously so given the simple purpose I want place think idea into a practical context on so let's talk about distributed hash table so again that start with the obvious question what is distributed hash table while I'm assuming you know what's a hash table is because it's always the same thing as victim parts and we are not Europe path that role and I think it's reasonable to expect you to all understand what it is it's a simple peak value data on this is because it is distributed because the whole is built from several independent yet related part so it's a little bit like this abstract encyclopedia that the whole encyclopedia is made of performance that independent yet related to each of archaic distributed hash table is a structure that consists of many independent nodes that of the collaborate with each other over the network but this is also the centralized that's a new node is more important than than than the other so there are no client-server relationship engaged right solution it and that
would have no and so what is the hash table a DFT is its key value based on why would I want to implement 1 of these well it's a really interesting programming problem is fascinating properties as I mentioned there's no single point of failure of the GHG algorithm by look condemn on the efficiency of the huge number of nodes has good handling of fluid network membership note the joint work on a solid foundation for more complex services such as the ones that the whole the rest of this morning on this test in the real world of BitTorrent users to get any of the 3 methods as well and and obviously if you're if you like older it's an example of decentralized samples which a fascinating on more just technical and political levels what have so guess what distributed hash table Mozart have to conquer handle lots of asynchronous network based on my own which means that they think so we have a context of the how do they think I 0 how do I use a sink to make all this work so let's introduce some core concepts this 1 being given the common quite simply this encoded just keeps looping on each iteration of the loop basically just to think 1st thing is that it holds for all I events that occurred during the time it took to complete the previous iteration of the and the other thing that it does is that it it runs any callbacks that that need to be running during this current iteration of the of the loop also carries out various housekeeping the difficult facts that have yet to be executed that's not something reading and so it's important again this is sort of an articulatory took to define appalling and callbacks are so appalling is discovering the status of something external to the program on in a single I this is network related higher than that of callback is some code has to be executed with on event has occurred that's been detected by polling and this metaphor obviously the kids appalling are we there yet are we there yet are we there yet and and the mother is saying election when we get there and this creates a sort of call back in she is promising do something when some condition is met by that gets the intervention and again a graph
models so it's important to note that polling takes place once during each iteration of the loop i of and I discovered by Paul violence discovered by the calling determine with cold packs to execute join in the current iteration of all pending callbacks executed on after that the other of the success of the existing 1 and only can't continue while this is happening it is brought so the next iteration cannot start until all the sequence to be executed callbacks then
in some sense so you will be asking that there's something wrong here is
that sound very unfortunately concurrency is also a problem and there actually more than 1 way to do it so it's worth looking at was taken sometimes examine why
works in the way it it does so in this case additional fitting model several problems and have task is the record of the the records by change retrieve data in different ways task the rights of changes that task rights exchanges what's the problem here while task a sobering other radical containing of these changes so we have lots of data which is something that we want to avoid so why don't wait until about 1 task is finished before continuing so 1st you may then be followed by c and so on and this is Eastwood standard sort of terministic but what happens if they need to wait for something such as a call to another machine on that 1 while the program has to wait until the eighties network all complete out and in that situation that we can't get along with others that this you still hanging around the age so given the situation this slide programmers brothers blocks and this is an exchangeable but with writing code in history and the parliament to network-based events which is precisely the sort program that basing it on this has been designed to help with and so you probably ask yourself why not just get on with task B and C while we wait for the results in the network over and above the view thinking and then actually you've you've described quite succinctly
what they think just so what is the most important side of the tool that you're paying attention so singularities of interest and this means that never faced by is non-blocking answer with well the program does not wait for a reply from the locals for continuing program and well if I call back be and when the result of that report is that in the meantime the program continues to hold In responsible network related events and callbacks execute drawing iteration of the of the of the week after the lecture that I have is detected
confused while you should be because actually this is how we as humans think about the conference so in the real world we make plans all the time the washing machine finishes is an expected events and how close tries to call for when these expected event happened how committees stop efforts to the washing
basket and also we multitasking the same way that same kind does was to between the things we need to do while we wait for other things happen we know we will have time just please the orange juice Weil-Petersson the execute while we make breakfast we don't just stick something bridge in the market where put on for 5 minutes and then watch what I don't see you probably as with other things this is exactly the sort of thing that I'm talking we move on to this in 1 of those that anyone think eventually it's it's that this is how we get the world around us so much so that the what was on the micro it out so long my last month of the microwave goes paying and I is the expected events and
take project sons and so a single avoid potentially confusing and complicated competency will retain the benefits of string the country code and the other is the fundamental advantage of using but we plan ahead expected events by defining qualified to be called when such events you UK and in the meantime and we sequentially handle the callbacks relates with that made happened in the fact that questions before we actually start off power and synchronous 100 tasks creating how do such tasks waiting for non-blocking network-based and how callbacks to foreign to handle
the eventual result of to answer these questions we need to understand reading futures and top so I'm going to attempt to explain curvature in another 3 slides encouraging in sense is object represents an error in resenting an activity that eventually complete In order to decorate function that returns an important thing to know about the protein so that they may be defend the yield from syntax and when interpreting suspended this allows the event to get on with the other thing that readings results generated so the lazily generate results according to coaching doesn't actually start execution the yield from other objects and that when the yielded from object as a result the current team continues from the yield from statement that suspended and this is called re-entry and at the end of the chain of view from this object that returned results already except rather yielding from some of the current
so that this node that they so yeah that's just crazy that OK so this is a decorative routine method that handles an incoming HTTP requests as part of my upstream something is giving from the current routine created by the functional see something with the response that generates a list of current loopholes by the building from the readings created by the payload book read called which is in the reads the raw data posted as part of the question you have to stop trying to carry out of 101 data derived the curve was again while waiting for the current routine created by the self-growth process data method of itself which 1 of the things which is called the data of among the task encapsulated by this protein is complete returns a response the books dreamcoat routine guess the return results resumes execution from where each unit from this it's so we
cannot asynchronous activity happens but how do we handle the results are handled result of the parents were radical
works well for this you need to understand fusion task which are trying explain in just 2 slides on so future if you don't twist it this will be the future of represents a result that may not yet be available yet callback functions are added to a sort to do is to be executed when is and the task is simply future that wraps parenting and the resulting object is realized when the when the carotene conflicts of the 6 future has resolved
which sounds overeager to so here's some example of exist so the 1st thing I do is I created back should handle also talk again anything does is called it looks the result of an example is a created task associated with a certain routine that will that will cause and then I actually that task the of the the callback divided by the and in the task
object is automatically resolve the co routine that's associated with a cold and at the end I expect go remember generators of lasting up and running the event itself OK so here's another perspective on this little bit mind-bending somebody marriage and like point and that we used to work with the 1st part of the part of the mean function and understand the value of the features and tasks are going to be like first-class function call we can also pass them into functions and return them as value which is modeled knows what the result is that we can start and callbacks so let's do a
quick recon before we come to the sort of the end of the talk of the wave of 5 minutes and so and evaluating this will become the pulse of Ireland managers event handling callback parity of object represents activity that it should be complete source decorated functions and the
subject of future represents a response that may not be able to get and the city callbacks executed when it results in a set of features of the mysterious got transferred of and tasks are boilerplate future losses that rock proteins and these are realised that every single it's so a give out the but very quickly look at how they they think it works in practice with DHT put 1st gets do that I need to very quickly explain
how DHT you so as as I mentioned before the is made of and the classic way to visualize this is where the context and you'll see
what each node has a unique ID that is within the set of all possible values for certain hashing function you want what hashing functions of fuel in all holders things that more than I in my case it might be used to shot fired from the idea value indicates the nodes position in the Gulf and in this way we can tell whether node is located in the abstract network work so we can tell who is close to and how far away the nodes of features so there is some sort of
notion of distance DAG and this is a key value pair key is turned into a hash again using the same fashion of a shot of the text and the value is stored and nodes used by these are close so the hash of
the key the so this is similar to understanding where to look things up in a more people in encyclopedia going back to the beginning of articles are words all keys in our case and associated given definitions of values of that is stored in volumes cover some alphabetical range within the global
encyclopedias so of all the loans the age where
several along the z can handle oops no matter what each node maintains something called the routine table that tracks the steps of this and its interaction with with its peers results in exchange of information about the nodes that has been talked to and that's how the rooting table is populated initially kept date because nodes keep bumping into each of them answers the tablespoons splits the cost-effectiveness of into buckets look it's contain same number of pairs that will look at the cover a smaller range close to the local and they are therefore the local no knows more close nodes as you you can see me or at about half of 1 of and the the nodes by knowing the
different book it's in that the score at each node i has some very simple rules they behave quote is a very simple rules it doesn't really matter what those rules on just put a few of them to illustrate what I mean so but the important thing about the state was immediately put value chain to be able to get energy to get this set and each 1 of those fundamental actions requires the look of the case of so that you know which nodes you want interact with along these interactions are asynchronous and lookups or also parallel concurrent because you can offset the appears at the same time the about information you need so how does the look work because of of the 6 degrees of separation again in which always features that they talk
about what it so that I want a value with the key to the hash is in a position at 6 o'clock as the targets of the 1st thing I do is I asked the nodes in my kitchen table that are closest to the uninvaded
with notes from there table of of of
the here and this keeps going recursively until I find nodes that and I actually had that there are actually close to the target until I can't find any nodes that are at the top the
list so the set requires a look and how is this handled in the role of facing higher well look what is future because it's something whose result because you know so we finished looking up we need to interrogate on that there of the
state of the look up the progress of finding diagnosed cloaked to talk is held within the look of instance and the lookup results because it's it's future when the result it's not on the result is you need to be of value or not found exception in the case of the text operations which can to be in 20 closest is known nodes in the reached in of the books a will be then is the 29 stole this election so whereas network out as a sing higher handled this network approach on how do nodes in the DHT handle down 1 aspect of the so called concepts 567 streams transport protocols streams are high-level abstractions that allow you to send and receive traffic down the network of from the network using we don't want transports are provided by a single I handle low-level network activity such as TCP and UDP and they have a low level of violent and offering and them the tools the ease for you so you shouldn't really have to enter the protocol of indel network protocols at the application layer so example page 2 p
next thing so this was a lot of fun so streams of flows of data that can be read from or written to and they are built upon the transports and protocols so what is the transport transport is concerned
about how still looms over the network where for article is a protocol what so what do we know that arise from the network so it turns out that it works and how to turn the raw bytes it's something we have such as the next request http called and really units work with with streams and perhaps protocols and things like that so my final thoughts of with DHT I've managed to get 100 % unit coverage and this because they think I is part of Python and testing is is normal using the built-in into library more on and so that's how you organize your code is this is a key factor that but this is in contrast with for example twist which has its own test run on its own unit test classes to work on the same way the as the 1 in the standard library which can be confusing and my DHT implementation is small and simple so area 871 lines of code as of this morning interdigitated itself this because they think it makes it easy of about 1 minute of it makes it easier to think about concurrent problems OK so uh underneath the abstractions made he to write a simple cut this short and competence on and finally on attention the difference between buyer and CPU-bound don't use a single value if you need to do something with lots of CPU overhead because of course that's going to block the event if you have of that
however that's the sweet spot so the future of the color changes for better in in part the of 5 carotenes with the a sink and the weights and to which we don't mention gets very his favorite
features that must be good I said if you remember his security the ieee church in earlier on I simply bought is taken at the front end and the correct the was in fact few 5 on this a proper native proteins and they're not to be actually computer-generated would like remain at the new basing syntax outside the codebook 5 everything but not of some internal yields from actually makes it easier to to get to the end of the and also the features that it awaits context manager and iteration protocols involved I think the best option it and so had fallen to his predecessor of the features that of have test this characters of the average purposes only on the Finnish but you know what few years the project is probably with the given B uh any questions you as anybody have a short question short question like HTK capture no questions that all OK thank you very much
Gewicht <Mathematik>
Hash-Algorithmus
Modul
Flächeninhalt
Hypermedia
Hash-Algorithmus
Vorlesung/Konferenz
Projektive Ebene
Speicherabzug
Information
Softwareentwickler
Minkowski-Metrik
Tabelle <Informatik>
Resultante
Bit
Multiplexbetrieb
Relativitätstheorie
Modul
Code
Socket-Schnittstelle
Computeranimation
Client
Informationsmodellierung
Zustandsdichte
Festspeicher
Speicherabzug
Server
Vorlesung/Konferenz
Primitive <Informatik>
Tabelle <Informatik>
Bit
Hash-Algorithmus
Datennetz
Datenparallelität
Interaktives Fernsehen
Ein-Ausgabe
Kontextbezogenes System
Term
Code
Computeranimation
Message-Passing
Knotenmenge
Prognoseverfahren
Datennetz
Code
Hash-Algorithmus
Mereologie
Optimierung
Datenstruktur
Message-Passing
Tabelle <Informatik>
Funktion <Mathematik>
Folge <Mathematik>
Hash-Algorithmus
Punkt
Pauli-Prinzip
Fluid
Zahlenbereich
Iteration
Aggregatzustand
Element <Mathematik>
Code
Gerichteter Graph
Computeranimation
Eins
Übergang
Metropolitan area network
Loop
Informationsmodellierung
Knotenmenge
Algorithmus
Reelle Zahl
Stichprobenumfang
Hash-Algorithmus
Speicherabzug
Optimierung
Ereignishorizont
Peer-to-Peer-Netz
Tabelle <Informatik>
Softwaretest
Schlüsselverwaltung
Graph
Datennetz
Kategorie <Mathematik>
REST <Informatik>
Güte der Anpassung
Einfache Genauigkeit
Systemaufruf
Knoten <Statik>
Kontextbezogenes System
Ereignishorizont
Quick-Sort
Konditionszahl
Speicherabzug
Ext-Funktor
Tabelle <Informatik>
Lesen <Datenverarbeitung>
Differential
Resultante
Subtraktion
Programmiergerät
Datenparallelität
Mathematisierung
Familie <Mathematik>
Oval
Code
Computeranimation
RFID
Task
Virtuelle Maschine
Informationsmodellierung
Datensatz
Task
Datennetz
Vorlesung/Konferenz
Optimierung
Addition
Sichtenkonzept
Datennetz
Mathematisierung
Systemaufruf
p-Block
Systemaufruf
Quick-Sort
Ereignishorizont
Rechenschieber
Programmfehler
Rechter Winkel
Ablöseblase
Reelle Zahl
Standardabweichung
Fitnessfunktion
Resultante
Programm
Datennetz
Stellenring
Turing-Test
Iteration
Ereignishorizont
Systemaufruf
Computeranimation
Singularität <Mathematik>
Iteration
Datennetz
Optimierung
Ereignishorizont
Verkehrsinformation
Metropolitan area network
Virtuelle Maschine
Task
Automatische Handlungsplanung
Vorlesung/Konferenz
Bridge <Kommunikationstechnik>
Quick-Sort
Ereignishorizont
Computeranimation
RFID
Resultante
Lineares Funktional
Befehl <Informatik>
Sichtenkonzept
Krümmung
Code
Ereignishorizont
Computeranimation
Task
Objekt <Kategorie>
Rechenschieber
Task
Verkettung <Informatik>
Datennetz
Code
Speicherabzug
Projektive Ebene
Generator <Informatik>
Ordnung <Mathematik>
Fehlermeldung
Zeichenkette
Lesen <Datenverarbeitung>
Resultante
Prozess <Physik>
Euler-Winkel
Gebäude <Mathematik>
Anwendungsspezifischer Prozessor
Wurm <Informatik>
Strömungsrichtung
Mailing-Liste
Computeranimation
Endogene Variable
Gewöhnliche Differentialgleichung
Task
Message-Passing
Knotenmenge
Einheit <Mathematik>
Code
Verweildauer
Mereologie
Endogene Variable
Vererbungshierarchie
Reelle Zahl
Kurvenanpassung
Große Vereinheitlichung
Lesen <Datenverarbeitung>
Resultante
Lineares Funktional
Logarithmus
Versionsverwaltung
Information
Quick-Sort
Computeranimation
Web log
Task
Rechenschieber
Objekt <Kategorie>
Metropolitan area network
Task
Loop
Verweildauer
Vererbungshierarchie
Speicherabzug
Mini-Disc
Resultante
Bit
Punkt
Wellenlehre
Computeranimation
Task
Puls <Technik>
Datenmanagement
Task
Perspektive
Code
Koroutine
Ereignishorizont
Lineares Funktional
Systemaufruf
Quellcode
Objektklasse
Ereignishorizont
Quick-Sort
Systemaufruf
Ranking
Objekt <Kategorie>
Generator <Informatik>
Funktion <Mathematik>
Loop
Gerade Zahl
Mereologie
Task
Einfügungsdämpfung
Task
Funktion <Mathematik>
Menge
Loop
Code
Endogene Variable
Anwendungsspezifischer Prozessor
Ereignishorizont
Textbaustein
Computeranimation
Lineares Funktional
Hash-Algorithmus
Schlüsselverwaltung
Datennetz
Ortsoperator
Abstraktionsebene
Computeranimation
RFID
Metropolitan area network
Knotenmenge
Menge
Hash-Algorithmus
Abstand
Schlüsselverwaltung
Spannweite <Stochastik>
Wort <Informatik>
Spezifisches Volumen
Schlüsselverwaltung
Computeranimation
Tabelle <Informatik>
Trennungsaxiom
Resultante
Subtraktion
Gruppenoperation
Interaktives Fernsehen
Abgeschlossene Menge
Zahlenbereich
Schlussregel
Peer-to-Peer-Netz
Computeranimation
Überlagerung <Mathematik>
Energiedichte
Spannweite <Stochastik>
Knotenmenge
Minimalgrad
Verkettung <Informatik>
Menge
Endlicher Graph
Objektorientierte Programmiersprache
Maskierung <Informatik>
Information
Aggregatzustand
Tabelle <Informatik>
Knotenmenge
Ortsoperator
Hash-Algorithmus
Ablöseblase
Wiederkehrender Zustand
Extrempunkt
Schlüsselverwaltung
Computeranimation
Tabelle <Informatik>
Resultante
Metropolitan area network
Message-Passing
Knotenmenge
Menge
Mailing-Liste
Hill-Differentialgleichung
Computeranimation
Resultante
Nichtlinearer Operator
Datennetz
Protokoll <Datenverarbeitungssystem>
Benutzerfreundlichkeit
Abstraktionsebene
Kartesische Koordinaten
Ausnahmebehandlung
Datenfluss
Transportproblem
Maskierung <Informatik>
Computeranimation
Übergang
Homepage
Streaming <Kommunikationstechnik>
Knotenmenge
Arithmetische Folge
Speicherabzug
Idealer Punkt
Streaming <Kommunikationstechnik>
Aggregatzustand
Instantiierung
Mittelwert
Subtraktion
Gewicht <Mathematik>
Komponententest
Klasse <Mathematik>
Mathematisierung
Implementierung
Zentraleinheit
Code
Gerichteter Graph
Computeranimation
Streaming <Kommunikationstechnik>
Lesezeichen <Internet>
Einheit <Mathematik>
Programmbibliothek
Kontrast <Statistik>
Schnitt <Graphentheorie>
Gerade
Softwaretest
Protokoll <Datenverarbeitungssystem>
Datennetz
Abstraktionsebene
Ereignishorizont
Teilbarkeit
Flächeninhalt
Mereologie
Kantenfärbung
Overhead <Kommunikationstechnik>
Zentraleinheit
Softwaretest
Codebuch
Protokoll <Datenverarbeitungssystem>
Computersicherheit
Iteration
Kontextbezogenes System
Computeranimation
Endogene Variable
Konfiguration <Informatik>
Motion Capturing
Gewöhnliche Differentialgleichung
Message-Passing
Datenmanagement
Debugging
Zoom
Projektive Ebene
Gravitationsgesetz

Metadaten

Formale Metadaten

Titel Lessons learned with asyncio ("Look ma, I wrote a distributed hash table!")
Serientitel EuroPython 2015
Teil 136
Anzahl der Teile 173
Autor Tollervey, Nicholas
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/20173
Herausgeber EuroPython
Erscheinungsjahr 2015
Sprache Englisch
Produktionsort Bilbao, Euskadi, Spain

Technische Metadaten

Dauer 24:40

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Nicholas Tollervey - Lessons learned with asyncio ("Look ma, I wrote a distributed hash table!") This talk introduces the asyncio module. I'll cover what it's for, how it works and describe how I used it to write a real-world networked application (a distributed hash table). We'll explore the event loop, coroutines, futures and networking with examples from my code. This won't be an exhaustive exposition. Rather, attendees will grasp enough of asyncio to continue with their own studies. By the end of this introductory talk I hope attendees will want to learn more about asyncio and perhaps give it a try in their own projects.
Schlagwörter EuroPython Conference
EP 2015
EuroPython 2015

Ähnliche Filme

Loading...