How async and await ended up in Python
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 132 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/44899 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
EuroPython 2018106 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
SoftwareIntelComputervirusTwitter <Softwareplattform>Twitter <Softwareplattform>HackerSoftwareentwicklerRechenschieberComputeranimation
00:32
Einfach zusammenhängender RaumUnrundheitNotepad-ComputerCoxeter-Gruppe
01:19
Exogene VariableGeradeSystemaufrufCodeProgrammiergerätExogene VariableServerProgrammierungComputeranimation
01:45
Ordnung <Mathematik>ProgrammiergerätExogene Variablep-BlockGrenzschichtablösungGeradeCodeMultiplikationsoperatorCodierungProgrammierung
02:11
ThreadCodeMultiplikationsoperatorOrdnung <Mathematik>BefehlsprozessorThreadLoopSynchronisierungWarteschlangeSoftwareDatenverwaltungKontextbezogenes SystemEreignishorizontPunktKonditionszahlVerklemmungProgrammierungKartesische KoordinatenTaskGruppenoperationProgrammbibliothekSystemaufrufParallelrechnerSchreiben <Datenverarbeitung>Computeranimation
04:03
Funktion <Mathematik>MomentenproblemCodeEreignishorizontLoopGemeinsamer SpeicherSpeicherabzugSynchronisierungZweiComputeranimation
05:03
ComputervirusZweiTaskHochdruckp-BlockProgrammierungComputeranimation
05:35
ProgrammierungKoroutineSpeicherabzugp-BlockGewicht <Ausgleichsrechnung>SoftwaretestMultiplikationsoperatorArithmetischer AusdruckZweiMomentenproblemTaskTermComputeranimation
06:07
Komponente <Software>MultitaskingKoroutineProgrammComputerZusammenhängender GraphPunktMultiplikationURLMultiplikationsoperatorProgrammierungTouchscreenComputerunterstützte ÜbersetzungKoroutineMultitaskingGoogolComputeranimation
06:44
Ordnung <Mathematik>Funktion <Mathematik>KoroutineOrdnung <Mathematik>KoroutineGerichteter GraphErzeugende FunktionSpeicherabzugGewicht <Ausgleichsrechnung>GeradeMinimumAusnahmebehandlungIterationTypentheorieProtokoll <Datenverarbeitungssystem>Message-PassingComputeranimation
07:47
HochdruckMessage-PassingBefehl <Informatik>Funktion <Mathematik>GamecontrollerHochdruckPunktComputeranimation
08:23
IterationObjekt <Kategorie>AusnahmebehandlungMultiplikationsoperatorWärmeübergangZeichenketteFunktion <Mathematik>Streaming <Kommunikationstechnik>Klasse <Mathematik>SystemaufrufGamecontrollerLoopMAPComputeranimation
09:57
IterationZahlenbereichAusnahmebehandlungComputeranimation
10:28
Wort <Informatik>MultiplikationsoperatorIterationLoopAusnahmebehandlungInstantiierungErzeugende FunktionFunktion <Mathematik>Konstante
11:45
Funktion <Mathematik>Funktion <Mathematik>Erzeugende FunktionPunktDatensatzHochdruckObjektorientierte ProgrammierspracheZeichenketteIterationComputeranimation
12:45
Funktion <Mathematik>PunktFunktion <Mathematik>Kontextbezogenes SystemGenerator <Informatik>Suite <Programmpaket>ZweiIterationAusnahmebehandlungHochdruckComputeranimation
13:44
ComputervirusGeradeArithmetischer AusdruckVariableMultiplikationsoperatorIterationAusnahmebehandlungLoopStetige FunktionHilfesystemGenerator <Informatik>ZeichenkettePunktComputeranimation
15:26
HochdruckHochdruckInstantiierungBitZeichenketteErzeugende FunktionComputeranimation
15:50
Komponente <Software>MultitaskingKoroutineProgrammComputerPunktZusammenhängender GraphGenerator <Informatik>MultiplikationsoperatorZeichenketteAusnahmebehandlungMultitaskingFunktion <Mathematik>Leistung <Physik>Primitive <Informatik>MultiplikationURLZweiIterationTopologieCodeObjekt <Kategorie>Interface <Schaltung>KoroutineOrdnung <Mathematik>Computeranimation
17:56
KoroutineProgrammierungGenerator <Informatik>MereologieThreadEreignishorizontCodeLoopTaskBildschirmmaskeDatenparallelitätResultanteGerichteter GraphPunktDifferenteFunktion <Mathematik>ParallelrechnerKoroutineTypentheorieFahne <Mathematik>Computeranimation
19:20
Gerichteter GraphRechenschieberGewicht <Ausgleichsrechnung>Arithmetischer AusdruckObjekt <Kategorie>ProgrammbibliothekVersionsverwaltungSynchronisierungBitProzess <Informatik>Interface <Schaltung>PunktProgrammierungÄußere Algebra eines ModulsEin-AusgabeSpeicherabzugOrdnung <Mathematik>TabelleKlasse <Mathematik>IterationTeilbarkeitVerkehrsinformationMultiplikationsoperatorFunktion <Mathematik>Projektive EbeneSoftwareentwicklerFormale SprachePhysikalismusTypentheorieFluss <Mathematik>CodeGenerator <Informatik>ParallelrechnerBitrateMixed RealityMatchingMomentenproblemSchnelltasteCASE <Informatik>Kette <Mathematik>Lokales MinimumÄhnlichkeitsgeometrieKoroutineDokumentenserverSchlüsselverwaltungSoftwareDifferenteComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:08
My name is Pavlin, and I work as a software developer in Hacksoft, which is an outsourcing company, and I do mainly Python, which is Django, and JavaScript, which is React.
00:28
And you can find me in Twitter, just like this. So, I want to start with this slide was not initially in my presentation, and you might already know, but my talk was scheduled for Wednesday, but, suddenly, my flight got delayed,
00:45
and we missed our connection, so I want to thank, firstly, to Alexander Handorf, and for the assistance, and Owen Campbell for stepping in and taking my slot, which made
01:01
this talk possible, so if you can do a round of applause for them, it will be really good. And a little disclaimer, my talk was scheduled for 40 minutes, but now I have only 30, so bear with me, and let's get started. So Python is synchronous.
01:22
What does that mean? It means that each line executes in order from top to bottom, and knowing that, let's take a look at the following example. You have a line of code that makes a call to a remote server, and this means that
01:40
the programme is doing nothing while it's waiting for the response. It just waits. And the programme blocks and waits for the response in order to continue, so what if you have several lines of code that do the same? Each line will execute when the previous line is done waiting, and, in a short, you
02:01
will wait more, and you will be blocked more time. So how can we solve this? How can we speed up the programme? Can we make it non-blocking? Of course you can. And one way to do this is with threads. Threads are dead. If you haven't seen this talk by David Beasley, I highly recommend you to do it.
02:23
So you can speed up multiple threads, each thread doing one thing at a time, and when multiple threads are running, each CPU can run one thread at a time, so in order to allow if you have more threads, in order to allow all threads to share resources,
02:41
the CPU very often context switches between the threads in a non-deterministic order. And threads are great, but sometimes they can make the code more complicated and harder to understand, and there are also race conditions and deadlocks.
03:00
And about the non-blocking thing, there is another way to make it. We can make our code asynchronous. With asynchronous programming, the software, e.g. our application, manages the threads and the context switching rather than the CPU, which means that the context is going
03:21
to be switched at defined points rather than in non-deterministic interval. And one way to do asynchronous programming is with an event loop. So there is an event loop and there is a queue of events, so-called tasks, and the loop constantly pulls tasks from the queue and runs them.
03:43
One way to do asynchronous programming in Python is with AsyncIO. AsyncIO is a standard library which was added in Python 3.4 and provides an event loop. So let's take a look how can we use AsyncIO to write asynchronous code in Python.
04:04
So this is what we have to do. In this example, we import AsyncIO at the top, and then we define a coroutine using the async dev keywords. So this function is the coroutine.
04:20
It's like a normal function but it has async in front of the dev and it has an await expression and we're going to talk about that in a moment. So what this coroutine does is it prints hello, it waits several seconds, and then prints well with the respective delay, and then finishes.
04:42
So the next thing we do is we get the event loop, we create tasks and shadow them in the event loop, and then we run the event loop forever. Let's see what is going to happen if we execute this code in Python 3.6.3.
05:04
So hello will be printed twice, as we saw. We have hello at the top. So hello will be printed twice, after one second, world with delay one will be printed, and after another second, world with delay two is printed.
05:20
So both tasks start simultaneously and the second task finishes one second later than the first task. So this is exactly what we wanted, right? I mean the program didn't block. So the program didn't block. We shadowed two tasks and they started at the same time.
05:41
The second task didn't have to wait for the first task to finish, and we achieved that by defining a coroutine with async dev keywords, and we used an await expression inside it, which we're going to talk about in a moment. But what exactly is a coroutine?
06:01
What does that term mean? Easiest thing to do, let's ask Google about that. So, Google tells us, a coroutine, the coroutines are computer program components that generalize subroutines for non-preemptive multitasking by allowing multiple entry points for suspending
06:21
and resuming execution at certain locations. Well, my understanding about this definition hasn't changed since the first time I read it. I mean, I remember looking at my screen just like the cat in the picture, and my first thoughts were, what if I have to explain this to someone else?
06:43
I mean, this is a picture of me and my niece, Ralitza, she's two years old and she's really cute, and I love her so much. So in this picture, she's teaching me how to draw here, and believe me, she's better
07:01
than me, even though she's two years old. So what do I have to explain to her what a coroutine is? I will have to start with the basics. In this talk, we are going to talk about order of execution, the iterator protocol, generator functions, we're going to talk about use from, and then we will make a
07:25
definition of a coroutine in Python, and then we're going to look at the asyncio.coroutine and types.coroutine decorators, and finally, last but not least, we're going to talk about async and await. So order of execution, as we already said, in the beginning, each line executes from
07:45
top to bottom in order. So until an exception is raised, for example, in this example, the message after the raise will never be executed. Another way to end the function's execution is when a return statement is reached.
08:01
In this example, the print message will never be executed. So return implies that the function is returning control back to the point from where it was initially called from. And another way to, another way to, yeah, there's yield.
08:29
I mean, yield implies that the transfer of control is temporarily and the function expects to regain it in the future. We will get back to yield in a minute, but before that, we have to mention a few
08:42
more things. First, iterable. What's an iterable? The iterable is an object that is capable of returning its members one at a time. For example, a list, a tuple, dict, string, et cetera, so an iterable is an object
09:04
of any class that has defined the underscore, underscore, iter, underscore, underscore method. So in a short, an iterable is something that can be used on the right side of the for loop.
09:21
So iterator, this was iterable, this is iterator. What's an iterator? That's an object representing a stream of data, and repeated calls to the iterator's next method returns successive items from the stream, and when no more data is available,
09:43
a stop iteration exception is raised instead. Iterators are required to have an iter method that returns the iterator itself. Let's check out an example. So this is a pretty useless iterator class.
10:02
Here, that iterator will return the numbers from one to three, and as we already said, we have an iter method that returns self, and in the next method, we return the numbers
10:20
from one to three, and if the next number is higher than three, we raise a stop iteration exception. Let's see how can we use it. So first we have to make an instance of our iterator class, and then we can manually call the next method on it. The first time we call next, it returns one, next time it returns two, third time
10:41
it returns three, and if we call it again, it will raise a stop iteration exception, which is really nice. So another way to get its values is to iterate over the iterator with the for loop. So what the for loop does is it calls the iter method and then consecutively calls the next method of the iterator, and when a stop iteration exception is raised, the loop is
11:04
exited successfully. So let's get back to the ute keyword. So what does that word mean, or what does that word do? Let's talk about generator functions. So if the body of a function contains ute, that function automatically becomes a generator
11:23
function, and once we call the generator function, it returns generator iterator, and we can get the values from the generator iterator by calling next on it, and once the generator is exhausted, it will raise a stop iteration, and you can consume the
11:42
values of a generator only once. Let's check out an example. So this is a normal function, but instead of returning, we return values by yielding them, and since the function has ute inside it, that function is a generator function.
12:00
Let's see how can we use it. So when we call the generator function, nothing will get printed. It will return a generator iterator, and from now on, when I say a generator, I will be referring to the value returned after calling the generator function.
12:22
So gen is generator iterator, and I will call it a generator. So how can we get the values? We have to call next. If you saw the function, you will see that it prints a string, returns one, and what
12:43
happens is that, let's see the function again, what happens is that at the point where it returns one, the function pauses. It's not destroyed, it pauses and it keeps its context. So at this point, when it returns one, when we call next, it will resume from the place
13:06
where it was paused, e.g. the first ute, then it will print ute in second value, it will return two, and finally it will raise a stop iterations exception. So generators have a send method.
13:21
What does the send method do? Well, it resumes the execution of a generator and passes a value into it. I mean, sending none in the send method is equivalent to calling the next method of a generator, and the send will return the value either by the generator just like
13:42
next does. Let's see an example. Everything is the same except this line over there. So what we've done is we've assigned the ute expression to a variable, and the value that we're going to pass to the send method will be assigned to that variable.
14:01
Let's see how it works. We have to instantiate our generator, which returns the generator, which I call a generator, and we can call send with none, which is equivalent to calling next on the generator,
14:22
which is going to print, going to ute value, and it's going to return 42. So the next time we can call the send method with a string, which will be assigned to the variable received, and it will be printed. And the next time we call send with none or with a string, it doesn't matter,
14:42
a stop iteration exception is raised. So in Python 3.3, a new feature comes in play, ute from. The ute from expression, so if we loop over an iterator and ute each value, this will
15:02
be equivalent to yielding from the iterator, and when the iterator is a generator, it will be exhausted first, and it will get every value from that generator, and then the execution continues. How is this helpful?
15:20
It's helpful because ute from allows us to chain generators, which is really cool. Let's see an example. This is a little bit complicated. We have a generator for a generator, which utes one, prints some string, and utes two. And in the second generator, we make an instance of that generator function, we ute from it,
15:43
we ute from the generator, then we print the string, and we ute three. So what happens when we try to run that code? We get the second generator, and the first next code pauses the second generator at the
16:01
point of ute from, and goes into the first generator, which returns one, so the next time we call next, the string from the first generator is going to be printed, which is in the middle of the first generator, and two is going to be returned, and the next time we call next, in the middle of the second generator, three is returned, and
16:22
then it raises the exception. Knowing all of this, I mean the functionalities, the capabilities of generators, let's take a look at the definition from the beginning. All routines are compute and program components that generalize subroutines for non-preemptive
16:42
multitasking by allowing multiple entry points for suspending and resuming execution at certain locations. Now, it looks like a coroutine is an object which implements the methods of a generator, and so suddenly all generators implement the coroutine interface, but the generators
17:08
were not meant to be in use that fashion, so with its arrival in Python 3.4, I think AsyncIO provides a useful decorator in order to solve this issue.
17:22
So what this decorator does is, well, it is used to label a function as acting as a coroutine that was meant to be used with AsyncIO, and AsyncIO required all generators
17:41
that are going to be used with AsyncIO to be decorated with this decorator. So let's check out the example with AsyncIO which was from the beginning and was using Python 3.6.3. This is the example. Same thing.
18:00
So let's compare it with how AsyncIO was used back in Python 3.4 when AsyncIO originated. The only difference here, as you can see, is that we are not using AsyncDef, but using a normal function which is a generator because it has yieldFrom, and we decorate
18:23
it with AsyncIO, and instead of awaiting, we use yieldFrom. And with generator support and an event loop in the form of AsyncIO, at the point of Python 3.4, there was enough to support asynchronous programming in Python in the
18:42
form of concurrent programming. What does concurrent programming mean? It means that with concurrent programming, we write code that is supposed to be executed independently of other parts, but it all executes in a single thread. I mean, the event loop is running in a single thread, and we create tasks and shadow them
19:04
in the loop, and when the task is monitored by that loop, and when the task is done, its result is sent back to the coroutine via the send method that we already looked at. And in Python 3.5, the types coroutine decorator comes, which what it does is it flags a
19:31
generator as a coroutine, just like AsyncIO.coroutine does, and it looks like finally we have arrived.
19:43
I guess all of you are here for this slide and the next one. So the Async keyword goes before def, I guess I have to click, yeah, the Async keyword goes before def in order to show that a method is asynchronous.
20:02
And AsyncDef is used to define a function as being a coroutine. And the key thing about Async and types.coroutine decorator do is that they tighten the definition of what a coroutine is. It takes coroutines from simply being an interface to an actual object.
20:23
And if you use the inspect model and use the function isCoroutine, it will only return true for the functions that are defined with AsyncDef, and this makes the distinction between a generator and a generator that is meant to be a coroutine a lot more strict.
20:46
So the next slide, await. The await expression also comes in Python 3.5, and the await keyword is only valid inside AsyncDef functions.
21:01
Await operates much like yield from, but the acceptable objects to an await expression are a little bit different. When you call await on an object, that object needs to be awaitable. What does that mean? Well, I guess you can...yeah.
21:23
It was easy. I mean, awaitable object is object of a class that defines the await method. And the await method should return an iterator, but that iterator should not be a coroutine itself. I mean, coroutines are awaitable and they are awaitable objects and they can be
21:44
used on the right side of the await expression. The key thing here is that await will not accept a generator that is not flagged as a coroutine. I mean, even though generators kind of support the coroutine interface, this is the key
22:03
thing because the await expression protects us from accidentally using a generator instead of a coroutine. But there's a catch, as always. First, you cannot use yield in async-def function. And the only way to return a value in an async-def function, which we can call coroutine,
22:26
is by using return, either return or await. So async allows us to distinct coroutines from generators and the await expression makes sure that one does not mix and match objects that match the same interface, I
22:45
mean the API of coroutines, and makes it more clear that these objects are waiting for coroutines to finish. And the await that we saw is the switch point.
23:00
I mean, this practically means go do something else while I'm waiting for something to finish and when I'm done, come back here and continue. And async and await in general make it more clear that the code is asynchronous. So once methods are not confused with generators.
23:24
And the follow-up question is, what's next? I mean, async and await provide us an API for asynchronous programming in Python. And the good thing is that people are using it.
23:43
David Beasley, he made an awesome talk. And he made Curio, which is an alternative for AsyncIO. And from what I saw in his keynote, which I highly recommend you to watch, is that what he did there was based on Curio.
24:01
And Nathaniel Smith made Trio, which is another library for asynchronous programming. And the good thing is that AsyncIO is a core library in Python. And it's adapting, and it's evolving, and it's looking. And Curio and Trio are helping the asynchronous Python world to evolve.
24:29
And for me, what it means is that I have to make another talk. Because as a software developer, my job is to write good code and to improve it and evolve it
24:44
and support all functionalities. And this is what I do in my work project. And this is what Python core developers are doing with the language that I'm using in order to write my project in work.
25:00
And that's the definition of software development, actually. I don't know if I said it, but I'm a physics graduate. I haven't been in a software development school. And when we learn new things, we apply them and we refactor our code. And while we refactor the code, we learn new things and repeat again the process.
25:26
I mean, learn something new, refactor, learn something new, refactor, evolve, improve. And personally, I observe how Python is evolving with great anticipation.
25:41
And I really can't wait to see what's next. And thank you. Thank you, Pavel. Yeah, you can find all the code snippets in this repository.
26:04
Okay, we have time for one, maybe two questions. Anyone? Here, okay. Hi, great talk. If we want to call out to something outside of Python, like C++ code written and bound
26:27
with something like Pybind or something, can we also use asynchronous functions and how do we go about doing that? Well, let me check if I understood the question correctly.
26:42
You are asking how to call asynchronous code from a synchronous code. Asynchronous from synchronous or the other way around? I'm asking how to make an asynchronous function that calls out outside of Python, calls something outside.
27:01
It could be making an HTTP request. The thing that it calls, the synchronous function has to call another asynchronous library. I mean, your best bet is to have a library written in an asynchronous manner. That's what I'm asking. How do I do that? You have to write your library to be asynchronous.
27:23
And how do I do that? I don't know. Okay. Any other questions?
27:41
Hi, thanks for the talk. I'm curious if there's like a use case for using the generator way to do it or to use the decorators or the sink ioc decorator or the types decorator or are those just obsolete and we should just always use async and await?
28:01
The current moment you should always use async and await and I don't have much experience with asynchronous code, I haven't written a lot of code with it, but at the moment you have async and await and you should use that. The generators were there because they were needed but they are not needed any more.
28:22
One more quick question. A similar question about use cases. I was curious, what is the use case for using yield from as opposed to using tools chain? Can you repeat? Using tools chain and using yield from, what is the use case for using yield from?
28:44
I'm not sure. I don't know. I can't answer that question. Okay. Thanks. Thanks.