Asyncio Evolved: Enhanced Exception Handling with TaskGroup in Python 3.11
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 141 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/68642 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2023130 / 141
8
17
22
26
27
31
42
48
52
55
56
59
64
66
67
72
73
77
79
83
86
87
95
99
103
105
113
114
115
118
119
123
129
131
135
139
140
141
00:00
Staff (military)Exception handlingMultiplication signTask (computing)Group actionComputer animationLecture/Conference
00:29
CodeSoftwareTask (computing)Group actionCore dumpError messageCoroutineSoftware developerImplementationMultiplicationException handlingBinary filePauli exclusion principleGauge theoryExecution unitFunction (mathematics)Inclusion mapScheduling (computing)Loop (music)Computer virusLetterpress printingLatent heatGroup actionException handlingFunctional (mathematics)Task (computing)SequelMultiplication signPower (physics)Software developerGraph coloringShared memoryLibrary (computing)Motion captureCASE <Informatik>Operator (mathematics)Point (geometry)Sampling (statistics)Single-precision floating-point formatAndroid (robot)Context awarenessSimilarity (geometry)AuthorizationData managementDifferent (Kate Ryan album)Computer configurationCodeTwitterProcess (computing)Core dumpPressureAxiom of choiceBitSoftware engineeringRevision controlCoroutineFerry CorstenPairwise comparisonFocus (optics)DatabaseFreewareLevel (video gaming)WritingConcurrency (computer science)System callComputer animation
09:12
Error messageData typeException handlingTask (computing)Personal digital assistantConcurrency (computer science)Computer configurationMotion capturePauli exclusion principleGroup actionTask (computing)Process (computing)Core dumpResultantFrequencyLetterpress printingElectronic mailing listFunction (mathematics)Functional (mathematics)BitVapor barrierMusical ensembleSingle-precision floating-point formatThread (computing)Exception handlingGroup actionTypprüfungCASE <Informatik>Set (mathematics)Error messageCodeCategory of beingDifferent (Kate Ryan album)Shooting methodPauli exclusion principleComputer configurationMultiplication signAreaForcing (mathematics)WeightGraph coloring2 (number)Boundary value problemMathematicsConstructor (object-oriented programming)Content (media)SubsetLevel (video gaming)Key (cryptography)Moment (mathematics)System callType theoryCoroutineFlow separationComputer animationLecture/Conference
17:54
Task (computing)Personal digital assistantException handlingFunction (mathematics)GodError messageCone penetration testProcess (computing)Letterpress printingIndependent set (graph theory)Code refactoringComputer virusThermal expansionMach's principleCodeCalculationData compressionFunctional (mathematics)AreaCodeComa BerenicesException handlingTask (computing)Level (video gaming)Process (computing)Video gameSpeech synthesisStandard deviationError messageOcean currentPoint (geometry)Single-precision floating-point formatFunction (mathematics)Core dumpParameter (computer programming)DiagramOrder (biology)Group actionResultantSinc functionCASE <Informatik>DivisorFerry CorstenData managementContext awarenessCode refactoringData miningBitSystem callMultiplication signSubsetWritingSweep line algorithmCompilation albumBeat (acoustics)Complete metric spaceComputer animation
26:37
TwitterComputer animation
Transcript: English(auto-generated)
00:05
I will talk about asyncio-above enhanced exception handling with Task Group in Python 3.11. This is my first time in Prague. It's such a wonderful city.
00:21
I want to thank the staff for this wonderful conference. My name is Junior Fukuda from Nagano in Japan. And you can find me on Twitter on JuniorFFF. In English, my name often gets confused with Junior.
00:41
But it's not Junior, it's Jun-ya. If it's hard to pronounce, feel free to call me Jun. Please feel free to take picture and tweet. I'm a software engineer at Grubex Inc. And I also write and translate English book.
01:06
Currently, it's only available in Japanese. So it's an excellent choice for those wanting to learn Japanese. Well, today, I would like to introduce a new future of asyncio and new, hello-ish world.
01:23
This is our goal for this talk. Introduce Python 3.11's new API, AsyncIO Task Group. Compare and explain the difference between Python 3.10 and Python 3.11.
01:42
We can write exceptions easily, so this is how cancellation works. I hope this talk will inspire you to give the new API AsyncIO Task Group a try. I will guide you through a reliable and safe way of writing with AsyncIO Task Group.
02:08
And I hope that together we can reach this new hello-ish world. Why? Am I giving this talk, you might ask? There are too many reasons.
02:23
So I can cross that tweet by Python Core developer, Yuri. He's one of my heroes and the author of the asyncawait-pep. When Task Group was merged, he kept tweeting with excitement.
02:43
The first reason is that I wanted to understand what was so amazing. At first, I didn't understand why I was so excited. The second reason is this. There is my favorite book, Using AsyncIO in Python, published by O'Reilly.
03:03
There was a code I didn't understand. This code is rather long for a quick start. I want to simply the code for the hello world with every asyncio. I found that AsyncIO Task Group can do that.
03:21
I'm giving this talk to share simple and powerful asyncio with all of you. Our agenda for the discussion includes the following key points. First, we will explore the concept of capturing concurrent exceptions. We will turn our attention to how we can cancel tasks when an exception arises.
03:49
This is an important step in maintaining the smooth operation of our asynchronous code. Finally, I will try to reach AsyncIO's new hello world.
04:05
Here's what I'll be talking about today and what I won't be. What's that in my mind? Let's get started. I have a question for you all.
04:22
Are any of you using AsyncIO? If so, thanks. Thank you very much. Many people I accept are using it. We are all in this together.
04:40
I apologize what I prepared is for the case where many people are not using it. Please listen with the mid-test of someone who hasn't used it yet. Let's quickly go over AsyncIO. AsyncIO is Python's standard library for writing asynchronous code.
05:03
It enables handling multiple tasks concurrently. Especially effective when dealing with IO-bound tasks such as database access or HTTP requests. With every new Python version, AsyncIO continues to evolve.
05:20
High-level APIs have become more and more solid and very user-friendly to write code for. Sure. With that in my mind, why don't we use it? It is really so. Let's think about it a bit. So I've given some thought to why we might not use AsyncIO.
05:46
Here's a quick rundown. Today, the focus is here. Async processing seems like it might behave unproductively. What happens if your exception occurs in this process?
06:05
Have you mentioned or been asked this during a review? Seems like it might behave unproductively. I'm concerned about the behavior of the coroutine when it is executed.
06:31
A new feature that solves this issue. That's where AsyncIO task group comes in.
06:40
Aided in Python 3.11. The basic feature of AsyncIO task group is multiple tasks and handled concurrently. All at the same time. Similar functionality was available even before Python 3.10. Such as AsyncIO gather or AsyncWaitATC.
07:05
Restart with the official document. Here is some code taken from the official documentation. AsyncIO task group uses a context manager to run multiple tasks at the same time.
07:20
In this example, it's run the asynchronous function, some call and another call at the same time. The tasks are finished when the context manager is executed. However, you can also do this with Python 3.10.
07:43
I tried writing the same code using AsyncIO gather. Actually, this looks simpler. So what makes AsyncIO task group special? Task group has two major features. One, capturing concurrent exception and canceling tasks when exception occurs.
08:06
This API excellent when dealing with exception and cancellation. Let's look at each feature one by one. First, capturing concurrent exceptions.
08:22
When handling multiple tasks together, naturally exceptions can also occur concurrently. Let's compare Python 3.11 and 3.10. Let's see how we could capture exception in version prior to Python 3.10.
08:43
The AsyncIO gather function has an option for exceptions called returns exceptions. Let's explain this. Today's talk has a lot of code. It may also be a little hard to see.
09:00
Check it out here. Let's get set up. There's an asynchronous function to run concurrently. The first is coroutine call success, which returns success. The second is a coroutine call value error, which raises value error.
09:23
And the third is a coroutine call type error, which raises type error. Run these at the same time to see how they proceed, process. Please note that we have omitted await here for the sake of explanation.
09:46
Let's take a look at the care when. Use specially false for this option. This is what the code looks like. Execute the three async functions.
10:02
This created area at the same time. Let's execute this code. I will only handle value error and cannot catch type error. I will catch the first exception that occur like this.
10:24
Next, let's take a look at the scenario when the return exception option is set to true. In the case when the return exception option is set to true, you will reiterate the result as a list after executing all the tasks.
10:47
Therefore, you need to check the list for the result. This is what the code looks like. I will run tasks together using gather.
11:01
The difference from before is that return exception set to true. Let's see how this is handled. The result will be returned as a list. Therefore, it is necessary to have a process to check the list.
11:25
I will add a process to check the result list. The code is simple. The purpose is to check for errors. Now let's run the code again.
11:42
To handle errors properly like this, it is necessary to perform checks. When return exception is set to false, only the first exception can be called. All the occurred exceptions cannot be captured.
12:02
When return exception is set to true, it is necessary to check the return list of values. All tasks must be performed before a result can be rigid. By setting return exception to true, you will receive a list of results
12:22
that includes both the successful results and exceptions that occur during the exception of the task. To handle errors or process the results correctly, you need to iterate over the list and handle each item accordingly.
12:46
Additionally, all tasks will be executed regardless of whether an exception occurs during their exceptions. For example, let's consider a scenario
13:02
where you are making 10 HTTP requests concurrently. Suppose an exception occurs in the first task. In this case, you would need to wait until the remaining 9 tasks, which might also fail, are completed.
13:22
This means waiting for all 10 results, even if they all end up with the same error. The solution is asyncIOTaskGroup, and the new syntax accepts the start syntax.
13:45
I will execute the same async function as before. There are three async functions. We will modify asyncDFMake. First, let's simply execute multiple tasks and receive the results in a list.
14:03
Each will look like this. Now, how should we write the exception handling? Add try, add accept star, value error, type error. Using accept star, we can catch each exception separately.
14:24
Now, let's execute this code and see the result. You can see it catching both of the two exceptions, value errors and type error.
14:43
It can catch exceptions occurring concurrently. It is written in a way that is easy to read, similar to synchronous processing. This is the first feature of asyncIOTaskGroup.
15:02
The key here is accept star. This is a new syntax in Python 3.11. Let's take a look at accept star for a moment. Accept star is a new syntax in Python 3.11, accepted in PEP 6054, exception group and accept star.
15:29
Use when you need to write multiple unrelated exceptions. PEP 6054 lists five motivations.
15:43
Among them, there are concurrent errors. It mentions a change of asyncIOGather, which does have a good way to handle exceptions thrown by multiple tasks. The C Python code for asyncIOTaskGroup.
16:03
During a dander I exist, its boundaries are all errors and throws a base exception group. It is the content of the error, only subclass of exceptions. The base exception group's constructor returns an exception group.
16:26
See this reference for the exception group. Next. Canceling tasks on exception. What happens to the remaining tasks
16:43
being executed concurrently when an exception occurs? Depending on the situation, the remaining tasks may just continue to run. Let's look at an example
17:01
of when asyncIOGather is used to use and tasks are left over. As an async function, wait a bit. There are three functions. As an async function called call along,
17:22
asyncIO slip and print. Execute three async functions in asyncIOGather. We are done for clearly when it is executed. Now let's execute. Sorry.
17:46
It will output barrier and then done. After done, there will be an output from the added asynchronous function call along saying call along task is still running.
18:03
In asyncIOGather when an exception occurs, other tasks remain running. This make is difficult to predict behavior. You may want to stop or roll back what you do.
18:24
Cancellation comes into play here. In some cases, the remaining tasks are executed as they are. Cancellation is not new feature. You can also cancellation with asyncIOGather. Let's modify the previous code.
18:44
There are three points of modification. First, we add a postpress processing to the call along async function that gets canceled. Next, we check for remaining tasks and instruct them to cancel.
19:02
Finally, we wait for the completion of the post-cancellation process. We use task cancel function to cancel a task. It sends a cancellation request to the task.
19:23
Then asyncIO canceled error exception is sent to the canceled task. Therefore, you need to cancel asyncIO canceled error within the async function. Also, you need to send the asyncIO canceled error
19:41
back from within the async function. I'm a little confused. Let's try to implement it. Let's try to modify the async function call along. We need to catch the canceled error. We will use try except to catch the canceled error.
20:08
Next, we add the post-cancellation processing and then we throw the canceled error. When we throw, the task will be treated as canceled.
20:23
When using asyncIOGather, it's necessary to check remaining tasks, instruct them to cancel. And then wait for the processing to complete. For each of three, we use task cancel,
20:41
task done, and asyncIF. Let's modify the code. Here's the code from earlier. Easy to check task status, the process is the same. Unfinishes tasks with task done.
21:03
We run task cancel for unfinished tasks. Then we wait for compilation with asyncIO sweep. The overall picture looks like this. Review beats the output in case a task remains.
21:24
This is code before modify can show. Other tasks will call in to execute when exception occurs. Here's the modified code. We should see the output column task was canceled
21:42
and the entire process should complete right. When canceled, the async function gets the cancellation.
22:03
This is the basic for cancellation. When using asyncIOGather, it was necessary to check remaining tasks. Tell them to cancel and wait for processing.
22:21
Let's examine the case with asyncIO task group. With asyncIO task group, all remaining tasks get canceled automatically when the context manager is exited. And it waits for the cancellation process to finish.
22:43
Simply put, there's no extra work required. The tasks get canceled on their own. Now, let's refactor the code we just wrote. Here is the code from earlier. Refactor here first.
23:03
Turn off gather and review the task creation. Put the result in one of the get result. Put answer next to the accept. Checking the task status and waiting for the cancellation process are not necessary.
23:24
So we will remove them. It's the same as using the task group normally. With this, we can cancel the tasks. Let's give it a run.
23:44
The order of catching exceptions and processing after cancellation has been switched. This is because we are doing the cancellation process after catching exception before. Task group properly manage the tasks within.
24:05
Since everything is wrapped up within the task group, it results in more productive behavior. Now, that we have seen features of AsyncIO task group,
24:21
exception and cancellation has been simplified. Finally, write AsyncIO code in reliable and safe way. The new standard how it works. First, we import AsyncIO. Next, let's define our Async function.
24:43
After that, we line up the Async function. We want to process concurrent using a task group. Oh, and don't forget to consider exceptions. We don't have to worry about cancellation here.
25:03
It's handled automatically. Finally, we execute it with AsyncIO run. How about this for Harrowish World at Python 3.11? Considering cancellation and...
25:20
Sorry. Considering exceptions and cancellations. Quick start introduced first. It was checking the status of the task and waiting for the cancellation process. Some function and argument are deprecated,
25:43
so if I went to do this in Python 3.11, I would probably do the following including exceptions. The book is excellent. Please check it. Catching and canceling exception with AsyncIO task group was introduced.
26:08
I'd be delighted if this talk serves as an opportunity for you to consider your current code
26:21
or speak interested in using AsyncIO for the first time. Please enjoy a more comfortable AsyncIO life. Hello? Sorry. We have to stop now.