We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Parallelism Shootout: threads vs asyncio vs multiple processes

Formal Metadata

Title
Parallelism Shootout: threads vs asyncio vs multiple processes
Title of Series
Part Number
143
Number of Parts
173
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language
Production PlaceBilbao, Euskadi, Spain

Content Metadata

Subject Area
Genre
Abstract
Shahriar Tajbakhsh - Parallelism Shootout: threads vs asyncio vs multiple processes You need to download data from lots and lots of URLs stored in a text file and then save them on your machine. Sure, you could write a loop and get each URL in sequence, but imagine that there are so many URLs that the sun may burn out before that loop is finished; or, you're just too impatient. For the sake of making this instructive, pretend you can only use one box. So, what do you do? Here are some typical solutions: Use a single process that creates lots of threads. Use many processes. Use a single process and a library like asyncio, gevent or eventlet to yield between coroutines when the OS blocks on IO. The talk will walk through the mechanics of each approach, and then show benchmarks of the three different approaches.
Keywords