We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Python's Parallel Programming Possibilities - 4 levels of concurrency

Formal Metadata

Title
Python's Parallel Programming Possibilities - 4 levels of concurrency
Subtitle
multiple Machines, Processes, Threads and Asyncio - what they are and why you might use them
Title of Series
Number of Parts
118
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
I'm going to talk about the 4 main levels of parallelism in modern Computing: multiple (virtual) machines multiple processes multiple threads multiple green threads, aka asyncio Why you might use each of them, how to go about doing so with python and some of the pitfalls you might fall into along the way. To do so, I'll give short examples in code of achieving each level: leveraging multiple hosts using RQ, and also the possibility of RPC with HTTP multiprocessing and threading using their respective modules from the python standard library asyncio demonstrated with AIOHTTP That sounds great, but there are ""gotchas"" you should know about before you get started, for example: multiple machines can actually be multiple virtual machines on the same host effectively communicating between processes is hard, how can we go about making it easier? the limitations of threading and the GIL runemin/emexecutor - do we ever really need to use multiprocessing or threading directly again use of asyncio when dealing with both networking between hosts and between processes - you end up using two different kinds of concurrency at the same time. That can be confusing, but also awesome. I'll finish of by showcasing a library I built, arq which is a job queueing and RPC library for python which uses asyncio and Redis.
Keywords