We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Speed Up Your Data Processing

Formal Metadata

Title
Speed Up Your Data Processing
Subtitle
Parallel and Asynchronous Programming in Data Science
Title of Series
Number of Parts
130
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
In a data science project, one of the biggest bottlenecks (in terms of time) is the constant wait for the data processing code to finish executing. Slow code, as well as connectivity issues, affect every step of a typical data science workflow — be it for network I/O operations or computation-driven workloads. In this talk, I will be sharing about common bottlenecks in data processing within a typical data science workflow, and exploring the use of parallel and asynchronous programming using concurrent.futures module in Python to speed up your data processing pipelines so that you could focus more on getting value out of your data. Through real-life analogies, you will learn about: 1. Sequential vs parallel processing, 2. Synchronous vs asynchronous execution, 3. Network I/O operations vs computation-driven workloads in a data science workflow, 4. When is parallelism and asynchronous programming a good idea, 5. How to implement parallel and asynchronous programming using concurrent.futures module to speed up your data processing pipelines This talk assumes basic understanding of data pipelines and data science workflows. While the main target audience are data scientists and engineers building data pipelines, the talk is designed such that anyone with a basic understanding of the Python language would be able to understand the illustrated concepts and use cases.