We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Building reproducible distributed applications at scale

Formal Metadata

Title
Building reproducible distributed applications at scale
Title of Series
Number of Parts
130
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Packaging in Python is hard. Packaging is particularly hard when code needs to run in a distributed computing environment where it is difficult to know what runs where and which parts of the code are available to run there. In this talk we will present different ways to ship Python code to a compute cluster, what Python's "pickling" feature has to do with this, what self contained executables are and the challenges we met when shipping Python code to a cluster with 1000s of nodes running 1000s of jobs like TensorFlow or Spark. As an example, we will show how one can run a PySpark job on top of S3 storage using PEX as a self contained executable artifact. Finally we will explain how those ideas generalize for different Jobs (like Tensorflow, Dask), different virtual environments (like Anaconda or vanilla Python virtual envs) and different distributed storage's (like S3 or HDFS). The auditor will get an overview of the challenges of Python packaging for distributed applications and see code samples that can be applied in his own project.