We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Building reproducible distributed applications at scale

00:00

Formal Metadata

Title
Building reproducible distributed applications at scale
Title of Series
Number of Parts
130
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Packaging in Python is hard. Packaging is particularly hard when code needs to run in a distributed computing environment where it is difficult to know what runs where and which parts of the code are available to run there. In this talk we will present different ways to ship Python code to a compute cluster, what Python's "pickling" feature has to do with this, what self contained executables are and the challenges we met when shipping Python code to a cluster with 1000s of nodes running 1000s of jobs like TensorFlow or Spark. As an example, we will show how one can run a PySpark job on top of S3 storage using PEX as a self contained executable artifact. Finally we will explain how those ideas generalize for different Jobs (like Tensorflow, Dask), different virtual environments (like Anaconda or vanilla Python virtual envs) and different distributed storage's (like S3 or HDFS). The auditor will get an overview of the challenges of Python packaging for distributed applications and see code samples that can be applied in his own project.
BuildingScale (map)TensorDataflowMachine learningComputing platformUser-defined functionSeries (mathematics)Arithmetic meanType theoryModule (mathematics)Task (computing)Scheduling (computing)Level (video gaming)Cache (computing)CodeInstallation artRevision controlScaling (geometry)Goodness of fitWindowLogic gateVirtual machineTwitterMusical ensembleInterpreter (computing)CountingGroup actionTransformation (genetics)Workstation <Musikinstrument>NeuroinformatikWave packetSoftwareComputerElectronic data processingSoftware engineeringFactory (trading post)Trojanisches Pferd <Informatik>FlagObservational studyCASE <Informatik>Data miningSoftware frameworkGateway (telecommunications)Data storage deviceFile formatSemiconductor memoryCuboidLocal ringWordFunctional (mathematics)Binary fileRoboticsVirtual realityProcess (computing)Machine learningDistribution (mathematics)Series (mathematics)Digital photographyAuthorizationEndliche ModelltheorieFatou-MengeComputing platformSpeech synthesisPoint cloudIndependence (probability theory)Metric systemRevision controlElectronic visual displayData storage deviceYouTubeAnalytic setXINGDesign by contractAssociative propertyCrash (computing)TouchscreenProxy serverBitDataflowImplementationGastropod shellIntegrated development environmentFrame problemReal numberMessage passingVector spaceFunction (mathematics)Graph (mathematics)Visualization (computer graphics)Office suiteHookingTask (computing)Link (knot theory)Instance (computer science)Point (geometry)System callModule (mathematics)Default (computer science)Standard deviationScheduling (computing)Meeting/InterviewComputer animation
Vertex (graph theory)CodeExecution unitInstallation artRevision controlSubject indexingRepository (publishing)Computer fileCache (computing)Local ringLink (knot theory)Distribution (mathematics)Scale (map)BuildingMachine learningData modelDataflowTensorDeterminismSoftware frameworkWeightSource codeVirtual realityIntegrated development environmentMixed realityPlastikkarteRhombusData managementLocal ringInternetworkingDataflowProcess (computing)ImplementationRevision controlWeb-DesignerDigital photographyMusical ensembleControl flowOpen sourceDivisorCondition numberCASE <Informatik>Multiplication signInstallation artInterpreter (computing)Software bugSet (mathematics)Coma BerenicesVirtual machineData conversionServer (computing)Wave packetResultantIntegrated development environmentBEEPVirtual realityDifferent (Kate Ryan album)Scripting languageGradient descentHookingRootRepository (publishing)Parallel portLinear regressionWeb 2.0Logistic distributionCycle (graph theory)Video gameCache (computing)Software frameworkSource codeWeightEndliche ModelltheorieVariable (mathematics)Cartesian coordinate systemMachine learningAsynchronous Transfer ModeTask (computing)Library (computing)Level (video gaming)Scaling (geometry)Computer animation
Integrated development environmentRandom numberInstallation artStandard deviationVirtual realityDistribution (mathematics)Computer fileLibrary (computing)Content (media)Binary codeDefault (computer science)Asynchronous Transfer ModeClient (computing)Configuration spaceFunction (mathematics)CodeGlass floatSeries (mathematics)Arithmetic meanVirtual realityDifferent (Kate Ryan album)Mathematical optimizationRevision controlInterpreter (computing)Source codeCartesian coordinate systemMultiplication signComputer fileFlagDefault (computer science)Resolvent formalismLink (knot theory)Computer configurationFunction (mathematics)Common Language InfrastructureImplementationStandard deviationEquivalence relationJava appletPauli exclusion principleFunctional (mathematics)Vertex (graph theory)CodeData storage deviceIntegrated development environmentProcess (computing)Endliche ModelltheorieContext awarenessRight angleTask (computing)Distribution (mathematics)Interactive televisionInformation securityContent (media)Flash memoryMedical imagingMaxwell's equationsPhysical systemIndependence (probability theory)Goodness of fitYouTubeFood energyWater vaporMathematical singularityFactory (trading post)State of matterWorkstation <Musikinstrument>Keyboard shortcutWordSystem callMusical ensembleOcean currentMereologyComputer animation
Point cloudFunction (mathematics)Series (mathematics)Arithmetic meanCodeInheritance (object-oriented programming)Context awarenessAsynchronous Transfer ModeRevision controlData storage deviceVertex (graph theory)Source codeOpen setReading (process)WritingComputer fileConfiguration spaceInterface (computing)Generic programmingInstallable File SystemArrow of timeData storage deviceGlass floatLink (knot theory)Digital photographyContent (media)Distribution (mathematics)Streaming mediaFunctional (mathematics)Process (computing)Default (computer science)Vertex (graph theory)Product (business)Level (video gaming)Physical systemInterpreter (computing)Different (Kate Ryan album)Interface (computing)Point cloudData storage deviceComputer fileIntegrated development environmentCodeRemote procedure callDomain nameImplementationFile systemSampling (statistics)Asynchronous Transfer ModeLibrary (computing)Message passingFlagFile archiverWeb pageMultiplication signVirtual realityMathematicsDistributed computingInformationElectronic mailing listCASE <Informatik>Cartesian coordinate systemHecke operatorModal logicSoftware frameworkWebsiteSeries (mathematics)Forcing (mathematics)Endliche ModelltheorieAlgebraHand fanWorkstation <Musikinstrument>Service (economics)Open setMereologyMonster group1 (number)Absolute valueMusical ensembleUniform resource locatorCore dumpExpected valueBefehlsprozessorDynamical systemYouTubeFatou-MengeGodComputer animation
Meeting/Interview
Transcript: English(auto-generated)