We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Things I wish I knew before starting using Python for Data Processing

00:00

Formal Metadata

Title
Things I wish I knew before starting using Python for Data Processing
Title of Series
Part Number
55
Number of Parts
169
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Miguel Cabrera - Things I wish I knew before starting using Python for Data Processing In recent years one of the ways people get introduced into Python is through its scientific stack. Although this is not bad, it may lead to learn solely one aspect of the language, while overlooking other idioms and functionality included in Python as well as some basic software development good practices. I will share some useful tricks, tools and techniques and software design and development principles that I find beneficial when working on a data processing / science project. ----- In recent years of the ways people get introduced into Python is through its scientific stack. Most people that learned Python this way are not trained software developers and many times it is the first contact with a programming language. Although this is not bad, it may lead to learn solely one aspect of the language while overlooking other idioms, standard and common libraries included in Python as well as some basic software development good practices. This may become a problem when a data science project is moved from an experimentation phase to an integration with technical environment. In this talk I share some useful tricks, tools and techniques and as well as some software design and development principles that I find beneficial when working on a data processing / science project. The talk is divided into two parts, one is Python centered, where I will talk about some powerful Python construct that are useful in data processing tasks. This include some parts collections module, generators and iterators among others. The other I will describe some general software development concepts including SOLID, DRY, and KISS that are important to understand the rationale behind software design decisions.
DemonProcess (computing)Electronic data processingCoefficient of determinationInternet forumCycle (graph theory)AlgorithmHypermediaProcess (computing)Uniqueness quantificationLecture/Conference
EmailAreaInformationLecture/Conference
SoftwareStack (abstract data type)Machine learningMachine learningSoftware engineeringWave packetVirtual machineObservational studySource codeXML
CodeMathematicsObject-oriented programmingVirtual machineLevel (video gaming)IterationBoss CorporationInformation engineeringRandomizationPoint (geometry)Laptop1 (number)Direction (geometry)Projective planeModule (mathematics)Scripting languageInformationGenderUniverse (mathematics)Interactive televisionMultiplication signRight angleSoftware developerGoodness of fitGraph (mathematics)ProteinMedical imagingAbstractionMathematical analysisStudent's t-testDifferent (Kate Ryan album)Lecture/Conference
CodeMachine learningImplementationInformation engineeringSoftwareSoftware developerLecture/Conference
SoftwareCodeComputerCodeScripting languageComputerProcess (computing)XMLLecture/Conference
SoftwareComputer programmingStreaming mediaMereologyScripting languageParameter (computer programming)Code1 (number)Cellular automatonEndliche ModelltheorieDynamical systemObservational studyCodeSoftwareComputer animationLecture/Conference
Exception handlingModul <Datentyp>Social classAerodynamicsType theoryFormal languageCodeAttribute grammarProgramming paradigmObject-oriented programmingAttribute grammarObject-oriented programmingCore dumpOperator (mathematics)Matrix (mathematics)XML
HTTP cookieObject-oriented programmingSocial classCASE <Informatik>Different (Kate Ryan album)Template (C++)Lecture/Conference
HTTP cookieInheritance (object-oriented programming)Line (geometry)Curve fittingSystem callFunctional programmingConstructor (object-oriented programming)Social classObject-oriented programmingCASE <Informatik>Time zoneTemplate (C++)HTTP cookieMultiplication signAttribute grammarType theoryKey (cryptography)Duplex (telecommunications)Orientation (vector space)Instance (computer science)Expert systemComputer programmingAlpha (investment)Right angleStatement (computer science)AdditionLaptopComputer animationLecture/Conference
Object-oriented programmingPlanningWritingComputer programmingScripting languageLecture/Conference
Representation (politics)Physical systemSingle-precision floating-point formatOpen setInterface (computing)Inversion (music)Degree (graph theory)Key (cryptography)Object-oriented programmingComputer fileMultiplication signProcess (computing)Social classXMLLecture/ConferenceSource code
Object-oriented programmingParameter (computer programming)Table (information)Combinational logicInformationRow (database)Software developerBit rateSystem callCASE <Informatik>Computer programmingCodeComputer animationLecture/Conference
Rule of inferenceSelf-organizationCodeStandard deviationCodeSpacetimeRule of inferenceRight angleComputer fileTask (computing)
TendonCASE <Informatik>Text editorMaxima and minimaConfiguration spaceLecture/Conference
Data structureSoftware testingCodeVideo gameSoftwareCycle (graph theory)Software testingProjective planeRevision controlData structureMultiplication signCore dumpSoftware developerSource codeXMLLecture/Conference
Programmer (hardware)Formal languageAreaSoftwareWebsiteDescriptive statisticsComputer animationLecture/Conference
CodeOrientation (vector space)Object-oriented programmingRight angleCodeAlgorithmLibrary (computing)Perspective (visual)FrequencyData dictionaryBlock (periodic table)BuildingStatisticsCountingModule (mathematics)Source codeXMLLecture/Conference
Generating functionArithmetic meanOperator (mathematics)Default (computer science)Real numberIntegerCASE <Informatik>Computer animation
Field (computer science)Social classFlow separationInformation securityElectronic mailing listOperator (mathematics)FreewareDefault (computer science)CountingLecture/Conference
Total S.A.Inheritance (object-oriented programming)Social classFunctional programmingStatisticsInformation overloadWritingWahrscheinlichkeitsfunktionLecture/Conference
Data dictionaryProcess (computing)Special unitary groupCodeLecture/Conference
MathematicsPoint (geometry)CASE <Informatik>Social classAttribute grammarEquivalence relationContext awarenessData structureVideo gameSource codeXML
Social classCodeQuicksortInstance (computer science)Public domainOntologyMereologyLecture/Conference
RankingSocial classCodeMereologyHypermediaDerivation (linguistics)Right angleData storage deviceIterationXMLLecture/Conference
Letterpress printingSocial classGraph (mathematics)Data dictionaryDifferent (Kate Ryan album)Online help
Graph (mathematics)Directory serviceElement (mathematics)WordCASE <Informatik>NumberIterationElectronic mailing listLecture/ConferenceComputer animationProgram flowchart
Socket-SchnittstelleImplementationInfinityFunction (mathematics)CASE <Informatik>Electronic mailing listIterationObject-oriented programmingCodeType theoryBit rateData dictionaryComputer fileData storage deviceLecture/ConferenceProgram flowchart
Local ringFunctional programmingElectric generatorCASE <Informatik>Social classIterationParametrische ErregungComputer fileModule (mathematics)CodeLecture/Conference
Price indexLine (geometry)Order (biology)InferenceComputer fileLine (geometry)MereologyBit rateIterationSheaf (mathematics)Data dictionaryException handlingElectronic mailing listGraph (mathematics)Computer animation
Electric generatorParameter (computer programming)FrictionIterationExpressionElectronic mailing listNumberSquare numberGenerating functionArmFamilyMultiplication signLecture/ConferenceComputer animation
Square numberData typeNumberSemiconductor memoryElectric generatorExpressionMiniDiscObject-oriented programmingFactory (trading post)Square numberMultiplication signFunctional programmingCASE <Informatik>LaceBinary multiplierElectronic mailing listLecture/ConferenceXML
ExistenceElectric generatorFunctional programmingNumberGenerating functionSequenceCodeComputer programmingFibonacci numberLecture/ConferenceComputer animation
CASE <Informatik>SubsetNumberFunctional programmingSequenceLecture/Conference
Open sourceLetterpress printingOpen setPrime idealGenerating functionReading (process)CASE <Informatik>Beta functionPhysical systemIterationFunctional programmingXML
Letterpress printingOpen sourceOpen setServer (computing)Open sourceElectric generatorLine (geometry)System callLocal ringLoop (music)Process (computing)Lecture/ConferenceXML
Data streamElectronic data processingSemiconductor memoryStructural loadIterationSocial classCASE <Informatik>Electronic mailing listEvent horizonResultantField (computer science)Performance appraisalLecture/Conference
Open sourceStreaming mediaDigital filterLine (geometry)Streaming mediaServer (computing)Level (video gaming)outputExtreme programmingProcess (computing)Object-oriented programmingSheaf (mathematics)Standard deviationOpen sourceOrder (biology)XML
Open sourceIterationObject-oriented programmingGenerating functionFunctional programmingInformationClique-widthOptical disc driveLecture/ConferenceXML
ArmProcess (computing)Software developerMedical imagingMultiplication signClosed setAutocovarianceElectronic data processingProduct (business)Object-oriented programmingCodeEndliche ModelltheorieSoftware engineeringLecture/Conference
InformationMereologyIterationLibrary (computing)Table (information)XMLComputer animationLecture/Conference
Multiplication signLecture/Conference
Transcript: English(auto-generated)