Things I wish I knew before starting using Python for Data Processing
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 55 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21189 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201655 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
DemonProcess (computing)Electronic data processingCoefficient of determinationInternet forumCycle (graph theory)AlgorithmHypermediaProcess (computing)Uniqueness quantificationLecture/Conference
01:05
EmailAreaInformationLecture/Conference
01:31
SoftwareStack (abstract data type)Machine learningMachine learningSoftware engineeringWave packetVirtual machineObservational studySource codeXML
02:01
CodeMathematicsObject-oriented programmingVirtual machineLevel (video gaming)IterationBoss CorporationInformation engineeringRandomizationPoint (geometry)Laptop1 (number)Direction (geometry)Projective planeModule (mathematics)Scripting languageInformationGenderUniverse (mathematics)Interactive televisionMultiplication signRight angleSoftware developerGoodness of fitGraph (mathematics)ProteinMedical imagingAbstractionMathematical analysisStudent's t-testDifferent (Kate Ryan album)Lecture/Conference
04:56
CodeMachine learningImplementationInformation engineeringSoftwareSoftware developerLecture/Conference
05:43
SoftwareCodeComputerCodeScripting languageComputerProcess (computing)XMLLecture/Conference
06:13
SoftwareComputer programmingStreaming mediaMereologyScripting languageParameter (computer programming)Code1 (number)Cellular automatonEndliche ModelltheorieDynamical systemObservational studyCodeSoftwareComputer animationLecture/Conference
06:54
Exception handlingModul <Datentyp>Social classAerodynamicsType theoryFormal languageCodeAttribute grammarProgramming paradigmObject-oriented programmingAttribute grammarObject-oriented programmingCore dumpOperator (mathematics)Matrix (mathematics)XML
07:41
HTTP cookieObject-oriented programmingSocial classCASE <Informatik>Different (Kate Ryan album)Template (C++)Lecture/Conference
08:14
HTTP cookieInheritance (object-oriented programming)Line (geometry)Curve fittingSystem callFunctional programmingConstructor (object-oriented programming)Social classObject-oriented programmingCASE <Informatik>Time zoneTemplate (C++)HTTP cookieMultiplication signAttribute grammarType theoryKey (cryptography)Duplex (telecommunications)Orientation (vector space)Instance (computer science)Expert systemComputer programmingAlpha (investment)Right angleStatement (computer science)AdditionLaptopComputer animationLecture/Conference
09:31
Object-oriented programmingPlanningWritingComputer programmingScripting languageLecture/Conference
10:04
Representation (politics)Physical systemSingle-precision floating-point formatOpen setInterface (computing)Inversion (music)Degree (graph theory)Key (cryptography)Object-oriented programmingComputer fileMultiplication signProcess (computing)Social classXMLLecture/ConferenceSource code
10:50
Object-oriented programmingParameter (computer programming)Table (information)Combinational logicInformationRow (database)Software developerBit rateSystem callCASE <Informatik>Computer programmingCodeComputer animationLecture/Conference
11:46
Rule of inferenceSelf-organizationCodeStandard deviationCodeSpacetimeRule of inferenceRight angleComputer fileTask (computing)
12:15
TendonCASE <Informatik>Text editorMaxima and minimaConfiguration spaceLecture/Conference
12:53
Data structureSoftware testingCodeVideo gameSoftwareCycle (graph theory)Software testingProjective planeRevision controlData structureMultiplication signCore dumpSoftware developerSource codeXMLLecture/Conference
13:20
Programmer (hardware)Formal languageAreaSoftwareWebsiteDescriptive statisticsComputer animationLecture/Conference
13:48
CodeOrientation (vector space)Object-oriented programmingRight angleCodeAlgorithmLibrary (computing)Perspective (visual)FrequencyData dictionaryBlock (periodic table)BuildingStatisticsCountingModule (mathematics)Source codeXMLLecture/Conference
15:08
Generating functionArithmetic meanOperator (mathematics)Default (computer science)Real numberIntegerCASE <Informatik>Computer animation
15:49
Field (computer science)Social classFlow separationInformation securityElectronic mailing listOperator (mathematics)FreewareDefault (computer science)CountingLecture/Conference
16:24
Total S.A.Inheritance (object-oriented programming)Social classFunctional programmingStatisticsInformation overloadWritingWahrscheinlichkeitsfunktionLecture/Conference
17:07
Data dictionaryProcess (computing)Special unitary groupCodeLecture/Conference
18:05
MathematicsPoint (geometry)CASE <Informatik>Social classAttribute grammarEquivalence relationContext awarenessData structureVideo gameSource codeXML
18:32
Social classCodeQuicksortInstance (computer science)Public domainOntologyMereologyLecture/Conference
19:08
RankingSocial classCodeMereologyHypermediaDerivation (linguistics)Right angleData storage deviceIterationXMLLecture/Conference
20:01
Letterpress printingSocial classGraph (mathematics)Data dictionaryDifferent (Kate Ryan album)Online help
20:33
Graph (mathematics)Directory serviceElement (mathematics)WordCASE <Informatik>NumberIterationElectronic mailing listLecture/ConferenceComputer animationProgram flowchart
21:35
Socket-SchnittstelleImplementationInfinityFunction (mathematics)CASE <Informatik>Electronic mailing listIterationObject-oriented programmingCodeType theoryBit rateData dictionaryComputer fileData storage deviceLecture/ConferenceProgram flowchart
22:39
Local ringFunctional programmingElectric generatorCASE <Informatik>Social classIterationParametrische ErregungComputer fileModule (mathematics)CodeLecture/Conference
23:20
Price indexLine (geometry)Order (biology)InferenceComputer fileLine (geometry)MereologyBit rateIterationSheaf (mathematics)Data dictionaryException handlingElectronic mailing listGraph (mathematics)Computer animation
24:01
Electric generatorParameter (computer programming)FrictionIterationExpressionElectronic mailing listNumberSquare numberGenerating functionArmFamilyMultiplication signLecture/ConferenceComputer animation
24:49
Square numberData typeNumberSemiconductor memoryElectric generatorExpressionMiniDiscObject-oriented programmingFactory (trading post)Square numberMultiplication signFunctional programmingCASE <Informatik>LaceBinary multiplierElectronic mailing listLecture/ConferenceXML
25:50
ExistenceElectric generatorFunctional programmingNumberGenerating functionSequenceCodeComputer programmingFibonacci numberLecture/ConferenceComputer animation
26:54
CASE <Informatik>SubsetNumberFunctional programmingSequenceLecture/Conference
27:22
Open sourceLetterpress printingOpen setPrime idealGenerating functionReading (process)CASE <Informatik>Beta functionPhysical systemIterationFunctional programmingXML
27:47
Letterpress printingOpen sourceOpen setServer (computing)Open sourceElectric generatorLine (geometry)System callLocal ringLoop (music)Process (computing)Lecture/ConferenceXML
28:15
Data streamElectronic data processingSemiconductor memoryStructural loadIterationSocial classCASE <Informatik>Electronic mailing listEvent horizonResultantField (computer science)Performance appraisalLecture/Conference
29:16
Open sourceStreaming mediaDigital filterLine (geometry)Streaming mediaServer (computing)Level (video gaming)outputExtreme programmingProcess (computing)Object-oriented programmingSheaf (mathematics)Standard deviationOpen sourceOrder (biology)XML
30:09
Open sourceIterationObject-oriented programmingGenerating functionFunctional programmingInformationClique-widthOptical disc driveLecture/ConferenceXML
30:37
ArmProcess (computing)Software developerMedical imagingMultiplication signClosed setAutocovarianceElectronic data processingProduct (business)Object-oriented programmingCodeEndliche ModelltheorieSoftware engineeringLecture/Conference
31:54
InformationMereologyIterationLibrary (computing)Table (information)XMLComputer animationLecture/Conference
32:28
Multiplication signLecture/Conference
Transcript: English(auto-generated)
00:00
talk of this session. It's titled, Things I Wish I Knew Before Starting Using Python for Data Processing. So ladies and gentlemen, please welcome our next speaker, Miguel Cabrera. So welcome to my talk.
00:22
I remember this room being smaller last year. I don't know. So my talk uses a clickbait title, but I won't show any ad today. So my name is Miguel Cabrera. I'm gonna talk to you about some things I learned
00:41
in the last few years. I've been working with Python, and I would have preferred to know before starting using it for data processing. So quick introduction. I'm Miguel, I'm from Colombia. I live in Berlin. I work for a company in Munich called TrustU. We do data processing for hotels,
01:04
and as I said, I've been doing Python just for a couple of two years, so this is more like a beginner to beginner talk. However, I think if you're starting with Python, and in particular in the data science area, you're gonna take some good stuff from this talk.
01:22
That's my contact information. So the priors for this talk, where I think you are. So you are relatively new to Python. You are used Python mostly. You're in the scientific stack, NumPy, SciPy, so on.
01:42
Your work, or your desired work, has the data, were in it, or machine learning. You are not necessarily a trained software engineer. So if you are, if you have years of software engineering experience, you're probably gonna get bored in this talk.
02:03
So it's your opportunity to walk away. I can give you two minutes now. No, just kidding, but you can leave if you want. So who's who? I wanna know, so who wants to be, or who is a data scientist? Please raise your hand. Okay, data analyst, data engineer here, yeah.
02:27
So machine learning developer, that's a cool, okay. That's, I hope I'm not gonna bore, like. So for developer, yeah, so you're being warned. It's really basic, really high level.
02:40
So if you already have experience, you might get a bit bored. So this is a really basic. Other title, okay. So the agenda, we're gonna talk about some basic concepts and practices, in particular object-oriented programming. Then I'm gonna talk about some goodies of the collection module.
03:02
So something about iterators and iterables. As this was like a collection of things I wanted to show, I have many more things prepared, but because of time, I do have to pick the ones I like the most. As there are different things, there are different level of abstraction,
03:20
so be warned that we're gonna switch to a really high level to some code, and we're not gonna get in-depth in any of those. However, I will give you some points and some direction where you can get more information about it. So this is like a meta-talk, so to say. But let's start with a story. This talk, as I said, is based on my experience,
03:41
but it's also based on experience of my colleagues and my interaction with them. So let's talk about David. David just graduated from the university. He's a math PhD. He mostly uses R and MATLAB, and he comes to work for a company where he has to do mostly Python.
04:01
She started to write a code to classify some documents, text documents, for example. He uses NLTK and exactly learn, which is, I assume you are somehow familiar with it. Thing is, he writes a really nice, I Python Jupyter notebook, sorry, with the code, it runs, nice wraps, and so on.
04:23
This is a random image, so don't try to get anything from it. And then my boss tell him, okay, you have to integrate the code in our code base. So he had to go from a Python notebook to a really big project with a lot of dependencies,
04:40
a lot of script and so on, and of course, he is lost. So he tries the best to do that with not knowing what's going on, and he ends up writing what some people call spaghetti data science code, which is same as spaghetti code, but for data science.
05:03
So if he has to integrate the code, it's gonna be really bad for him as well, and someone else has to integrate the code, that person's gonna hate him forever. Almost forever. So how we prevent that to happen when we're gonna start doing data science
05:21
and we're gonna actually integrate data science and machine learning so I can learn coding to our code base. So the thing that I can say is going back to the basics and in a nutshell, I think data engineers or scientists have to become more software developers and get into the middle point. And how do you do that? Well, first we have to have a distinction
05:42
between code and software. I take this from a talk from Daniel Muset in PyData Berlin this year, and I really like how he put it. So code is something that runs in a computer. So when you write a script or you write a Python notebook, you're probably writing code.
06:03
Code might not have tests or follow any convention or documentation, you just write the code and it does the job, it's okay. Software, on the other hand, some people think it's just a programming test, text inside a deliverable. Some people think it's the whole thing, including all the script, deployment script,
06:23
testing, documentation, even customer support or technical support are inside the software. And you want to do, create the software that is maintainable, testable, deployable and all the apples that you can put in. So the question is, what's the way to
06:45
transform my code into software? So let's go back to the Python, to the basic. What is Python then? If I'm gonna work in Python, what is Python? The important thing from this is, I got it from the documentation of Python. So this is Python's object-oriented programming language.
07:01
So as a data scientist, you should be able to know what is an object and how to use objects. So this is like the first thing I'm gonna give you is, as a Python data scientist or data engineer, learn how to use objects.
07:20
And for that, I'm gonna give you a really quick and dirty introduction to what are objects. So objects, three main concepts. The objects, they have data, they call attributes, and they have some operational data that are called methods, that's in a nutshell. And how does an object look in Python?
07:41
Well, before going into that, I just want to raise a distinction between cookies, cookie cutters and cookies, sorry, classes and objects. So what is a class and what is the difference with the object? A class is kind of like the template that you use to create more such objects. In the case of cookies, the cookie cutter, if you're from the UK, you're gonna call it psquid.
08:03
And you take cookie cutter to create many, many, many cookies and you eat them, hopefully afterwards. Not all because it's gonna be bad for you. So in Python, this is how an object looks. It has a name, a class, sorry, that's the template for creating cookies,
08:22
or the cookie cutter in this case. It has any construction function that you, that is called every time you instantiate it. And it has some data, that's the attributes and some methods. Right now, you're already expert in object orientation in Python.
08:40
If you want to create a cookie, you just instantiate the cookie class. If you want to, one of the key concepts in object oriented programming is that you can subtype, so you can extend one object or one class to make do something special. In this case, I, alpha is just a type of cookie
09:00
that is eaten in Spain, also in South America. And I just, with this example, I extend the cookie class and I just add additional attributes, for example. So, who's familiar with scikit-learns? Just raise your hand if you use Jupyter and IPython notebook. Yeah, so, not so much.
09:22
But when you're working in IPython or Jupyter and you're calling scikit-learn for examples, you're writing a statement. Many of those statements look like this, and what you're doing there is you're actually calling objects and creating objects and interacting with objects. So, you have to be aware of what you're doing
09:41
and how you can use that in your advantage. So, how do I write good object-oriented code now that I know how to write code? This is, that's a really tough question and I don't plan to answer it today, but I'm just gonna give you some tips. There's some basic ideas in the object-oriented world and actually in the programming world that you want to know.
10:01
One is, don't repeat yourself. So, if you're writing scripts and you feel you're repeating, repeating code, copying from one file to another, the same code, you might want to create an object out of it and reuse it, and that's one of the key features of object-oriented or the target of using object orientation is to reuse things. Keys, always keep it simple.
10:22
Don't try to put a lot of things inside objects. And also, use the solid principles. These are really abstract principles. I'm not gonna go into details, but basically it's that one class and one object should do only one job and you have, well, I'm gonna skip the rest out of time issues,
10:42
but my recommendation is to check that and to check the link below. It's really important, it's really nice to know these concepts. So, first thing, I think it's important if you're gonna start doing serious data science with Python is learn object-oriented
11:01
programming in Python mastery. So, the next thing that you have to learn once you already know how to organize your coding objects is there's things called conventions. This convention is like table manners for developers. So, you're sitting at some things you want to do so you don't annoy other people
11:20
what you're doing, what you're eating, or what you're programming in this case. When data scientists try to integrate the code, one of the things that annoy people the most is that they have no idea of conventions and you have to always return the code to fix it or you fix it yourself. Why conventions?
11:40
Well, what is, well, conventions are important because readability comes. And they are small details, actually. Things like fuel you space or fuel you tabs. What are the indentation rules? How do you organize the code in a file? PEP-EQ is a defective standard so you should learn it.
12:01
There's some resources online to you to check. This is a nice user-friendly way of learning. PEP-EQ.org is an example of a right versus wrong way to doing things. There are many details in these conventions and you might get, oh, so many things to learn.
12:22
I just want to go. Well, you can help yourself with your editor. In this case, it's Emacs I want to use but sorry for the BI guys or other editors. You can configure probably to help you not only with checking that you're following the convention of your company or if you're using PEP-EQ, following PEP-EQ, but also to help you detecting
12:44
things that might go wrong. Like for example, in this case, a variable is never used and your editor can help you detect such things. Other topics that I would have loved to mention or to go into more detail but I don't have the time because I want to show you more cool stuff.
13:03
It's a project structure testing. There's no test in a data science project. Generally, they don't come with tests. Versioning and branching, namely learn how to use your source control. Core reviewing in general, the software development lifecycle. There's some books that I recommend you to read.
13:21
They're really general. They're not that specific to one language but if you want to get closer in this side of the data scientist area and become more of a software developer, those are good books to start with. Also, I was reading the description on your Python website and there are some talks that I think are relevant
13:42
and they probably talk about these issues. If you go to any of them, please tell the guy that I sent you there and he's probably gonna buy me a beer for that or something. So let's go to some now, go into code right now. It's been a really theoretical part
14:01
and you're probably boring and you want to see some code. So let's do it. So the tips and tricks I would have loved to know before starting doing in a code sense. Particularly, in a nutshell, the collection module. It's incredible how few, when you start using Python
14:20
from the data science perspective, how few you know about things that are in the standard library and one of those things is the collection module. And let's start with basic thing, counting. Counting is kind of like the basic building block for many statistical algorithm. If you start from basic NavAce to Word2Vec,
14:41
they are based in counting. However, I don't think data scientists know how to come properly in Python. Let's see. Let's start. How do you count in Python? The first attempt, you use dictionaries. Who has written such code to count stuff? Oh, you particularly know your stuff, apparently.
15:02
Let's see. So the actually more Pythonic way, if this one, when you don't ask for permission but for forgiveness. I think it's something like that that it means. Who has written something like this? But we can do better.
15:21
Let's use collection default it. Who's familiar with default it? So some of, yeah, that's good. Some of you. So it's basically the same. You use default it that has a, basically you pass a default value or a default generation function. And in this case, it's an integer
15:40
and by default it will be zero. So I don't have to do any check. But let's use the counter. Who's familiar with the counter? Few of you. So counter is really cool. And it's just a default it that is already prepared for counting. And it's for free.
16:02
And that's how you use it. I just pass only the list of items, an iterable there, and I just get the count. However, come from some extra goodies like you can get some, the most common, some values and do some step operation on them. And I found that pretty cool.
16:20
But remember, counter is a class and I just mentioned that you can take classes and extend it and add your own behavior. And for example, you want to calculate the probability and for some items. I can extend the class counter at a normalized function and I already have the probability mass function for that.
16:43
Easy peasy. If I want to overload the initializer to go normalized as soon as I have all the items in the counter, you can do it also. So when you're counting things in Python that you want to do when you're using a statistic
17:01
and you can use things like pandas or scikit-learns, probably sometimes when you're building the features, you should totally check out counter class. And there's a really nice article from Trey Hunter about how historically the counting process been developed in Python and it's really good to read.
17:21
Name topples. So, name topples are a thing that is, I discovered recently and they're kinda hidden more or less. Who's familiar with name topples?
17:41
Most of you, some of you, yeah. So the thing is that when you're writing code, you use, people use a lot of dictionaries, lists, topples, and when you start integrating that into a large code base, you see that code and you see it's a dictionary and you don't have what to expect.
18:01
So it's really, it makes the code hard to read. In this example, if I remove this, you have no idea what I'm talking about, what is PT in this case. You might, out of the context maybe. So just by using name topples, you can make the code clear. So name topples are basically sort of like a class generator online with the particularity that the attributes are read-only.
18:23
So they are basically a nice struct in Python as if you're familiar with C, that's more or less the equivalent. And so you can create class on the fly. It has cool methods also. If you really need to use Dict, you can transform it into a dictionary and you can create one out of an interable.
18:43
So, and I think it's a nice way when you're writing code to organize it and to create sort of like domain classes that represent things in your code and your ontology. In this case, we worked a lot with hotels, so I created a hotel base and a total distributor
19:01
and I actually inherit from it and add a method to calculate something. So and I pass the class or instances of this class around my code and that makes it, in my opinion, more readable. So let's go to the more needy part
19:24
and this is really interesting because it's really confusing. And for me it was, and actually I remember that during my first, my first interview for the company I'm working, right now I was asked something about iterators and iterables and I think I answered correctly
19:42
but it was out of luck. I don't think I, then I discovered, oh, I did it right but I didn't know why. So let's talk about this. So when you see a code like this,
20:00
you're probably familiar with it, how to iterate through a list. But what is happening underneath? Why you can do this? How can you do this and why it works and how can you write your own classes that have the same behavior?
20:20
I was confused and was looking for ways to, what's the difference between iterate or iterables? Is it a list, dictionary and so on? And I found this nice article by Beeson Driesen when I use it, I use this graph from him. You should totally check it and we're gonna start kinda like exploring the concept
20:43
using this graph. So two concepts, maybe abstract for you right now, it's iterable and iterator. So an iterator, sorry, an iterable is something that you can call the iterator method on and it will return an iterator. And an iterator is something that produces a value
21:01
when you are called next. Abstract, okay, let's go into more detail. So a comprehension, for example, produces a container. And that container, for example, can be a list, a dictionary, a tuple also. A container is something that you can check whether something is inside the container.
21:24
That's where more or less the name comes from. In this case, I checked that one number is in the list. In this case, it's set. And a container is typically an iterable. So you can go through all the elements one by one. So in this case, this is a list
21:40
and I call the iterator method that gives me X and Y that are iterators. So I can call, if I see the types of both, one is a list and the other is a list iterator. And I can call the method next on those items and obtain laterally the items from the list.
22:03
Now, when you do this code underneath in the by code level, that's what's happening. Python gets the iterator from the list and it start getting the values. So this is more or less like syntactic sugar in some way.
22:21
So in a nutshell, iterables is any object that can return an iterator. That includes container like list dictionaries, files. They have to implement the, if you want an object to behave like that, you have to implement the iterator under method. Some of those things might not be finite.
22:42
They just can generate value forever and when I see an example of that. There's a module in Python called iterator tools that have a lot of functionality to working with them in iterables and iterators in generative. So how do I implement my own iterator?
23:00
So for kind of like parametric reasons, you can implement both the iterable and the iterator in one class. So you have an iterator method that return itself and then you implement the next method. In Python 3, it's like a dunder method. In this case, it's just a code that reads a file
23:24
and then iterates in inverse order. So it start from the last line up. When there's no more lines to return, it will raise top iteration, which is the exception that is called to stop the four. You can use it easy.
23:41
You instantiate it and then you do the same as if it were a list. So now we covered the green part of this graph. So we know that we can get iterable from things like lists and dictionaries and files and from there we can get iterators that will produce values in a lazy way.
24:04
But there's another way to get that iterator and it's by using generator. Who knows who's a generator? Fewer of them, okay. So let's start. You can get a generator from a generation expression
24:22
or from a generation function. Both, as I said, are generators. So from a generator expression, let's start with a non-generation thing. It's basically a list comprehension and I'm generating 10 numbers and then I'm creating, sorry, a list of 10 numbers
24:41
and then I create the same list of the square of those numbers. So if I check the type, it's a list. What these are, only 10, but what if it's a billion of numbers? Probably I won't have enough memory to store them in my RAM or in my disk. You can do the same with generation expressions
25:03
and this is not a tuple, just although it looks like it, it creates a generator object that will produce the squares, in this case from the list number, in a lazy way. So each time I call next, it will calculate the square and return it.
25:24
Think about it as a factory of items and the factory uses, in this case, the function, the square function, multiply X by itself. So if I want to do the same,
25:41
I do with a list, I get, I generate the squares and the lazy squares, I can print the items and it will be only generated when the four internally calls the next function. Before that, those number don't exist.
26:02
So a generation function, it's the same idea, but it uses a magical work or yield that works in a nice ways. When you create this, you call the function fib, you will obtain also a generation, a generator.
26:22
Then you call next, what will happen is that the code is gonna be executed, then this yield will return the value back to the program and will continue only after the next, well, the next next is called.
26:44
In this case, I'm calculating the Fibonacci sequence that you might be, you are familiar, I'm sure you're familiar with, and I can just call next and next and next. Something to be aware here, this is an infinite generator. You see the while through there, it won't stop.
27:01
If I put this into a for loop, it will go forever. It will be generating, generating, generating numbers or the sequence. I can use some, one of the methods or the functions of either tool to just obtain just a subset of that,
27:20
and in this case, I just get the first three using four. You can also implement your iterators or iterables using the yield keyword, namely replacing the either function instead of returning itself. You just return a generation function.
27:41
In this case, I'm reading a file, for example, from HDFS. I distribute the system. Imagine just one server located somewhere, and imagine I pass a source that has the method open, and I just start iterating through it.
28:02
This open method might be even an iterable or even a generation generator, for example. I do something with the line, and then I pass it back, and I just can't call that as I do with the for loop and process of the line. So that's more or less the iterables and iterator.
28:21
So I can think that, hey, this is supposed to be related to data science and data processing and so on. What is this, what is the relation? Well, sometimes you cannot load all your data into memory. And if you're working into the big data field, that's probably your situation.
28:44
As I say, you might not have enough memory to store all the data you want. That will happen when you use a list. So you can work in such cases by using data streaming. In data streaming, you can get it by lazy evaluation, which is what I just showed, generating or processing things
29:01
as long as they are available or needed. And you can create some such in-memory data processing pipelines. Using iterables by changing them. So my example, I just showed you this class that gets some obtained line from a server
29:21
and just send it, do some processing, maybe split something. I can create another that takes that and then check whether it's a, say, Python comment or some random comment, and passes over. So it's kind of like a stream that gets processed and then sent forward.
29:42
And you can change, first create with the source, create one object and pass it as the input of the other, for example. And you can just call it in a for loop. And inside, you're gonna be processing in kind of like an extreme fashion.
30:01
So think about Inception, the movie. You're going different levels. First level, do something. The second level, do something. They are sending data up. You don't have to write an object for this. You can just get a generation function and replay the whole thing with the function. I just do it as an example how I like to do it.
30:23
There's a talk in your Python which we'll get into more detail. It's on Friday. If you got really, probably you didn't get the whole idea. It's clear as I want, but you can for sure get more information in this talk.
30:41
So to finalize, some conclusions or closing remarks. Data scientists, engineers, developers, you name it, you should learn, in my opinion, start with the collection and data tools, models that are basic, that are your best friends. You should teach iterables, iterators,
31:03
and build your data processing pipeline using them. Use object-oriented programming for organizing your code. That will help you not only to make your code more maintainable, but when you go to integration and you're working in large teams, you will have a better time
31:22
getting your code into the code base. And finally, you're gonna have to start moving to be more a software engineer instead of being just a scientist or a data juggler. You will have to become more of a software developer
31:42
when you want to get your solution into an existing product. Some credits, the images I use, I base most of my talk in a couple of articles,
32:01
in particular in ideas coming from Radim Trejurek, he's the creator of a library called Jensim, and he also, I really like how he, in this article I'm linking here, how he talks about data processing pipelines using such iterators and iterables.
32:20
As I say, we're for trust you. We are hiring. You want to know more about the company I'm working for. We have a small table where you can get some goodies. Just drop by and talk to us. And if you want to talk to me about the talk after the Q&A session, you also are welcome there.
32:44
So a question, comments, remarks, or you want to trash my talk, be welcome too. Thank you. I think everyone is hungry, so I don't expect many questions.
33:01
We have time for questions anyhow, if you want. If we don't have questions, we thank you again. And good lunch.