EuroPython 2016: Lightning Talks V
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 162 | |
| Number of Parts | 169 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21235 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2016162 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
SoftwareMobile appVideo projectorSoftware engineeringMultiplication signCartesian coordinate systemLecture/ConferenceXML
00:44
Hill differential equationRevision controlMultiplication signCartesian coordinate systemPoint (geometry)Computer fileService (economics)DatabaseRoundness (object)Lecture/ConferenceComputer animation
01:26
Virtual machineDigitizingVirtualizationCuboidCondition numberInstance (computer science)Drop (liquid)Sampling (statistics)Power (physics)Configuration spaceFirewall (computing)Lecture/Conference
02:11
Distribution (mathematics)MultiplicationFirewall (computing)PermanentCategory of beingPoint cloudDistribution (mathematics)DivisorCross-correlationCivil engineeringInstance (computer science)Configuration spaceComputer fileArray data structureDataflowScripting languageGroup actionLecture/ConferenceXML
02:54
Firewall (computing)PermanentRule of inferenceGroup actionPhysical systemDrop (liquid)Degree (graph theory)NeuroinformatikConfiguration spaceComputer fileInstance (computer science)Virtual machinePoint cloudLecture/ConferenceComputer animation
03:35
Cartesian coordinate systemScalabilityScheduling (computing)Queue (abstract data type)Goodness of fitLecture/ConferenceComputer animationMeeting/Interview
04:15
Computer configurationProjective planeVideo projectorRight angleMultiplication signFrustrationIntegrated development environmentJava appletLecture/ConferenceComputer animation
05:10
Computer configurationSoftware testingBit rateProcess (computing)Different (Kate Ryan album)Insertion lossFocus (optics)Task (computing)Letterpress printingWritingMultiplication signCASE <Informatik>Software testingConnectivity (graph theory)Computer animation
05:52
Element (mathematics)Electronic mailing listSocial classField (computer science)Link (knot theory)Different (Kate Ryan album)Software testingWeb pageSingle-precision floating-point formatCartesian coordinate systemGoodness of fitObject modelSampling (statistics)Pattern languageElectronic mailing listLecture/ConferenceComputer animation
06:52
State of matterElement (mathematics)Metropolitan area networkDevice driverSoftware bugMoving averageWeb pageCartesian coordinate systemMeasurementInheritance (object-oriented programming)Data structureOrder (biology)Element (mathematics)Video gameString (computer science)Selectivity (electronic)QuicksortConnectivity (graph theory)Function (mathematics)Right angleEndliche ModelltheorieLibrary (computing)Set (mathematics)Similarity (geometry)Text editorCodeCuboidComplete metric spaceWeb 2.0Lecture/ConferenceComputer animation
08:29
Metropolitan area networkElement (mathematics)Inclusion mapMenu (computing)DataflowUniformer RaumInsertion lossMaxima and minimaNewton's law of universal gravitationGrand Unified TheoryComputer-generated imageryEndliche ModelltheorieContent (media)Connectivity (graph theory)BitCASE <Informatik>Library (computing)LogicWater vaporLecture/ConferenceXMLProgram flowchart
09:15
Device driverEndliche ModelltheorieSample (statistics)Library (computing)Endliche ModelltheorieMultiplication sign2 (number)TheoryWechselseitige InformationPresentation of a groupLaptopComputer iconConnected spaceLecture/Conference
10:04
Data modelGraph (mathematics)InfinityState of matterPhysical systemSoftware testingEndliche ModelltheoriePixelSoftware testingGraph (mathematics)Goodness of fitModellbasiertes TestenCartesian coordinate systemPhysical systemNumberVector potentialState of matterLecture/ConferenceComputer animation
10:45
Physical systemGraph (mathematics)State of matterData modelInfinitySoftware testingSoftware testingCartesian coordinate systemNumberVector potentialPhysical systemState of matterObservational studyMereologyCondition numberDeclarative programmingMechanism designGroup actionGraph (mathematics)Finite-state machineConnectivity (graph theory)BitSoftware bugLecture/ConferenceComputer animation
11:39
LoginSoftware testingMobile appWeb pageEndliche ModelltheorieDifferent (Kate Ryan album)State of matterPosition operatorGraph (mathematics)Group actionError messageJava appletLecture/ConferenceComputer animation
12:19
Java appletLibrary (computing)Computer fileInterface (computing)Java appletGraph (mathematics)Vertex (graph theory)Element (mathematics)CodeGroup actionSet (mathematics)Software testingAutomatic differentiationMaxima and minimaLecture/ConferenceComputer animationSource codeJSON
13:37
RankingRaw image formatScaling (geometry)Near-ringLevel (video gaming)Lecture/ConferenceSource codeXMLComputer animation
14:21
Scaling (geometry)Logical constantWeb 2.0Data conversionGraph (mathematics)Point (geometry)Volume (thermodynamics)Pattern languageAlgorithmNeuroinformatikMultiplication signScalabilityScaling (geometry)Lecture/Conference
15:26
Scheduling (computing)Gene clusterScaling (geometry)Multiplication signStructural loadOnline helpPolar coordinate systemProcess (computing)Lecture/ConferenceComputer animation
16:06
Scheduling (computing)Process (computing)Different (Kate Ryan album)Speech synthesisStapeldateiPhysical systemMultiplication signoutputParameter (computer programming)Instance (computer science)Reduction of orderOverhead (computing)Type theoryQueue (abstract data type)Data storage deviceCuboidLecture/ConferenceComputer animation
16:48
SupercomputerScaling (geometry)Presentation of a groupDatabasePerturbation theoryProcess (computing)Touch typingSoftwareLocal ringMiniDiscLecture/ConferenceComputer animation
17:29
Scaling (geometry)Inverse elementContent (media)Single-precision floating-point formatScaling (geometry)ComputerComputer fileFile systemPhysical systemMetadataOperator (mathematics)Pairwise comparisonObject (grammar)XMLComputer animationLecture/Conference
18:18
Scaling (geometry)MetadataLogicSoftwareQuicksortScaling (geometry)Figurate numberBlock (periodic table)File formatLecture/ConferenceXMLComputer animation
19:06
Goodness of fitPlotterLecture/Conference
20:00
Open setIdeal (ethics)SoftwareSoftware developerSlide ruleSoftware developerPoint (geometry)Multiplication signProjective planeLecture/ConferenceMeeting/InterviewComputer animation
20:48
FrequencyComputer animation
21:32
Multiplication signProgrammierstilPoint (geometry)Video gameCuboidForm (programming)Source codeComputer animationLecture/Conference
22:29
Set (mathematics)NumberRoboticsComputer animationLecture/ConferenceMeeting/Interview
23:09
Boom (sailing)SummierbarkeitTotal S.A.Multiplication signRight angleLecture/Conference
23:51
WordPoint (geometry)Distribution (mathematics)InformationSoftware developerMachine visionHigh availabilityMereologyRight angleCentralizer and normalizerSoftware developerCodePoint (geometry)Peer-to-peerLecture/ConferenceComputer animation
24:42
Electronic mailing listWorld Wide Web ConsortiumWhiteboardProcess (computing)FAQView (database)EmailNoise (electronics)Electronic mailing listStandard deviationInformationWeb portalArithmetic meanLecture/ConferenceComputer animation
25:24
Context awarenessProbability density functionWeb portalWordEvent horizonQuicksortStandard deviationLecture/ConferenceComputer animation
26:06
CodeCodeLattice (order)Real numberLecture/ConferenceDrawingComputer animationMeeting/Interview
26:51
Maxima and minimaCivil engineeringEvent horizonLecture/ConferenceComputer animation
27:30
SineWordDifferent (Kate Ryan album)Image registrationMultiplication signGame controllerSelf-organizationRight angleNear-ringLecture/ConferenceComputer animation
28:22
Heat transferTuring testMixed realityFreewareAreaCombinational logicBus (computing)Heat transferType theoryCodeWhiteboardGame theoryMusical ensembleForceLecture/ConferenceComputer animation
29:02
Web 2.0Electronic mailing listInformationWebsiteSystem callGradientDataflow2 (number)Row (database)XMLLecture/Conference
29:50
Sign (mathematics)Event horizonMereologyRow (database)Universe (mathematics)Event horizonSpring (hydrology)Power (physics)TrailGoodness of fitSystem callMoment <Mathematik>Shared memory2 (number)JSONUMLLecture/Conference
30:44
Event horizonSign (mathematics)Mereology2 (number)Self-organizationStandard deviationPressureBitMultiplicationDifferent (Kate Ryan album)WebsiteMoment (mathematics)LaptopLecture/ConferenceJSONUML
31:57
Bit rateProjective planeInequality (mathematics)MeasurementLecture/ConferenceComputer animationMeeting/Interview
32:43
World Wide Web ConsortiumMetropolitan area networkIntegrated development environmentArmSoftwareSoftware developerPhysical systemSystem administratorOpen setGame theoryMusical ensembleFront and back endsService (economics)Server (computing)Mathematical singularityOpen sourceDuality (mathematics)Bit rateComputer programmingSoftware developerGame theoryMilitary baseNumberLecture/ConferenceComputer animationSource codeXMLMeeting/Interview
33:23
InformationAsynchronous Transfer ModeMetropolitan area networkBit rateLevel (video gaming)Pulse (signal processing)Repository (publishing)Mathematical singularityWeb pageAutomorphismArmTwitterFood energyProjective planeReal numberNumberMeasurementBitDatabaseCommitment schemeView (database)Computer animationProgram flowchart
34:29
Physical systemWeb pageInformation systemsPulse (signal processing)WebsiteProjective planeFood energyProcess (computing)DatabaseSoftware developerConnectivity (graph theory)Formal languageInterior (topology)Execution unitComputer animationLecture/Conference
35:11
Endliche ModelltheorieNumberVideo gameReal numberGoodness of fitRoutingLecture/ConferenceComputer animation
35:54
Data modelSound effectMetropolitan area networkVideo gameEndliche ModelltheorieInformation technology consultingDemo (music)Cartesian coordinate systemMathematical modelWater vaporForceDataflowLecture/ConferenceComputer animation
36:36
PredictabilityCASE <Informatik>SimulationMultiplication signMechanism designLevel (video gaming)Direction (geometry)Group actionTouchscreenMoment (mathematics)Connectivity (graph theory)Power (physics)Cartesian coordinate systemLecture/Conference
37:34
Data modelSound effectLine (geometry)Core dumpCombinational logicCartesian coordinate systemService (economics)Data conversionComputer fileCore dumpDatabaseRaster graphicsCalculationSpacetimeQuantumTheory of relativityCausalityPresentation of a groupCodeCASE <Informatik>Software developerWeb-DesignerComputer animationLecture/Conference
38:27
Data modelCore dumpNetwork socketInteractive televisionSimulationDensity of statesService (economics)Virtual machineData managementSystem callDependent and independent variablesSimulationServer (computing)Content (media)Network socketLevel (video gaming)CalculationRaster graphicsIntegrated development environmentVector spaceCore dumpWeb 2.0Lecture/ConferenceComputer animation
39:12
Term (mathematics)SimulationWater vaporElement (mathematics)Endliche ModelltheoriePhysical systemEvent horizonCurveCalculationLogical constantMultiplication signAreaNichtlineares GleichungssystemCore dumpComputer animation
40:20
Message passingEndliche ModelltheorieGenderIncidence algebraMetropolitan area networkReal numberFlagLevel (video gaming)Scaling (geometry)WordComputer animationLecture/Conference
41:14
Analytic continuationCodeSlide ruleGastropod shellVideo gameStapeldateiPatch (Unix)Lecture/ConferenceComputer animation
42:41
Gastropod shellMultiplication signDifferent (Kate Ryan album)ResultantVideo gameString (computer science)Web pageWebsiteLecture/ConferenceComputer animation
43:26
Asynchronous Transfer ModeBinary fileAerodynamicsStructural loadPlane (geometry)Chi-squared distributionComputer fileMetropolitan area networkGastropod shellWebsiteVariable (mathematics)Multiplication signSource codeDirectory serviceProjective planeFormal languageGastropod shellLecture/ConferenceComputer animation
44:42
Revision controlDirectory serviceGastropod shellLine (geometry)Mathematical singularityMetropolitan area networkCodeGastropod shellGoodness of fitScripting languageVideoconferencingBitTraffic reportingGroup actionLecture/ConferenceComputer animation
46:00
Gastropod shellException handlingCellular automatonVolume (thermodynamics)Field (computer science)Likelihood functionScaling (geometry)BitProjective planeMedical imagingLecture/ConferenceMeeting/Interview
46:49
Duality (mathematics)Ring (mathematics)Computer fileRight angleEndliche ModelltheorieScaling (geometry)Multiplication signTheory of relativityCodeComputer animationLecture/ConferenceMeeting/Interview
47:34
SummierbarkeitMetropolitan area networkRing (mathematics)CodeLibrary (computing)CASE <Informatik>Multiplication signCurveFunctional (mathematics)AlgorithmGene clusterWritingBitPower (physics)LaptopCluster analysisGoodness of fitLecture/ConferenceComputer animation
49:23
Metropolitan area networkVarianceFunctional (mathematics)LaptopGene clusterForestCASE <Informatik>WritingCircleLink (knot theory)Right angleNumberLecture/ConferenceComputer animationSource code
50:09
Metropolitan area networkHausdorff spaceSummierbarkeitCloud computingHash functionWindowMassSineRing (mathematics)Ultraviolet photoelectron spectroscopySystem callAlgorithmResultantCASE <Informatik>Traffic reportingCircleMathematicsMultiplication signArithmetic meanShift operatorGene clusterLecture/ConferenceComputer animation
50:58
Total S.A.Gamma functionMetropolitan area networkMenu (computing)Data acquisitionHydraulic jumpMultiplication signCellular automatonGroup actionBlock (periodic table)Lecture/ConferenceComputer animationMeeting/Interview
51:42
Multiplication signLengthLecture/Conference
Transcript: English(auto-generated)
00:00
Vitaly will take the first five minutes to deploy a projector app. He already tested it. If at first you don't succeed, Vitaly Vokhin, please give him a big hand.
00:26
Hi everybody, my name is Vitaly Vokhin and I'm a software engineer at Cloudlog. But in my spare time, I like to hack things together and try new technologies to build some applications.
00:41
And I really love Docker for this. So I like to Dockerize my applications and at some point of time you usually want to deploy your application somewhere. And Docker, it's not only about Docker engine, it's not only about running the containers.
01:05
Docker also comes with a bunch of few nice and handy tools. For example, Docker Compose, where you can declare your, define your application in a simple YML file. Like there is a database, there is a Redis, there is my application and it just runs all these services.
01:28
Another handy tool is Docker Machine. So Docker Machine is an interesting tool for orchestration and managing your nodes.
01:44
So it can create Amazon EC2 instances, it can create digital lotion, droplets, virtual box machines and so on. But what do you do next? So when you create a new instance, you usually want to provision it somehow, like install some tools like Htop or something.
02:08
You may also want to, for example, configure your firewall. And there is a handy technology for this, it's called Cloud Init.
02:21
And in the documentation they say that Cloud Init is a de facto multi-distribution package that handles early initialization of cloud instance. Well, I never heard about that tool until yesterday. Anyways, Cloud Init allows you to define a simple cloud config file where you say,
02:46
okay, I want to install the Htop, I want to upload this script and execute this script. I want to, it allows you to add users to the system, to add groups to define which users, participants of which groups and stuff like that.
03:08
So, and the only thing that you need, the only thing that is left is to create, for example, a new DigitalOcean droplet with a Docker machine and say, okay, use my cloud config YML file to provision the instance.
03:27
And that's all. In a couple of minutes you have a brand new shiny droplet with everything installed, with everything configured. And all that's left is to run your application with the Docker Compose.
03:41
That's all I have. Thank you. Thank you very much. Excellent. Wonderful. We need Jeremy to set up. How did you like today's lunch? I really liked, it was pinchers and I think it was scalable microfood.
04:01
Which we could all see because we had zero food queues to do the scheduling. Very good. Still setting up. So, if at first you don't succeed, skydiving is not for you. Also, lightning talks is not for you today because we have to be very quick and very thorough with the projector.
04:25
So, give a big hand to Czerny. So, hi again. First of all, my name is Czerny Makoshek. It's not that hard, come on.
04:40
So, yeah. A couple of months ago I was working on a Google project and being a Google repository, today you of course want to make everything right the first time. There was a lot of things going on and as a team we learned really a lot. But today I wanted to share one of my frustrations with the Selenium environment, the community,
05:09
not only the Python community but also the Java community and what I'm trying to do about it.
05:20
So, as someone that is, okay, so the task was to write end-to-end tests and Selenium seemed like a good tool for that. And when you're writing tests there are a lot of different problems you have to solve. And one of them is that things are changing all of the time.
05:44
So, there are cases where your front-end engineer is changing a component for various reasons. And of course once it is changed the test will fail and because the tests are failing you can't merge things and so on.
06:05
And there are a lot of different problems or other problems that you have also, that you have to solve. For example, single page applications are popular lately so you have to model that somehow.
06:22
There are good practices like page object models and so on. And yeah, once you get all that knowledge and you know the good patterns there are still exciting acrobatics you have to master.
06:41
And there are funny things to model like infinite lists. Here's an example. Yeah, and of course what will you do if the application is flaky? Will you measure that? Will you ignore that? So, yeah. And once you get to model the page you have to of course structure the test as good as possible.
07:08
And one of the steps are to model the elements themselves. And on the left side are the examples and on the right side is the code that actually works.
07:21
So, you start with simple things like that but the problem with that is that you can't find reusable components on the web anywhere. At least I didn't find it. So, I started to write a library and I'm planning to publish it. And so, as said, simple things like that are not a big problem. Although they take like half an hour to implement but it gets interesting.
07:48
So, imagine you have to model like a text box where you have to filter a string and you have auto completion and you have to, once you enter the string you have to select one of the members.
08:04
Not so trivial anymore. And it gets more interesting. So, here you have rich text editor because why not? And here you actually have to play with iframes which is pretty interesting with Selenium.
08:24
Yeah, so you end up with models like that. Here on the left side is a model for data picker which depending on your front-end engineer who is using either material UI or React components or whatever might be trivial to implement or not.
08:46
But in any case you have to do it because there isn't a library for models anywhere. Yeah, so a bit more things. So, some of the models require really a lot of logic.
09:04
For example, you can select something only when it is clickable. When Selenium gives a signal that you can actually or the user can actually click on it. So, you end up not only with a library for models but also for interesting acrobatics with Selenium.
09:24
This one in particular is very useful and it took really some time to get to this solution. Okay, five more seconds. So, that's the future where we want to go, have models for all the things and here it will be published. Thanks.
09:41
Thank you very much. David, Terry, and Anja, could you please connect with Fakunna and also Anton, David, connect with him so that you can put the presentation on one laptop.
10:02
We have PiCon Poland, we have PiCon Germany. Terry, set up. He's going back and back. Go up. Do it. Do you know, Rick Astley has a very good collection of Pixar movies and he's very generous in lending them, except one.
10:24
He will never give you up. Model-based testing with graph worker, David Terry, big hand. Hey, so my name is David and I'm going to talk about model-based testing.
10:44
One of the challenges for me when I'm testing application is that as the number of potential states in the system increases, the number of paths that a user can take through this application increases in a kind of like really fast way, exponential or something. So it's not really feasible to write handcrafted tests for every one of those paths, but what you can do is
11:04
you can model the system or more realistically like a component in the system as a graph or a finite state machine and then you just generate your tests by walking over that graph until some kind of end condition is reached. This means that the tests are more declarative, like you're just defining what should be tested rather than the actual mechanics of testing it.
11:25
The tests are less coupled to the application code, so if you change a state or some other bit of your application, you just need to change the tests in one or two places rather than lots. And if you walk over the graph in a random way, you can actually find new bugs with your tests, which I think is pretty cool.
11:41
But it also does mean the tests are not necessarily always the same. So to demonstrate this, I made here a little kind of test app. It's got a login page. You can do a good login, you can log out, a bad one, and then from there you can log in again.
12:01
So then this, if it's modeled as a graph, looks like this. You have your nodes, which are the states that need to be verified. The first page, the error page, and the logged in page. And then all of the edges are the actions to move between these states. So GraphWorker is actually a Java library, sorry, but you can combine it with Selenium and it's still pretty practical, I think.
12:25
And also I just like the idea. So yeah, you model your graph as a GraphML file, and then GraphWorker will take this and it will generate an interface for you to use, which looks like this.
12:48
You have just a method for every edge and a method for every vertex. And then when you want, you can see it here, when you want to write the tests, you just overwrite each of these methods.
13:01
So here this is just Selenium code for selecting an element and clicking on it. And down here, selecting an element and asserting that the text is a certain value. And then when you run the tests, it looks like this. Yep. Yep. There. So 100% edges, 100% vertexes. Yeah, that's it.
13:47
Thank you very much. Excellent. Rafael, please consider to scale up here. Where's Team Koala? Is Team Koala near enough? Okay. Be prepared to scale after Rafael.
14:06
Yes, she may. Oh, wow. Oh, wonderful. It's already there. Yep. Excellent. Give a big hand to Rafael. But considerations at scale.
14:21
A little bit of virtue. So a lot of breath has been spent at this conference talking a lot about technologies where you have active compute going over your data, all of the web technologies. But what happens when your data is too big for that? I'm talking about after Spark has fallen over, which it did at about 10% of our daily volume.
14:42
At that point, you're dealing with petabytes of data. You're dealing with hundreds of users accessing your data in a very, very, very concurrent fashion. So how should you tailor your access patterns so that you don't fuck it over for everyone else? Based on some work by Robert Grossman at the University of Chicago, we've come up with a basic idea of what a scalable algorithm looks like.
15:07
If you have X amount of data and you have X amount of compute resources, it should take that much time. But if you double your data and double your compute resources, time should be invariant. That is a data scalable algorithm. It will work over any size of data.
15:26
If the amount of time goes up, you should refactor your algorithm or buy a Cray. And if the time goes down, I really wonder what you're doing. Now, scheduling the work. If you're working with large scale clusters like at the national labs or at big companies, you often have a scheduler to help spread the load over thousands of PCs.
15:45
And when you're working with those, there are a few things you should do to be a nice person. First of all, if the time that your job takes is less than the time it takes to schedule your job, you are doing something wrong. I cannot express to you how many times I see people who have jobs that run at
16:04
a tenth of a second and yet it takes two tenths of a second to schedule the job. If you are never actually filling any of your nodes, you should think about batching things differently. Speaking of batching, most scheduling systems offer ways to take several jobs that are the same type with slightly different input parameters and simply pass in the job once and then an
16:25
array or whatever else to pass into the job many, many, many times for all of its instances. This will reduce scheduler overhead. Also, don't fill the queue. The queue is not a storage place for all the work you ever intend to do. You can knock over schedulers. I don't care how big you build your box. A million
16:44
jobs is still a million jobs over your 10,000 slots. It's a lot of work. If you want to be really sexy, you can use something like DRMAA, or as you pronounce, drama, to programmatically do all of this. I have a quote. It makes this presentation feel deeper.
17:03
Now, scaling data. Your approaches to data. It's nice when you have tons and tons of salary workers doing little things against your live active databases when, no offense, hot jar, but when you're at hot jar size data just because you guys decide to publish how much data you have.
17:20
Now, when you're working with far larger amounts of data, remember that it is always faster to go across the network than it is to go to disk. I don't care if it's local disk. Now, we're scaling in kind of the inverse of TV. TV, we already have 4K televisions, and maybe next year we'll have some bloody 4K content. But for now, we already have tons of people promising the world to every single investor about big data,
17:44
and maybe next year we'll have the kind of compute that can make you run Redis at five petabytes. But until we do, remember, you're going to be using a file system, and when you use file systems, you want to be moving data and not metadata. Sorry, Hansel dude, that's not going to work at scale. And so, when you do this, you will fuck yourself over on metadata operations.
18:04
There's no way GPFS scales this. You can try this with object stores, but you're just moving stuff that doesn't matter to what you actually want to be doing. Please, use a packaging format, use HDF5, use something where you're going to be having fewer metadata operations in comparison to the amount of data you're moving. Don't put your logic in your metadata.
18:21
Lastly, think about how you're coalescing your I.O. You can read, read into RAM somewhere on the network, push it around, do the work with MPI ranks, great way to do this. Other sorts of multiprocessing, there's all sorts of ways to do this. We did find that when we tried to run with all of our data at scale over Spark, 95% of it was pickling.
18:44
There are better ways to do that, but, you know, figure out what works well for your data, and then bring it all back and write it once. That way, you coalesce I.O., you use I.O. that is large enough to make sense. Most blocks on devices are four megs, so if you're doing four kilobyte I.O., yeah, think about it.
19:02
And you will be a much better grid user. So, thank you very much. I'm 27, I started on Fortran, and it's great to be on Python. Very good. Thank you very much, Raphael. Our next speaker will be Team Koala. Please set up.
19:23
We have three conference announcements here. Anja, Anton, and Fazumna, so we blocked you, clustered you together, and you'll get six minutes. You can fight with each other. Who will get how many minutes? I was told on the Telegram channel that was possible. What?
19:41
Sound. There is a plug-in for sound if you still have this. Yeah, I have it. Thanks. Very good. That will be harder if it's console and we have to go via USB-C. We can't hear anything, though. Oh, thanks. All right. Okay. You're good? Okay.
20:00
So, give a hand to Team Koala. So, we actually had somebody setting up at 7 to deliver this lightning talk to you. So, this one is about crazy ideas, about those kinds of ideas where everybody is against it, but because it just seems not realistic, such as developers doing marketing.
20:27
So, yesterday evening at 9 p.m., we had such an idea, and everybody was against it. But we did it without any experience, without any equipment, and without time. And actually, this was also how the Koala project started.
20:41
So, when you have such an idea, try to do it. And here's what we did just to prove the point. And because it's awesome.
21:35
His coding style was really bad. And when I say bad, I mean like really bad.
21:43
Well, I guess at some point he just couldn't stand it anymore. He was really pissed. But, little did he know, that we were working on a tool that would change his life forever.
22:24
Able to take on the world to the box. Cheers!
23:07
Thank you very much. Please turn up the sound. Turn off the sound. Turn up the beat. Please, the three, and I will set up the timer to six minutes.
23:22
What? Why eight? Yes, three times two is six. We have three times two. We have two minutes per conference. And we have Armin Rigo, who has a doctorate in... I'm not talking about a conference.
23:41
I'm not talking about a conference. PyCon, Argentina? No, no. PyCon, Argentina, Falcunda, Batista? Python, Argentina. Ah, Python. Wonderful. So, you indeed have nine minutes. Excellent. Falcunda, Batista, Python, Argentina. I go first. Ha!
24:01
And he cheated. Now it's Py... Okay. Give a big hand to Falcunda. Hola. Hola. This is working. It's not working. Hola. Yeah. Hola. So, Python, Argentina. Right. So, we presented part of these previous years.
24:21
You kind of know what we do. We just try to communicate everybody in the country to be the central point of reference. Everybody trying to find what Python needs. Try to contact other Python developers to have code peers or to hire people or to
24:43
sue people or whatever. If you pronounce Python in Argentina, the idea is for you to find what you need. So, we have a big community. We say that in Argentina Python is about the people. We have a mailing list with a lot of people that luckily doesn't have a very good noise
25:07
information ratio. We have a channel in Freeno that we call the channel of love. Not because we are kind people, but also because PR in Indy means love.
25:21
So, that gives us funny situations. We have a web portal with the standard things, show board, et cetera, and we have the official tutorials for Python 2.7, 3.5 and Django translated to Spanish, published in HTML PDF
25:41
and sometimes we print them to distribute to people, which is very nice. So, some words about last big events we had. We have a PyCon last year. It was our seventh. This is a standard conference, people watching, people talking, people talking, people talking,
26:03
people in the halls. We also have a PyCamp. This is more fun. We have this ninth this year. This is for a lot of kind of 30 people going to somewhere just to code for four days.
26:23
Code, code, code. People coding, people coding. People designing stuff. People talking about the design. People also having meetings. We have a meeting there with, of course, wine. And we have, of course, people launching together.
26:40
That's a PyCamp. Nice ways, nice places to take a walk. And, of course, learning how to fight for real. What we will do next? Well, we have a PyCon Argentina this year in November. It's a very nice place. You can go.
27:01
If you know Spanish, you will enjoy it a lot. If you don't know Spanish, you will learn. So, we will have a PyCamp next year. We still didn't planify, but it's one of the best events of the year. So, we will have a PyCamp. And we are finishing in the following months the nonprofit.
27:23
So, we will become an official legal entity that the idea is just to help the community to handle money, et cetera, et cetera. So, thank you very much. Thank you very much. PyCon.
27:40
PyCon Poland. Okay. Now, Anja will talk about PyCon Poland. And I will get down the time to two minutes. You be kind. Hi, I'm here on behalf of organizers. And I want to tell you that PyCon Poland is actually the fourth oldest PyCon in Europe.
28:07
And there are plenty of attendees coming. Already, a lot of them registered from all around Europe. And it's so-called that even Europe... Leidenhaupt is so-called that even Europe follows that. And it happens in mid-October in Oslo, near Warsaw.
28:24
Everyone gets an accommodation in the same hotel, surrounded by forest. It's really lovely there, but worry not about getting there, because there are gonna be conference bus transfers from Warsaw. So, only worry about getting to Warsaw, which is not that hard, I guess. There are five trucks. At least half of them are in English.
28:42
So, you're gonna understand most of it, I guess. There are plenty of other activities, including coding challenge that finishes in early morning hours. We play board games. We have, like, amazing live music, even. Hotel offers a lot of attractions itself.
29:03
We also have social dinners. They're lovely. These are the current prices. We also are gonna have a beginner's workshop, where we're looking for sponsors, especially participants and mentors. Go to the website, check it out if you're interested. If you have any questions, this is the list of ambassadors you can always ask for details.
29:25
But also, visit our website, plpicon.org, for more information. Thank you. Wonderful! Excellent. One minute and 30 seconds. That's the new record for a conference announcement.
29:40
Great! Now, the Germans will prove that they're even more efficient. Anton! Hi, guys. It's really great to see you here. Thanks for making it through the five days of intense talks and still attending this one.
30:03
I will not take much of your time. I'll have another record on the lightning talks, I hope. So, as you might guess already, I'm here to bring one more awesome event to the Python universe. It's PyCon VE, and it will be held on the last weekend of October.
30:20
It's Friday, Saturday, Sunday. Friday would be a workshop day and spring day, and Saturday, Sunday would be talks. We'll have three tracks. We'll have German track, English track, and we're working on a business track. Obviously, it's German and it's Bavaria, so everyone is invited for some good beer, good quality talks, some sharing of the engineering power, and of course, of developing Python.
30:43
So, guys, I guess I would contribute the last few seconds to just reminding that our community is awesome, generally. You see this conference, how well it's organised. It sets quite high standards for doing other conferences, so we are kind of under pressure now.
31:02
But I think my last five seconds was a bit of applause for EuroPython organisers. German efficiency, done. Thank you very much, Anton. Now I need Shai Efrati. Shai? You're here? Please set up.
31:24
On the other hand, you have different fingers. Took your time. So, I wanted to use a moment when somebody is struggling with his laptop to talk about offence.
31:41
There was an English philosopher, Jimmy Carr. He said offence can't be given, it can only be taken. So, give a big hand to the Krihelinator. So, I'm Shai, and I'm from Israel, so we don't say Krihelinator, we say Krihelinator.
32:12
I'm actually not showing my own project, I'm showing Tom's project, which is a friend and a colleague. And this project is already live, and I will start talking about it now.
32:23
So, GitHub stars. Do you use it? Yes, you use it. Do you use it for bookmarks, or do you use it to show appreciation to other people? Both. Okay. But are they useful as a measurement tool for other people's projects, if they are good or not?
32:42
Yes? I don't know. Okay, so I had a discussion with Tom. This is Tom. And I had a discussion with him, and I asked him, how come Meteor has more stars than Django, or Flask more than Django? Like, what kind of measurement tool is it? And we asked Meyer, which is this guy, and Meyer is a really professional developer at our company,
33:03
and he said, I don't use stars at all. And we asked him, so what do you use? And he said, I use pull requests, and I use GitHub issues, and I use, yeah. A number of commit, a number of contributors. So, yeah, let's do that.
33:24
So, okay, GitHub trending. Okay, GitHub trending. We all know that, we all know it, and we saw the stars, and it's only ranked by the stars. It's not a good way of measuring project quality. So Tom created this project, which is called the Crichellinator.
33:43
And the Crichellinator is measuring projects by their, like, real quality. Like, number of contributors, number of commits, number of merge pull requests, proposed pull requests, closed issues, and known, new issues, sorry.
34:00
And unfortunately, or fortunately, it is written in Elixir and not in Python, because he really likes to learn new technologies. If you want to know how the measurement is being done right now, so these are the factors. It's a bit random, we might change it in the future. If you have any suggestions, you can put it on the GitHub project, which is here.
34:31
So about the technology of the project, it's scraping the GitHub website every six hours. It takes random projects, and it compares them to the 500 good projects that are already in the database, and if they are better, it replaces them.
34:50
And the 16 scrapers that are doing the job, because it uses, like, a high efficiency of the language, and actually, that's it. Thank you.
35:03
Thank you very much. Thank you very much, sorry. Lars Clausen. Lars? You're already up here. That's good. That's excellent. So, if he's already up here, I need, oh, you renumbered. Does anybody remember the command renumber?
35:25
Nobody developed on Commodore 64? Anyway, Lars Clausen about live hydrological modelling with 3D eye. Give him a big hand.
35:41
Hi, guys. I talked to a couple of people the last days, and what I do, or what I've been doing for the last one and a half years, and encouraged me to give a lightning talk, because they thought it was interesting, and I thought I'd give it a shot. So now I'm here, terrified. Anyway, I'm going to talk about live hydrological modelling.
36:03
I work for a small consulting firm in the Netherlands. It's called Nelens Hormans. I put the name up there so you can actually understand what I just said. And I'm going to give you a quick live demo. So this is our application, and it allows you,
36:22
well, maybe I have to go back quickly and tell you what hydrological modelling is anyway. So I tried to make it real quick. It's a mathematical model of physical forces that affect water and how it flows. And we use it basically for predictions. But the cool thing is you can start your own simulation,
36:46
but you can also follow along a simulation that has already been started. So in this case, I asked my colleague Matijn to run a simulation. So this is based basically on Reddy's pops-up mechanism.
37:04
And all the action he's taking are also communicated into my directions. So you can see he's interacting with the map. I just can follow along. I don't have any powers to do actions myself.
37:21
So you can unfollow. You come back to this screen. You can start your own session. I'm going to start this model. It's going to take a while because I'm going to show you some main components of the application.
37:42
We have a service. It's called Impi. And it converts basically a spatial light database file and some raster files into files the calculation core understands. So what's the calculation core? That's where the actual calculation is done.
38:05
So it's called 3D I-Core. And it's written in Fortran. So I think today is a good day to give this presentation anyway because of the talk keynote this morning. Because, well, our application actually is a case where web development and scientific coding comes together.
38:28
So we have a machine manager. When I started the simulation, I did a call to the machine manager. And that service is responsible for starting, killing, and running docker containers.
38:42
So we also have a socket server for more static content. And we have the docker container that is an isolated environment for the calculation core actually. We also have a web map server that serves all the, well, the web map layers like rasters and vector data.
39:10
And all Impi. Of course not, but first let me first give you an impression of what you can do.
39:24
Because on the background, well with the play button you can start the simulation. So you see some elements start to respond to it. You see pumps that are pumping water through the system. You see water flowing that are the little balls.
39:44
But you can also interactively influence the modeling. So you can, for example, put some constant rain intensity onto, into the simulation. You can ask for historical data.
40:01
And, well, we have some predefined rain events that occur so and so often within, I don't know, a period of time. And that is being translated live into an Impi area which is then passed on to the calculation core that does all the equations.
40:21
Well, it's not all of Python, but our stack is, well, consists, it's not complete, but we use a lot of Django, GeoDjango, ZMQ, Redis for caching and the PubSub, Angular, and Leaflet. Thank you very much.
40:40
Thank you. Excellent. I need Leonardo Santagata to come on the stage real quick. I need the lady or the man who proposed to talk about how to scale Python for Excel users and who can't afford a name to come to the stage to be prepared.
41:02
I need Hugo Herter. Was it you with the Excel users? Excellent. That's Fabio. And Hugo Herter just as a backup. Where's Hugo? Okay, be prepared after Fabio. Maybe we can get you. So, give a big hand to Leonardo about.
41:25
It's about conch. Conch. Yeah, I'll just get to that. So, someone told me it's better to have code on slides, so I did, it's only code now. So, for, I guess, 10, 20 years of my life, I've been doing Bash as my shell, and I've been quite happy, and then I'll be doing like this.
41:48
I'm Santagata. That's my last name. And then I see this kind of code, and I have to fix it, and then I get crazy, and I say, no, let's do something better. So, let me get out of this.
42:01
Okay, so I switched to Z shell, which is much, much better, and oh, my Z shell, if you're still using it, it's good. It does a lot of things, like a cool prompt. If I go to something like this, it'll show me some stuff.
42:23
So, that's good, but when I have to fix it, and I have made patches for it, it's terrible again. And then I said, no, I need something better, so let me get out of this. And I moved to Fish. Fish is an amazing shell.
42:42
Yeah, it's going to get better, so don't clap now. It's even better. So, Fish is nice, but I got tired of fixing the shell I'm in every time in a different syntax, and with little different things. And I was like, no, I'm not fixing Fish, so I went back to Z shell for a while, but now I got something better.
43:05
So, I'm like, oh, one of the nice things is that Fish has live syntax highlighting, so you're typing this, it knows it's a command, and if you put an if, it'll know a string, it knows it's a string, so nice, but nah.
43:22
Okay, so, conch is a very nice shell, I'll just show the website in a while. It's a very nice shell, it's all made in Python, it's amazing, it uses prompt toolkit that was presented before, so it does have live things, it is very nice to try to help you in everything that you do,
43:46
and then I can show you, I started with, oh, this was supposed to be bigger, but it's fine. So, now we are in Python land, which is awesome, so variables have types, oh, that's so new.
44:03
And you can concatenate stuff and do all the kind of stuff you want to do, but never could in shell. For example, it's my first time I actually implemented something to go to my Go source directory into my projects.
44:21
It's a very deep path, if you know how this works. And I even did autocomplete, which is completely crazy in anything besides conch. So, finally this works, and I'm happy, it's Python, so I don't have to learn a new language. The project is amazing, the people doing it are really nice, I already did like three pull requests,
44:47
they actually actively told me to always be nice to other people when reviewing code and doing pull requests and everything else, which is really nice, so you see everyone is trying to help you to get your code in, and not like every other old shell project, so that's good.
45:06
Installing it is quite easy, just pip3 installed, it's a Python 3 only thing, which is also good, I think. Hopefully you also think that's good. So that's the website, conch.shone.sh, I guess.
45:25
So you can go there, read more about it, see a little video about this, and yes, it's very nice, I still have a minute, so let's see a little bit of people that actually know how to use it using it.
45:41
So yeah, very nice, I hope you all switch to it, or anything bad is still not happening on it, please report the bugs, we can try to fix it. And that's it, I hope you have fun and stop having terrible shell scripts everywhere.
46:01
Thank you. Thank you so much. Please set up, Fabio. I got a little feeling like in Inception when he went out from one shell to an outer shell to an outer shell to an outer shell, and I still fear we are living in some kind of shell. Can somebody please exit?
46:21
Now, give a big hand to Fabio about how to scale Python for Excel users. Cool, so I want to talk a little bit about the project I'm recently working on. So we saw Raul's keynote today, and in the data science world, there's one thing that struggles a lot of people,
46:48
and it is probably the most used tool for data science, or everyday data science, is actually Microsoft Excel.
47:01
Even my mother can do some stuff there. It actually has the right UI, and it's the right tool for those kind of problems, but it really doesn't scale for really interesting problems and really hard problems. So every time, how many people here work with related data science or pi data tools and stuff?
47:26
Not many. How many here actually had someone asking them to use Excel files and run some Python code over it? Okay, that's quite a lot. Who here has heard about Jupyter Notebooks?
47:46
Cool. So basically, that's one of the first things you say to someone using Excel. Like, try Jupyter Notebooks, and you try to write some small piece of code, and you just use some of the Python libraries that consume Excel, and that's it.
48:02
That's not really the case, because most of the time, Excel users are not even by far related to programming. That's really hard, and there's a really hard learning curve for them. So in an ideal world, you have something like Jupyter and somehow it's solved,
48:22
and you would write something for them, or they would just write themselves. So in this case, let's say I have a function that basically just uses scikit-learn and clustering algorithms to find clusters in the data.
48:43
And since I'm writing this for my mom, I'm actually writing some nice documentation telling how to use it. Sorry. How to use it and explain a little bit more.
49:01
So in an ideal world, we would have somehow a plugin, and they just could say, okay, let's use this thing, and it will load the notebooks I have somewhere, and I could consume it. But since Excel users, they don't have really nice, good feeling with code,
49:25
you could present and expose functions that are written in the Jupyter notebook to the Excel users. So in this case, I have two functions. The one I showed before is clustering. And you can check documentation and see what I've written.
49:44
It links to external links, so you can check stuff here. Oh, nice, it uses clustering. You can actually check and say, okay, let's try this thing with some Excel data here. And so let's say I have noisy circles here.
50:02
All numbers, because we like numbers, right? I don't, I'm not sure I have enough time. So say I want to take some data from Excel and say take selected data, call it circles, cool, confirm.
50:28
And then I say, okay, I want to use this clustering thing with circles and use mean shift algorithm. And very likely here, I would tell just to use random change,
50:43
so every time I change the interface, it runs again. And here we go, I can run, it executes, and this runs clustering algorithms on a kernel, and then gets back the result. In this case, the result is actually a bokeh plot, because we like bokeh plots, right?
51:02
And I can change stuff, it will execute every time I change the data. We can change and see, okay, let's use the blob, that's nice as well. So I can add here, cell blob, value is from six three to G, I think it's this.
51:28
Confirm, new blob data, since I had to, okay. And that's it. Thank you, thank you.
51:44
That sadly, sorry, let's us run out of time for today's lightning talk. Thanks to all the speakers, thanks to the audience. Give you a big hand.