EuroPython 2016: Lightning Talks II
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Part Number | 33 | |
Number of Parts | 169 | |
Author | ||
Contributors | ||
License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
Identifiers | 10.5446/21258 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
00:00
PlotterTheory of everythingRow (database)NumberMedical imagingSound effectExecution unitGoodness of fitElectronic mailing listComputer animationLecture/Conference
00:57
ExplosionGoogle ChromeSheaf (mathematics)Digital filterSet (mathematics)Point (geometry)Metropolitan area networkRow (database)Structural loadElectronic mailing listClosed setRemote procedure callLecture/ConferenceComputer animationXML
01:47
VirtualizationData transmissionComputer fontBoris (given name)CircleSet (mathematics)Electronic visual displayForm (programming)Closed setLecture/Conference
02:37
Metropolitan area networkBit rateBitMultiplication signJSONXMLUMLLecture/Conference
03:47
Public key certificateProcess (computing)Data miningRule of inferenceContext awarenessOpen sourceProjective planeQuery languageMereologyRow (database)Client (computing)Network topologyVideo gameLecture/ConferenceXMLComputer animation
05:02
Point (geometry)GoogolHill differential equationGoogle ChromeDean numberExplosionPowerPointEmailFront and back endsOpen setExt functorMereologyStructural loadClient (computing)Physical systemPort scannerWeightInterface (computing)NP-hardRow (database)Vector spaceFront and back endsTerm (mathematics)Traffic reportingDebuggerClient (computing)Associative propertyMereologyQuery languagePoint (geometry)Computer animationLecture/ConferenceSource codeXML
06:03
Parallel portServer (computing)Open setSoftwareRead-only memoryMiniDiscPort scannerModul <Datentyp>Multiplication signDiagramWordMiniDiscMoment (mathematics)BefehlsprozessorSemiconductor memoryBand matrixLecture/ConferenceXMLComputer animation
07:03
QuadrilateralUrinary bladderRead-only memoryNewton's law of universal gravitationMetropolitan area networkWritingReading (process)Software frameworkSlide ruleQuery languageSource codeTask (computing)Semiconductor memoryMiniDiscPairwise comparisonMultiplication signInstance (computer science)Data acquisitionVirtual machineScaling (geometry)Lecture/ConferenceXML
08:01
Set (mathematics)Single-precision floating-point formatData typeSystem callKernel (computing)Google ChromeSineCellular automatonRead-only memoryMiniDiscMetropolitan area networkPowerPointPoint (geometry)State of matterService (economics)Open setPort scannerProcess capability indexMulti-core processorVirtual machineSemiconductor memoryMoment (mathematics)Core dump2 (number)LaptopCASE <Informatik>Direction (geometry)Projective planeSource codeLecture/ConferenceComputer animationSource code
09:00
Goodness of fitBitDynamical systemRight angleLecture/Conference
09:52
ComputerIntegrated development environmentVirtualizationProjective planeScripting languageComputer animationLecture/Conference
10:41
Binary fileScripting languageCellular automatonVarianceCASE <Informatik>Escape characterScripting languageOracleModule (mathematics)Phase transitionACIDCommitment schemeLecture/ConferenceComputer animation
11:57
Scripting languageVirtualizationMultiplication signInstallation artPrisoner's dilemmaInterpreter (computing)Phase transitionDevice driverRight angleLibrary (computing)Lecture/ConferenceMeeting/InterviewComputer animation
12:47
Scripting languageBinary fileGoogolInformationInheritance (object-oriented programming)World Wide Web ConsortiumRevision controlMultiplication signFlow separationError messageLatent heatComputer programmingPairwise comparisonRiflingWater vaporScripting languageRankingView (database)AngleTask (computing)Annihilator (ring theory)Physical systemShared memoryProjective planeLecture/ConferenceComputer animation
14:24
Revision controlProjective planeSystem administratorArithmetic meanLatent heatIterationMappingSpacetimeRow (database)Lecture/ConferenceComputer animation
15:12
SpacetimeStructural loadOnline helpGodPower (physics)Goodness of fitBitDisk read-and-write headForm (programming)Presentation of a groupCASE <Informatik>Lecture/Conference
16:01
CodeSoftwareFreewareCodeThermal conductivityStandard deviationConnected spaceBitNumberDifferent (Kate Ryan album)Representation (politics)OnlinecommunityLecture/ConferenceComputer animation
17:00
CodeTerm (mathematics)Metropolitan area networkCodeProcess (computing)Thermal conductivityOpen sourceSheaf (mathematics)BitMereologyTraffic reportingElement (mathematics)Row (database)1 (number)Lecture/ConferenceXMLComputer animation
17:58
ExplosionTrailEmailGroup actionEvent horizonSound effectCodeMetropolitan area networkService (economics)Inclusion mapRouter (computing)Total S.A.Point (geometry)Traffic reportingRow (database)Single-precision floating-point formatSelf-organizationEvent horizonProcess (computing)Shared memoryNormal (geometry)Web pageDecision theoryOnline helpIntegrated development environmentStatisticsNumber2 (number)Figurate numberLecture/ConferenceComputer animation
19:16
CodeThermal conductivityCodeProcess (computing)FeedbackObservational studySlide ruleRight angle2 (number)Lecture/ConferenceComputer animation
20:13
Point cloudWordOrder (biology)BitProjective planeLibrary (computing)Similarity (geometry)QuicksortWorkloadLecture/ConferenceXMLComputer animation
21:20
CodeScripting languageOverhead (computing)Host Identity ProtocolInformation systemsHigher-order logicPort scannerLibrary (computing)Web 2.0DatabaseWordPoint cloudWeb pageVideo gamePartial derivativeMereologyScripting languageBitOnline helpCodeLecture/Conference
22:10
Line (geometry)Metropolitan area networkBinary fileDiscrete element methodDensity of statesCrash (computing)ForceInclusion mapHill differential equationLink (knot theory)Power (physics)MereologyHost Identity ProtocolKnotCloningCovering spaceArtificial neural networkInsertion lossMaxima and minimaPointer (computer programming)Division (mathematics)Port scannerCodeString (computer science)ParsingKey (cryptography)Goodness of fitFunction (mathematics)Instance (computer science)WordXMLLecture/ConferenceComputer animation
23:01
Personal area networkComputer iconPort scannerForceMetropolitan area networkHost Identity ProtocolMusical ensemblePermanentTuring testChi-squared distributionValue-added networkWhiteboardPower (physics)Menu (computing)ExplosionLink (knot theory)Maxima and minimaCodeReal numberExt functorLTI system theoryComputer fontString (computer science)WordBitElectronic mailing listCountingGame theoryFrequencySocial classMultiplication signPoint cloudOrder (biology)Lecture/ConferenceXMLComputer animation
24:05
Real-time operating systemMaxima and minimaCurve fittingGraph (mathematics)Computer networkMathematical analysisExt functorMusical ensembleAuditory maskingLibrary (computing)Bookmark (World Wide Web)Medical imagingInstance (computer science)Order (biology)Machine learningFrequencyWordComputer animationLecture/Conference
24:59
Data miningMaxima and minimaLaptopGodSinc functionAreaNeighbourhood (graph theory)Multiplication signPerfect groupSquare numberFerry CorstenFilm editingLecture/ConferenceComputer animation
26:24
Port scannerMobile appSocial classService (economics)Uniform resource locatorMultiplication signGroup actionWebsiteMessage passingCountingLecture/ConferenceComputer animation
27:31
Green's functionMessage passingGraph coloringGreatest elementLaptopRandomizationNumbering schemeAreaPrisoner's dilemmaProcess (computing)Multiplication signDifferent (Kate Ryan album)Volume (thermodynamics)Lecture/ConferenceComputer animation
28:22
BitGraph coloringWeb 2.0Right angleOffice suiteStandard deviationProper mapLecture/ConferenceComputer animation
29:16
Formal grammar2 (number)Multiplication signInformation securityUniverse (mathematics)Lecture/ConferenceComputer animation
30:16
Extension (kinesiology)Self-organizationLecture/ConferenceComputer animation
31:16
Event horizonExtension (kinesiology)Inheritance (object-oriented programming)WordSlide ruleService (economics)BitSoftware frameworkTelecommunicationPattern languageInternet service providerQuicksortMathematicsMereologyCommunications protocolMiddlewareInstance (computer science)NumberExtension (kinesiology)Goodness of fitEvent horizonLine (geometry)InjektivitätSystem callLecture/ConferenceComputer animation
33:14
Service (economics)System callNumbering schemeInternetworkingCodeStandard deviationCommunications protocolSerial portService (economics)ImplementationSystem callQuicksort1 (number)Queue (abstract data type)ResultantLecture/Conference
34:10
Service (economics)Social classMereologyCodeSoftware developerJava appletObject (grammar)Parallel portPolar coordinate systemPlanningSocial classWordSoftware testingComputer animationLecture/Conference
35:12
Service (economics)Gastropod shellEvent horizonSocial classType theoryBroadcasting (networking)MereologyService (economics)Gastropod shellInteractive televisionProduct (business)Event horizonBroadcasting (networking)Event-driven programmingCodeoutputType theorySocial classStudent's t-testPoint (geometry)Lecture/ConferenceComputer animation
36:54
Level (video gaming)Resolvent formalismMoment (mathematics)TouchscreenOpen setUsabilityMusical ensembleLecture/Conference
37:58
Line (geometry)Interface (computing)Form (programming)Data typeInformationCuboidVirtual realityMetropolitan area networkPhysical systemInteractive televisionGastropod shellCursor (computers)Presentation of a groupCartesian coordinate systemFocus (optics)UsabilityLibrary (computing)Normal (geometry)Demo (music)Line (geometry)CodeComplete metric spaceBlock (periodic table)Programmable read-only memoryArrow of timeInsertion lossWindowGastropod shellTouch typingCore dumpPoint (geometry)Functional (mathematics)Revision controlKey (cryptography)Direction (geometry)Reverse engineeringCodeSoftware developerRight angleElectronic mailing listMultiplication signImplementationNumbering schemeNetwork topologySoftware testingInstance (computer science)Interactive televisionOrientation (vector space)Sound effectPoisson-KlammerSpacetimeLecture/ConferenceComputer animation
42:02
Real numberDebuggerLine (geometry)Euler anglesMaxima and minimaClient (computing)Hill differential equationData Encryption StandardMetropolitan area networkPointer (computer programming)Uniformer RaumPersonal area networkInclusion mapElectronic mailing listCombinational logicProgrammable read-only memoryDatabaseInteractive televisionClient (computing)Real numberImplementationProof theoryFunctional (mathematics)Multiplication signCloningTwitterInstallation artCartesian coordinate systemInferencePoint (geometry)Video gameUsabilityFamilyTheoryLecture/ConferenceSource codeComputer animation
43:20
CausalityVideo gameMetropolitan area networkBitSummierbarkeitTorusClosed setLecture/Conference
44:11
Discrete element methodField (computer science)Graph (mathematics)Pauli exclusion principleString (computer science)InterpolationFunctional (mathematics)Symbol tableCartesian coordinate systemParameter (computer programming)Letterpress printingLecture/ConferenceComputer animation
45:16
IterationGastropod shellInsertion lossVariable (mathematics)Line (geometry)Incidence algebraInterpolationString (computer science)Virtual machineRun time (program lifecycle phase)WordComputer programmingCodeTransformation (genetics)ExpressionRight angleType theoryParameter (computer programming)Spring (hydrology)Coefficient of determinationFile formatReal numberBitCASE <Informatik>Lecture/Conference
46:53
IterationMetropolitan area networkCodeAlgorithmGradient descentIterationMachine learningNumberFile formatOcean currentString (computer science)Logical constantComputer programmingStatement (computer science)Axiom of choiceText editorSign (mathematics)Letterpress printingEquals signTransformation (genetics)CodeLecture/Conference
48:01
IterationCodeMortality rateProjective planeLibrary (computing)ParsingoutputComputer fileMereologySymbol tableExpressionForcing (mathematics)Square numberLecture/Conference
49:39
Right angleArithmetic progressionSlide ruleContext awarenessCellular automatonConstructor (object-oriented programming)SpacetimeEndliche ModelltheorieLecture/Conference
50:39
Division (mathematics)MassHypermediaBefehlsprozessorDivisorTheoryLecture/Conference
51:30
Group action2 (number)Modal logicEigenvalues and eigenvectorsTransport Layer SecurityHypermediaLecture/ConferenceProgram flowchart
52:23
ResultantEvent horizonForcing (mathematics)Image registrationCombinational logicMultiplication signLecture/Conference
53:28
FamilyWater vaporHuman migrationProjective planeBitInformationMedical imagingQuery languageMultiplication signLattice (order)SequenceBit rateExtension (kinesiology)Virtual machineSequelVariety (linguistics)Line (geometry)Lecture/ConferenceComputer animationSource codeJSON
55:28
NumberFamilyMoment (mathematics)Software testingLine (geometry)Table (information)CodeProjective planeArithmetic progressionSheaf (mathematics)Human migrationReplication (computing)Lecture/ConferenceUML
56:28
Table (information)CodeMathematicsMetropolitan area networkLinear regressionDiscrete element methodSoftware engineeringHaar measureTelecommunicationAuthorizationRow (database)Query languagePoint (geometry)LaptopCodeInstance (computer science)Computer configurationMathematicsControl flowVideo gameSoftware testingParallel portDatabaseRouter (computing)NumberHuman migrationPairwise comparisonResultantApproximationDifferent (Kate Ryan album)Server (computing)Table (information)Position operatorKey (cryptography)Product (business)Service (economics)Configuration spaceBitLecture/ConferenceComputer animation
58:25
Expert systemHuman migrationDatabaseSequelBit rateProcess (computing)Endliche ModelltheorieOrder (biology)Metropolitan area networkSummierbarkeitTouchscreenLecture/ConferenceJSONUML
59:52
Library (computing)System callSoftware testingHypothesisWave packetSource codeSoftware bugMaterialization (paranormal)Unit testingMereologyProduct (business)StatisticsEndliche ModelltheorieCASE <Informatik>Module (mathematics)Software frameworkExpected valueCodeDifferent (Kate Ryan album)Bit rateStandard deviationLecture/ConferenceComputer animation
01:01:46
NumberWebsiteGoodness of fitPoint (geometry)Software testingCodeProjective planeModule (mathematics)Multiplication signHypothesisEndliche ModelltheorieDampingElectric generatorGraph (mathematics)Mathematical optimizationRight angleType theoryLinear regressionComputer programmingPattern languageWater vaporCASE <Informatik>Complete metric spaceElectronic mailing listFrequencyLecture/ConferenceJSON
01:03:24
Red HatMultiplication signDivisorImage registrationFlow separationVideoconferencingGoodness of fitRight angleLink (knot theory)Lecture/ConferenceComputer animation
Transcript: English(auto-generated)
00:00
Alright, come in everyone, lightning talks start at 5 o'clock, the best place to sit for the lightning talks is right up at the front, especially if you're a speaker. If you're a speaker in the lightning talks, I definitely need you up at the front. If you're not a speaker, there's still room at the front. Or the lovely second row, mmm, secondy, mmm, second.
00:21
Okay, remember how squirrels pack nuts? Fill up the rows from the middle first. First person on my list is Cast Vochies. Are you here, Cast? Good. You wanna come and start setting up? Great.
00:48
Number 2 will be Fondo Battista, or something to that effect. Fondo, are you here? Very good. Alright, I'm just gonna go back over here and check number 3 now. Some people will check several at once.
01:02
Paul Hallett, Paul are you here? Yes. I can keep doing this actually. Raphael Schulze. Yeah, okay, you're pretty close to the front. I'm gonna allow it. Alright, you don't have to sit in the front row. That's great, thanks Raphael. Ben Foxall, Ben Foxall, Ben Foxall.
01:28
Alright, someone's gonna get bumped off the list. That's great, we've got loads of lightning talks for you today. Come closer to the front everyone if you've just walked in. There's loads of room up at the front. When you're closer to the front you can see better.
01:41
The person giving the lightning talk feels like you're more engaged and more curious about what they have to say. It makes them happy. A happy speaker delivers a better talk and a better talk informs you more and makes you more happy. In a virtual circle of knowledge transmission, passion, community and fun. Come close to the front basically is what I'm saying.
02:07
Yeah, that person is right. If you close the display settings app that little warning will go away. Or we can leave it. I hope, I don't know. Lightning talks are a succession of talks interspersed with people heckling from the audience about how to fix your display settings.
02:29
Come on, lots of room near the front everyone. The front is the best place to be for the lightning talks anywhere. You get to see more, you get to learn more, you get to feel more. The whole thing is more engaging. The day ends on a high note. You go home happy and the whole conference turns into way better value for money which we can't possibly argue with.
02:47
Come up to the front, come up to the front. Lots of room at the front. Also a bit of room here in the middle in a little enclave behind the cameraman. That's probably because people don't want to sit behind the camera. Oh okay, I'm looking at a camera. I wanted to look at a talk.
03:02
Come on in, we're going to start in about one minute. So there's not a lot of time now. If you're going to come to the front, it's right up here. Come on, walk, walk, walk, walk, walk, walk. Walk fast and safely towards the front. Now we don't start until exactly the time.
03:22
You can, no it's not, my watch is the only watch that matters. You can start heckling me when the official lightning talks have started but it's not fair to do that before. Get out you.
03:40
Hi, okay, we're about to start. So if you're now walking in, I need you to start walking in much more quietly. So tiptoe to your seat from now on as we're about to start. Same lightning talk rules as Harold. One hand for little finger claps, two hands for final claps.
04:02
You have up to five minutes, you don't have to use the whole five minutes. Karst, take it away. Thanks, everyone can you hear me? Okay. I'm Karst Varyages, I'm here to raise awareness for an open source project that we developed.
04:23
It's called BeQuery and basically I want to use these minutes to explain what it does, why it's really nice and I think more people should be aware of it and possibly use it. I co-founded my company three years ago and we started and we had some problems. So we had actually clients from the start.
04:42
They were retailers with billions of records of data, but we were hardly any money and we still had the need to be able to respond to that. So we had this technical issue that we had to address. And that's how BeQuery in the end came to life through various other parts in between.
05:03
What does it do? It basically runs everything which you see here in the back. I have actually something like 20 million records here in the back. And it runs through it, it aggregates basically on the back end and makes it possible for us to be very efficient in terms of aggregation and reporting.
05:28
So that's basically what BeQuery does. As you saw, the front end is HTML. It's what our clients use. And the whole entire back end is basically Python.
05:41
So that's what we do there. It's part of a larger part which we'll also outsource later on. But in the start I want to start with BeQuery. Why did we make it? There are several solutions out there like, of course, Hadoop. Spark did not exist three years ago yet. But there are other solutions but especially it's a very small startup.
06:03
They are very hard to administrate. They could be very expensive, use a lot of resources. And at the same time there were more technological developments in the background. I think the lowest rose, the most important one, which you see actually with MongoDB, WireTiker,
06:21
and they're moving towards compressed on disk saving. That's also what's happening in the Python community because of these two men here who made bicles. Unfortunately they're not here at the moment but they also helped us a lot, really great guys. And what bicles is, it's basically compressed data container.
06:41
So it takes data, puts it compressed on disk, which means that it sends it zipped to your CPU, it unzips it there. And the idea is that modern CPUs are so quick that you actually overcome the memory bandwidth issue and are nearly as quick as it will be in memory.
07:00
And that's actually true. The only problem with bicles is that it does not aggregate anything. It's basically bare bones framework for compressed data and for reading and writing that compressed data. So that's where we made bquery. bquery is basically on top of bicles and it makes you, well this is, I will post the slides on slides here
07:23
so you can read this later on. But basically what it does is it's aggregation framework. And it's rather fast. So we have comparisons with Pandas for instance. So compared to Pandas in memory, it's like 1.5 to 2.5 more times slower. That's of course comparing your own disk aggregations
07:40
compared to what Pandas does. There's also, the sources are down there, there's some examples of a New York taxi data set with DAS where you can see basically it's being spread over a Hadoop cluster of eight machines with 30 gigabyte per machine, eight processors, etc. And what it basically does, and I hope it scales well,
08:03
but what it does is what my own quad core machine now is doing in 1.8 seconds, it does that in 0.5 seconds with the same eight machines. And what you cannot see at the moment, but it uses less than one gigabyte of memory
08:21
and runs this fast on my own laptop. So that's basically what I thought was maybe interesting for more people. And it's downloadable. Only if you have PIP-8 it won't work at the moment because of some strange reason. If you have PIP-7 you can still install it
08:41
otherwise from the source, from GitHub directly. We're still working on it, but you're, of course, free to look at it and also join the project if you want to. So that's basically my short introduction of BigQuery and what it does. Any questions?
09:11
Thanks very much. Next is Fondo. Good. How's everyone enjoying their conference so far?
09:23
It's cheap, but it works. Like a free whoop. Just do it again! Are you ready to go, Fondo? Without further ado, Fondo would like to give him a big hand. Thank you.
09:41
So raise your hand if you use virtual end. Right, a lot of people. Virtual ends are awesome, right? I mean, you have a lot of benefits. You can install whatever you want without making dirty your installation and you can reproduce environments from one computer to the other, etc.
10:04
I don't want to sell you virtual ends. What is the problem with virtual ends? We need to manage them manually. So it's not a problem, really, for when you're working in a big project because you enter in the virtual end and you're there eight hours. But what do you do for the scripts?
10:22
You have 35 scripts in your computer and do you install the dependencies in one virtual end for all the 35 scripts? Or do you have 35 virtual ends, one for each script? And you have to remember which virtual end was
10:41
before executing the script and remember to enter the virtual end before executing the script, getting out of there and entering, etc. What do you do in that case? It's a problem. It's a mess. So, Face is here to help us. With Face, it's very simple. You only indicate the dependencies.
11:03
That is all you care about. You care about your script and the dependencies. You only need to tell the dependencies, execute it, and nothing else. How do you execute it? Well, very simple. As any other script, we're calling it with Face,
11:21
or put Face in the shebang, or even as a Python module if you want, you can call it and you can execute with your script, and you only need to do specify the dependencies. How do you specify the dependencies? Well, it's very easy. So, for example, if you want to use a script
11:41
in a virtual end that has request, you just call face-d request, and Face will execute your script in a virtual end that only has request installing it. If you call this, if you are in a clean machine, it will create a virtual end, it will install request calling pip, et cetera,
12:02
and execute the script there. The second time you do it, it will be super fast because it already has a virtual end with request installed. If you want to execute another script with only request, it already has the virtual end. You don't need to care about anything. If you call face-d your dependency
12:24
and don't specify a script, it opens for you an interactive interpreter. This is the quickest and easiest way to try a new library. Did you try this library? No, I didn't. Oh, face-d library, and you have an interpreter, and you have an interpreter
12:42
that lives inside the virtual end with that dependency installed, and you just try it. You don't need to do anything else. If you like to use a Python, you just tell it to use a Python. If you want to use any specific Python version, you just tell it to use any specific Python version.
13:01
If you have several dependencies, you just call dashd several times. If you have, I don't know, if you have to put one of the dependencies, or several of the dependencies in a specific version, or greater than, or less than, whatever, you just specify it. If you have a requirement.text,
13:22
you have to tell it with dash error. So, that was the simple one. Let's level up. The simplest way to work with a script is this one. You have your script, you put face in the shebang, and how do you specify the dependencies? See that magic commentary face in the import?
13:42
Face will execute this script in a virtual end that only has request installed because you tell it through a commentary. You can tell the commentary in several ways. You can put it in the doc string, you can put it in the face shebang line, et cetera. There are several ways.
14:01
For example, another complicated task, to start a Django project with a version of Django that you don't have installed in your system. How do you do that? Well, with face, it's easy because you call face, telling it that the dependency is Django 1.8,
14:20
and with dash X, you execute something inside the virtual end. So, you're executing the Django admin of the version from the virtual end, and this way, you can start the Django project in that specific version. Also, for example, you can, if you have a specific PIP requirement because you have a proxy or whatever,
14:41
you can also tell it to face to teach PIP how to work. So, keep calm and use face. You can install it very easily. If it's already in Debian, it's already in Ubuntu, it's already in Arch, you install it and use it, and no more virtual end manually handling.
15:01
Thank you very much. All right, come on in. There's still a few people coming in, everyone, so make some space at the end of the rows. If you spot there's space in the middle of the row, everyone move two seats towards the spaces, make it so that people can actually come in, make yourselves helpful there. There you go.
15:20
That's one space liberated there. That's fantastic. There you go. There's loads of people sitting down at the back of the room. They could easily be given a bit more space. Oh my God, the power! Everyone is doing what I say! There you go. All right, next is Paul Hallett. Paul, you're here somewhere?
15:41
There you go. Yep, that's good. And then after Paul is Mr. Schulze. And then after that we're going to have Ben Foxall. Ben, did you arrive? Good, all right, come a little bit closer forward. Are you ready to go, Paul? I am. In that case, give him a hand.
16:05
Hi, everyone. So my name is Paul Hallett. Does anyone here use Django? Yep. Does anyone know that the Django Software Foundation has a code of conduct committee? A few people. Okay, so I'm a member of the Django Software Foundation
16:20
code of conduct committee. I call it just the code of conduct committee from now on. I have an announcement to share from Django and the code of conduct committee today, but for those of you who didn't put your hand up, tell me a little bit about what we do. Pretty simply, we are responsible for making sure the Django community provides a harassment-free experience for everyone involved.
16:41
We have members that are global and representative of you, the community, so we've got people from different backgrounds and different experiences, and, as I said, we have something quite big to announce today, and that's our code of conduct documentation. This isn't the Django code of conduct. I'm sure you're all aware, if you've ever been to a Django conference,
17:01
that we do have a code of conduct, but this is documentation on how we actually deal with the processes of receiving issues and making sure that the Django community is friendly and safe and inviting. Before I get into why we decided to open source this, I want to tell you a little bit about what's involved. The first section is about membership,
17:22
how we elect members, how members can maintain their status. We are trying to promote, like, a non-burnout-style membership, so people opt in, and they say, I would love to be part of this for six months, and then after that, they have no obligation to stay. We also have the most important part, which is how we handle reports. We take each report as seriously as the next one
17:43
and the previous one, and we have processes where exactly how we go through this, so people who feel like they're uncertain about making reports can see the exact processes we go through to make sure that we handle those issues seriously and justly and fairly. We also mention how we keep records of reports.
18:01
There's no point in us actually running this committee if we aren't able to collaborate with Django organizers and other organizers of Python events if we don't keep a record of those reports, and that includes some processes around anonymization as well. You may not realize this. Not many people put their hand up that actually knew we had a committee,
18:21
so very few people even realize this, that we actually work with the organizers of every single official Django conference to make sure that we share reports between them, so they share with us the attendee list, and if there's any reports, we're able to share them with other communities. And finally, you can also find
18:40
our transparency and statistics there. This is the actual figures of the number of reports we've received over the past two years. As you can see, we've gone from 11 two years ago to seven so far this year, so hopefully we'll be able to bring that down and ultimately make ourselves redundant and make the community friendly and inviting. The reason we did this, there's three reasons.
19:01
The first one is obviously to hold ourselves accountable to you to make sure that we're providing this safe environment for people and also to let you understand how we make these decisions. The second reason, which is closest to my heart, is to help other tech communities who have not been able to adopt a code of conduct to show them that you can do this safely,
19:22
you can do it fairly, that we don't witch hunt. This is actually done through a very democratic process. And finally, to get feedback from you to understand if we're doing a good enough job, if we're making it more well understood, and generally to get your feedback on how we can run this better. So a big shout out to these people.
19:40
The reason my name here isn't because I'm pretentious, actually it's because Ola Sutarska wrote these slides and she's announcing this exact same lightning talk right now in Philadelphia in the USA at DjangoCon US, so we're launching this right this second. So it's there, go and have a look, give us your feedback, and I look forward to hearing it. Thank you.
20:08
Alright, who wants to hear a story about a squirrel? I tried to, I thought we could just do jokes, but I haven't actually got any new ones since last year.
20:21
So instead I've got a funny story about a squirrel that I'm going to try and read in an entertaining manner. And it starts off, and we're not going to have it for long, I never dreamed slowly cruising on a motorcycle through a residential neighborhood could be so incredibly dangerous. Little did I suspect, dot dot dot. That's your intro. Let's move on to Star Wars word clouds.
20:47
Hi everyone, I'm Raphael. This is a fun project that I recently worked on. Why? Just because I like Star Wars and I like to play around with data, so I thought it was a good idea. I also think that these very small projects
21:00
are a neat way to just explore basic data science concepts and the capabilities of Python libraries. So in particular, I guess with this, I just want to encourage maybe newcomers to Python or to data science to do something similar. In particular, I wanted to test this library,
21:21
which is the word cloud library. I don't know if you know it. This is from Andreas Müller. If you don't, then check it out. And so the first step that I did was, of course, to get some data. For this, I went to this web page, which is basically the database which contains movie scripts,
21:41
and this is a partial screenshot of how this looks like. Of course, what we need to do here is extract data that is actually being spoken by the people because we want to create word clouds from Star Wars characters. So with a bit of introspection of the HTML and the help of a beautiful soup,
22:01
we can just parse the HTMLs. This is an example code for episode five and you hope. So we basically get the HTML using requests, parse it using beautiful soup, and then we iterate over it, extract the code for each character that is being spoken,
22:21
and then we end up with a dictionary where basically the keys are the character names and the values are the strings of the text that they spoke. I'm not going to go into details here. This is an example output of Darth Vader from episode four. As you see, this is raw text,
22:41
so the next step is basically to go ahead and clean this by basically doing things like removing punctuation, removing stop words, for instance, lower casing it. You do this for each character across all episodes, and then you end up with something like this,
23:01
which is a nice string of just words. Again, the example of Darth Vader. There's bits and pieces that you still need to do, like, you know, merge dictionary entries that denote the same character, like Luke and Luke's voice or C3PO and 3PO,
23:20
stuff like that. And this is the list of, the top list of the characters by word counts, and then we're ready to create our word clouds. For that, it's as easy as instantiating this word cloud class with basically a list of words and their frequencies.
23:43
And so let's play a game. Who's this? Come on. It's Luke, of course. Who's this? That's easy. 3PO, exactly. What about that one?
24:02
Someone said Yoda, right? That's Yoda. Well, now an easy one, easy one. This one, right? Han Solo. Okay, last one. Who's this? Of course, our all-time favorite character, Star Wars.
24:22
Exactly. So what we can do is basically do, we can also pass, for instance, image masks to this library and create more beautiful stuff. So we come up with something like this. You recognize this. Here's Yoda.
24:41
This is Padme. That is Obi-Wan. And, of course, Darth Vader. Of course, this is just the beginning. There's a lot more you can do with this data. For instance, use TFIDF instead of just word frequencies. Apply machine learning and now try to classify maybe dark and bright side, whatever.
25:03
Do some network analysis of the interconnectivity of the different characters in the movies. I am new cortex in GitHub. If you want to check the notebook, it's here. Star it, fork it, clone it, run it, whatever you like. And thanks a lot for your attention.
25:28
All right, Ben Foxall next and then Christian Stefanescu. Christian, are you here? All right, you're ready to run up. After Christian, we'll have Jonathan Slenders. Jonathan, are you here? Very good.
25:41
I was on Bryce Street, a very nice neighborhood with perfect lawns and slow traffic. As I pass an oncoming car, a brown furry missile shot out from under it and tumbled to a stop immediately in front of me. It was a squirrel. And it must have been trying to run across the road when it encountered the car.
26:01
I really was not going very fast and there was no time to break or avoid it. It was that close. I hate to run over animals and I really hate it on a motorcycle, but a squirrel should pose no danger to me. I barely had time to brace for the impact. Animal lovers never fear. Squirrels, I discovered, can take care of themselves.
26:22
Are you ready? Yeah, give him a big hat. Thanks for that intro. That was perfect. Cool, so I built a little app over the last couple of days and I thought I'd show you over here. I actually built the last commit in seven minutes ago, so I've not seen if that works, but hopefully it will.
26:45
So this site is a kind of a multi-device site. So what I want you to do is get out your phones and stuff and visit this URL, which is eupy16.herokuapp.com. And we can see this device count going up
27:01
and this is a tiny little flask app that's using our service, which I won't talk about just now. So we're getting 21-ish devices. It's going up and down. It's kind of weird. And what you should see is you should see this kind of greyed out logo. And what this tool that I've hacked together is, it's basically a kind of logo designer, I think.
27:23
So what we can do is each of our devices is connected to this flask app and through our infrastructure and I can send out messages. So basically let's choose the first color of this logo. So let's choose green. Whoo! Cool, so that kind of works, which is good.
27:43
Wasn't totally sure about that. And yeah, we can choose these other colors as well, so maybe green and blue. And you should see that updating on your phones and laptops. I've put some buttons down the bottom which have emoji in them.
28:01
So the first emoji, this chooses a random color scheme for those two things. So you should see a different color scheme for each of your, from your neighbors. And I can press this a few times and you can get some new ones, new ideas for your logos, right? This speaker button, what I'd like you to do is put up the volumes on your devices.
28:28
Cool. You'll get more of that. Okay, so this is using the Web Audio API to synthesize a note that's randomly chosen on your device. Right, okay, slightly linked to the last talk,
28:42
we're going to make this a bit more real. We're going to use these two choosing colors and sounds to select a winner, okay? And the thing you're going to win is this BB-8. Whoo! Cool. So what's going to happen is everyone's phones are going to change colors
29:03
and they're going to play notes and they're going to get a bit more kind of chaotic and then eventually one person will have the kind of, like the proper logo colors, everyone will also have gray and that person will have one. So if you hold up your phones and turn them to the center of the room, that would make it kind of cool, right? So is everyone ready for this?
29:24
Uh, whatever. Cool. So let's start this then. Is this going to finish before the heat death of the universe? Just have you checked? You'll be like 40 seconds left.
29:43
Cool, so it's kind of in time. What is happening? It's a bit long.
30:41
And we're going to have a session tomorrow at lunchtime if you're interested to show how we build this, or whatever, have lunch. Thank you very much. Thank you. Christian?
31:01
One of the organizers there, did you want to say something, Fabio? Did you want to say something? I saw you creeping up, no? Okay. Oh, you have a talk? That's even better. There you go. Are you just ready to go, Christian? Yeah. Then take it away.
31:22
First of all, sorry, I'm super nervous. I usually don't speak in front of more than three people. My name is on the slides, and this is Namako. Namako is a Japanese word. I looked it up. It basically is a fungus which you use to make miso soup.
31:43
So I'm sorry I pretty much gave it all away. You know I'm going to talk about the microservice framework. Yeah, so a bit of background. We had in our company a big change in which we decided to go for the fancy stuff you build today,
32:02
an API and some microservices. Then we were looking around for the fitting framework to build our microservices, and we found Namako. Although I'm quite nervous still, I hope I can convey some of the excitement I had when I found Namako. So Namako can do RPC or event-based kind of communication.
32:25
I'm going into these patterns in more depth soon. If you don't know what that means, it uses eventlet under the hood for async worker handling. It uses dependency injection,
32:42
and it uses it for so-called dependency providers. These are all sorts of things you can plug in to your Namako services like logging or think of it like some sort of middleware. And then it uses extensions,
33:01
and it uses extensions, for instance, to define the protocols to transport messages. So let's look at RPC calls, and this is the main part we're using Namako for. Okay, so I already said microservices, so in the grand scheme of things,
33:21
there's a method somewhere out there on the Internet, probably in some sort of service which I want to call, and I'd like my code to be nice and readable, and this is actually doable with Namako. So this could be the code you could be writing tomorrow.
33:41
Under the hood, in the middle there, let's think of Namako as, or actually, Namako does this. It serializes your call to JSON, sends it over RabbitMQ, that's the AMQP protocol implementation, and calls the service, then creates a new queue
34:02
which will hold the result and returns it. So that's it for RPC. Now let's look at the code, and this is the part I like the most, I think. This is just a class, and this is just a method, and I think the only thing that's kind of unusual
34:22
besides the import of some fungus is the decorator at RPC, which transforms this plain Python method into something that actually works remotely. I was thinking about the parallel to Java,
34:43
but I quickly dropped it, so Java developers call POJOs, or so-called plain old Java objects, but I think this doesn't work for Python objects. So anyway, it's a simple Python class, and if you're like me and you like testing,
35:03
you probably like this as well. Look at how nicely the method doesn't contain anything about the transport. Okay, now for the RPC part, I also like the tooling a lot. In the upper left corner, we have a helper
35:22
I can call from the command line, which will run my service. It will bring it up, and I can start calling it from, and that's the part in the bottom right, a shell. So this is basically a Python shell which is configured to talk to this said service.
35:41
So interaction is quite fast, and you can test stuff, and you can also use this in production to talk to your services in production. Now, I said events are also possible, so for the scenario, I have an event handler somewhere in my service. I can write this kind of code.
36:01
I instantiate an event dispatcher and send it a hello, and I also can pass a payload, and then, yeah, sorry, there's no response, of course. I'm just sending the event. Let's look at the code again. Again, the input is missing also,
36:23
but the decorator is the only thing that distinguishes it from a simple Python class. There are three types of event handlers. I'm gonna put them all there so you can see them. So maybe singleton is the easiest one. You want exactly one delivery.
36:42
Broadcast means you reach all, and a service pool is actually just one out of a cluster of services. It can do HTTP, but I wouldn't.
37:02
Thank you very much. That was Christian. I've got Jonathan coming up on stage. Fantastic. After that, it will be Mathias Rav. Inches before the impact, the squirrel flipped to his feet. He was standing on his hind legs and facing my oncoming victory cross county tour with steadfast resolve in his beady little eyes.
37:23
His mouth opened at the last possible moment. He screamed and leaped. I'm pretty sure the scream was squirrel for banzai or maybe dye you gravy-sucking heathen scum. The leap was nothing short of spectacular. He shot straight up, flew over my windshield, and impacted me squarely in the chest. Instantly he said upon me,
37:41
if I did not know better, I would have thought and sworn he brought 20 of his little buddies along for the attack. Snarling, hissing, and tearing up my clothes, he was a frenzy of activity. A frenzy of activity very much like Jonathan, who will give us a short lightning talk about prompt toolkit.
38:00
Thank you. So this is a very short presentation about prompt toolkit, which is a library for building command-line applications in Python, with a very strong focus on usability. So for that, we go back to the normal Python shell, just to have a short demo.
38:20
So let's start a Python shell and do some Python coding. So for instance, we can do an if-test and then print hello world, just as a demonstration, right? Everyone has done this. Now the problem is, suppose we have to execute this again. What do we have to do? We have to press the up arrow three times, like this.
38:42
We have to press the up arrow again three times. There we go. Once more, and now we can execute it. This is a bit annoying. Even more annoying is if we're at this point and we have to insert a line. We cannot insert a line right below df true, right?
39:00
The only thing we can do at this point is press Control-C, like this, to interrupt it, fetch the lines again from the S3, and while fetching them, insert the lines. So that's really annoying. Now, coming back to Prom Toolkit, it's a library I've been working on for the last three years, and about one year and a half ago,
39:20
I released a tool called PT Python. So it's a Python shell built on top of Prom Toolkit. Now, let's do the same thing here. We print, we do an if true, we print hello, like this, we print world, and you see, as I type, I have syntax highlighting, right?
39:41
And also, I have very nice code completion. Now, if we have to execute this again, the only thing I have to do is press the up arrow only once, like this. There we go. Now, even more, we have multi-line editing.
40:03
That means we can navigate in two directions. We can move the arrows up and insert lines in between here, and we can just execute the whole block in one keystroke. So Prom Toolkit is a library that implements those kind of things
40:22
for people who want to implement interactive applications at the command line. Prom Toolkit does most of the baseline functionality, so redline is the library that's used by the native Python shell. It's used by many command line applications, and most of the redline functionality, like VI key bindings, Emacs key bindings,
40:42
reverse incremental search, all those kind of things that you expect, they're implemented in Prom Toolkit, so we can search back in history and execute from the history. Coming back to my presentation, we have seen this. So lately, I got in touch with the IPython core developers,
41:03
and we collaborated a lot, and this resulted in IPython 5, so maybe you have seen it. It's released a few weeks ago, and IPython 5 has a front-end built on top of Prom Toolkit, so that means that the functionality that you have just seen in PT Python,
41:20
like the syntax highlighting, like the multi-line editing, it's all present in the latest version of IPython 5. With Windows support, Mac support, and Linux support. Further, the only thing I haven't said yet is that we support bracketed paste, so that means if you're pasting a chunk of Python code,
41:42
it's recognized as being a paste, and it won't execute it yet. It's inserted as one blob of Python code in a multi-line buffer, so you can edit it before you execute it, and also means that the tap, so the indentation will be kept like it was.
42:02
And further, we have Moi support. In the last year, many tools were created using Prom Toolkit, so this is a list. The list keeps growing. Many people start creating tools on top of Prom Toolkit. This is one of the last ones, HTTPPrompt, which is a combination of HTTP and Prom Toolkit,
42:21
which is a nice tool to do HTTP GET and POST requests. These are a few others, so there we see a few database clients, like PGCLI and myCLI, and AWS Shell to do interaction with Amazon. I've even two full-screen applications, so PyVim, for instance,
42:40
which is an implementation of VI in Python. It's more as a proof of concept, but if you want to play with it, it's fully functional. Not all the functionality of real VI, but it is usable, and we have PyMux, which is a clone of Tmux and pure Python as well. That one is very usable.
43:01
I use it all the day as a replacement for my Tmux, and every time when I miss some functionality, I add it to Tmux or to PyMux. So that's it. If you want to find me, you can find me on GitHub, on Twitter here at the conference. As a reminder, we have the PIP installed, so you can PIP install a Python, PIP install PT Python.
43:23
Thank you. Thank you. Next is Matthias Rave. After Matthias, we've got Juan Luis Cano. Snarling, hissing, and tearing at my clothes,
43:41
he was a frenzy of activity. As I was dressed only in a light T-shirt, summer riding gloves and jeans, this was a bit of a cause for concern. The furry little tornado was doing quite some damage. Picture a large man on a huge sunset red touring bike dressed in jeans, T-shirt, and leather gloves puttering along at maybe 25 miles an hour
44:00
down a quiet residential street and in the fight of his life with a squirrel. And losing. Matthias, you ready? Yeah. Hello, everyone. I'm going to tell you about a new feature of Python 3.6,
44:23
which you may have heard of, maybe not. It's mostly for those of you who haven't heard of it. So a quick show of hands, how many are reading the Python 3.6 release notes draft? Okay. That's nice. So maybe half of you will learn something new now.
44:41
It's called Literal String Interpolation, and it's PEP 498. So suppose you have a function, a simple hello world function. You can use string interpolation to insert arguments into the stuff you're going to print. So it says, hello world, hello Harry.
45:02
You should probably know this. And the thing about string interpolation is if you have an application where you're doing logging in the real world, maybe some of your log lines are really long and have several things to interpolate.
45:22
Basically, we would like to be able to do the same thing as we can do in the shell and in Perl and in PHP, that is, insert variables inside the string literal and have them be inserted where they are. So my second example here uses the dot format,
45:41
which looks like what we want. But again, the problem is that I have to specify the variables to interpolate next to the string literal and not inside of it. So the solution is the Literal String Interpolation feature of Python 3.6, and you basically put an F in front of the string literal,
46:04
so it's called an F string, and you can put any kind of Python expression that you want, and it will just be evaluated at run time. So in this example, the name greeting will be taken from one of the arguments,
46:22
and target will be taken from the arguments, and we call the title string method to abecase the word. So I started playing around with this a bit, and I've made a small program that will automatically apply this transformation
46:42
so you can upgrade your code to use Python 3.6 as soon as it comes out, if that's what you fancy. So I took a real-world example from one of my machine learning hand-ins. I have some gradient descent algorithm I've implemented, and I'm writing out how long into the method are we
47:06
and which iteration is it and what's the current cost, and blah, blah, blah. And basically, I have a program that can automatically turn this into the F string below where you can see how we do number formatting
47:21
just like the old format method of strings. And it has... Currently, I have just integrated this with a vim because that's my idea of choice. You can blame me all you want. It works like this.
47:41
You can select the text you want to transform in your editor and press the equals sign that does this transformation, and there you go. It has turned the print statement into something that uses F strings, and we can sort of tweak it from here.
48:01
And that's all. If you're interested, the code is on GitHub. I'm mortal on GitHub, and you can go check it out, and it's a small project using the AST library that does a simple parse of your Python input file and simply looks for the percent formatting expressions.
48:21
That's all. Thank you. Juan Luis. After Juan Luis is Andrea, Andrea Crotty. What did I say? Okay. Good-o. And after Andrea is Dave McIver.
48:42
I grab for him with my left hand. After a few misses, I finally manage to snag his tail. With all of my strength, I flung the evil rodent off to the left of the bike, almost running into the right curb as I recoiled from the throw. That should have done it. The matter should have ended right there. It really should have. The squirrel could have sailed into one of the pristine kept yards
49:01
and gone on about his business, and I could have headed home. No one would have been the wiser. But this was no ordinary squirrel. This was not even an angry ordinary squirrel. This was an evil mutant attack squirrel of death, twisted evil. Somehow, he caught my gloved finger with one of his little paws, and with the force of my throw, swung around and was a with rounding sump
49:23
and an amazing impact, landed squarely on my back, and resumed his rather antisocial and extremely distracting activities. He also managed to take my left glove with him. The situation was not improved. Not improved at all. A big hand.
49:44
Thank you very much. This is going to be quick, because I'm nervous as hell. Not because I don't speak usually in front of hundreds of people, but because I don't usually prepare slides in five minutes. My name is Juan Riscano. I'm the chair of the Python Spain nonprofit,
50:00
and we organize the PyCon in Spain. But before getting into detail, let's put some context here. Okay, this is the country that you are now so far this week, and if we zoom to the southeast of the country, you can see this white thing over here, which is the only construction made by humans that is visible from space.
50:23
It's actually one of the biggest greenhouse cultivation in the world. If we zoom even more, we can see this Fort Bravo scenery that was used a lot in the 70s and the 80s to film some spaghetti westerns. And if you zoom a little more, you can see Clint Eastwood over there when he was young.
50:43
Well, we are celebrating the PyConess this year in Almería, which is the city in the southeast that I was talking to you before. This is going to be our fourth edition, and we're aiming at 400 attendees. And what are you going to get if you come to the PyConess? Well, you're going to get to see a very beautiful city with a lot of Arabic heritage,
51:04
lots of very beautiful monuments, and also one of the most beautiful beaches in Spain. We even celebrated it in October instead of November, so you can relax a bit and go to the beach because it's going to be very good weather still. This is the other thing that you're going to get
51:20
because the food there is very, very good. Maybe not as sophisticated as here in the vast country, but I can assure you that the quantity is going to... Yes. This is called the tapas thing here in the south. And you also get to know the wonderful Spanish community.
51:43
The Spanish community is so friendly, and we've been celebrating this PyConess, as I told you, for four years already, and it's always amazing to get to know these amazing people. This is a picture of the third edition, which was last year in Valencia. Again, like 400 people or so.
52:02
And this is the group picture at the end of the second edition in Zaragoza. So the column for papers is still open, so Clint Eastwood wants you to come. Thank you very much. All right, thank you. Andrea, you're next. Then David McIver. And then Anselm Lingnow.
52:21
Anselm, are you here? Anselm, yeah. Okay, very good. How's our timing? We've got 10 minutes. A quick announcement. If you haven't picked up your ticket yet for the social dinner and you want to go, you have to pick it up before 6.15. So the lightning talks are not going to run over. We'll finish at 6. You've then got 15 minutes to pick up your ticket
52:41
from the registration desk, after which it's closed, and then you can't go to the social event. There you go. Another thing that might be booing would be this guy on his motorbike. His attacks were continuing and now I could not reach him. I was startled, to say the least. The combination of the force of the throw, only having one hand, the throttle hand,
53:00
on the handlebars, and my jerking back, unfortunately put a healthy twist through my right hand and into the throttle. Now, a healthy twist on the throttle of a victory cross-county tour can have only one result. Talk. That is what the victory cross-county tour is made for and she is very, very good at it. The engine roared and the front wheel left the pavement.
53:23
The squirrel screamed in anger. Andrea! Hi everyone, I'm Andrea Crottin. I work for a company in London called Iwoka and one of the projects I worked on in the last few months
53:40
is a migration of a big codebase from Jungle Project from MySQL to Postgres. So I just want to share with you a little bit of what we encountered and what were the problems and try to help you not to do the same mistakes. So, why first? Well, I think this image is amazing, so that's enough information.
54:08
I didn't... I don't take credit for that, but I think it's great. So, the reality is... Well, originally, the beginning of why they chose MySQL was kind of a coincidence because they had to go live
54:20
and the only thing installed on the CTO machine was MySQL, so they decided to go for MySQL. So, after a few years, we wanted already to switch to Postgres, but then we found out of an actual real problem we had of some query which we need to do all the time, everywhere, which involves a self-join and some other ugly stuff
54:43
with MySQL and has to be more or less done in raw SQL or with some crazy hack with the ORM. However, Postgres has this nice thing called distinct on, which MySQL doesn't have, which allows you to write this query in a very simple, non-nested thing, which can also be written in the ORM in just one line.
55:02
So, amazing. So that was kind of the final reason to do the actual switch. So, how you actually do that? Well, all you have to do is to change the settings. The data will be moved automatically by Django,
55:22
and then... And that's it. Well, not really. I think that didn't work, so... So, just a few numbers to understand how big the project was. It's like a big project, so 190,000 lines of Python code, more than 100 apps,
55:41
383 tables to migrate, and at the moment we have around 3,000 tests, which takes quite fast, but yeah, a lot of tests. So, the plan, we are in progress. Actually, it's not finished, but we're almost there. It's to first adapt all the code, then actually do the data migration itself,
56:00
and then we're done. Data migration actually was not as easy as I thought, because I found this project called PG Loader, which is great. It's fast, it's smart and everything. The only problem is that you don't really get any live replication of the data, so you kind of have to do the switch in one go,
56:22
which is kind of scary, and yeah, it's not a very nice thing. But yeah, that means we have to try it many times first, and be sure it works, and only when we are sure everything is fine, we have to do it one go. PG Loader, however, is a really nice tool, and we also had great communication with the author
56:41
that helped us a lot, and also managed to convince the company to sponsor him to do some features we needed, so that's also a nice thing. Another problem we had was that we have very big tables, just a few of them, and we actually don't need to port them at all, and we will move them somewhere else completely. So to do that, to kind of handle the situation,
57:02
we just drop the foreign keys, change the queries, and then do a database router to just keep all these things where they are now, and we will handle them later. So that's another option we can use. The changes in the code, first of all, get everything running on your CI, like Jenkins or whatever.
57:21
Be sure that the new code doesn't break postgres anymore. Look for all the places where there are row queries that you want to fix and test them, and then you have to do a lot of manual testing, maybe and check more things. So one thing which I had to do, for example, since we had a lot of APIs which were not tested at all, I wrote like an IPython notebook
57:41
to run three parallel Django instances connecting to two different servers on different configurations, and then with this notebook connect it to all of them, and then do the comparison of the results doing some approximation of the floating point numbers we get back, so it was a lot of work,
58:02
but that's mostly because there were no tests for this thing, so yeah. So these are just the tips of the things you probably should do. Really use migrations for everything. Luckily we do that. We don't have anything in the database schema which is not in a migration, and that helps a lot. Test all your code and all your queries
58:21
because you will make life a lot easier when you have to do anything like that. Never rely on implicit ordering of databases. That's really a bad thing, but because MySQL always orders on ID anyway, everything is fine, but Postgres doesn't do that. And then try to make your Django apps independent
58:41
from each other so they don't import models all over the place and it's all like a tangled mess. And then split your monolith as soon as possible because that would really help with anything like that. Yeah, conclusions. This is a sentence I stole from this morning. Yeah, and that's it. Thank you. Thank you, Andrea.
59:01
Our final talk today will be by Dave McIver. David McIver, that's the next one. Anybody who did sign up and didn't quite make the slot today, please come and sign up again tomorrow. If you're keen, you'll get there super early and then we'll know that you desperately wanted to do your lightning talk. And that's about that. Now where were we?
59:21
We had a man there. We had a man screaming. The squirrel screamed in anger. The Victory Cross County tour screamed in ecstasy. I screamed in... Well, I just plain screamed. Now picture a man on the huge sunset red touring bike dressed in jeans, a slightly squirrel-torn t-shirt wearing only one leather glove and roaring at maybe 50 miles an hour
59:41
and rapidly accelerating down a quiet residential street. Hi there. I'm not a squirrel, sorry. I'm David McIver. I just want to tell you quickly about the testing library that I write called...
01:00:00
I'm afraid you've already missed the training course that was yesterday morning, but hopefully I can intrigue you enough to check it out anyway. The basic idea of Hypothesis is that it makes your tests smarter and less work to write, mostly less work to write, by adding a source of generated data to them so that it can try probing your test code
01:00:24
with a whole bunch of different values and find the one that you inevitably forgot because we all rate our tests when tired. This is what it looks like. It's a simple decorator library. It works with any testing framework you like. It's not a test runner. I use py.test.
01:00:40
It works with unit test. It works with nodes, whatever. This is an example from when we were testing some Mercurial code. Mercurial represents a bunch of stuff internally as a UTF-8B, which is a way of, for some reason, taking arbitrary binary data and turning it into valid Unicode. And that works about as well as you'd expect.
01:01:03
I don't know how widely that you'd use this one is. I want to claim that this is a great bug because Mercurial has been in production and widely used for about 10 years, but I suspect the reality is that this is in a relatively small corner of their code. This is fixed now, by the way. We found a bunch more bugs in the same code
01:01:20
once they fixed this one. It's also found bugs in the Python standard library. This is from the new statistics module in Python 3. One of the many nice things you get when you finally get around to switching to Python 3. And it even works now. It didn't then. To be fair, this is a horrible little thing
01:01:40
to do to any statistics module. In this case, we're calculating the mean, and it doesn't work well with very large numbers. And it works with most of the things we're going to want to use. This is a test using NumPy. This one's not actually a bug. It's just floating point numbers being awful again.
01:02:03
And so if you're doing scientific Python or anything with NumPy, it will slot right into your testing. I'm sure you're all testing your scientific code. It works with Django too. It works really well with Django. It will automatically take a look at your modules and just go, oh, you need these types.
01:02:21
I know how to generate those. Here, let me generate them all for you from scratch. In this example, we're overriding a bunch of things, but you don't have to do that. The out-of-the-box generation is pretty good. This one won't make sense without the full example, but basically, in this case, it tries to add the same user to a project twice,
01:02:42
and the code didn't expect that. This is one I don't have time to go into properly, but one of the really cool hypothesis features is that you can give it a complete model of your API and tell it what should work, and it generates random programs against your API and eventually finds one that breaks. This one's a bit experimental
01:03:01
and slightly harder to use than the rest, but it's pretty cool when it works for you. That's more or less all I'm gonna say. I hope you check it out. Go to the website is hypothesis.works, and it's got a whole bunch of introductory articles. It really is quite straightforward to get started, and we've got a good community and who, if I'm not around,
01:03:23
they'll be happy to answer your questions as well. Thanks again. Thank you very much, everyone. That is us finished actually properly on time. Thanks to each and every one of you for coming to the Lightning Talks, for proposing Lightning Talks, for being speakers,
01:03:40
for paying the registration fees, for being here for free, for going out in the evening, for having dinner, for being lunch. See you all tomorrow. Let's have another several days of conferencing. Everything's gonna be brilliant. Hooray, good night! Oh, and I'll post the link to that squirrel story, which I was about 20% of the way through. I think that's maybe better read on its own time.