A faster Python? You Have These Choices

Video in TIB AV-Portal: A faster Python? You Have These Choices

Formal Metadata

A faster Python? You Have These Choices
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
A faster Python? You Have These Choices [EuroPython 2017 - Talk - 2017-07-13 - Arengo] [Rimini, Italy] Python was never intended as a fast language but many modern uses of Python require high performance computing, particularly in data science. This talk explores your options for squeezing maximum performance out of critical Python code. This talk provides a succinct summary of the options you have: C extensions, Cython, CFFI, PyPy and many others. It also shows the trade-offs between execution performance and the cost of writing and maintaining code with each choice. Each option is also explored for maturity and ease of use for Python programmers. A real world programming problem is coded and benchmarked using each of these techniques. All the code used in the talk is available on GitHub. At the end of this talk you will be better place to decide on which technique to use to make your code run 100x faster
Point (geometry) Metropolitan area network Focus (optics) Algorithm Octahedron Point (geometry) Sheaf (mathematics) Bit Data management Term (mathematics) Natural number Self-organization Set theory
Axiom of choice Presentation of a group Greatest element Hoax Run time (program lifecycle phase) Code Multiplication sign Real-time operating system Formal language Medical imaging Array data structure Benchmark Computer configuration Different (Kate Ryan album) Semiconductor memory Electronic visual display Aerodynamics Endliche Modelltheorie Boss Corporation Algorithm Concurrency (computer science) Reflection (mathematics) Electronic mailing list Bit Benchmark Virtual machine Root Exterior algebra Process (computing) Computer science Right angle Ranking Data structure Writing Point (geometry) Slide rule Dataflow Algorithm Token ring Virtual machine Reflection (mathematics) Parallel computing Computer Number Revision control Moore's law Performance appraisal Profil (magazine) Operator (mathematics) Data structure Computing platform Mathematical optimization Numerical taxonomy Computer architecture Pairwise comparison Information Interactive television Line (geometry) Domain-specific language Software maintenance Computer programming Performance appraisal Numerical taxonomy Personal digital assistant Pauli exclusion principle Object (grammar) Mathematical optimization Abstraction
Point (geometry) Slide rule Freeware Overhead (computing) Perfect group Code Multiplication sign Connectivity (graph theory) Mereology Disk read-and-write head Graph coloring Software maintenance Formal language Product (business) Software bug Revision control Programmer (hardware) Mathematics Different (Kate Ryan album) Library (computing) Numerical taxonomy Overhead (computing) Standard deviation Axiom of choice Projective plane Physical law Electronic mailing list Code Maxima and minima Line (geometry) Software maintenance Benchmark Formal language Frame problem Mathematics Type theory Arithmetic mean Personal digital assistant Quotient Revision control Video game Right angle Quicksort Perfect group Library (computing)
Type theory Information Code Multiplication sign Moment (mathematics) Set theory Line (geometry) System call
Slide rule Functional (mathematics) Just-in-Time-Compiler Code Multiplication sign Typinferenz 1 (number) Translation (relic) Compiler Number Mathematics Root Computer configuration Operator (mathematics) Square number Soliton Mathematical optimization Form (programming) Area Beta function Standard deviation Interface (computing) Projective plane Moment (mathematics) Physical law Bound state Shared memory Code Drop (liquid) Software maintenance Benchmark Formal language Performance appraisal Mathematics Type theory Numerical taxonomy output Self-organization Right angle Lipschitz-Stetigkeit Quicksort Library (computing) Row (database)
Continuum hypothesis Functional (mathematics) Just-in-Time-Compiler 1 (number) Sound effect Compiler Mereology Shape (magazine) Number Compiler Subset Pattern language Compilation album
Classical physics Point (geometry) Functional (mathematics) Game controller Code 1 (number) Sheaf (mathematics) Focus (optics) Graph coloring Subset Number Formal language Mixture model Goodness of fit Mathematics Type theory Extension (kinesiology) Computer-assisted translation Library (computing) Form (programming) Social class Beta function Projective plane Moment (mathematics) Code Basis <Mathematik> Line (geometry) Control flow Formal language Frame problem Hand fan Concordance (publishing) Type theory Film editing Exterior algebra Function (mathematics) Interface (computing) Summierbarkeit Right angle Writing Library (computing) Extension (kinesiology)
Module (mathematics) Link (knot theory) Moment (mathematics) Projective plane MIDI Memory management Menu (computing) Bit Line (geometry) Rule of inference Exterior algebra Resource allocation Hooking Read-only memory Intrusion detection system Software testing Extension (kinesiology) Writing Extension (kinesiology)
Code Interface (computing) Projective plane Boilerplate (text) Electronic mailing list Code Bit Line (geometry) Formal language Formal language Number Hooking Extension (kinesiology) Abstraction Physical system Extension (kinesiology)
Purchasing Email Slide rule Functional (mathematics) Greatest element State diagram Strut Graph coloring Formal language Template (C++) Revision control Different (Kate Ryan album) String (computer science) Energy level Extension (kinesiology) Macro (computer science) Library (computing) Module (mathematics) Projective plane Line (geometry) Formal language Similarity (geometry) Angle Right angle Library (computing) Extension (kinesiology)
Module (mathematics) Code 1 (number) Code Flow separation Graph coloring Formal language Formal language Interface (computing) Right angle Software testing Extension (kinesiology) Extension (kinesiology)
Axiom of choice Area Axiom of choice Data storage device Sheaf (mathematics) Code Graph coloring Formal language Performance appraisal Numerical taxonomy Performance appraisal Numerical taxonomy Spacetime
Axiom of choice Standard deviation Statistics Group action Functional (mathematics) Code Observational error Constraint (mathematics) Decision theory Multiplication sign Mereology Event horizon Number Subset Product (business) Twitter Measurement Revision control Facebook Goodness of fit Performance appraisal Benchmark Bit rate Arithmetic mean Computer configuration Average Software testing Library (computing) Standard deviation Constraint (mathematics) Structural load Projective plane Moment (mathematics) Core dump Staff (military) Statistics Benchmark System call Error message Revision control Self-organization output Pattern language Resultant Library (computing)
Point (geometry) Greatest element Implementation Computer file Code Outlier Range (statistics) Tube (container) Student's t-test Number Product (business) Benchmark Bit rate Average Different (Kate Ryan album) File system Cuboid Software testing Maize Dot product Graph (mathematics) Information Sound effect Cartesian coordinate system Benchmark Category of being Subject indexing Geometric mean output Software testing Right angle Game theory Table (information) Reading (process) Library (computing) Geometry
Information Structural load Range (statistics) Bit Benchmark Twitter Number Performance appraisal Performance appraisal Benchmark Universal product code Read-only memory Semiconductor memory Software testing
Module (mathematics) Code Software developer Multiplication sign Projective plane Electronic program guide Usability System call Software maintenance Product (business) Formal language Revision control Goodness of fit Operator (mathematics) Revision control Determinant Proof theory
Axiom of choice Axiom of choice Projective plane Sheaf (mathematics) 1 (number) Plastikkarte Benchmark Product (business) Product (business) Heegaard splitting Numerical taxonomy Performance appraisal Benchmark Self-organization Proof theory Self-organization
Axiom of choice Benchmark Computer program Code Video game Mathematical optimization Abstraction Product (business) Form (programming) Self-organization Measurement
Group action Length Code Outlier Multiplication sign Decision theory View (database) Source code Mereology Mathematics Hypermedia Different (Kate Ryan album) File system Series (mathematics) Endliche Modelltheorie Block (periodic table) Keyboard shortcut Moment (mathematics) Bit Maxima and minima Variable (mathematics) Flow separation Cognition Category of being Data mining Arithmetic mean Website Heuristic Quicksort Metric system Reading (process) Writing Row (database) Point (geometry) Purchasing Computer file Online help Number Revision control Authorization Address space Standard deviation Graph (mathematics) Information Projective plane System call Compiler Subject indexing Word Hybrid computer Geometric mean
minus for also if HL which is a systematic hedge fund in London is a little bit about as we've been around 30 years this year which is quite long different terms we um trade in about where systematic sets the algorithms to do the trades not discretionary traders were active about formal markets around the world that we gather about 3 billion market data points every day and we've focus or text all you can find it on the nature of the public on Oct it's been very successful 1st have across small organization 34 what we do but we all speak Python so this talk is really based on our experience in using Python as a systematic hedge fund and the things that we've had to do when performance isn't what we would like to score 3 sections this told what is just
to introduce the concepts and the scope out tokens performance is such a big subject to economy will come in 45 minutes and then there's another technical solutions out there you can have to make Python fast your code run faster and it's somewhat confusing the hero different solutions I tried a critical technology taxonomy really had to make sense of all these different options you have and so you can reason about intelligently and finally we'll talk about evaluation criteria how you evaluate the different solutions are on on display here so 1st of real interaction so firstly what this talk is is really collected tore through some other Python all transistor Python this is a general purpose computing became more specialized computing and so all the solutions are you so general-purpose solutions to compute power the so Segovia way of evaluating alternatives so you can decide which 1 is good for you and is also a reflection on the trade-offs you inevitably going to make between things but performance and maintainability and cost so the so but as they what this is not about can I will mention on private this is not about number and and so if your problem domain is 1 that can be solved by vectorized operations on fixed-size arrays telephone number on pandas because that's where you're going to get your best performance and if your problem is solved by concurrency well then you know what to do and on talk about that also the caching well belong to make any definite recommendations to and I has some advice that are common recommendations you because I don't know what your particular problem is an all problems have different solutions but this is not a real comparison of a benchmark comparison of the different solutions solar benchmarks to see this presentation a fake that is to say that not real benchmarks the case so much for flaky wise talking exists in what we have to talk
about improving the performance and this is perhaps why Python is how it is so the reason why Python is a legislature will perceive this like is is of the annotated language every line has to be valid so every time we directly into a plastic valve same on time is dynamically typed as well so the real time assessed each time uh noted that access Israeli version machine which is a great way of abstracting away from the platform but abstraction comes at a cost runtime cost of Python that image it yet woman Python 3 6 1 of some this fighting 3 which opens the door to possibly putting Egerton and we'll talk about 2 solutions as well and and Python itself takes very little optimization opportunities with the code compared with say aggressive seeking + + compilers Mr. to reflect that the 1st 3 but this list is 1 of the reasons why Python slow but they're also the reason what Python is so productive white so quick to write Python kind what Python is a valuable and the 2nd to also perhaps why Python slide but then in be facing the hard problems to solve in computer science so if this hasn't and you're ready to have to be many times your boss comes to you and says this Python criterion is far too slow Unified way speeding it up so fast enough while they goddess of number questions is really is actually what I saying isn't it possible that Python is fast enough for the job at hand so we're actually billions of dollars each day we can do that Python because the way we tried Python is fast enough and then we can take advantage of all the other aspects of Python the making very quick and cheap to develop new ideas which is more important to us than the speed of of operation and if you are like you get faster not going from making faster overall you profile in the course of profiling is a the dark art profiles of online and those rated talk earlier in the year in the tutorial earlier in the week of profiling had a lot of information of it you don't have to do that but you get your profiling skills outside which bits and the whole point of doing that is so the spend the right amount of money in the right place because ranking there's we do things cheaply OK then have a look because the algorithms a well-formed passed by using our algorithms while using right data structures give often overlook that and I was involved in a problem recently where people make building very large lists of Python objects and adding to the list of course I felt appreciate that the list has to be contiguous in memory when you start adding stuff to it becomes a point where it has to be copied the new memory that found for the whole continues this this is very expensive operation and some architectures like a linked is made much more efficient just changing that the destruction made a big difference in runtime but I put up this famous quote to the bottom of the K to remember that when you change your code and have happened since you memory so some of the introduction of the flow talk about the options eHealth technologies exist out there for speeding up Python code and how it can color caste crises to make more sense because was a huge model choice and that there is
some of the and the projects that are aimed particularly at making Python code or the code to sort of Python run much faster and I'm sorry if your favorite project is not that that there is the so you can to it latest if you make but when you're considering any of these things in my we were starting or what would the perfect solution be what we really really like as a solution to making Python the faster and so will his Michael wish list
here this is perfection so they can run like I think I directly out enough change capital cases no effort on 1 because in court lazy programmer 1 change occurred so the perfection would run like a directly uh to be no maintenance overhead my half a half and it worked for Python versions all library code on the Standard Library also party candidate was using my and my and some of that it be free of course fully supported uh when and bugs in it but a perfect debug story uh NAB 100 times faster in later so not being greedy with a shopping list I could a thousand times faster but lingustic 100 components of so this is what I wish for so the thing is this doesn't exist OK this solutions out there that exist parts of this list so this is where you can make the trade off if you gonna go from maximum performance Republican incurs some cost we'll debug story Michael a hell or something of that says worth reflecting on what would be perfect when you really make those trade-offs and as blindly choosing 1 solution your implicitly making this trade if you don't realize of you might get in trouble later so the color taxonomy of chosen is based about how much code you have to change them before using haptic technology but what into a pretty little Nico change uh some small Monaco change already rewriting your covered in the different language yeah so this of the law exists for the first one little make a change and this flight project on the top head of Python itself and is aligned benchmark of a tall typically these projects will give you 1 tolerance tight times speedup varies enormously depending on the problem the 1st Python the most solid thing which is very well projects the I am I gonna right siphoned not optimize code now show you what I mean there at the middle pipelines racing type I talked yesterday and I think assailant tomorrow it should look at those frames in solution and the other notable products quotients can apply some look at briefly the case of 1st or even an I write some code here given site and see what happens is code computes a standard deviation of a array flights so I put siphon as on right in Python slide I put in siphon expects life work magic and sigh thing goes about 1 . 3 times faster which is really no performance improvement at all so Python is rubbish is in a sorry sight is rather is not solution told was going CCY siphon struggles to this point is actually goes we're helping siphon the toll free just take out 1 line here uh where computing the mean and we look at what siphoned does away sigh from what is it's a takes your Python code I generates the code compiles and all the guys and if you look at the C code for that 1
line here is the siphon uh C code uh his line we're talking about and if we dig down through this rather complicated can we see that this line some divided by lenders expressed by these 3 calls
into the sea Python API these calls are highly generalized to hone in on specific not specific any time and the reason why so I think all make an improvement on what the improvement on all code is because we haven't inside type information will revisit Fox siphoned in a moment to see what happens when we can fix that because that involves changing our code called the the next 1
is pipeline so to believe that benchmarks all R A 7 times of surrogacy Python uh implantation is a just-in-time compiler is funny drop-in replacement for 2 7 3 5 code the but it doesn't really have a C Python API but it supports see confirmed function interface which I'll mention moment uh so that's a way into seems suppose plus that is not completely compatible with certain libraries uh I think in Fhloston pillar work up a lot it's a common age but is a it's incredible on the slide here that should make the pause although it does have the work probably in it so I guess what I say is I change up to something like we should certainly try this because it has been very successful in some areas consortia pipeline and so skin is a project he does automatic type inference in translator Python code to C code it's quite elderly it only supports up to point 2 6 uh and there's been very little activity for the last past some underdeveloped delve more into that is also 1 piston uh which is LLP in place to compile and this is this backed by Dropbox or should say was spectral books is a python 2 7 and the project was suspended in January this year and offer these last 2 ones just as an example of the problems you have when you choose of the technology to go for performance your making quite committed move to that technology and if that project is runs in the salmon stocks going maintain the news of the goblet the technical debt this is going input beamformer me look at the evaluation criteria for which the project to use for your code does this so much a little code change more hands we accept the we some code change we can make some countries recordings projects of what options you have that and well here's some and now we get speeding up of 2 so 10 to 100 and that can arrange a sort of siphon optimized the number parakeet Python on through them so siphon optimized basically that you do this kind of
thing on the left is the standard deviation code which is what the Python code largest law so if and then get anywhere thought on the right is much more heavily optimized soliton code and I and here on the functional polymers I've actually declared entities the num pi arrays so important employ active siphon hollered opportunities that on declaring the local types as the size T and so on that and putting a decorator missing bank check the bounds of this array which is also a cost a rather tall on importing from the lips C math library and under the square root from there rather than the Python Lassparri which is problem whatsoever so if I do this in size and I get it running 62 times faster which is like a really good improvements from just improvement the consider this how maintainable is a code on the right this is a kind of left and we're talking about a really simple operation here square root on right so this is 1 about trade-offs is a if you really want sixty-twofold performance improvement implicitly you gonna put up with a maintenance problem and and Bobby your code is much more complicated and standard deviation so siphon optimize so I think can rapidly become failure will be because this is really weird hybrid code it's not C or C + + is not Python it's is in between thing and is quite a lot in tuning siphon together make some forms of it has been a black art and often it's and share of in organizations
so quantum number this is backed by Continuum Analytics which is always good to have you back up and it's a j compiler it works is pretty much a a Python num pi and basically you just annotate your uh functions we refer to this brings a whole technology and you run it and uh the uh jet will get to work and try to optimize that based on what it seen in the past the number the sagitta compilers and 1 the early ones
parakeets still around same kind idea you decorate your code and and then it'll it'll it'll trijet freight uh parakeet is quite old underscores pattern 2 7 it's Ramachandran known highest part effect that but has been little activity of the last 4 years 4 years ago a UFO parakeet was the best thing and you start writing on a and sorry you know so the of Python
is the other 1 laser a slightly different technique they base the annotate the function were common and you can see the types and the commoner and then you base want to play fan generates C + + code and then you run that it really aims at 7 scientific computing its color subset of Python that it interprets but it can produce really powerful so high performance at some cost but so that's
so some code change section uh the alternative filter interview 1 really hard forms the basis right in a different language they like C + + of some of that and so what kind of projects out there that do that and this is where you really get this color 100 X performance improvement typically even better maybe here the sum of C C + + base forms uh we have the original C Python C extensions before talk about the moment this first one is that it's too many for me to go through them but I'm gonna cut it out of it and is also rusting fulfillment out in a thing if that's your thing actually go for the K I want to talk about any of those but let us look at some of the frame of the C C + + based ones which is right a c extension she Afify and apply . 11 they will take different approaches to basis same problem is how do you get into the writing and C so firstly the classic C extension the now and these give you a mixture of joy and agony when you are involved in C extensions so what's a joint while it's written in C see very easy 32 keywords and see so what could possibly go wrong you can make him a C + + that whole works nicely you have really precise control over your performance to this point and there's a lot a good number of libraries shall we use a lot because C extensions concordant and employ a dresser during see annex inquiry efficient fast and if you're writing for the standard library you have to be here because striking often 1 Australian often library requires you to write in C in fact see 89 there was some little extensions to 98 a things on 2 thirds thousand 17 we're stuck in CAT 9 which is French me of my bi-gram anyway so as a joy riding the extensions or what's the agony well this is a little class problem 1st class you write in Python with something like this this is taken from the Python documentation from the tutorial about how you write your 1st extension and base you make kill class that contains the 1st name last name and has a method we can pull out combined names that's in place and what's a lot when you want to see extension well it's this is are remind lines of C code addressing a very complex and sophisticated and well documented API can and um often has a low lines occurred where is the real agony and this while he is the real agony
it's in C and you have to do the reference counts they had run that gifted money memory management yet understand how Python does is memory management is a very specialized skill lighting in the C API and it's quite expensive to write government lines with is a handful of testing is rule problematic with Python modules are returned to that in a moment some debugging a c extension is kind of a real blackout as well um GDP what's following if your hook up a might be if you like the body like the uh that can be done is a bit tricky to set up but just to show how it can be done so here's a
screenshot debugging UCI Extension next elect that is a link that 1 of my projects that she had to do that and so if we don't like being C extensions of we find that 40 painful OK what alternatives so here are different
languages so let's move away from Python C extensions number to see if FFI which is a reducing project and there was a a told yesterday about it that was very good I give you a list of the talks I thought were valuable here and electron SocietĂ  if I you writing
a Python it allows you to call that C carried out from within Python and its sea-based because hook up to c plus plus a little bit of work the abstract away much of this boilerplate as 190 lines of C extension which is basically the interface code and it also abstracts where the bill system so if you remember this is what we're trying to do this is what we have to do in C extensions this would be a sort of
fairly crude CFR if I equivalent importing Afify like right CDF with some string in it which is that you seek out that gets compiled up like Creighton you want I can assign those and the 1st and last names to my own name like an extract about in some fashion that and that carried on well fits someone 1 slide and it just works and it's been executed in sea land so a different approach is pi bond 11 which I think is also fast a project things of talked more about pi bond level from different and angles see satisfy 1 language uh Azadari C + + library so you're writing C + + and it makes a very easy right the extensions is color similar to boost points and if any of you use them and they're wonderful it's a professor Levinson's column modern version of C + + kind easy to write purchase the suppose so here we go here's just 1 you this is a cluster from a create this is what we have to in this extension this is happily
behind works in C + + Ukraine struct with it has constructed it has a single method namely concatenated names it as a 1st and last names as members and then you polar you include pipelined 11 and I gives access to a whole lot of functionality templates and macros that basically the creating of the module is displaced 4 lines at the bottom the very same project OK and then you've got your the shared library that you can run version for that and d and so that's all I gotta say about
those things are other C and C + + 1 so choose the ones that you your shot but to say that if you're working in a 2nd and in a separate language from C + + try and arrange your code to look color like this
so on the right is audio C + + code for example that doesn't include Python dij as my interest something else that is pure C + + code if you like on the left is perhaps of Python code or axis the K and they always have a lot to do carry this let's see extension code be the pi 11 code would be the CfA phi K and the promise of but in the middle is the 1 that's really hard to test so she can physically and logically separated code left and the right as far as possible you can then test them independently for correctness and performance performance what was is before after if you put too much code in the middle it affected comes untestable so I'd recommend some physical and logical layout that is something like this
that so that's basically the taxonomy which was the 2nd so section of the store and now I wanna talk about evaluation criteria that you might use to judge which of these solutions is suitable for you so again here's a choice you make so
how do you work that we might be able to divide these out now with this taxonomy and understand which is the 1 that would give us the right trade-offs about how do we evaluate even within that taxonomy which is better so I suggest that there are 3 color things areas you need to consider when evaluating these which is this space he who you are
and what company you work for organizations in what he in what skills you have some about and as a technical criteria I will probably mostly engineers here only like technical criteria cause put numbers to them the numbers are our lifeblood uh but it's also technical criteria as well but argue that are equally important when you make a choice die so 1st of all the well thank guessing if they here on this you know Facebook or Google Microsoft really that so these companies tolerancing projects that publish white papers and heavens so staff in the same way using this particular project and it's really fantastic us and some about the temptation is because they're using it the you should use it but you're not than k they have their own problems which may not be your problems they have their own culture and a constrained by that you have your own continual constraint but that may not be constrained by that what they did so I understand what they're trying to do what you're trying to a different site than be to overly impressed us because when the big companies doing it and that's what you should do land because it's the companies back something that is also a plus for project because it's likely to increase its longevity and quality so consider also what skills you have to you and and maintain this kind of all what skills you can get all what skills you might lose as an organization because if you've ever gone down the rate for example of a highly optimized Cython and you go a couple people got really skillful at doing that and your whole code is running re Snavely conveys to people leave then these kind of knock on a holiday would the so the silicon solar technical criteria with facet other this stuff engineers and what we might macaque OK these events of input numbers 2 and compare 2 different options against each other so what might that be technical criteria we might look at all these projects depend on to that kind of things that we don't really wanna depend on what different versions of we don't use we can put numbers to that what versions of Python 2 they support this we see some these projects unusable older versions of Python someone struggle to move make the transition to Python 3 so many Sopot support part of the call the place and all was part of the standard library and some parts the third-party libraries you might depend on support so you can go and find out and comparative and his very easily and a we get benchmarks because people are obsessed with benchmarks of parts of obsessed that benchmarks carry a lot more white and always think they should they deserve but mainly because there wrong and as I said will eventually to see here lies to start real benchmarks so 1 become the obstacles to benchmarking accurately uh to make rational decisions about to what various options you have well this simple measurement errors this is measuring wrong thing measuring the wrong the bad statistics and I went the go through benchmarking loads of good material out there about how to bid for public baths is this is an interesting 1 because in benchmarking you gotta take a small subset results and then you got a reason about the wider world with statistics based on that small set and if you do bad statistics will liquor coverage the moment time then you're going to see yourself have been talking and then actually human cognitive biases uh confirmation bias you might be overly formed 1 library and therefore you might have implicitly be starting to uh look for posted benchmarks Netlibrary we might get fixated on 1 small part of the problem ignoring the wider aspects of problem so these all obstacles to objective and so this is an entropic falls on the
mentioning here but we're running a test on 8 times we got library see legali the dealer comparing them and but also uh we did taking the average of the timings of them and they've been very good statisticians overtaking the standard deviation and then the whole they saying so that fight library C library D all of them that exactly the same but if you don't like the the has seen pattern to it the 1st time you run the test it takes 18 the subsequent times a very consistent aid this is maybe characteristic of some college it kind of behavior and so I'd say that if you're running uh this in production of many times that lie BB is deftly about of choice if you're only of operate that function maybe once and for the worst choice so you gotta look at the trends as well as the raw data From there is another common
fallacy is i got number tested independent tests of 2 libraries GNH of combining and take the average and the average looks like the the side OK if you notice a testing widely different towlines um and is actually quite misleading take the average if I add another column on it right divided h by g you see that in the 1st test H take slightly longer proportionately but all the other test is twice as fast as a as this J. so how come the averages so misleading saying the same is because what you need to do when combining widely different numbers is take the geometric average not them 1st every geometric averages of product will numbers divided by the and 3 now we see the geometry average library H is much faster than G effect tables a really bad way tubes and benchmarks as well because it is numbers is a much better way presenting a bench this is a real
world 1 and not from a company I currently work for and the basically this is the graph along the y axis on the x axis is the growth rate is a false or is going up to 100 megabytes what we're doing here is this is a sequential file but we want to randomly access it and so we're creating indexed by read seeking through a to find interesting points and then we can randomly access it much faster than the sequential file that the red box the original place in kind the talk also the Y. axis is how many milliseconds per megabyte file it takes to create the index that we want to pay the red dots are the racial Python implementation the green dots are writing it in C as a wealth of data in this file for 1 thing you see is a big range of inputs a file sizes the testing of from very small to very large this about t decade improvement general on the C code the fact that it flattens out means it's I when and this where missing stuff happening with some outliers up the top and reflected in the bottom if you look at the bottom right hand side the some some files remarkably taking much much faster than others even with the Python code and as reflected in the C code as well when we investigated those files we found another technique because they have a particular property the weakest point for all follows the games other 10 x improvements so presenting data like this gives you a whole wealth of information you can make reasoned decisions and compare that with the table how achieved
good how much information going to get a laugh the
OK if you benchmarking to speed is glycolysis stuff you should do because memory might be more important Ohio or the trends of benchmarks and what's eventual due to put interaction and forget about it where you what we've been talking a production code as well the
so the last thing evaluation of love and technical criteria so these things he called put numbers on caring and sometimes they show a sharp away from this because you complement numeric value on it is a bit Hodges solution what argues equal is important so here's some non technical
criteria polymer consider these installation deployment if it's a large deployed again dependencies is of writing the code and maintaining it was so debug story module 2 story for analyzing your code all reporting things out of production and is the project you chosen be a pipeline of siphon whatever it is is a how future-proof is and that's a reader for judgment call because of course the past is a guide for future that
so we got so how can you try and predict whether a project as we see several projects just some the halted so uh let's take the your solution doesn't do that leaving you with it and before that that these are the kind of things you want look at probably is what Python version supported moving operations in a timely fashion what his development status and if it's all of that may be a good thing because it's been around for a long time this material on the other hand it might be using very old technology and your project might use more the you a language features and be better than that is it maintains the they have good instant backers like a large company or determine company a fixes quick so they take the hours that kind of thing so he's using it
and uh or in anything is a consultancy around because of his money to be made it his money splitting around a project is more likely it'll lost the 1 that's just been given out for free was taking for free that's so
those are the kind of 3 sections so I have here sought just finish up by just making a summary this is what I can say that it is
the takeaways who is just choose use least solutions chooses solutions as Pope of your organization your skill set your product is your choice just recognize you gotta be making trade-offs and if you don't realize that the implicit tradeoffs come back and bite you Citron be explicit about what we regard as important the benchmark if you must uh bench what wisely when you can not argue these non-technical criteria require for the longevity of your card and then the technical ones but just as a shout out to switch shot out here for this book I have
nothing to do with this but it's all except on I when the also slightly and um but idea it's high-performance Python I got Carlisle out of it and I recommend if your they comes a lot more territory than covering this for and but if you're instant high false Python this is that we want to go and get and also just don't listen to 1 person the me mean consider other
opinions and his summit talks here in Europe life and Monday and wednesday we had so these torso into I go out from that Cfa profiling siphon so that uh tomorrow we got to talk some difficult o 2 1 and see + + 11 you 1 on pipeline uh so listen to lot opinions when it comes to a forms not just 1 and that's about it and all answers your questions as best I can do this is if you want to
restore me on social media this is about as far as again social media which is get help also HO close to somebody can produce word you go pick up and you may want posters for happy if you have a look at all to the work to the random that's it for me so pools your questions was the stricken the things that few questions it the it hoped the wrong information used in testing the use of the difficulty in contributing government I have how do you go around confirmation bias I suspect humans have been struggling with this problem for a room of millennia really but I don't know I think it's on I mean I think when you want wanna presented material to persuade other people but you have to come and make that material falsifiable there must be enough information in there so that someone can form a contrary opinion so if you just say this is fast you're not giving anyone any purchase to say that that they can't deny you because you're not given information if you say I believe this is faster and his data for FIL giving us some sense of falsifiability because they can make that they say all you taking the athletic mean of geometric mean and therefore you're drawing a wrong conclusion so summaries cognitive biases can be helped by presenting your information to a large number of people and they might be other sports something that you can't spot the soft it's a difficult To avoid confirmation bias yourself but it pumps if you about about it you ask yourself the question that may help as well yeah it the he thinks the top of the 2 main technologies for mining namely by live inside of what is your view regarding advantages and disadvantages of it can often OK I'm while I use I think on a lot time the body from where projects and work we depend on a quite well it is for it is 1 of the most mature um the project out there it's used all over the place and you I know that standard example be pandas heavily depends on siphon Anderson about so it's gotta be a stop and the the pipeline 11 is much hold newer project have but I think it's very interesting because perhaps at some about addresses some of the issues that I've come across some time Thomas siphon severe like debugging Insight and is really quite a and it can be quite a challenge uh the fact that siphoned optimized code is in this strange so hybrid form I understand why that be so but it also makes a clear hard to read and reason about pipeline apart from the bindings European Workshop writing in C + + which should be easier to reason about that so I think it really depends on who you are what your capabilities are likely if your strong C + + shop for example than pi bond 11 would be a very interesting and read too tight but almost siphoned off 2nd block you can get into into 2 classifiers I'd say performancewise is probably not a big deal between them but I have noticed sometimes the source and generated code that and you can make small changes in the p y x siphon file and you get radically different changes in performance which is not necessarily predictable and is just the way they use a lot heuristics to generate a C code in some changes and so from noticeable small between versions as well something that by if find projects the upon becomes down to this point know much choose between for performance and that it very much depends on what kind of choppy you already and and what the personal preference of the be wrong to choose either 1 going back to the graph on the following look at the data graph of formants you mentioned that you identified something there have been accident and holds the increased or at least native might as well but the news common our kind of going down the whole organization and you know as is of new intelligent beings and maybe see probably think of you know this is a momentous anything you have any tips for how we can avoid you know going all where immediate annex improvement but if you need a part of the this is own within this 300 data points of each of these can it turned out was something like 8 so sigh data points into the outliers which occur at 1st sight might think just statistically uh not significant but we dig and actionable Academy from initiating so this is based sequential file where a series of records of written 1 of the other and to read to any point in the folly of to read everything up to because of variable length records OK and so you can't is going someplace and the reason the connecting indexes the index the start of the she's verbally freckles why it turned out that most of them the variable-length records have a maximum size of 1 k 2 lots of small records in and it turned out the outliers had no record size of up to 64 K so that much furor consecrating the index of being much more seeking much less reading concept read ahead read from so that accounts really for the top 10 x improvements what we realize we can actually rewrite all of files from 1 carrycots 64 K records and that was how we got the extra 10 x improvement but without it being presented in a graph like this without call the curiosity to say who those outliers might be interesting and I'm not to dismiss and we would have found that 10 improvement so I guess we could on a rabbit hole because you couldn't just be statistically significant but we did start off by doing a little bit more statistics and few more runs and that kind of thing and discovering these are actually real allies rather than to statistical moments and I rabbit holes is something that I seem to Mary dive into I'm afraid so Proxmire vise of backing on a rabbit hole is more 1 of experience of a good decision making more questions is the a and thank you at the it nobody everything with their articles performance yeah have using who's playing yeah and it has to have a sites which are of the compulsory called time and it was lot of these very people to false so he's pipeline 11 improving on on these metrics on the other hand probably a bit outside my ideas and expertise here so this is probably some speculation um that I have used this path and for car tall and I really model this project it's getting on quite a it now and it is common got all sorts of um uh stuff that probably reflects history rather than in modernity and pi 11 but from what I understand from the authors was really a was inspired by these points but it's a complete code of refresh starting uh writing a fresh the whole thing and in C + + 11 which would make me think that given a modern compiler you would get less variability with pi 11 and we've talked big so it might be worth I said recommend you gotta look a pipeline 11 and see if you get the same kind of problem but I'm not going to predict the would my guess is that you wouldn't pumps get that problem the problem 11 but i'm sense speculative at that but I'd recommend you can try OK so means the with all the persons who above all in the end that few