Can Rust make Python shine?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 43 | |
| Number of Parts | 173 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/20102 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Bilbao, Euskadi, Spain |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201543 / 173
5
6
7
9
21
27
30
32
36
37
41
43
44
45
47
51
52
54
55
58
63
66
67
68
69
72
74
75
77
79
82
89
92
93
96
97
98
99
101
104
108
111
112
119
121
122
123
131
134
137
138
139
150
160
165
167
173
00:00
Integrated development environmentJava appletElectric generatorProgramming languageLogical constantSoftware developerIntegrated development environmentType theoryScaling (geometry)Active contour modelJava appletProcess (computing)Virtual machineCodeLecture/ConferenceComputer animation
00:51
Programming languageBitCASE <Informatik>Profil (magazine)Moment (mathematics)Lecture/Conference
01:39
Hoare logicData typeMeta elementPattern languageAdditionIcosahedronSign (mathematics)Divisor (algebraic geometry)Natural numberPrime idealImplementationRevision controlStandard deviationCartesian coordinate systemControl flowMultiplication signProjective planeMultiplicationImplementationLibrary (computing)Thread (computing)Cartier-DivisorProfil (magazine)Pattern languageNumberServer (computing)Software developerPrime numberTypinferenzYouTubeAlgorithm3 (number)Product (business)Point (geometry)Online helpWeb browserProgramming languageSet (mathematics)Observational studyVolume (thermodynamics)Event horizonMoment <Mathematik>Spezielle orthogonale GruppeAbstractionComputer animation
04:53
ImplementationGrand Unified TheoryInclusion mapMetropolitan area networkElectronic data interchangeComputer fileWindowCodeArmRevision controlVideo game consoleEvent horizonLogarithmArc (geometry)Chi-squared distributionRemote Access ServicePC CardProgramming languageRight angleComputer programmingAlgorithmSet (mathematics)Condition numberTask (computing)AreaEndliche ModelltheorieGame theoryRange (statistics)NumberSemiconductor memoryMultiplication signResultantSpeech synthesisLengthFunction (mathematics)RandomizationData structurePrime idealProgrammer (hardware)MultiplicationTheoryPointer (computer programming)Java appletArray data structureAdditionError messagePairwise comparisonBitFunctional programmingCountingArrow of timeParameter (computer programming)TupleLibrary (computing)Line (geometry)Standard deviationComputer animation
11:59
VarianceKernel (computing)LogarithmEvent horizonVideo game consoleWindowComputer fileCodeElectronic data interchangeMetropolitan area networkCloningOpen setInsertion lossComputer programmingData structureProgram slicingVector spacePointer (computer programming)Type theoryCompilerIntegerSlide ruleWorkstation <Musikinstrument>Point (geometry)Computer animation
14:03
Metropolitan area networkArmComputer fileCodeElectronic data interchangeWindowCountingCuboidDrop (liquid)Arc (geometry)Real numberHausdorff spacePC CardSinePort scannerFunction (mathematics)Interface (computing)Sample (statistics)String (computer science)Module (mathematics)Portable communications deviceOverhead (computing)Type theoryFrame problemTask (computing)Event horizonLoop (music)CompilerError messageCompilation albumPrime idealComputer programmingMultiplication signFunctional programmingLibrary (computing)Event horizonTelecommunicationSystem callStatisticsOverhead (computing)FrequencyFinite differenceRhombusString (computer science)Beat (acoustics)Pairwise comparisonInternetworkingLoop (music)Profil (magazine)Tracing (software)Residual (numerical analysis)Thread (computing)Database normalizationInterface (computing)Motion captureContext awarenessSemiconductor memoryCompilerPointer (computer programming)AreaAbstractionPhysical systemStress (mechanics)DampingEndliche ModelltheorieCartesian coordinate systemNeuroinformatikSoftware testingNatural languageFrame problemData conversionSet (mathematics)TupleLevel (video gaming)Standard deviationTask (computing)CodeImplementationReading (process)Object-oriented programmingComplex (psychology)Computer animation
19:45
Arc (geometry)FingerprintData typeEvent horizonExt functorEmulationMathematical singularityState of matterJava appletFrame problemMetropolitan area networkNewton's law of universal gravitationWechselseitiger AusschlussInformationSoftware testingData structureLine (geometry)CompilerNumberFlow separationSampling (statistics)Different (Kate Ryan album)Frame problemShared memoryObject-oriented programmingFunctional programmingThread (computing)Ocean currentIterationCondition numberComputer programmingStatisticsState of matterMoment (mathematics)Multiplication signPrimitive (album)Right angleComputer fileMotion captureRootModule (mathematics)Reading (process)Single-precision floating-point format
22:35
Total S.A.StatisticsGrand Unified TheoryGamma functionMetropolitan area networkTable (information)Value-added networkMaxima and minimaCodeSymbol tableOvalCommodore VIC-20Type theoryCodeRow (database)ExpressionFrame problemFunctional programmingPointer (computer programming)Block (periodic table)Object-oriented programmingMappingData structureAnalogyOvalPoint (geometry)Computer animation
24:07
Metropolitan area networkUniform boundedness principleLine (geometry)CodeSymbol tableCloningCodierung <Programmierung>Total S.A.String (computer science)Information systemsInterpreter (computing)Protein foldingCodeBlock (periodic table)WordData structureLibrary (computing)Data conversionMultiplication signDifferent (Kate Ryan album)Video gameLine (geometry)String (computer science)Point (geometry)CASE <Informatik>UnicodeTelecommunicationGoodness of fitLecture/Conference
25:58
String (computer science)Interpreter (computing)Mathematical singularityModule (mathematics)Metropolitan area networkString (computer science)Source codePhysical lawExecution unitAbstractionCASE <Informatik>Line (geometry)Representation (politics)Library (computing)Profil (magazine)Interpreter (computing)UnicodeCodeModule (mathematics)Macro (computer science)Computer animation
27:02
Metropolitan area networkBit rateModul <Datentyp>Computer fileCodeElectronic data interchangeWindowTotal S.A.CodeMultiplication signLine (geometry)Function (mathematics)Profil (magazine)Natural number3 (number)Video gameCartesian coordinate systemComputer animation
29:33
Link (knot theory)CodeCompilerDistribution (mathematics)Library (computing)DampingInstallation artCompilation albumExtension (kinesiology)Type theoryPhysical systemQuicksortComputer animationLecture/Conference
Transcript: English(auto-generated)
00:05
Okay, let's start. First, I briefly introduce myself. My name is Dmitry Trofimov, I work for JetBrains. I am team leader and developer of PyCharm IDE. But today, I won't be speaking about PyCharm in this talk. If you want to discuss anything about PyCharm,
00:21
come to JetBrains booth. PyCharm team will be there during all the conference, ready to answer all your questions. Don't be shy. Mostly, I develop in Java and Python. Java has been my primary development language for many years. And Python, for me, is more than a language that I use.
00:43
It's like a subject of constant investigation, like a snake that I examine, which type of scales it has, which teeth, and what is inside of it, how to debug it, how to profile it. And also, I'm curious about new programming languages. And half a year ago, I started to play with Rust a bit.
01:02
I knew nothing about it at that moment. Still, I don't know much about it now. But I hope that I will be able to introduce it to those who are not aware of it at all. By the way, who of you have heard about Rust? That's pretty much. How many of you have tried it already, developed some stuff?
01:21
Oh, that's cool. For those who have tried it, I hope it will be also interesting, because I will show some corner cases. Yes, so, what is known about Rust? Yes, and answering one more question. Why did I pick Rust? Why did I start to invest my time in learning it?
01:43
I found it interesting, and also, I wanted to develop profiler for Python and to make it work fast. So, okay, I will tell about it. Rust. Rust is Mozilla project, and they actually are using it already
02:02
for the new browser engine called Server. The project started in 2010 as a side project of Mozilla employee, Graydon Hoare. And the version one was released on May 15 this year. So now it is 1.1.
02:21
And before version one, things were changing, at very high pace in Rust, breaking compatibility. And now it's still changing. Standard library is not polished yet, and ecosystem around it is just starting to emerge. But now it has backward compatibility,
02:41
and this allows to develop production applications in Rust. So, what is Rust? What did I know about it when I started to learn it? What exactly captured my eye? They told that it is fast, prevents nearly all sick folds,
03:00
guarantees thread safety, close to the metal. It has zero cost abstractions, pattern matching, and type inference. And that sounded very cool. I thought it would be very interesting to learn it. First I started to listen some talks on YouTube. Nothing became clear from them. Then I found some specification. It turned out that it is already outdated.
03:22
The language had just changed. And then I found Rust book. For all of you who is interested, I recommend this book, The Starting Point. It's online and very well written, and I don't think that a talk can help you to teach the language. That's why today I won't explain basics of Rust at all.
03:42
I don't have a goal to teach you the language, but to give you a feeling of it. Okay, so let's start with a small but real problem. And as the speed is main advantage of Rust, let it be a computational problem, like computing primes. So the problem is to compute prime numbers between two and n.
04:01
And prime number is a number that has no divisors except itself and one. So like two, three, five, and so on. And we will solve our problem with the help of an algorithm called Repetensive. Algorithm is simple. The work is this way. We first take all the numbers from two to n.
04:21
And then iteratively throw away those who have divisors. So something like that, we start with take two. And then we throw away all events. And then we proceed to the number three. We take as a prime and throw all the multiples of three. And then it will be five. And all multiples of five are thrown away,
04:42
and seven, and so on. So here we see a Python implementation of Arabathian C. It's quite beautiful, isn't it? Here we initialize our non-primes as an empty set. And then we iterate through a range. And if the current number is not in the set,
05:01
we increment our counter and update all the multiples. We put it in the set as a non-primes. And then return it from the function. Okay, so let's run it. This is our function. And also we have here main function, which takes a common line argument as n,
05:22
and then executes our function and prints output. Okay, so it's something like, okay, so we have four between two and 10. And for 100, we have something that seems to be correct.
05:42
Okay, let's measure the speed, the time of this program. We will comment out the output, because it always slows down the execution. And okay, and for one million,
06:04
make something like this one. Yeah, it's more than half a second. It's pretty fast, but what if our task is, what if we do care about speed?
06:20
What if our task is to implement the algorithm as efficient as possible? One of the obvious solutions is to use C programming language, because everyone knows that C is very fast and it's efficient and very well, very good programs are written in C.
06:40
For example, C Python is written in C. So let's implement this algorithm in C. So the program now is a bit longer, but it's pretty much the same. So we have our defense function. Unfortunately, there is no set in the standard library
07:02
of C, so we use array of primes, and where all items mark as one, this means that it's prime, and when it's zero, it's not prime. And then we iterate through it, and if it's prime, we implement our counter, and then update all the multiples of zero,
07:24
and then we put our array into the result structure. And unfortunately, there is no tuples in C, so we need to have this result structure, which holds a counter and our array.
07:41
So in here, we execute it, and we print the counter, and then we want to print all the primes. We first put them into another array, which is the length of the counter. So we iterate again, we put them, and then we print them.
08:01
So it looks quite similar. So we need to compile this, and we run it for 10.
08:22
Okay, it works correct. Now we run it for 100. Oops, and we have a segmentation fault. Anybody knows where the error is? Yeah, I know that it's difficult, but let's see, it managed to print the count of primes.
08:46
So probably the error is somewhere here. Let's examine those lines again. So we have all primes array, and which is the length of the total primes, and then we iterate through our first primes array,
09:05
and if the number is prime, we increment this counter and put it here. So probably we could think that we go out of range here, but it's impossible because we start from zero, and we increment it with the same condition
09:24
where we increment it here when this is true. So it shouldn't go out of the range. And here we just print the data out of the array. What is the problem? And actually that kind of problem
09:41
that could be called like a beautiful journey in C, because you have no idea what's going on. And I'll tell you what's the problem. The problem is not here. The problem is here, because when we returned our primes array, we thought that we are returning array,
10:01
but actually what we do in C is that we return the pointer to this array. So it's the pointer to array. It's not array itself. And array was allocated at the beginning of the function on the stack. And actually it was valid only in the scope of the function.
10:22
And after we return the pointer to this array and we go out of the scope, this array, it's just no more. It has ceased to be. It expired and gone to meet at maker.
10:40
So, and we still have a pointer to it. And that is a problem very common for C programmers that's called a dangling pointer, the pointer that points to some random memory. And that's why we have a segmentation fault here.
11:09
So our task was to get our primes as fast as possible. Solution was implemented in C, but C is not the solution. I know that there are programs implemented in C
11:21
and probably there are people who are convenient with C and who know how to use C efficiently and probably they don't make such errors. But something still tells me that sometimes they do. And I personally, after years of Java and Python, just can't imagine how to live in the world
11:41
where you can suddenly become a pointer to a random data in memory. So let's carry on to Rust finally. Let's implement the same in Rust.
12:11
What we will do now is just to re-implement our C program, but in Rust.
12:21
And we will see how Rust compiler handles this situation. So here we have our C program
12:41
and here we have quite the same Rust program. If you don't understand little syntax details, it doesn't matter because I just want you to understand one basic concept. For example, what we do here, we have the same structure
13:02
and actually it denotes the same as in C. We have counter that is integer and this is the pointer, the pointer to array. There is no pointers in Rust, but this denotes like it's called slice in Rust.
13:21
So it doesn't hold the data, it doesn't hold the data, it just points to some data external for that structure. So it's actually the same as in C. And here we allocate our vector, we initialize it and we iterate and we increment the counter
13:42
and then we do the same as in C and here we return our vector as a slice. Okay, so let's compile that.
14:00
Oops, I messed up with the typing. Yes, and we have compilation error. And Rust compiler tells us that primes does not live long enough. So what it tells us that exactly what we have here,
14:23
that hey man, you cannot compile that. I won't allow you because you just want to return the pointer to the memory that will expire after we leave the context of this function. And actually this seems to be very strict,
14:40
but what is better to get this error just in time before you run your program or to debug some mysterious segmentation fault just in the weeks after you deployed your program on the, to your users for example. I think this is much better.
15:02
But let's run it, let's make it work. We won't fix this exact copy because it doesn't make sense. Instead of that, we just re-implement it from scratch in more idiomatic Rust because Rust has a set
15:25
and it has tuples like Python, so we have much shorter solution and it resembles us Python. And okay, let's run it.
15:48
Okay, and for one million, something like, so yes, it's 20 milliseconds, so it's like 25 or 30 times faster than Python.
16:13
And the concept that helped Rust compiler to deduce the error that we had is called lifetimes.
16:23
If you are interested about it, read Rust book. So, concluding our comparison. Python is 25 times slower than Rust. And C doesn't work just, so Rust is fast and safe,
16:42
but that is exactly what they told us in the beginning, nothing new. And returning to our main topic, can Rust make my Python shine? Yes, but if you're searching the internet about communication between Rust and Python, you'll quickly find some tutorials about foreign function interfaces.
17:02
You will even find examples like this. These examples are quite clear and simple and they work if you try. So, this allows you to call Rust code from your Python code, but it's not enough.
17:23
What if I want to access C Python internals from Rust? What if I want to convert Python string object to Rust string? What if I want to return a complex object from Rust? What if I want to make Rust library impotable as a model in Python?
17:40
And actually, that is what is needed in the real applications. For example, a Python profiler. By the way, who of you have ever used a profiler for Python? That's cool. But for those of you who hadn't, I'll tell what profiler is. A profiler is a program that measures frequency
18:02
and duration of function calls of another program. And normally, the less overhead it has, the better. So, let's make a Python profiler in Rust and to see how it goes. Actually, that was my initial idea when I started to experiment with Rust to try to make, for example,
18:21
a simple, tiny Python profiler. There are two major types of profilers. Tracing profilers and sampling profilers, also called statistical profilers. And statistical profilers, they periodically capture frames of a running program.
18:40
And normally, it has less overhead than tracing profiler, which traces all calls in the program. Let's see how to implement statistical profiler in Rust. But here, we won't go step by step and implement all the program because we don't have so much time. We'll just focus on two important aspects and maybe we'll learn something along the way.
19:01
And the aspects are periodically and frames. So, how to run task periodically? There is no timer in Rust standard library yet, but there is a wonderful library called MIO, Metal IO. MIO is a lightweight library providing effectively
19:22
different operational system obstructions like timing. And we just create a event loop, set up a timing event in it, and then we run a new thread and we pass there our event handler. And what is interesting here is an event handler.
19:43
Our handler will capture frames and save them to statistics map. That is a sampler object that we created. It's called sampler. And as our timer works in a separate thread, that means that our sampler is a resource that is shared between different threads.
20:01
So it's a shared mutable resource, which is believed to be very dangerous, as everybody knows that a shared mutable state is the root of all evil, but not in Rust. Rust guarantees you a safe shared mutable state, which sounds like a lie, but it's true.
20:23
What we do is we just put our sampler into Mutex, a mutual exclusion primitive useful for protecting shared data. When you create a Mutex, you transfer ownership of the data into the Mutex, immediately giving up the access to it.
20:41
And then any access to the data through the Mutex will block threads waiting for the log to become available, thus making the data accessible only through the Mutex by one thread at a time. And to pass the reference to another thread,
21:02
we wrap it with the ARC. ARC provides reference counting through atomic operations, and it is also saved between threads. And having done all that, Rust compiler guarantees us that we won't have any race conditions, never.
21:20
It's just impossible. And not having done that, well, you can't access that object from different threads. Rust compiler won't allow you to do that. It won't allow you even to pass this mutable data to another thread, so it will be single thread usage only.
21:43
So compiler guarantees you that your program will work. And to understand this better, read about ownership in Rust. So capturing current frame. For simplicity, we'll capture only current execution line,
22:02
as we are not interested in call three at the moment. There are three pieces of information, file name, function name, and line number that we will collect at every tick of our timer. In Python, there is a function in module Cs
22:21
that is called current frame. Under the hood, it uses a function by thread current frames. Looking into the C Python internals, we will find out that the structure that we need is called actually underscore frame. So we have this underscore frame structure
22:41
that points to some pipe code object that we also need. And what we need now? We need to convert it somehow from C structure to Rust structure to be able to use it in Rust. And that could be sometimes hard
23:00
because some C types are not very obvious how to map to Rust types. There is no strict mapping, no direct mapping because they are just absent in idiomatic Rust. So there are special Rust types to fulfill that gap.
23:21
For example, C void is analog of void and asterisk mute is a special type that reflects C pointers. And normally there isn't null in Rust at all, but to check this row pointer so we have special method is null.
23:42
So knowing that, we write our code. And remember that when one tiny thing which is important, when you are calling C function or using row pointer, Rust can't guarantee safety anymore.
24:01
All such expressions should be within unsafe block. And I think that is what they mean when they say that Rust prevents nearly all sick folds because that is the word nearly means that it doesn't prevent sick folds in unsafe blocks when you work with C code.
24:22
So knowing these mappings, knowing how to use unsafe blocks, we just create our structures in Rust. And seems that we are very close and there is everything that we need. But how to convert Python string to a Rust string?
24:44
Funny, but that was nearly the hardest problem I faced because actually it was difficult. And at some point I came up with something like this. So it's just to convert the Python string
25:01
to the Rust string as the last line. And it did work sometimes. It didn't handle some differences between byte strings and unicode strings and I already started to implement that. But then I came across a library called Rust C Python
25:22
by Daniel Grunwald and my life became much more easier. That is a beautiful library and I highly recommend it. Actually it appeared that the most things that I needed to communicate Rust and Python were there. I only needed to add some details for my specific case
25:44
and also it's a very good example of Rust code. For example, a string conversion using this library looks like that and also it handles all the cases with unicode string representation.
26:03
And also it provides very, very important abstractions like a special lock corresponding to the global interpreter lock in Python. Yes, and this is how you can expose your native Rust library
26:21
of the Python module using the Rust macros. It's amazing. Read the source. It's very, very cool. So it's just this line. There are a lot of more under the hood and they are very interesting.
26:41
So enough. It was much code and very much details but I'm not finished yet. Now as we have a couple of minutes, let's see how our profiler works.
27:10
So we're profiling this at the sense.py
27:29
that we have written at the first time. Okay, that's not interesting. Well, I thought that it will happen at some point.
27:47
It's impossible to make live demo without fails. Yeah, so we comment out our print, pythons, make matter, yeah, let's start with, okay.
28:11
And it was fast. Let's make it for 10 millions.
28:20
Okay, it's very simple now. It's very basic. But what it tells us that the 85% of our time went in the light eight and 14% in light seven. Actually, nearly 1% was for output of this line.
28:43
So line eight is this one. So what we are doing here is we are updating our set in array. And what we are doing here, 15, 14% is we're incrementing counter inside the array. It looks believable. So it looks very, very logical.
29:07
Oh, yes. And if you still care about performance in your Python applications, but you don't want to dig that deep into the native code,
29:20
I recommend you to listen to the talk of my colleague, Katrina Tuzova, that will take place on the 23rd of July and Thursday. She will show you how to write performant Python code without using any C or Rust. Thank you for your attention.
29:50
Questions?
30:02
Is it possible to distribute Rust code as a Python package which can be installed using people? Yes, I've came across a little library that allows you easily to write your setup.py.
30:22
This way, it will compile your Rust code if you have Rust compiler installed. Maybe it even downloads it. I don't know. I never tried it. I wanted to try it, but had no time before. You just type setup.py, build or install,
30:41
and it builds your Rust from scratch. Like for C, setup.py allows you to install C extensions, so there is a library that allows it for Rust also. As you said, only one question.
31:01
Yeah, that was good. Thank you again. Thank you.