Python vs Rust for Simulation
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 118 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44773 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
EuroPython 201998 / 118
2
7
8
15
20
30
33
36
39
40
45
49
51
55
57
58
63
64
66
68
69
71
74
77
78
80
82
96
98
105
107
108
110
113
115
00:00
Computer simulationGoogolExecution unitFreewareSoftware repositoryType theoryContinuous functionEvent horizonDiscrete groupMixed realitySystem programmingPhysicsKolmogorov complexityComputer networkCommunications protocolGame theory1 (number)Software frameworkFront and back endsVirtual machineSoftware developerComputer simulationProduct (business)Library (computing)Formal languageEvent horizonMultiplication signForestSimulationType theoryVideo gameDiscrete groupGame theoryInstance (computer science)Physical systemSurjective functionDataflowPoint (geometry)LogicSound effectNatural numberSource codeWeb serviceComputer programmingFinite-state machineMathematical modelImplementationDifferent (Kate Ryan album)Office suiteLogistic distributionScalabilityWater vaporRight angleAbstractionAnalytic continuationNetwork topologySoftware frameworkTerm (mathematics)Process (computing)Thresholding (image processing)StapeldateioutputComputer clusterMixture modelLecture/ConferenceComputer animation
03:39
Execution unitCASE <Informatik>Computer simulationoutputSoftware frameworkInstance (computer science)SimulationDampingLibrary (computing)1 (number)Different (Kate Ryan album)Programming languageSource codeClosed setLecture/ConferenceMeeting/Interview
04:25
Point (geometry)FreewareComputer simulationReal-time operating systemMaxima and minimaMobile appSoftware frameworkPoint (geometry)Right angleInstance (computer science)Multiplication signForcing (mathematics)Scaling (geometry)Service (economics)Computer simulationLibrary (computing)Software developerSimulationNear-ringPresentation of a groupCASE <Informatik>Extension (kinesiology)ScalabilityPoint cloudComputer animation
06:32
Multiplication signPrototypeAtomic numberLibrary (computing)Software testingFormal languagePhysical systemComputer programmingInstance (computer science)Object (grammar)Point (geometry)Adaptive behaviorPresentation of a groupRevision controlCodeSemiconductor memoryScalabilityBus (computing)SimulationAdditionDataflowLogicSoftware developerExecution unitCASE <Informatik>2 (number)Different (Kate Ryan album)Device driverData structureElectric generatorType theoryReal numberWritingLecture/ConferenceMeeting/Interview
09:31
Parameter (computer programming)Loop (music)LogicVideo gameInstance (computer science)Social classDevice driverMultiplication signLoginUtility softwareArithmetic progressionFunctional (mathematics)ImplementationComputer configurationMathematicsUniqueness quantificationSimulationFingerprint
11:08
SineConvex hullAttribute grammarOcean currentLimit (category theory)Maxima and minimaMultiplication signCycle (graph theory)Arithmetic progression2 (number)Functional (mathematics)Arithmetic mean
11:40
StatisticsConvex hullComputer-generated imageryArithmetic progressionGroup actionArithmetic meanState of matterMultiplication signMaxima and minimaLogicPresentation of a groupLoop (music)Computer programmingRandomizationSlide ruleFreeware
12:50
Line (geometry)Read-only memorySlide ruleMultiplication signSoftware testingPrototypeComputer programmingSemiconductor memory1 (number)CodeDataflowCoefficient of determinationVideo game consoleNumberCASE <Informatik>2 (number)Physical systemDifferent (Kate Ryan album)Revision controlHost Identity ProtocolState diagramComputer animationDiagramProgram flowchartXML
14:49
Line (geometry)FluidMenu (computing)MathematicsRun time (program lifecycle phase)Execution unitSequelCartesian coordinate systemFingerprintTotal S.A.Normed vector spaceElectronic visual displayFunctional (mathematics)Data structureImplementationMacro (computer science)Utility softwareLevel (video gaming)Default (computer science)Mathematical optimizationLetterpress printingMultiplication signSocial classStrategy gameRun time (program lifecycle phase)Electronic visual displayFile archiverElectronic mailing listVector spaceHash functionDifferent (Kate Ryan album)Field (computer science)Pairwise comparisonCodeCASE <Informatik>Computer configurationTerm (mathematics)Constructor (object-oriented programming)Element (mathematics)Similarity (geometry)RandomizationDistribution (mathematics)Root
18:23
Line (geometry)Read-only memorySoftware testingData managementComputer fileSemiconductor memoryMultiplication signCompilerWeb serviceUniform resource locatorImplementation2 (number)BitComputer programmingMacro (computer science)Instance (computer science)Order of magnitudeCodePairwise comparisonLine (geometry)Domain nameType theoryResource allocationMemory managementRight angleHash functionMetropolitan area networkComputer animationProgram flowchart
20:34
Pairwise comparisonCodeCASE <Informatik>Computer programmingVisualization (computer graphics)Multiplication signLecture/Conference
21:04
Pairwise comparisonCodeRevision controlSoftware testingRead-only memoryPoint (geometry)Polarization (waves)CASE <Informatik>InternetworkingServer (computing)Local ringProjective planeComputer programmingImplementationFormal languageInstance (computer science)Semiconductor memorySoftware developerGoodness of fitDampingSimulationRevision controlGraph coloringWritingScalabilityPhysical systemPattern language
23:03
Multiplication signCodeSimulationInstance (computer science)CuboidComputer programmingService (economics)Order of magnitudeCASE <Informatik>Complex (psychology)Software developerPiComputer simulationPhysical systemMathematical optimizationTerm (mathematics)Decision theoryNP-hardElectric generatorWordMobile WebDifferent (Kate Ryan album)Open sourceLibrary (computing)Software testingVirtual machineoutputDevice driverResource allocationSet (mathematics)Software development kitPoint (geometry)Flow separationPresentation of a groupExtension (kinesiology)Row (database)Cycle (graph theory)Uniform resource locatorFunction (mathematics)WebsiteLecture/Conference
Transcript: Englisch(auto-generated)
00:04
Hello everybody. Since recently I'm actually a machine learning engineer at Freenow or former my taxi and I do a lot of backend development and putting models from our data scientists on to production but not only just putting into web service or into some batch jobs
00:22
but trying to optimize them so that we have certain threshold on our performance. This is also why I stumbled across Rust and as previous speaker he had told you about pyro3 I also tried it out, and it was very promising, but right now. I'm working on simulation
00:40
for our business purposes and the question of what should be executed how fast how Expensive in terms of scalability or maybe development time all different points They are Kind of important right now. This is why I decided to present this topic as well here
01:04
So what is a simulation? In a very simple manner simulation is an abstraction of Some event so you can think a state machine is technically a simulation an example would be
01:22
For instance How the water flows through the pipes in a system or how the blood goes through our Veins and then for instance you have an aneurysm how the blood Flow behaves there so all of this can be created in a simulation, so it's just a program that recreates some
01:42
real event The there are different types of simulation normally you will see continuous simulations, which are implementation of Mathematic models for instance that are continuous they define the whole flow of the logic a very well-known example would be a game of life or
02:04
For instance in chemistry how different chemicals react with each other and for instance Whether you're going to get an explosion or not this can be also played in a continuous way in the simulation a Second type is discrete event simulation Which is that it covers most of man-made systems for instance
02:25
Post office so event occurs and something happens only when somebody is in the post office I as a customer or manufacturing pipeline Logistics systems and stuff like that can be used in a discrete event simulation
02:41
But right now There are a lot of papers and most of the production ready simulation frameworks are mixed where you can define certain Event dispatching system that says okay these resources are dispatched And then the system decides how to proceed further or if there are any other side effects that can be described in continuous way
03:01
for instance forestry is a good example because you have a natural way of forest to grow or Recover from a fire for instance, but again you can dispatch discrete events like men planted forests or men created fire, so it's a mixture of both and
03:21
Obviously it requires more Development time involved There are numerous tools in all languages you can imagine for instance frameworks libraries game engine simulation Technically can be seen as a game engine without manual or human input and
03:42
Of course people build it from scratch with different programming languages There are just some Examples that you can use for instance you can use unity to simulate how people will learn By providing some model that will replace the human input for instance The model will learn how to drive a car in Mario Kart or stuff like that that can be considered a simulation
04:07
but in my case like there are different things you need to consider when choosing whether you want to go with a framework or a library or how you want to use the simulation whether you need the simulation in your business
04:20
So four main things are cost if you go with closed source thing it might cost you a lot Or it might result in a lot of development hours that also directly is Translated into cost then speed of the simulation you probably don't want to simulate something with Real-time or near real-time speed right like why then bother the second the third point is scalability
04:47
For instance you want to run different scenarios You cannot just wait for one to finish and then start another one so you want to for instance Horizontally scale it so you want to put it into a service you want to put it into cloud you want to provide
05:01
Some resources you need to know how the framework that you've chosen or you're written behaves in this way as well the force and one of the most important points is Extensibility if you are using something that is already there like a backbone library or even a ready Framework you might run into situation where you have a very
05:24
Important business use case which is not covered By this framework so you either have to pay the company to extend the framework or you have to do it yourself and Again, it translates almost directly into cost So whenever you decide to do something you need to keep an eye those form and for this presentation
05:44
I decided to go with very similar situation that I have on my work Which is we gonna simulate dispatch of taxi to a request or a customer So we're gonna have a world that has that can spawns with a certain chance of P a
06:01
Request we can have only max and requests which can be described as you can only have Thousand passengers in using your app and one passenger one request at a time It's not possible for a passenger to request a taxi to a point and B point Because how would you do it because you have only one physical body. What would you do would you split?
06:24
I don't know then request can be assigned to a free car only we don't have something like Shuttle buses, it's just pure car one request one car Request can be cancelled after a certain amount of time if they didn't don't get assigned
06:41
Which again is the real situation from taxi business passengers are not waiting longer than let's say 15 minutes They will just get a bus or a different taxi driver Cars can be either free or occupied We do not have any other use cases and we will simulate one day, which means we will simulate
07:04
24 multiplied by 60 60 ticks So one second is our atomic unit of time that will be driving the whole simulation Criteria so the talk is called Python versus rust So we have to somehow compare them objectively or semi objectively and these are criterias
07:24
I came up with the objective criterias are amount of code you need to at least prototype your flow then testing simplicity How many packages they are for testing? Is it simple to write a test? Is it just how much time you would spend on it?
07:41
documentation generation and documentation available Because probably you're not gonna write everything yourself You might need some additional libraries and crates performance obviously is important memory usage is also important because of cost and scalability points and Ecosystem will play later a big role once you have your prototype in place
08:04
For instance, if you have already existing business system that has for instance hadoop and coal You might run in the problems where your simulation has no Official adapters or connectors to hive or padoop itself or like there's just no craze
08:21
and you have to either write it yourself or rewrite it again in the language that has the libraries and language versions the previous speaker mentioned it already, but it Sometimes play a crucial role. Sometime it doesn't so you also have to evaluate this risk The subjective two points are called simplicity
08:41
Obviously if I have more experience in Rust I will say ah, it's easy. I don't understand Python at all I will spend one hour in Rust and one day in Python and vice versa and The second one is development speed. They are connected but development speed is Python is notoriously known for allowing people to fast prototype with other languages, especially aesthetically type languages
09:05
You have to think first what? Structures you want to use how you want to present your program How is the flow not of the logic but the whole flow of the program where objects are going memory? Collection all that so you have to put some time in advance
09:22
Saying all that let's go to Our presentation so I will show you a couple of codes We will start with Python first So the implementations are identical with a slight change or
09:41
Unique things that only rust has or only Python has Otherwise you have the same struct or class which is a request request has person you You ID which is its unique ID driver ID that can be assigned. It's technically an option So there is none driver or there is a driver
10:01
Remaining lifetime and fulfillment time this arch parameters that will Make our request either be cancelled or being fulfilled and is alive is a utility function that will tell us that yes Still kicking still working Another is a taxi
10:20
Which has ID so that we can see in the logs what request was assigned to what taxi if we have Slightly more complicated logic of assignment other than random to random and Then we have our world world is just a container that binds it all together So it has this main loop that says until I have time
10:41
Or in this day of simulation do this and within do this will happen spawn creation if we can assignment if there are free cars update so This Ticking for instance all the requests that are pending will start decaying and all requests are in progress will start being fulfilled and
11:04
The cleanup that will say this one is already done free the taxi Allow it to be used in the next cycle in the assignment and put this progress request into cancelled or finished so you can see all of this in
11:20
The Attributes and functions so the world will have runtime. This is amount of seconds. It will run for age is current step Requests pawn chance is again this random chance to spawn a request Max active request is the limiter of how many active requests we can have at once active mean
11:42
both in progress plus pending and And taxes taxes as I said before can be free and occupied and Requests are spread into four groups pending progress finished and cancelled by the state and execution Then we have maybe spawn request which is obvious if we can't spend spawn a request
12:05
It's not we have not reached our max. We will do it with random chance Then we have distribute if there is a taxi in free taxis we will try to attempt or we will assign this car to a first
12:20
Non-assigned request so there is no special logic yet The update request is exactly that tick down Either remaining waiting time or fulfillment time the cleanup is just put into next state if I have to and the main loop is While we have time
12:41
Do all the steps one after another I'm not going to run this program for the whole day because it's actually slow So I will be going back to the presentation slide Sorry for time So all of our
13:00
Criterias will translate into quite small amount of code to prototype our flow It's actually 94 without documentation our performance is for one day without printing into a console is 210 seconds I did it with hyperfine which
13:21
Gives it like several runs and uses average. So it's believable number It's not like I run it once with couple of different programs running in the background Memory usage just just remember this number So for this execution Python allocated on the hip about 35 megabytes and it's small and
13:42
Tests I don't have to tell you about Python tests. It's gorgeous. It's cool You can either use unity test or dog tests if you don't have if you don't want to bother or you can have Some additional like pi test packages or something completely different like nose tests The ecosystem is amazing. Tell me at least one use case that is not applicable to Python
14:06
The version of Python should not be a problem at all starting 3.6 sometimes you might run into rendering problems, but it's a slightly different issue and it dependent on the OS that you are running it on
14:22
I've spent about an hour writing this program, so I will say it's quite fast The simplicity of the code you can judge for yourself, but I will say it's also quite cool It's easy to understand what it's happening and not only by the structure but like from the Python code itself Okay, so now we'll have the second contender, which is rust as you can see
14:48
It looks very similar. You have to use imports this use std something is just an import to use certain Structure or function from a crate
15:01
Then you have these things are called macros. You probably saw it from the previous speaker They will give you all the implementation of this functionality into your struct just Simplification so we have the same request you can see right now struck as a class That has ID the remaining waiting time assigned taxi, which is now option. So again, it can be none or
15:25
in this case UID and fulfillment time Then we have our request Implementation it's in rust you can create a constructor directly and this is let's say a utility function where you also can provide some default values if you want to I
15:44
Have it here. So and is alive The next one would be taxi which looks almost the same way as the Python one With implementation of the new function and then we are at our world that has the very same
16:02
Fields that Python class as well has runtime age request ponchans Max active request taxis is a vector like list in Python would be a vector in rust Then active requests and archived requests. I have them separately and not united as a one
16:21
Hash map you could do this as well. But this was the idea to go with at the first so it's not completely optimal implementation and rust but something that works Then you have the new function for the world. You have the RNG for random. You have to pass it through In comparison to Python that can just use it as a global
16:44
Then we have a utility function print and Implementation for that then maybe spawn request you already seen it's very similar Distribute unfulfilled request is slightly different. You now can see this eterm root find which
17:00
In rust you have a concept of mutable and immutable You probably all know that and you have to think in advance Whether you want to change certain structures or not where you want to reference them or not and stuff like that You normally get used to this after a couple of days writing rust But it is something that is very different from Python just to notice
17:21
Then update request is the same and then we have cleanup requests which is slightly different in comparison to Python due to this concept of borrowing and that borrowing and reference checking so you cannot just say
17:42
One a vector from an element from vector just append to a different vector because there's gonna be movement and you Have only one owner at a time. Just ask me later about technical details so that we don't Spend too much time on this but it's just the way it is. It still is kinda understandable from the code that you can see
18:03
Then you can have like you have your run till done, which is the same this display function is just a implementation for print Rust does not always by default has print possibility for every struct you have and then we have our main
18:22
So what does Rust tell us It results in hundred sixty lines of code with our documentation documentation was with three slashes and performance with one day with no logging is 154 milliseconds
18:41
versus Python 200 seconds the memory usage is a Magnitude less than Python it actually is Keep a location and not stack. So there are some Shady business done there. I Say I must say but in pure keep a location was just bits Python dead
19:06
The different criteria were the simplicity of the code so I Would say it's subjective and you would understand from reading the code twice or three times if you never did it before But it's still Python wins in this regard then amount of time I've spent on writing
19:25
This one is one day because my first implementation actually I managed to make it quadratic time so it never ended I runoff patients before that So you really have to invest time in advance thinking of okay. What am I doing here?
19:43
Also, you will fight with the compiler a lot in the beginning, but he is your friend So if it compiles it works the other one was test cargo is the Manager package manager for instance and everything manager in rust you can russ around just cargo tests
20:05
with and write the test in the same file as your normal program or in a different file and just annotated with Hashtag test which is a macro for tests then ecosystem is Not that good at all in comparison to Python. It's
20:23
Can be in a certain domains like embedded programming or for instance low-level programming in general Right now it's catching up on web services as well, but Python is very much ahead and Again, the situation where you have something like presto and Hadoop or hyphen Hadoop on your business use case
20:44
Just forget up. There is no crate in rust for using those SQL layers Yeah, so let's have a more visual comparison So amount of code rust is about two times more code
21:01
It can be even more or about the same amount if you know how to write your program But for beginners Python is a clear clean advantage there the simplicity is in both cases very simple both languages are trying to keep it very simple and usable
21:21
Documentation I would have said that it's good in both cases but rust has one killer feature for me and which is you can have offline documentation served on your local server for instance HTTP Python server accessible offline if you have it cached with your cargo so you can build your
21:40
Project and you have it offline doesn't matter whether you have internet or not Which is as far as I know not the case for Python. This is why for Python I left no Mark or color for this one and Russell said like yes Memory efficiency da Performance you saw it
22:01
Ecosystem, by the way, there were no polarization in the program so it can run even faster and quite cheaper for that matter Ecosystem Python is a clear advantage there. There is no denying that versions In Python, you should have no problems if you got rid of your Python 2
22:21
hopefully And in rust it will depend whether you have to use nightly or not stable has most of the features nightly has Really nice features that you might need or might not Depending on how deep you want to go in your implementations since we are using a writing from scratch use case and
22:41
Development simplicity and development speed. I am a Python developer not a rust developer. So This is biased for me. So Would you write rather in rust or in Python your simulation if you need one To be honest, I don't know because you have to consider your pain points first like cost scalability
23:04
Your extensibility and speed because at some point if you go for Python you have to do optimization like it always happens and Optimization can take quite a lot of your resources in terms of development time in terms of even getting some Libraries that are not open source or whatever at the same time if you go with rust you will be very slow in the beginning
23:27
Maybe even very frustrated, but the end results always worth it. You won't be able to reach This performance with Python pure code without any site or whatever usage So, I'm sorry for not giving you the exact answer on this question
23:43
You have to consider for yourself depending on what simulation you're writing How fast should it be how scalable should it be how simple should it be in the end for other users to use it? if you want to have simple okay fast and
24:00
Right now simulation then go with Python if you are able to invest time then consider rust Thank you. So I think we have time for questions if there are any
24:26
Did you try to run your simulations pi pi? not yet, but In previous years. We actually Had a similar program and we me I tried to use
24:41
pi o3 C Python and pi pi pi pi from the box gave me 30% of Performance improvement which is still not the magnitude or several magnitudes And also pi pi is not compatible with all packages for instance. I kid if as again. I'm a machine learning engineer It's my biggest pain we use scikit and we cannot use pi pi for that matter
25:03
To be fair rust probably will always have problems with that as well But yes, I tried by pi in a similar situation. It didn't perform as well as rust. Thanks for the talk. They were interesting
25:23
Tell me about the business of writing simulations. Is that's a big thing Like is it critical to the business how many people work on that? I will say Simulation is the next big buzzword in industry for instance. I can give you a couple examples from mobility industry Tesla uber my taxi
25:42
We all writing simulations for slightly different purposes for instance Tesla has a giant and very complex simulator for Autopilot to test some new features before they roll it out into the streets So to avoid certain fatal flaws or to catch them slightly earlier
26:01
Uber has simulator for marketing purposes and In my taxi, we are developing it in the first row for Development purposes meaning we want to speed up our development cycle. For instance, we have a team that works on a location So connecting driver and a passenger and they have a brilliant idea
26:21
But until they put it into production and run an a B test for some time They don't know whether it was worth it or not. We can even lose people if it was a bad decision So we are one what we are doing. We are building a simulation that will allow them to at least cut the bad ideas Also, you can test certain things. Like how would my system react if we have a strike suddenly we have
26:48
Small supply like we have Instead of 200 cars one car We still want to make money so we will probably have to serve our best customers first and stuff like that You just cannot replay it in the real world. Also from your historical data is kind of hard
27:06
Because it's just historical and not flexible and this is where simulation kicks in also simulations are very used For data generation purposes, for instance, if you want to not only test your system on robustness like okay
27:21
This service works the request is coming in but you want to produce some edge cases and you want to test Like end-to-end content wise then simulation can also help you Thanks. Hi. Thank you for a presentation. You've mentioned about data generation and I know that there are papers for
27:45
Using machine learning models to speed up simulations. You're basically using input and output from the Simulation as then data set and data set for our model. Have you Heard or considered using that approach?
28:01
Yeah, the thing that we are building and it's quite like quite famous is agent based simulation so we have trained models of drivers or passengers or for instance your baristas if we're talking about coffee shop or even customers that are Learned from historical data
28:22
for instance I have a customer agent that learned that he's and sorry for the word asshole and he likes and leaving no tips and Spilling coffee everywhere so you would have something like this in your simulation. So I Don't know exactly. How would you use models to speed up simulation itself?
28:42
But using models in the simulation is quite a very often technique. Thank you. Thank you for your talk If you have any more