SimPy
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Alternative Title |
| |
| Title of Series | ||
| Part Number | 81 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/20032 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Bit rateEvent horizonComputer simulationDiscrete groupComputer simulationProcess (computing)Event horizonSelf-organizationRoundness (object)Library (computing)PlanningDiscrete groupAuthorizationTask (computing)Software testingComputer animationMeeting/Interview
01:11
Event horizonDiscrete groupComputer simulationFood energyIntegrated development environmentLoop (music)Process (computing)Task (computing)CoroutineSemaphore lineFunction (mathematics)State diagramObject (grammar)Process (computing)Event horizonType theoryDiscrete groupObject (grammar)Physical systemSocial classState of matterSoftware frameworkInferenceIntegrated development environmentComputer simulationMultiplication signSoftwareTelecommunicationInteractive televisionDialectReal-time operating systemStrategy gameProduct (business)Computer virusProjective planeCuboidSymbol tableSimilarity (geometry)Frame problemTask (computing)Ocean currentContrast (vision)AreaProteinGenerating functionPlanningLibrary (computing)Formal languagePoint (geometry)Data miningConnected spaceSoftware maintenanceElectric generatorCore dumpGroup actionRevision controlSmartphoneProgrammschleifeInstallation artCoroutineFunction (mathematics)Software bugComputer hardwarePhase transitionLoop (music)XML
05:08
Mountain passEvent horizonExecution unitProcess (computing)Interrupt <Informatik>Condition numberRange (statistics)CausalityLetterpress printingReceiver operating characteristicQueue (abstract data type)Shared memoryObject (grammar)Cone penetration testVarianceConditional-access moduleArmControl flowValue-added networkMathematical optimizationIntegrated development environmentType theoryComputer simulationPoint (geometry)Event horizonCondition numberInternet forumNormal (geometry)Process (computing)HierarchyOcean currentInterrupt <Informatik>VarianceControl flowMultiplication signCAN busObservational studyMessage passingRight angleDataflowConfiguration spaceBitSource codeData storage deviceSimilarity (geometry)Object (grammar)ProgrammschleifeLine (geometry)BuildingResultantSlide ruleCategory of beingFitness functionBlock (periodic table)Virtual machineFunctional (mathematics)Shared memoryWater vaporCASE <Informatik>Operator (mathematics)SummierbarkeitElement (mathematics)Queue (abstract data type)Analytic continuationRandomizationIterationLogicLoop (music)2 (number)Sinc functionLecture/Conference
12:03
Special unitary groupExecution unitUniform resource nameVenn diagramWide area networkMetropolitan area networkSimultaneous localization and mappingMenu (computing)Regulärer Ausdruck <Textverarbeitung>Computer clusterPhysical lawView (database)Weight3 (number)CAN busVarianceComputer simulationBuildingSocket-SchnittstelleSimulationManufacturing execution systemPort scannerFunction (mathematics)Electronic mailing listSoftware testingTelecommunicationNetwork socketSummierbarkeitVisualization (computer graphics)Message passingMereologyProjective planeTelecommunicationChannel capacityField (computer science)Condition numberMathematicsLevel (video gaming)Web browserTheory of relativityBit rateComputer simulationVector spaceControl flowOrder (biology)Function (mathematics)Multiplication signFunctional (mathematics)Electronic signatureMathematical optimizationMulti-agent systemElectronic mailing listSelf-organizationStatement (computer science)Data managementNumberSoftware testingContext awarenessNatural numberResultantMultiplicationProcess (computing)Library (computing)Different (Kate Ryan album)ArmIntegrated development environmentBlock (periodic table)Physical systemSpeech synthesisVariable (mathematics)Configuration spaceEvent horizonProduct (business)Physical lawInterpreter (computing)Electronic program guideDecision theoryComputer configurationOrbitCuboidAreaCoprocessorFormal languageAdditionAeroelasticityPlanningSoftwareEvent-driven programmingSocket-SchnittstelleNetwork socketRepetitionAlgorithmExtension (kinesiology)EmailSystem callSuite (music)Program flowchart
18:44
Event horizonCondition numberProcess (computing)Computer simulationException handlingBit rateForm (programming)Video gameStatisticsFocus (optics)Distribution (mathematics)Service (economics)Shared memoryTracing (software)Differential equationMultiplication signBand matrixSummierbarkeitComputer fileProbability distributionType theoryIndependence (probability theory)Table (information)Right angleInternet forumSampling (statistics)Presentation of a groupDiscrete groupWeb browserEstimatorVisualization (computer graphics)Physical systemField (computer science)Data miningCartesian coordinate systemArmLoop (music)FamilyLibrary (computing)Standard deviationState of matterCASE <Informatik>Interrupt <Informatik>Electronic mailing listArray data structureBinary codeComputing platformBitPoint (geometry)Analytic continuationServer (computing)Differential (mechanical device)JSONXMLUMLLecture/Conference
Transcript: English(auto-generated)
00:15
Next up is Stefan Scherfke. He works at the Office Institute in Oldenburg
00:23
and is a researcher there, and he works on simulating the smart grid. He's also one of the authors of SymPy, which he is going to talk about now, so please give a warm round of applause for Stefan.
00:41
Hello, and thanks for being here. So, planning big events like Europy is no easy task and requires lots of planning, and the organizer team this year has done an amazing job so far. And to test your assumptions that you have to make beforehand, to test them, you can build a model of the conference
01:01
of all the attendees and speakers and then run a simulation on that and see how your plans work out. And if you want to do that in Python, you can use SymPy. This is a discrete event simulation library. And SymPy can, of course, not only be used for planning conferences, but also for all other kinds of planning problems,
01:20
for example, production planning or hospital process planning. It has been used to simulate multiprocessor real-time systems. I've been using it to simulate electric vehicles and their charging strategies. And last year, I heard of a guy who used it to simulate the spreading of smartphone viruses via Bluetooth. So you can do really interesting things with SymPy.
01:43
And SymPy is by now relatively old. It was created in 2012 by Klaus Müller and Tony Venio. And it was designed as a process-based discrete event simulation framework that means that you model the actors that do something or that change their state over time as processes. And these processes schedule events
02:01
at a discrete point in time to indicate a state change or an action that they want to perform. And in 2008, Anche Luenstorf, a friend and colleague of mine and I started to contribute to SymPy. And in 2011, we became the project maintainers because Tony and Klaus wanted to retreat from the project.
02:23
And the currently active version is SymPy 3, which we released in 2013 together with a shiny new logo. And it's only been in the recent years that I realized that SymPy has lots of similarities between other asynchronous frameworks like Async.io from Gido or Chvistad or Tornado.
02:42
What we call an environment in SymPy is usually called an event loop in the other frameworks. Our processes are very similar to tasks or coroutines. And events in SymPy are really like a future or promise. And finally, we have resources in SymPy and the resource is very much like a semaphore.
03:03
But in contrast to the other frameworks, we have multiple types of environments, events and resources. And since SymPy is so similar to other asynchronous frameworks on here, so why not put asynchronous networking on top of it? So we created a SymPy.io. With that, you can use network communication
03:22
in your simulation and you can simulate network communication, which is not possible with the other frameworks, but more on that later. The core idea of SymPy is to use generator functions to model processes. Who of you does not know how Python generators work? Nobody, nice, okay.
03:41
So I'll just skip over it. One note, usually people say working on other frameworks that the concept of using generators for asynchronous coroutines is a new concept, but SymPy used it from the beginning, so it's not really that new, but like 12 years old. So here's a simple example how a SymPy simulation
04:01
can look like to start by importing SymPy after doing pip install SymPy, of course. Then we'll define a simple clock process which gets in reference to the environment, the event loop, because in SymPy we use explicit event loops and no global object because global objects are always not that good and may introduce hard-to-find bugs.
04:22
Then we have a name and a tick and our clock will repeat, printing the current simulation time and then waiting for some time. To start our simulation, we create our environment, then we start two processes, a slow clock and a fast clock. The fast clock will print the time every half simulation time step
04:41
and the slow clock will print it every simulation step. And then we can run our simulation until we reach simulation time two and there we can see the output. Apart from the normal environment that performs the simulation as fast as possible, we also have a real-time environment and that can be synchronized with wall clock time
05:00
to realize real-time simulations and that can be useful if you have hardware in the loop or if you have human interaction with your simulation or if you just want to analyze the real-time behavior of an optimization algorithm, for example. And of course, you can create more environments if you have specialized requirements.
05:21
So I said before, SymPy has several event types. The most important event type for simulations is the timeout event because it lets time pass. So here's an example for a conference speaker which will hold a talk at a certain point in time and it starts by yielding a timeout to wait until we start off the talk
05:42
and then it will yield another timeout event after 30 minutes to indicate the end of her talk. So this is really the basic event type in SymPy. And processes in SymPy are events too. That is very useful if you want to model process hierarchies. For example, here we have, again, a simplified speaker
06:01
that will just yield a timeout for the end of her talk and then return a handout. And we now have a moderator process that will spawn speaker processes and wait for the speaker process to finish the talk and after that, start another speaker process until the session finishes. So in SymPy 3, you can use processes like normal events,
06:22
start them, wait for them to finish and then they can also pass a return value that you can print, for example. Now, not all speakers may just talk exactly 30 minutes. Some speakers may talk slower or faster so there need to be some way for the moderator to interrupt a slow speaker
06:40
and SymPy also provides a concept for that. Here, our speaker talks for randomly between 25 and 35 minutes and can be interrupted and our moderator now starts, again, a speaker process but then yields its own timeout for 30 minutes and if a speaker may talk longer than 30 minutes, like in that example, the moderator just interrupts
07:03
the speaker and stops the current talk. But this example has a problem. If our speaker spoke less than 30 minutes, the interrupt would fail so we should fix that and we can do that by condition events that are also new in SymPy 3.
07:20
With a condition event, you can say, I'd like to wait for that event or that event or for that event and that event and that event so you can concatenate events with logical operators. We have, again, the same speaker process as in the last example but our moderator now yields a condition event that says, okay, I'd like to wait until the speaker process
07:42
has finished or until a timeout of 30 minutes have occurred and then you get results back. The results is a dictionary that contains all events that have been triggered and if our speaker process is not in the results, it has not finished yet so we interrupt it. And instead of the pipe as logical
08:01
or you can also use the end character to say, I want to wait for two events to occur both. Yes, that's event markers on the timeline. So another way for processes to interact with each other are shared resources.
08:22
SymPy has three categories of resources. The first one is called resources and the main tab of that is also called resource and as I said before, it's very much like a semaphore. The main resource has two slots. So our resource here has two slots
08:41
and we have a queue in red where processes queue up and the first two processes can then just go and acquire the resource and the third process has to wait until the slot becomes available again and then use that slot. There are specialized types of resources.
09:00
One is the priority resource where processes in the queue are sorted by priority and another specialization is the preemptive resource where processes, important processes, can kick existing users out of the resource. The second category of resources is called restore. Restore is similar to the resource
09:20
but instead of storing process tokens, it can store arbitrary Python objects and it has two queues. One for processes that want to put something in and one for processes that want to get something out. So in that example, the process puts an object into our store and another process can then grab that object and get it out of the store.
09:41
We also have a filter process where processes can define a filter function to only get certain types of objects from the store. For example, if you store machines in the store, you can say I only want to use machines that are currently functioning and not broken, for example.
10:01
And finally, there's a container. A container stores a discrete or continuous amount of some homogeneous abstract matter so it can be used to store apples or water, for example, and in that case, a process may put two liters of water into our container. Another process wants to get three liters out
10:22
but has to wait until another process puts two more liters in so that it can get the requested amount out of the container. So these are the three types of resources that we have in Ximpi. You can, of course, extend that with other types of resources if you need them.
10:40
And now that we know the building blocks of which you can compose a simulation, I'd like to show you a little bit more complex example of a conference attendee, how you can model that with Ximpi. We start, of course, by importing stuff and then doing some configuration. In our world, we have a continuous flow of talk sessions
11:03
and each session consists of three talks and each talk is 30 minutes long and after three talks, we'll have a break of 15 minutes. And our attendees will also receive an environment again. They will have a name and they start with an initial knowledge of zero and they are not hungry in the beginning
11:20
and during the talks, they gain more knowledge but also get more hungry and the hungrier they are, the less knowledge they gain because if you're hungry, you just cannot keep your attention up. So we start with a while loop that loops over sessions and within that while loop, there's a for loop for each talk, one iteration
11:40
and we get a random amount of new knowledge and we get more hungry and then we yield a time out for the end of our talk and after the three sessions, we just print how much knowledge we've gained and how hungry we are now. And then we go to the buffet, grab some random amount of food and reduce our hunger and wait again until the break is over
12:01
to go to the next session and we print a little status message again. Starting the simulation is easy. We just create an environment, start five attendee processes and run the simulation for a few hours and the output of that may look like that. You can see that four attendees gain different amount of knowledge and they get hungry and after the break,
12:21
when they've eaten something, they are not as hungry anymore and can gain more knowledge as you can see in the second block. So that was already relatively interesting but we didn't gain any new insights about organizing our conference so we should maybe model the buffet in more detail to see whether our buffet is big enough
12:41
or if people have to queue up and starve to death. So that's not what we want. So let's add a buffet to our simulation. We have two new configuration variables. One is for the duration of eating, that is three minutes and our buffet will have one slot so only one person can get food at a time and the others have to queue up
13:00
and the signature of our attendee process now also has a reference to the buffet. The part for visiting talks is the same as in the last example but now the part for getting to the buffet gets more interesting. We create a request for our buffet here in a context manager with a risk statement
13:21
and using the risk statement, we don't have to release our buffet when we are done. So you should always use risk. If you don't use risk to request a resource, you have to explicitly call resource release when you are done and to avoid that we use a risk statement and then we yield a condition event to wait either until we get to the buffet
13:41
or until 12 minutes of the break have passed because if we wait more than 12 minutes, we won't have enough time to eat our food and it makes no sense to wait any longer. And when we get resumed, we check how long we've waited and then check whether we made it to the buffet. If yes, we grab some food again and yield an event for the duration
14:01
of our eating process, I print a nice status message again and we are done. And if we had bad luck and didn't get to the buffet, we get a penalty of one hunger because we could just stare at the food but don't eat anything. And yeah, so sad. So and then we print a status message
14:22
and wait until the break is over and when we start our simulation, it looks like in the last example but we also create a buffet resource here with the capacity of one. And the output now looks again similar like in the last example but you can see that attendee number four didn't make it to the buffet,
14:41
couldn't reduce its hunger and thus gains the least knowledge in the following session compared to all other attendees. So yeah, and it's really hungry after the conference. So a very unhappy attendee and we know that we should add more slots to our buffet in order to make every attendee happy. And I think the organizer Kim Sissy
15:00
has made a great job with the buffet. Food delicious and enough slots, so yeah. Okay, that's the example. Now I'd like to get back to Simpa.io. Simpa.io is an event-driven networking library on top of Simpie and for that you can, for example, model multi-agent systems
15:21
and these multi-agent system can be distributed over multiple systems or processes and use network communication CCP sockets to communicate with each other but you can also just simulate the communication between your agents and for example, look how they behave if there's a big latency between messages
15:41
or if messages drop. So this is very cool for evaluating your distributed algorithms and Simpa.io also implements rep sockets so you can add a visualization for your simulation to make some nice graphics in the browser, for example. And Simpa.io is still in a very early stage.
16:01
It's already workable but doesn't have a lot of documentation yet and the APA may still change but roughly using Simpa.io looks like that. You just yield instead of a timeout a send event or a message receive event in order to say Simpie, okay, I'd like to wait until I got a new message or my message is sent. So it's not that difficult
16:21
and I've already used it in a production system and it's really cool and relatively fast. Yeah, our plans for the future, of course, keeping the community happy, helping everyone who has questions, implementing all the features that people want if they make sense, of course. We also have some minor optimizations in the pipeline
16:42
for some speech improvements. We are continuously working on our documentation and currently we're discussing adding some helper functions for helping you monitoring your simulation, especially if you want to monitor the usage of resources but there's nothing decided yet.
17:00
So some final notes. I hope that I could show you that Simpie is really easy to use and flexible. Simpie has lots of documentation. I'm really proud of our documentation because we have a tutorial, we have lots of examples, we have topical guides and an API reference. We have also a mailing list and of course, your stock Everflow where you can also ask questions
17:21
and we have quite an extensive test suite that makes sure everything works as expected and that can also be used as a reference of how things should work. Yeah, Simpie is written in pure Python and it supports Python 2.7 and 3.2 and above and it also has no external dependencies
17:40
so installing it is really very, very easy and fast. Since it's pure Python, we support all kinds of other interpreters like Json or PyPy, for example. PyPy can give you huge performance improvements if you have a longer running simulation and there's also a port of Simpie to Sison so you can get an additional performance improvements
18:00
by compiling Simpie to C and since a few months, there's also a project called SimSharp which has ported Simpie to C Sharp which is also kind of cool if people forget to another language. So yeah, and there's Simpie O that you can use for adding socket communication to your simulation.
18:20
So that's all. Go to Bitbucket or simpie.reads.docs.org to grab it and yeah, thanks for being here.
18:41
So if you have any questions, you may also raise your arm. In the field of computational chemistry, researchers use packages like Thinker, GAMES, Gromacs
19:00
and others to do molecular dynamic simulations or Monte Carlo simulations of hemical or bio-hemical processes. Do you know about the applications or uses of Simpie for computational chemistry
19:21
or if there are people that use this package for computational chemistry? No, not directly. A colleague of mine uses Monte Carlo simulation and he doesn't use Simpie for that because if you have just single processes that don't really interact with other processes or don't use resources,
19:41
you don't really need Simpie. So in that case, you may be better off with just using a while loop where your process updates its state after each time step or delta of time. So I think you may use, so you could use Simpie for that but maybe it's just too much
20:01
and you can just use plain Python standard library to do that. And if you want to use a continuous simulation with differential equations, Simpie also does not really offer anything to help you with that. So the main focus of Simpie really is discrete event simulation or discrete time simulation where processes interact somehow with each other or use shared resources.
20:22
That's where Simpie really helps you. Hi, and first, thanks for the talk and your work on this package. I like it. I have a question about interdependent events. What's the best way to reschedule them if needed? For instance, a server is sending a file
20:42
over a limited bandwidth and then it starts sending another file. And of course, this delays the arrival times of other files. So what's the best way to do it? So you want to know if I already scheduled an event and wait for that event
21:00
and now I'd like to wait for another event. Yes, if something happens that probably cancels that old event or just delays it or whatever. There are several ways to do it. One way to do it is interrupting a process that waits for an event and then tell them, okay, please wait for another event. Or that is the process that waits for the event itself,
21:22
cannot do that by itself, so it has to really wait for the interrupt. And if the process itself wants to become active, it should do something like yielding a condition event where it says, okay, I'd like to wait for that event and if it takes too long, then I'd like to for something else and for that you can use the condition events. So these are condition events
21:40
or interrupts by another process. Or maybe just refactor your simulation and just implement another process called a network cable which just handles all these delays. Which is because it has reference to all the events and can just say, okay, cancel event and just create a new event further in the future.
22:01
You could do that by implementing a new type of event but once the process yields an event, it really waits until the event occurs and then it gets resumed. The only exception to that is if it gets interrupted by the event loop. But if you create a special event type where the condition under which the event triggers can be changed depending on whether one thing happens
22:23
or another thing happens, then you can also achieve your goal. So it should be possible. Okay, thanks. Hi, I got actually two questions. The first one, do you have some tools to module the statistical discrete distributions
22:45
like points on something or am I supposed to use something like Scipy? Or, and the second one, do you have some built-in visualizations to the tools or should I use Mac? Could you please speak a little bit louder? Louder, okay.
23:00
So, okay, once more. Do you have some built-in tools for modeling discrete distributions, statistical distributions? Or am I supposed to use Scipy? Because Scipy is like real big package and if you try to implement it on the platform where you can't install that, it's like quite a huge thing. And it has a lot of binary dependencies and stuff.
23:22
It could be quite useful if I had some basic bunch of statistical distributions to model timeouts. Do you understand what I mean? Scipy doesn't have built-in any distributions but you can use the random distributions provided by the standard library or by Scipy.
23:42
So Scipy is really just a small package and tries to focus on the modeling of processes and event loop and everything that goes beyond that is left to library set up better in that than RIA. So why should we rebuild distribution, random distributions if they're already very well done
24:00
in Scipy or NumPy or Python standard library? Okay, I understand. So the answer is no. Okay, and I think you got the same policy for visualization or something. You've got the same policy for visualization. I mean, could be quite cool to see how your system develops with time and such stuff.
24:21
Yeah, Scipy used to have some kind of visualization built in but it really looked not that good as Matplotlib or D3.js in the browser. So we just said, okay, we exclude that from Scipy and don't maintain it anymore. And collecting data from a simulation is relatively easy. You just pass, for example, this or NumPy dictionaries,
24:41
NumPy arrays into your processes and then append data and after your simulation is done, you just pass these lists or arrays to your visualization like Matplotlib and then let it do the visualization. So that's the better approach. Okay, again, thank you. Yeah, any more questions?
25:01
No, then, thank you again, Stefan. Thank you.