Concurrent programming with Python and my little experiment
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Teil | 61 | |
| Anzahl der Teile | 119 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/19949 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produktionsort | Berlin |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
EuroPython 201461 / 119
1
2
9
10
11
13
15
17
22
23
24
27
28
41
44
46
49
56
78
79
80
81
84
97
98
99
101
102
104
105
107
109
110
111
112
113
116
118
119
00:00
CodeUnrundheitEndliche ModelltheorieDatenparallelitätEindeutigkeitAutorisierungEnergiedichteComputeranimationXML
00:40
GruppenoperationProgrammbibliothekProgrammierungDatenparallelitätFunktionalProzess <Informatik>Projektive EbeneEndliche ModelltheorieSpeicherabzugDatenverwaltungBenutzerbeteiligungServerVertauschungsrelationMusterspracheVorlesung/Konferenz
01:23
BefehlsprozessorParallele SchnittstelleMachsches PrinzipModelltheorieROM <Informatik>Message-PassingProzess <Informatik>PaarvergleichProgrammiergerätVariableCASE <Informatik>ProgrammierumgebungHalbleiterspeicherMultiplikationsoperatorFunktionalVirtuelle MaschineArithmetisches MittelModelltheorieCodeDatenparallelitätMereologieProgram SlicingParallele SchnittstelleComputeranimation
02:05
Virtuelle MaschineFunktionalLaufzeitfehlerEigenwertproblemProgrammierungPhysikalisches SystemDickeDemoszene <Programmierung>CASE <Informatik>Message-PassingXMLComputeranimation
02:48
Familie <Mathematik>Element <Gruppentheorie>FunktionalMereologieMessage-PassingDifferenteProzess <Informatik>Erlang-VerteilungCodeXMLComputeranimationVorlesung/Konferenz
03:30
StandardabweichungSingularität <Mathematik>LoopEreignishorizontMultiplikationsoperatorProgrammierungRahmenproblemProgrammbibliothekVektorraumEndliche ModelltheorieArithmetischer AusdruckProzess <Informatik>SchnittmengeSoundverarbeitungGüte der AnpassungRechter WinkelCodeDivergente ReiheAusnahmebehandlungSphäreFunktionalDerivation <Algebra>ParametersystemMaßerweiterungMusterspracheResultanteThreadMehrprozessorsystemProgrammschleifeMessage-PassingEreignishorizontDifferenteSchreiben <Datenverarbeitung>IterationLesen <Datenverarbeitung>GruppenoperationXMLComputeranimation
06:20
Natürliche ZahlProgrammbibliothekMultiplikationsoperatorGruppenoperationEntropie <Informationstheorie>Funktion <Mathematik>LaufzeitfehlerCodeDifferenteTopologieSummierbarkeitBinärcodeEin-AusgabeEreignishorizontKonfiguration <Informatik>Arithmetisches MittelLoopComputeranimation
07:35
DatenmodellKnoten <Statik>HochdruckLaufzeitfehlerSchedulingKomplex <Algebra>ProgrammbibliothekRahmenproblemKugelAlgorithmische ProgrammierspracheEndliche ModelltheorieAnwendungsspezifischer ProzessorEnergiedichtePhysikalisches SystemMultiplikationsoperatorDatenfeldHalbleiterspeicherDokumentenserverFunktionalPaarvergleichStapeldateiCodeMustersprachePatch <Software>Proxy ServerKoroutineThreadDatenparallelitätQuick-SortMetadatenGemeinsamer SpeicherMessage-PassingComputeranimation
10:20
PortscannerSinusfunktionImplementierungSingularität <Mathematik>Klasse <Mathematik>SystemaufrufBimodulPhysikalisches SystemRFIDMetropolitan area networkGleichmäßige KonvergenzARM <Computerarchitektur>SchnittmengeElektronische PublikationZeiger <Informatik>Software EngineeringLokales MinimumMehrwertnetzDualitätstheorieInklusion <Mathematik>SpeicherbereichsnetzwerkGammafunktionModul <Datentyp>SynchronisierungPhysikalisches SystemKonditionszahlNichtlinearer OperatorReverse EngineeringFunktionalNotepad-ComputerUrbild <Mathematik>MultiplikationsoperatorBildgebendes VerfahrenCodeRechter WinkelMessage-PassingAbstimmung <Frequenz>Spezielle unitäre GruppeSummierbarkeitInhalt <Mathematik>StellenringResultantePuffer <Netzplantechnik>Anwendungsspezifischer ProzessorSchnittmengeProgrammbibliothekEreignishorizontImplementierungHochdruckDatensatzTrennschärfe <Statistik>Termt-TestEndliche ModelltheorieMereologieArithmetisches MittelGewicht <Ausgleichsrechnung>LaufzeitfehlerBenutzerfreundlichkeitZeichenketteSpannweite <Stochastik>Physikalische TheorieLeistung <Physik>Güte der AnpassungSoundverarbeitungVorhersagbarkeitModelltheorieEinfach zusammenhängender RaumMengenfunktionZusammenhängender GraphCASE <Informatik>Funktion <Mathematik>MengensystemProzess <Informatik>Kontextbezogenes SystemStatistikKonvexe MengeZentrische StreckungBitrateHyperbelverfahrenSystemaufrufSchedulingAbstraktionsebeneThreadKoroutineAnalytische FortsetzungBimodulWarteschlangeSchreiben <Datenverarbeitung>Mailing-ListeBootenSynchronisierungParallele SchnittstelleDigitaltechnikÄhnlichkeitsgeometrie
19:40
DifferenteSingularität <Mathematik>CoprozessorMultiplikationsoperatort-TestProgrammbibliothekElektronische PublikationOrtsoperatorGewicht <Ausgleichsrechnung>Einfach zusammenhängender RaumDruckverlaufAnwendungsspezifischer ProzessorBimodulHilfesystemLaufzeitfehlerMehrprozessorsystemDokumentenserverDifferenteDigitaltechnikComputeranimation
20:30
GeradeVorlesung/Konferenz
21:06
Googol
Transkript: Englisch(automatisch erzeugt)
00:15
Next up is Benoit Chien-Woo, he's, did I correct, pronounce that correctly?
00:21
Okay, he is best known for being the author of G-Unicorn, but today he's going to talk about how he ported the Go concurrency model to Python. So please give a warm round of applause to Benoit.
00:42
Thank you everybody, I'm pleased to meet you and to be there today. I will present you my little journey in concurrent programming with Python and how I finished by experimenting to port the Go concurrency model in Python and the library I wrote for that.
01:01
So just quickly to introduce myself, I'm Jean, you can offer with the web server in Python. I'm also a CowsDB core committer and member of the PMC, which is the project management group of the CowsDB. I'm a Python Foundation fellow member, and I'm doing a process for a living. So first, what is concurrent programming?
01:22
Basically, concurrent programming is a way to run function simultaneously, but it doesn't mean formally in parallel, it doesn't have to be in parallel, it could be just running your function execute code simultaneously by sharing your process in multi-type slice
01:41
on which you will run from time to time a part of the code you want to execute. Or you can run it on a multi-CPUs or machines, and then it can be run in parallel in that case. You have mostly two models of concurrency. The first one is a char and memory, which is easier for the programmer, he doesn't have
02:03
to care about variable, he can use global variable, variable have passed two functions, and virtual machine, the runtime is managing it for him. It can be problematic, this is not a more efficient way to run your program concurrently,
02:21
because the runtime has to care about locks. When you want to access, for example, to global variable, the system will have to ask for the latest step of the variable, and then you will have to lock to runtime, check the variable, and pass it to the function, back and forth. And Python is not really efficient for that.
02:43
And the other way is message passing, so you have two models, you have the actor model, and the CSP model, I won't detail it, same, today this is mostly a way to say my function, my code is running independently in a part of my process, on the different
03:03
process and different machine, and I'm passing to them data via message passings, so they can discuss between themselves and exchange data. Erlang and Scala are very well known to handle them.
03:22
So nowadays in Python, you have different way to handle that, you have Freeway in the standard library, which is basically just an improved select, select allows you to wait on the read and write events on your circuit, I think it allows you to do that, but you
03:46
can register a function that will handle some code when an event happens. You have a more primitive way, which are threads in Python, I will come back later on that, but you have many problems with thread in Python, it allows you to run concurrently,
04:04
but doesn't run parallel, and are running consequently. You have multi-processing that allows you to launch code on different OS processes, and that have the same API threads, and on top of them, you can use a shooter, which
04:21
are now in Python 3, but are available as an extension, Python 2, and that allows you to execute code using a thread of pool, or thread of process, on top of multi-processing of threads, and you can execute your code on one thread or one process, and the result when you come back later in the future, you recover the result of the exceptions
04:44
you had. And you can also put some callback on that, on the results. You have Asyncio, which is a modern way in Python 3, which allows you to do evented and asynchronous programming, it basically allows you to yield to or yield from a generator,
05:04
and not really like Asyncio, because it's like you are doing message passing, because you can send to an iterator, or getting back some reading from an iterator, but this is not message passing, and this is only running on one frame in your threads, so it
05:22
just interrupts your code the time you get the data from an iterator or to an iterator, and then you come back on the code, so you can't really run mid-tips code concurrently, you just interrupt from time to time, and it's also running on one frame only.
05:42
You can also use external library, you have too many groups of external library, the one that are implicitly yielding, running asynchronous code, blocking code in background, you don't have to care about how it's run the code, it's hidden, you are using, you are developing
06:02
your Python code like you are usually do, each function are blockings, you pass a function, get the result, etc, etc, and the code is run in an event loop. gevent, eventlet, evergreen are good library for that, and they are all based on greenlets, they just are using different event loops for gevent, this is libevent today, and
06:25
evergreen is using pylibuv, and pyuv. The other group is event library twisted, like twisted, which is mostly working like asyncio, at least, it inspired a lot of asyncio, and this is, for me this is like
06:45
having nochi and somehow on Python, using spakety code and waiting for callback, passing for callback, eventually deferring some code for later, but this is not what I really wanted, and all this library, asyncio, gevent, eventlet, evergreen, twisted, all are based
07:06
on event loops, mostly targeting IO input output, so files, events, and sockets, events, with some options to use the time where your event loop is yielding to under some
07:22
other code, or to under a timer. So I decided to write a new library for that, to experiment some new code, and basically put in the runtime library, the runtime to python, offset means this is a small and
07:41
virtual complex of the plants, so that it's growing on another plant, and sharing the same at the end, they are both sharing the same at the end, but the other plant have its own life, and this is basically what is offset, this is making the go scheduler on top of python, on top of python VM, I have some goals, sort of, I want a simple
08:06
blocking up here, like gevent and eventlet, but that is not trying to patch in background my standard library, I prefer something explicit, I prefer to see, to know when my library
08:21
is patched, and I prefer to proxy standard library for that. Based on the concurrency model, so it used to be working on python 2.7, and still working on python 2.7 and 2.8, 2.7.8, but I'm mostly focusing today on python 3.4 and pypi, because
08:42
I don't have time to handle all the python today, and you can find it on my repository. The concurrency model is mostly a memory model, it's based on co-routine, each co-routine are a bunch of code that is running on one thread, it's basically a way to split your
09:04
thread in a, to pass function execution in the stack of your threads, and when you want to switch from one code to the other, the system is using one frame in the thread to execute, and go back in your code, etcetera, and it's basically just adding metadata to
09:25
your threads. Each co-routine doesn't know from each other, they are running independently from other, you can't pass global variable, obviously in Go you have global variable, but it's not advised to use them, and this is the same in offset, you can use it because you are in python, but I don't advise you to use them, because you will have again
09:45
concurrency issue. They don't share anything, and channel pipes are the only way to communicate between them, and also a way to unblock your code, because when you are waiting for a channel, the scheduler is able to run again, to say someone I will send to the channel, or to execute another
10:05
code, etcetera, and you can wait on native channels. So yeah, I decided to code that on top of python because it was easier, but it was I have some nightmares, so really I didn't sleep some time to find a way, and so, because
10:26
python has a drawback, and the main drawback is well known, this is the gill, and the main problem of the gill, the gill is here to allow you to make easy code, to have global variable, to easily pass variable to your function, like I said, to do that
10:43
the runtime needs to lock, and it takes time, and also, because of that, one of the OS threads is executed at a time. You may have native threads running in parallel in your systems, but you can only run one thread at a time to execute the code, and then you need to wait for the thread finish
11:04
to execute, to get the results. So, during this time, your system is locked, but it works well for IOBOON things, like accepting one socket, waiting for connections, reading an event on sockets, or reading a file, writing to a file, because all of that is in the background by your OS,
11:26
so using thread for that, and I will use for that, is very fine, and yeah, python has no implementation of the coroutine, and I don't want to try to do that on yield. I don't think it will work anyway.
11:41
So I decided to create some coroutines, to create the coroutine systems. Coroutine will always be executed in the same thread, in the same thread, and blocking call operations will be executed in their own thread, a python thread. To implement coroutine, I'm using Python Fiber.
12:02
Python Fiber is basically a port of continuous on C pythons from Sagul, and I contributed to Python Fiber to also have the same API on top of PyPy, so you can use Python Fiber on top of PyPy using the same API.
12:21
I created a module to do atomic locking, so instead of using mutex, et cetera, on the system, I'm using atomic operations that you can find on GCC and other implementations, and I've created the FFI for that. Everything is abstracted in Prose class, so if you want to use greenlets, you can
12:42
eventually write your own abstraction to build coroutines. So scheduling, so this is the main part of the offset library, there are three entities in the scheduler, the other threads, called F, because they will run in futures, in thread pools, scheduler context, which is called P, because you have one scheduler context,
13:04
which we maintain in thread two, in the main thread, and this is called P because of process, and this is processing your Python VM. And coroutines, which are called T, so how the scheduler works is pretty simple, in fact, at the end.
13:21
You have a process context, which is maintaining a run queue, a run queue of coroutines, and these coroutines are stacked in a queue, and the first one entered will be the first to be out, to be executed. At the time, you have only one coroutine that is running, the one that is put out of
13:42
the run queue, and when the coroutine has stopped to run, it can be placed, put back in the run queue, or it can be another coroutine from the run queue will be executed. Of course, you still have a syscall, a system call to handle any blocking call, like waiting,
14:02
having a timer, or waiting for connection, or reading a file, any blocking operation, mostly IO operations. So you are still in your run queue, and when your coroutine is detected as blocking, and this is done by patching, processing the standard library in the syscall modules,
14:23
in offsets, I'm tagging any blocking calling from the standard library in the syscall modules, and when a function is detected as blocking, the scheduler will put the coroutine out of the run time,
14:43
out of the run queue, and we create a call in the thread pools that will handle the call. So the call, accepting connection, or reading a file, will be handled in the threads, and when you come back, the scheduler recreates the coroutine and puts it back in the run queue,
15:01
and it puts it back on top of the run queue, so it will be executed in priority, and you will get the results in your code later. So by doing that, all your syscalls are able to run in parallel, and you are still able to handle the rest of the code in your run time.
15:22
Some examples. Here is a coroutine example. On the right you have the Python code, on the left you have the Go code, which are pretty similar. So what this code is doing is pretty simple, it's putting hello world five times, and the first time the main function is executed, so here you are seeing some kind of all code,
15:44
because the run function is about to be removed, I just need to commit the code. But right now it's how it works, so you have a coroutine that is saying words, and the main function that is saying hello, and the same function what it's doing is,
16:02
five times, it's printing the string that is passed to it, and it's sleeping during 100 milliseconds, and when it's sleeping, this is a syscall function, and when it's sleeping, the coroutine is put back in the run queue, and the next code will be executed.
16:23
So the first time we say hello, the main function will be put back in the run queue, and the next coroutine, which is a world coroutine, the first coroutine will print hello, will be put in the run queue, then the next coroutine will be executed,
16:42
we say hello, et cetera, et cetera, five times. And yeah, pretty similar to the Go code module of the syntax. Channels, so channels are fully implemented in offsets, so like in Go, you have all the features you have in Go language, you have them in offsets.
17:02
Channels are mostly pipes that connect concurrent coroutine, you can send value in a channel, into a channel from one coroutine to the other, and receive this value in other coroutines. And by doing that, when you are sending a value, the coroutine is put back in the run queue
17:21
to let the chance to another queue, to let the chance to a coroutine to send back the data to this coroutine. When the data, when the coroutine is sending, it will boot back in the run queue, and the next coroutine will be available, okay? And you will be able to read, so by doing that, we are mostly articulating the scheduler
17:42
and unblocking your code by passing message from one coroutine to the other coroutine. Channels can be buffered, that means that you don't have to wait that someone is receiving the first value you are sending, you can send many value at a time before blocking your code. And you can select on between channels.
18:02
Here is a simple example, on the right you have Python, on the right and the left you have Go again, and we are making a channel. Here the goal is to sum a list, and to sum the list more efficiently, we will split in two, and we are splitting the list in two, we are creating two coroutines that will sum this list. To this coroutine, we are passing
18:22
a channel which is created by the make chance function, and when the sum is done in the coroutine, it will send back the result in the channel, and the main coroutine, the main function is waiting for the result of these two coroutines, getting the result of the sums on x and y, and we sum the final result to print the final sum, and the code in Go
18:48
is quite similar. This is an example with buffering, and this is an example with the channel, so the channel is the select function, which allows you to wait on multi-channel,
19:02
and you can wait that someone is able to reset the data you want to send to him, or to wait that someone is able to send you some data that you want to reset, mostly like a select when you are waiting for read and write in your circuits. And this is done by the if send functions here, and you register some events in the
19:24
select function, so you are, here we are receiving a send, and we receive, sorry, and voila, this is a few monetary functions, not timed to date. You have other models implemented, you have sync functions which are atomic locking
19:41
that you can use in your library, in plus channels so you can do some atomic locking, and you have timer, and net and IO modules too, to handle the IO connection to the file and your circuits in an unblocking manner. What I want to do is to rewrite channel using a map,
20:06
this is precedence, spend on different processes, machine, which needs the first one to be done, channel, and make the run time switchable, like in Rust, so you can eventually switch to a run time if you want, or to use a multi-processing run time, or any other run time.
20:25
Any help is appreciated, and you can contribute to the repository. If you have some questions.
20:41
Yeah, if you have questions, please line up at the microphones, or raise your hand. No? No questions yet? Okay. Thanks again, Benoit.