Message-passing concurrency for Python

Video in TIB AV-Portal: Message-passing concurrency for Python

Formal Metadata

Message-passing concurrency for Python
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place

Content Metadata

Subject Area
Sarah Mount - Message-passing concurrency for Python Concurrency and parallelism in Python are always hot topics. This talk will look the variety of forms of concurrency and parallelism. In particular this talk will give an overview of various forms of message-passing concurrency which have become popular in languages like Scala and Go. A Python library called python-csp which implements similar ideas in a Pythonic way will be introduced and we will look at how this style of programming can be used to avoid deadlocks, race hazards and "callback hell". ----- Early Python versions had a threading library to perform concurrency over operating system threads, Python version 2.6 introduced the multiprocessing library and Python 3.2 has introduced a futures library for asynchronous tasks. In addition to the modules in the standard library a number of packages such as gevent exist on PyPI to implement concurrency with "green threads". This talk will look the variety of forms of concurrency and parallelism. When are the different libraries useful and how does their performance compare? Why do programmers want to "remove the GIL" and why is it so hard to do? In particular this talk will give an overview of various forms of message-passing concurrency which have become popular in languages like Scala and Go. A Python library called python-csp which implements similar ideas in a Pythonic way will be introduced and we will look at how this style of programming can be used to avoid deadlocks, race hazards and "callback hell".
Keywords EuroPython Conference EP 2014 EuroPython 2014
Message passing Concurrency (computer science) Bit Message passing
Slide rule Group action Ultraviolet photoelectron spectroscopy Single-board computer Mereology Coprocessor Food energy Computer programming Twitter Revision control Digital photography Moore's law Programmer (hardware) Causality Whiteboard Different (Kate Ryan album) Befehlsprozessor Core dump Computer hardware Form factor (electronics) Computing platform Default (computer science) Multiplication Software developer Projective plane Plastikkarte System call Sheaf (mathematics) Duality (mathematics) Computing platform Whiteboard Quicksort Multi-core processor Library (computing)
Pointer (computer programming) Message passing Concurrency (computer science) Code Sheaf (mathematics) Mathematical analysis Quicksort Whiteboard Demoscene Computer programming Formal language
Mountain pass View (database) Programmable read-only memory Mereology Computer programming Mathematics Graphical user interface Semiconductor memory Synchronization Process (computing) Hazard (2005 film) Imperative programming Abstraction Physical system View (database) Process calculus Bit Deadlock Variable (mathematics) Formal language Annulus (mathematics) Message passing Process (computing) Telecommunication Quicksort Router (computing) Software engineering Dependent and independent variables Syntaxbaum 3 (number) Maxima and minima Event horizon Computer Emulation Sequence Term (mathematics) Deadlock Energy level Message passing Form (programming) Multiplication Hazard (2005 film) Run time (program lifecycle phase) Code Transputer Line (geometry) Pointer (computer programming) Event horizon Error message System on a chip Synchronization Formal grammar Form (programming) Computer-assisted translation Library (computing)
Manufacturing execution system Mass flow rate Thread (computing) Concurrency (computer science) Code Multiplication sign View (database) Decision theory Sheaf (mathematics) Set (mathematics) Parameter (computer programming) Run-time system Mereology Arm Computer programming Variance Formal language Programmer (hardware) Blog Different (Kate Ryan album) Object (grammar) Hazard (2005 film) Process (computing) Endliche Modelltheorie Fiber (mathematics) God Physical system Touchscreen Building Moment (mathematics) Parallel port Bit Price index Deadlock Sequence Benchmark Demoscene Message passing Process (computing) Auditory masking Chain Computer science Quicksort Writing Reading (process) Point (geometry) Reading (process) Slide rule Functional (mathematics) Implementation Letterpress printing Maxima and minima 3 (number) Similarity (geometry) Student's t-test Law of large numbers Scalability Emulation Smith chart Goodness of fit Arithmetic mean Deadlock String (computer science) Operating system Ranking Quicksort Mathematical optimization Window Execution unit Just-in-Time-Compiler Hazard (2005 film) Run time (program lifecycle phase) Server (computing) Weight Projective plane Basis <Mathematik> Line (geometry) Error message Personal digital assistant Video game Abstraction Library (computing)
Implementation Thread (computing) Concurrency (computer science) Java applet Multiplication sign Time zone 3 (number) Mereology Special unitary group Computer programming Emulation Formal language Word Benchmark Operating system Metropolitan area network Raw image format Code Formal language Inclusion map Number Message passing Sample (statistics) 4 (number) Spherical cap Dew point Compilation album Quicksort Design of experiments
Axiom of choice Concurrency (computer science) Variety (linguistics) Code Multiplication sign Decision theory Source code Sheaf (mathematics) 1 (number) Insertion loss Mass Special unitary group Event horizon Computer Computer programming Formal language Writing Wave Term (mathematics) Synchronization Moving average Selectivity (electronic) Gamma function Message passing Dean number Chi-squared distribution Block (periodic table) Water vapor Bit Benchmark Formal language Inclusion map Type theory Number Message passing Exterior algebra Process (computing) Sheaf (mathematics) Spherical cap Order (biology) Formal grammar Data Encryption Standard Summierbarkeit Quicksort Reading (process) Library (computing)
Axiom of choice Meta element Mass flow rate Java applet Graph (mathematics) Mountain pass Programmable read-only memory Demo (music) Execution unit Parsing Bit rate Cohen's kappa Run-time system Computer programming Storage area network Data model Connected space Pointer (computer programming) Uniformer Raum Different (Kate Ryan album) Computer configuration Estimation Social class Metropolitan area network Programming paradigm Executive information system Software developer Interior (topology) Shared memory Electronic mailing list Milner, Robin Amsterdam Ordnance Datum Bit Trigonometric functions Formal language Inflection point Message passing Process (computing) BCH code Theorem Quicksort Optical disc drive Reading (process) Spacetime Wide area network Programming paradigm Slide rule Auftragssprache Syntaxbaum Maxima and minima Dreizehn Parallel computing Computer Computer icon Number Chain Goodness of fit Term (mathematics) Operating system Conditional-access module Surjective function Data type Graph (mathematics) Run time (program lifecycle phase) Chemical equation Computer network Core dump Coma Berenices Coroutine TDMA Power (physics) Mathematics Error message Medical imaging Software Integrated development environment Integral domain Estimation Personal digital assistant Network topology Infinite conjugacy class property Library (computing) Code Weight Multiplication sign Mathematical singularity Combinational logic Arm Java remote method invocation Formal language Mathematics Cuboid Process (computing) Injektivität Thread (computing) Type theory Array data structure Condition number Right angle Bounded variation Physical system Game controller Mapping Link (knot theory) Object-oriented analysis and design Virtual machine Electronic mailing list Emulation Sequence Aeroelasticity Message passing Quicksort Mobile Web Axiom of choice Java applet Scherbeanspruchung Human migration Number Intrusion detection system Lie group Sheaf (mathematics) International Date Line Form (programming)
Implementation Constructor (object-oriented programming) Maxima and minima Counting Protein Formal language Power (physics) Regular graph Dedekind cut Different (Kate Ryan album) Core dump Moving average Summierbarkeit Message passing Library (computing) Surjective function Metropolitan area network Continuum hypothesis Satellite Software developer Projective plane Parallel port Coma Berenices Bit Stack (abstract data type) Lattice (order) Message passing Keilförmige Anordnung Process (computing) Spherical cap Interpreter (computing) Summierbarkeit Queue (abstract data type) Quicksort Whiteboard Simulation Physical system Library (computing) Design of experiments
Point (geometry) Software engineering Slide rule Implementation Concurrency (computer science) Java applet Open set Event horizon Formal language Finite element method Semiconductor memory Different (Kate Ryan album) Term (mathematics) Selectivity (electronic) Physical system Identity management File format Software developer Bit Line (geometry) Message passing Kernel (computing) Process (computing) Exterior algebra Finite difference Order (biology) Quicksort Library (computing)
Concurrency (computer science) Multiplication sign Source code Scanning tunneling microscope Insertion loss Solid geometry Thermische Zustandsgleichung Computer programming Programmer (hardware) Mehrprozessorsystem Semiconductor memory Different (Kate Ryan album) Forest Single-precision floating-point format Process (computing) Enterprise architecture Moment (mathematics) Shared memory Amsterdam Ordnance Datum Mass Bit Mereology Instance (computer science) Message passing Arithmetic mean Process (computing) Right angle Quicksort Point (geometry) Reading (process) Implementation Connectivity (graph theory) Syntaxbaum Number Revision control Term (mathematics) Natural number Energy level Data structure Task (computing) Addition Multiplication Standard deviation Server (computing) Consistency Cartesian coordinate system System call Particle system Word Personal digital assistant Interpreter (computing) Abstraction Library (computing)
thank you very much so this is quite a broad overview of the talk is not so specifically about my own work and that is small so the idea in message-passing concurrency a little bit about how they used currently in Python about how they might be used in the future because of
course there's some great opportunities with a 3 so why multiple the reason for worrying about all of this concurrent stuff quite so much is largely today with the rise of multiple probably most of you have phonemes in your pockets that are that have quad-core processors and I'm sure I don't need to tell you about what Moore's law is that trend is going to continue and speed ups in programs and will start to be gained by using more cause more efficiently other than simply as buying a faster processor that's
really important what's also important is as hardware becomes cheaper and people do more interesting things with its different sorts of platforms are becoming available so that's all that you can see on this slide is called the parallel aboard and it's by a company called out of fever so this was a Kickstarter project and I'm afraid I have to go on with me but this is about the size of a credit card says the the same sort of form factors rather part and the aim of this company is to do for supercomputing what the Raspberry Pi is doing this to the general development so to make it cheaper to make it more readily available on the desktop to all sorts of people whether they be hobbyists of scientists or whatever so the interesting thing about this board is the it's it's kind of like a large group I've got the sort of features you'd expect of a of a single board computer this is a dual call on things chip but the the really interesting thing is this chip that other people make themselves which is a 16 core many core coprocessor and the idea here is if you want to speed up your program then use this typewriter coprocessor to help you do that so they got a 16 core version of that chip and 60 call 64 call version of that so that made Our multi processing very cheap very fast and also very low energy and that's really not that talented programmers support so the default libraries role in C. and if you want to be experimenting with a
large amount of data experimenting with making it difficult complex scientific analysis a lot faster probably going through the see workflow of my thing very carefully crafted code that doesn't blow up scenes you run it and compiling its and all those other things is is not really much fun people use this sort of thing often want to explore and explore
their data and explore their programs and that's why dynamic languages really come so to use Python on a board like that would be really fantastic but can we did it come at it natively in really nicely without so the message passing concurrency is not a
new idea it's a very very old idea and it came from 2 lines of work 1 line of work right practical has this old a trick called in most transputer and the idea was that you would have many of these strands features that like the CPE and you put them in a great and you would want them all together so this is like very early form of malt multiple the other line of work was very theoretical so CST communicating sequential processes or 1 way of mathematically formalizing concurrent and parallel processing there are many other ways and there will also the reasonably similar in terms of the ideas that are in them and these 2 things went together but they never became popular because we have threats instead so I'm not going to go through all of this sort of mathematical details of CST but I want to give you a flavor of what it's all about and why it might be important so in the process algebra view of the world computation has is made up of imperative commands that run sequentially and you have lots of those but you have them in processes which 1 concurrently and can communicate with each other so you have different processes which will learning the same that there is an imperative programs and they they share many variables or any data if they want communicate with each other will synchronize with each other and they need a special way of doing that and a common way of doing it but not the only way is to pass messages 5 channels so the way to think of this as a sort of broader used to think about Unix processes communication by part she different processes or running in parallel and if they want to communicate they do so by part because they don't share any memory with each other but on sort of lighting of the details that it is here is an abstract idea is a mathematical formalism so when I talk about CSP process but not necessarily operating system process and when C is he talks about events and synchronization that's not necessarily like this sort of events that you would see when you're programming give but it might be it might be an
ongoing talk about that in a bit more and detail later on so why would anyone have vessel so that this stuff well if you work with friends and you work with locks you know they're really tough to to get right practices important and when you're dealing with really low level things like locks and pointer arithmetic in all of this stuff it's hot and looking is hard and deadlocks and starvation inmates hazards unrelated things a heart and they don't sit well with also the very high level Pythonic view of the world which is to
use the abstractions of the language to make our life really simple and hide a lot of the really hot things which is a good way to the competing it's what computer science is all about so the good thing about message-passing concurrency is that message passing removes some of these possible faults that you can have with locking so you can't have erased hazards with message-passing concurrency trade have deadlock still say that you know is not perfect and hopefully if you have a big message passing language reliably than a lot of the difficult stuff is hidden away for you in the runtime system perhaps or in a library so all the cool kids that a message passing at the moment it's an idea come back around because of multiple and other things so the these the the next few slides some examples of message passing concurrency in different languages so I've already mentioned you next and pipes synopsis of the really simple part of idea of message passing and hopefully 1 most people familiar with so this is a simple so of hello world in God and the syntax here is a little maybe a little unfamiliar supply from people the idea is that here on this line but we're creating a chance to create a new channel that we can pass messages around with like all points and that'll be bi-directional like like a pipe is on the command line this thing that looks like a function is a function but this special the special keyword go and finds of it means that it's also guarantees so only seeing is like a currency it's this little light weight kind of thread is not a threat created by the operating system so it's much cheaper to create a reading or destroyed only thing up an operating system thread and then what we're doing on this line with this so the funny syntax is we're sending the string Hello World down this channel and my channel anemic and among the scalability straightaway system be running in the background the ball the ball around program so the sorts of funny bits the syntax of the fiber really conveyed but these are these ideas in languages and you'll see a lot of weird syntax the CSP syntax during this sort of thing is a playing an exclamation mark or to receive something down shower question mark so it a kind of hasn't got better through the ages in my view comes from and so on this line here what we're doing is receiving something down this channel from this channel and whatever we see with this printing out so this is a simple sort of how world in that language model must is another new language does similar things so we've got the same so that the idea here we've got a channel being created here we got a background process running here where we're doing some sending them we're receiving this value and printing about what slightly different with the mask is that if you are familiar with working with pi Unix pipes and see know that when you create unix pipe and see what the operating system gives you is 2 ends of the the park the ascending and in the receiving end and that's what's happening here with must we got the sending end of this channel from we got receiving and this channel and the idea there is to prevent you from doing silly things like sending down the receiving end of receiving down the sending and so you the programmer in must have to decide ahead of time what where in my program i don't want to send down this channel and where my going to want to see on this it's something we need to think about and at compile time so scholar being a JVM languages taking up the the whole screen with its revised and ready is it if you didn't have that this is exactly the same sort of thing so we have an answer which is similar to the sort of background processes that we were talking about and carry chains and so on and so we got this slide practices here was sending so this is really using CSP syntax with plain and here where receiving something and printing it and Python yes please applied consistently is mine library and as some of you in this room have been a really generous contribution to detect the stuff on over that In previous Europe items so this is an attempt to doing something like this in a point on it waits for Python but built as an add-on to the language as long as a as a library so here we've got CCS the processes so we're not saying in the code here how those processes all students right find whether they're currently scenes of threads or operating system processes Rigotti got processes here that can run in parallel with the decorators we got channels that can be shared between them and we can read and write with those channels and again we're just sending HelloWorld printing it out and then on this line so this is a much more so the CSQ way of doing things and perhaps the other the other examples we're saying well we're gonna take these 2 processes and 1 them in parallel and start them all and if we have a huge program with many processes we might decide well 1 them all in parallel or we might 1 of the then 1 a few moments sequence whatever we whatever we wish so we've got quite a lot of flexibility they're about how our prior parameters so as to the put together so the last example is biased to mind some joules and what he was looking at that was really interesting which was can we build a language like this that has this to guy will must stop channels and concurrency is only on Python toolchain to companies the the and Technology that pipe writing developed to do this so this is exactly the same Hello World example that we've got channels here we're gonna send down that channel Hello world and then we've got some sort of an unusual received the syntax there too received something from the channel and printed out and this function here is being run in the background as a sort of asynchronous code so that's a really nice project and it's a really nice way of working I think some of his is some was a vibrated student but I think it's a testament to the the yeah good engineering of the pipe writing then an undergraduate student can produce a working language like that in the small amounts of time for a for a final year project so
all not gonna talk in greater detail about optimization and speed and efficiency in those sorts of basis of issues that I just wanted to show you quickly uh 1 of sounds benchmarks which shows quite nicely that with the case with the trace and it now aligned perform well compared to other other languages of this sort so Segovia here and not camp which is a set of descendant of all can both compiled languages and now and compares pretty pretty reasonably well for them and this is only a small benchmarks so we perhaps should take it as the gospel but it's a good indication of this sort of way of working might might be positive on the other hand I haven't got here is saying benchmark with Python CSP but we looked at similar things Python CSP and that was engineered very differently so and I'll talk a little bit more about the the student design decisions that message passing concurrency implement might take in the next section of the talk but what we found Python CSP is our implementations of channels were very very slow very
very slow so you wouldn't expect so I wouldn't expect a Python implementation of this to be as fast as something like go that's compiled and has a lot of uh so a lot of engineering going into these features but and perhaps wouldn't expect I would hope that message passing would not be the bottleneck in any program and what we actually found was that of compliance is incredibly fast designed exactly for this but compared to other sorts of interpreted languages we looked at after the Javier thinking maybe united because I've JCS season built on Java threads Java threads OK but you know the operating system threads maybe you you get something like performance and we actually got sort of a hundred or so times worse and and didn't didn't do I wanna talk so there some lessons
learned and but there's some interesting stories about part of take away this is that actually it's very difficult to engineer that kind of that kind of performance if you're starting from an interpreted language that hasn't been built with this to the concurrency in mind so this
next section of the talk is it all about the source of varieties of message passing concurrency and that can be can be created and the different
decisions that implements it would have to make that if they were going to implement something like this in Python so 1 choices synchronous channels vs asynchronous channels so these since the way of thinking and in the so the process out of a way of thinking about so the a mathematical formalism and the idea is that all channels block on reading and writing so you don't move forward to the computation and to your reader your writers finish and some people including me think that the nice thing about this is is then very easy to understand what your program steering and reason about because you know exactly in Warsaw order everything's things going to happen you know this this piece of code will not be forward to let's finish this read and understand the best and then all the other things that are waiting on it will be able to move forward as well asynchronous channels there are quite common as well in different languages and some people have suggested that a bit faster and sometimes that seems to be true and certainly the benchmark shady before that showed that in that particular benchmark as sums asynchronous channels for now aligned that with a bit faster than his synchronous ones if he did have synchronous channels that you need to think a little bit about avoiding some of the common problems that people have concurrency like starvation and you don't necessarily want a process to block for a ride on time if it doesn't have to sometimes you might have to sometimes it might be waiting on long computation but if you don't have to to block you would probably prefer not to so a common feature of message passing languages and libraries is some way of selecting the next ready the events to process so if we're thinking in terms of events being channels message passing down channels if you have a loss of channels that you're waiting on you want to read from so for example if you've got a solar mass produced type problem all work a farmer type problem then you might say well give me the 1 that's ready 1st and that's schools alternation in sort of uh off-camera old-fashioned language or selection and more generally sequence select for me the channel is ready to and I usually if you're implementing that selection you do not rather the carefully because although you might want to select the next most of the next ready channels reform if you got a channel was always ready to reform and some of the the the the taking a little while don't want those other channels to not be processed so usually there's a little bit of that work goes into that to avoid starvation and
decent good light so that's 1 if the synchronous or asynchronous channels so you might have buffet on behalf of channels another if are your channels bi-directional the unidirectional so we saw in in must you get what you guessing in Unix C which is read and on the right and the the channel I must point to a common way of firm working with channels to avoid Miss avoid some mistakes in your in your code if you look at the James EST library which is a very nice slide because it's been engineered very well with a lot of thought going into its correctness the JCS the libraries by Java-like and job people don't mind having thousands of classes to choose from and large amounts of documentation and they don't mind pressing control space in their IDE and getting a long long long list of things and so JCS feature works without that's paradigm and he has lots of different channel types that all classes i haven't listed them all because the slightest small um but so you can have things like to have one-to-one channel that has 1 reader 1 my surprises attached to it at any 1 time you any-to-any channel that has any number attached to them at any time and so on and then you always have the reader the writer and that channel where you are and the idea here is to use the type checker to design at the loss of potential falls out there might creep into your code so but not for Java because it fits well with the to the job where the things it's what job of people would expect so when I write Python CSP and design that I made the channels any-to-any channel but I didn't get people to read and analyze tend I let them shoot themselves in the foot because it seems to me to be a bit more of the dynamic way of doing things and a bit more Pythonic but not full this not foolproof so there is there is there a couple of different design choices another is most in mobile channels so this is something that wasn't built into CSP originally from but it was built into different prices out of the cold called the pi-calculus Robin Milner and then the candidates that the team can University USA the took over but the development of all kinds created of camp on which to the fused together the pi-calculus in the US way of doing things so a mobile channel it's a channel that can be sent down another channel to a different prices and the idea of doing that is that you can think of your message passing program as being like a graph well the nodes of the graph by your processes and the box between processes your challenge that link those prices together at 1 time you may wish to change the topology of the graph and changes so that's the good reasons why you might do that 1 might be a bit to do with load-balancing if you have a computation that splits among a lot of processes you might find some of them are more active than others and you might decide to change the light balance between them which might also mean changing the topology of the graph and he's reporting their data that human and aggregating the data and so forth that's 1 reason another reason might be that you might be running these processes across the network to you might not only be working with 1 machine you might have some prices farmed out to another machine on your network and then you might have a few slides uh latency or we might have issues like network failure or whatever and not might make you think well during the running of my price at my idea might computation of like to change its apology to make the most efficient uses that network of machines so that's 1 reason so this is this is this leads us to the territories mobile channels can be great if you can if you can use them really well and you've got a good use case for them if you're in a situation where you you need to shut down there network and graph of running concurrent processes then you need to notify each node in in your class but that it needs to shut down and so during that safely is quite an important thing to do so in this the message passing world 1 way to do this is called poisoning which means that you tell a channel of the node that decides to shut everything down effectively that tells age channel all of its channels that it knows about that they need to start shutting down and they need to propagate the message that this program is going to hold and this is called poisoning so you poisonous channel and the idea is that it poisons the well of the whole program and each process shut itself down safely and not something that takes a little bit of care in a little bit of good engineering because you need to think well you know if I'm a process and you'll processes and I say I I want you to but I have an ongoing dialog with unit which then but that means the happening of Isildur Fico myself you might know what to do don't know united icon so here other the other we talked about channels which was probability difference of channels the other thing is how to represent the processes and there are a lot of different choices that the so in some languages about 1 CFP process is 1 protein and and that makes sense in some paradigm so I think this is kind of and JS works and that leads to very fast message passing because in the runtime system always processes share memory there will always in the same operating system thread say they can do a lot of things right very fast and they can pass messages down channels by very fast but then it's hard to take advantage of multicore fuel in 1 thread and you could have a one-to-one mapping where 1 CSP processes 1 OS Thread that gives you much slower message-passing because whoever implements that does have to deal with locking in all those later she's but then you can start taking advantage of the features the operating system has you could make 1 CSP process 1 I want friends and that's a really good choice if you thinking about migration processes around the network and running your code on more than 1 computer once so so the MPI some if you're into MPI or you can have some sort of multiplexed variation of all of those options so you have some prices yes prices the cohesines but that live inside and right and there are other CS the prices is that live inside another arrest thread but a really close readings themselves and all sorts of combinations there and this is really where at why Python CST was not as fast as we hoped because we were looking at taking advantage of multicore in the network so we're using the SOS one-to-one mappings which are not the best in
terms of speed so and that control huge amount longer pictures hopefully and we can have a good discussion that all wanted say a little bit
about message passing in python so there are lots of world but we don't have all that Python is not a
message passing language in the way that go is and Rusty isn't and accurate and all those other things python does sort of have a a lot of these ideas built into it he 1st to and sometimes in libraries sometimes in different implementations of the interpreter sometimes in all sorts of other ways so it's really please looking through the year Python scheduled to find that actually there are a lot of different talks in this conference that in some way have quite a lot to do with the ideas that have been talking about that so not necessarily straightforward implementations of message passing in the way Python CSP was but they take on some of those ideas either by implementing coatings on using coatings for using channels and and so on so in a sense message passing from Python is already here In this 2nd and and also of course In Python 3 . 4 we have proteins built in so that perhaps a big opportunity there to think about building these things in the a to the core language so if you're interested in this stuff than I would certainly be interested in talking to you on mind next step to this point and see if is considered in advance for the last 3 years while my day job is some take me to do different things but I think that the parallel the board will be coming out this summer side about a project working on that this summer's starting so the middle and we'll be looking at nite and hopefully efficient way ways of using Python for the power that ideally would use message passing in some way that we'll see how that works the duties rather constant on the employment CSV meeting behind into development sums language now lined up will be continuing so albeit the pipeline sprint this Saturday doing a little bit more on that and if you are interested in this stuff then please the common catchment
sometimes thank you very much the
desire to provide the user called so and I have seen that here we are always in reliance over the system layer for in the threats orders reviews see because in many ways in terms of employment processes become better 1 1 of the formation of that besides the us to use some of those other yeah so that those there is an interesting developments in the Open MPI library a few years ago when they found that their message passing was a little bit slower than they would like and Open MPI people tend to work on Linux slide the Linux kernel brought in a new way of doing that which is called cross memory attack and the idea crossed memory attached I think it is only in the next thing but the idea is is that we've got the different operating system processes and then rather than saying while I'm doing what you would do in applied which is the copy the memory you have to keep the memory in 1 place and in the past around and as so that handles that memory between crisis so that's much quicker way of doing it was built specifically for MPI but would possibly via a really good way forward for for any other implementation like an implementation point here is that interesting work the I wanted to ask what the patents the the library provides that stage event doesn't apart from the simulate and well so the question is a different API and identity it's simpler or not and applied consistently so they started because I wanted a line of Paul I wanted something like this in Python that the only things that are available would direct ports of the Java J. CSP language so that the idea of this is that it's really it's much more similar to the CSP way of doing things and than anything else so what's it provided provides processes which can be which can be various sorts of processes and it provides channels it provides selection alternation and it provides a small library built-in processes that might be useful so that the reason for that is that the way that CSP people tend to think about this is that the more concurrency have the better so rather than saying
well I've got my nice sequential program how do I say this is up to make it efficient so concurrent of sensible or whatever it is they say well you know it makes everything you possibly can concurrent so they tend to have libraries of the processes that do things like have 2 channels the damage to read 2 numbers from those channels and add them together and send them out down the 3rd channel so a process that dust those additions and enterprise that just those all the other method things so there support for that way of working if not way of working is something that's interesting to and I suspect that that way of working is properly and interesting to people who are interested in CSP for its side because it's not terribly pragmatic way of working so I know the answer to your question radiated by consisting of 2 is also the basic things he expects the message passing libraries it's just a matter of how it implements them and how well and I think it would probably score about 5 out of 10 to the heart of the moment but hopefully we'll get so quick question of consistency and multiple processes to this is a currently implemented with most of this disease something like multiple processes and multi-person how how we doing the message passing across processes is a term used people to serial so so that we've got 2 different ways of doing it 1 1 with threats and 1 with processes that I these multiprocessing and this year I stopped for and that's the thing so when the those words yeah so the idea that was that most multiprocessing
is really built for a particular way of working in a has a lot of internal kind but that supports that way of working that isn't so useful if you want to do things the CSP way for example I think when you when you 4 on a processing multiprocessing that crisis also forms a watchdog thread for that went to the In a CSP like we don't need that so the idea was to be just a little tiny bit more efficient by not having those multiprocessing intervals and I think in reality that if you compared eventually python CSP using multiprocessing along without you probably wouldn't find a vast amount of difference so you could easily do you might see these things using after processing because you got quite involved in in the in the library at so in that sense Python has some of these things build already was used for your school memory co-opted into the room to that's the the prices and then chemical and land use the solving from what was you must also supported at the end if think of it as a very good yeah absolutely convinced the means something like social members problem for all triples because you still have the question than who wins the reference task so some kind of library that could we could have a mutable data structures and when New York convention of that that the receiving channel the message is being passed so it's responsible for the the destruction and then you can that then you do some reliable message-passing between between channels yeah at the economics losses standards are say sorry the but the version of Python CSP that uses OS prices uses so the unique shared-memory trying things and that's still quite slow partly because I think shared memory is more efficient when your copying a large amounts of data or copying data many times through shared memory is not really intended to so the 1 off sentences see which is is kind of what you're doing when you do a message passing CST so it's not really the right the right tool for the job where something like cross memory attach might be the interest so where see future also message passing particle would be something like 1 for rather something like that the same guy you work so so wasting time on sort of does this kind of thing already in the fall the Iron processes that you want to run in the background and for that particularly in the case which is which is great but for the more general computational thing here be interesting to see message passing used together with Python 3 . 4 coating and see how that kind of thing I would be really interesting experiment the and really interesting to benchmark graphs and see if it could get really fast and useful for for those that have you know that this as it were the ordinary program above and some got particularly use case like like background so the point when I asked like would that be any the whole so the same pipeline ASEAN what is a is a fantastic piece of work and as I understand it the purpose of pipeline SCM is to make the call interpreter but component in a sense which means that you can then build these high-level sources concurrency that the program would see on top of that so I wouldn't expect I would expect that I hope my ICM is really successful I wouldn't expect that that would mean that ordinary pi program is all Python programmers rather use STM another applications thing that's kind of the wrong level of abstraction for the for the program so I think building message passing on top of pipeline would be would be really interesting was the question already mentioned sectors forests that you look at at the tools cool kids are insecticides that will tell tossed instances and journals in 1 of was so I didn't quite catch a solid sectors licenses of what the nature of the implementation of that holds supplies of data already what it's called genes and journals the message of single what's the strength of that you look into the yes yes they certainly my understanding is that status has a different implementation of the Python interpreter so it is not quite the ponds from what out which is why it's to loss actually it is the Poisson with some of the she's this is the accuracy and trying to get the last word by the company was once yes CS said it is here that also really interesting piece of work yeah you you are the


  560 ms - page object


AV-Portal 3.20.1 (bea96f1033d39fbe77f82542458e108105398441)