Running Python code in parallel and asynchronously

Video in TIB AV-Portal: Running Python code in parallel and asynchronously

Formal Metadata

Running Python code in parallel and asynchronously
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Running Python code in parallel and asynchronously [EuroPython 2017 - Talk - 2017-07-11 - Anfiteatro 2] [Rimini, Italy] My outline will be: 1) What does it mean to run code in parallel in Python? How does it differ from concurrency? Can they be applied at the some time? 2) GIL and why it complicates parallelism in Python (CPython), but only to some extent. 3) Difference between a thread and a process from the OS point of view. 4) When parallelism in Python is useful and when to avoid it. 5) Description of how to achieve parallel execution in CPython and how to do it properly. 6) Possible traps when using parallel programming in Python. 7) What happens if the code runs both in parallel and asynchronously? 8) Is it really beneficial? 9) How such execution can be achieved? As the outline shows I will focus on the parallel part as it is an important topic in our current time of multicore processors and multiprocessor systems. The topic has been discussed a lot of times but mainly from the scientific point of view, where it's been used for speeding up calulcations time. I will not go into these use cases (e.g. using MPI) but rather discuss it from web development point of view (e.g. multi worker applications)
Group action Synchronization Right angle Bit Mereology Computer programming Thread (computing)
Arithmetic mean Multiplication Antimatter Multiplication sign Antimatter Computer science
Axiom of choice Polar coordinate system Scheduling (computing) Injektivität Thread (computing) Serial port Code Multiplication sign Design by contract Mehrprozessorsystem Variable (mathematics) Neuroinformatik Web 2.0 Facebook Virtual reality Radio-frequency identification Different (Kate Ryan album) Object (grammar) Interpreter (computing) Befehlsprozessor Single-precision floating-point format Core dump Software framework Process (computing) Office suite Local ring Information security Multiplication Physical system Point cloud Service (economics) Antimatter Concurrency (computer science) Software developer Fitness function Shared memory Internet service provider Cloud computing Virtualization Variable (mathematics) Sequence Thread (computing) Process (computing) Befehlsprozessor Internet service provider Software framework Abelian category Physical system Resultant Spacetime Bytecode Software developer Parallel computing Mass Protein Rule of inference Product (business) Operator (mathematics) Uniqueness quantification Integrated development environment Spacetime Content delivery network Address space Multiplication Server (computing) Parallel computing Projective plane Content (media) Memory management Code Computer network Core dump Stack (abstract data type) Software Integrated development environment Interpreter (computing) Object (grammar) Local ring Address space
Email Scheduling (computing) Group action Multiplication sign Execution unit Set (mathematics) Function (mathematics) Mereology Computer programming Software maintenance Neuroinformatik Subset Medical imaging Synchronization Different (Kate Ryan album) Semiconductor memory Befehlsprozessor Shared memory Process (computing) Pixel Multiplication Area Concurrency (computer science) Software developer Complex (psychology) Fitness function Shared memory Infinity Thread (computing) Type theory Category of being Message passing Process (computing) Befehlsprozessor Telecommunication Order (biology) output Pattern language Abelian category Task (computing) Programmschleife Resultant Overhead (computing) Computer-generated imagery Parallel computing Vector processor Element (mathematics) Number Session Initiation Protocol Architecture Telecommunication Iteration Read-only memory Natural number Operator (mathematics) output Message passing Loop (music) Maß <Mathematik> Computer architecture Task (computing) Context awareness Overhead (computing) Serial port Dependent and independent variables Physical law Core dump Cartesian coordinate system Software maintenance Computer programming Loop (music) Personal digital assistant Interpreter (computing) Iteration Fiber bundle
Context awareness Thread (computing) Multiplication sign Primitive (album) Water vapor Parameter (computer programming) Function (mathematics) Mereology Semantics (computer science) Computer programming Data model Programmer (hardware) Roundness (object) Type theory Semiconductor memory Different (Kate Ryan album) Synchronization Object (grammar) Network socket Shared memory Logic Process (computing) Endliche Modelltheorie Multiplication Descriptive statistics Lambda calculus Physical system Mapping Block (periodic table) Shared memory Range (statistics) Maxima and minima Sequence Thread (computing) Demoscene Electronic signature Connected space Type theory Message passing Data management Process (computing) Befehlsprozessor Computer configuration Telecommunication output Right angle Queue (abstract data type) Remote procedure call Remote Access Service Task (computing) Resultant Functional (mathematics) Socket-Schnittstelle Server (computing) Mapping Virtual machine Lattice (order) Number Element (mathematics) Read-only memory Hybrid computer String (computer science) Ideal (ethics) Energy level Namespace Proxy server Condition number Task (computing) Socket-Schnittstelle Context awareness Scale (map) Dynamical system Default (computer science) Multiplication Server (computing) Paradox Letterpress printing Cartesian coordinate system Subject indexing Word Software Personal digital assistant Network socket Hybrid computer Calculation Object (grammar) Library (computing)
Complex (psychology) Multiplication sign Virtual machine Maxima and minima Planning System call Neuroinformatik Process (computing) Telecommunication Operator (mathematics) Core dump Logic Utility software Resultant
Socket-Schnittstelle Duplex (telecommunications) Duplex (telecommunications) Channel capacity Mechanism design Kernel (computing) Process (computing) Deadlock Network socket Buffer solution Process (computing) Sanitary sewer Data buffer
Context awareness Context awareness Demon Thread (computing) Code Similarity (geometry) Parameter (computer programming) Open set Variable (mathematics) Event horizon Thread (computing) Data management Process (computing) Event horizon Type theory Computer configuration Function (mathematics) Synchronization Partial derivative Pattern language Process (computing) Block (periodic table) Task (computing) Task (computing)
Context awareness System call Code Variety (linguistics) Parallel computing Mass Mereology Computer programming Architecture Programmschleife Universal product code Synchronization Computer configuration Object (grammar) Metropolitan area network Computer architecture Module (mathematics) Variety (linguistics) Concurrency (computer science) Moment (mathematics) Code Computer programming System call Type theory Computer configuration Personal digital assistant Order (biology) Synchronization Modul <Datentyp> Object (grammar)
Sound effect Software testing Computer programming Library (computing)
Module (mathematics) Architecture Variety (linguistics) Computer configuration Concurrency (computer science) Object (grammar) Synchronization Code Modul <Datentyp> Computer programming
System call Process (computing) Loop (music) Programmschleife Adaptive behavior Core dump Synchronization Ideal (ethics) Computer programming
few so everyone and you me well and OK and welcome to my talk I me home and they'll be I'll be speaking about running your Python coats in Apollo for most part of an asynchronous to be honest i've never spoken to such a big group of people so excuse me for being added overwhelmed and let me advertise other talks a little bit so you can see that asynchronous and parallel topics higher hot right now so even during that this conference we have several talks about them In they want you to understand what my talk is about an tool would not makes it has some other talks I also encourage you to see In so there are some asynchronous text parallel talks there's also talk about Python at which I think it would be interesting where I I think it's a poster session and this talk strives to be an overview of the topic so it's not and introduction that's worry they label to as advanced and also you might feel that i skipped some parts and the they want to put an end end to to put it together as an overview so you can later research what's interesting for you and tool and not book you with a lot of detail details OK so a few
words about mean so I worked at the LHC the experiment at CERN looking for antimatter for some time and later I decided to pursue a Ph.D. in computer science but then I've heard that if I drop out I will probably start a multi billion dollar business
but for some reason that hasn't happened yet yeah so and s and
currently in I work at a come combined
and I know as a developer were far too obviously stands for frameworks and tools so what my job is that by combining is to make sure that we use the best tools we can and so on how how do you define the best tools so sometimes you hear that Facebook use using some tools or that Google is using some other but doing it that do we have the exact same this scenario as them so my job is to create tools and to select tools which are the best fit for us
and so i come i itself uh is content delivery delivery network and cloud service provider so we're not very known in Europe for some reason but we have 1 of the largest for the largest and network of computers talking with each other and we also responsible of problem and between 10 to 25 per cent of all Web traffic we also have some security project in security protein products launched recently and we also have 16 officers in EMEA both sales and engineering OK there's a lot of mass when it comes to basic concepts in asynchronous and parallel programming so let me clarify something things 1st so when you have 1 pipeline and 1 worker 1 and working on it you have the serial or sequential execution When you have also want pipeline but multiple workers and they do all the work in the same time but not to be Apollo I would call it contract you need not agree with me and some people do not but let's assume that his for for this talk that is correct and also we have Apollo execution so we have multiple pipelines we have multiple workers and they actually do their things in polyp the um so the concurrency uh I usually when I think of concurrency are usually think about preemption uh how many of you know what preemption OK and have a few so let me just say that preemption as if it's red has the CPU time and the operating system's scheduler decides that there's some other threat that needs that time more so there the preemption a curious 1 threat is being stopped the other threat is being put into his place and they and then they switch roles until their job is complete so this is why you can sometimes see that things are a concurrent because you achieve results in a certain amount of time but they are a natural choice the polyp OK so how would you call this I would call this a headache are you might call it hollow and asynchronous so another thing and which I need to clarify what is the difference between the threads and processes because they are often mistaken nor processes are wrong the COLT bigger threats so threads are the place where your code is executed uh each processes each process has a threat and distract can be scheduled for execution it can get CPU time and all threads share of virtual address space and system resources of the process and they do not share stocks local variables and also but they do share process heap and process is an execution environment for threat so it has its outer space it has executable called it handles system objects so it and it brings it always necessary for threat to us the so and so IAEA I want to clarify that because sometimes people don't know why guilt in Python complicates things so how it applies to Python so in Python we we have multithreading and processing and when we talk about multithreading we have 1 process so this 1 environment we have many threats only 1 interpreter and due to there is a rule which says that in a Python process on the 1 Python bytecode instructions Spigs acutely at once so if you have many threats you cannot execute many bytecode instructions from different threads at once um but um with i all its of the different story because she you have a all and then it does not execute any bytecode instructions so he if you have threats and you do some I O in then you can actually see speedup of that's because it's not going through a Python interpreter and when we talk about multiprocessing we have many processes have many threats at least 1 thread per process we have many interpreters and all threads have their own interpreter and that's why they can execute in parallel the and so do we have Alex Martelli here the last it's always dangerous to be citing someone sitting in front of you said during a chat with rain on brain and had here he proposed the following classification which I think is a simple but nice so if you have 1 core usually 1 tool run a single process with a single threat so for a tool to 16 cost because that's how many courses you can get in the consumer PC PCs nowadays uh you can have multiple threads and multiple processes so why you should not use multiple threads on every on a single core that's because even those that I which might give you a speedup when it's done in in a threat it still needs some CPU time not a lot but it does so with only 1 course you want you should not achieve and speed the and also when you have 16
plus cost you usually have multiple CPU's so you enter the area of distributed computing and Alex proposes that as the time goes by and the 2nd category becomes less relevant as we are in the year of Big Data and even once fuel with 16 on or 32 course this is not enough I would argue that for some cases like back and toward it is but you can you can hear more about that in Raymond hitting the stock OK so you should have some knowledge about that now so when I is a reckoned developer think of speedup or performance boost which 1 I want to use hollow or asynchronous the yeah follow up because I want to all execute many things at the same time and I want and if I want to gain responsiveness and lower latency I'm choosing asynchronous OK so when running things in parallel it's useful the what when you have big data sets are complex computations when you have problems with part nature so-called coarse-grained I when you have want to work in applications yeah I O-bound problems are not a good fit for being paralyzed as they require a lot of I which is mostly 0 sequentially and also problems need to be complex enough so that Apollo overhead caused by uh process maintenance communication scheduling synchronization is negligible tool was going on inside the process the OK so who knows and slow the of some of you OK so i'm those law says how much speed up you can get when running in Apollo and when you you you you need to know what and how big parts of your program needs to be sequential and if you know that you can an approximate how much speed up you can get with a certain number of CPU's so let's say that we have a task that transport 10 minutes about 5 minutes of that time the sequential work like loading data so you can see that if we have given an infinite number of SIP use we can only achieve a speedup of 2 because that 2nd part if it's really uh running parallel than and that time Ghost was almost EUR but you still have that 5 minute time so it's so you really need to know your problem when you start working in parallel programming and just to give you an example of that some of you might say that it's a really trivial problem but in order to present you how that works they had to choose something like that and so here we have a small dataset and a really simple operation we have an input vector of 1 million elements and we want to calculate the outputs which are inputs plus 1 so we can run at sequentially and also we can run in Apollo in in different processes so how do you think how much the speedup will be you're running on for a course cost tool for non it it will actually be slower because the problem is really simple and data set is small and it's not enough to all of to have any gain and in fact you will actually do something and and even at 4 h cost or more it gets even more complicated and you get me even worse worse results OK so a common pattern in parallel programming is to put a problem so difficult more difficult by running into the followup so here we have a problem that's tool 100 times more complicated and how much speedup be now tool for almost it the yeah so that their the speed-up comes from using processes which they committed like I mentioned earlier we need needto have processes in Python to executes truly in parallel and arithmetic operations go through interpreters so so we need to put interpreters so here we have almost almost 4 OK so some some problems like that have uh pollen nature so here I was easily able tool and to divide the dataset into 4 subsets and I in the most parts of the program is running Impala so this I this type of thing has a part of nature so we usually do usually when we talk about pollinate nature we talk about coarse-grained problems so if we have a loop of groups if we have multiple images to process the to have multiple datasets on a really big set or maybe the dataset is not big about the operations we want to run on and on long so those problems are caused coarse-grained and those then there you can easily apply Apollo programming but for fine-grained problems there's a different story so when have iteration of a single an image or a single small small datasets you should not paralyzed that's at least not with the CPU because nowadays we can actually parlors fine-grained problems uh with massively parallel art architecture devices like use because they have really a lot of processing units and their threats are really like the OK so sorting further
programming we have different memory architectures and uh most known to on shared memory where each process has its own memory they where each process connects to a shared memory and the 2 works on the same dataset and we also have distributed memory AKT message passing so we need to all past data to processes and later get that back and that's why it's called message passing so how to apply them in Python
the and so for short memory we have type objects and those are objects created in shared memory and can be inherited by a child processes so if you import what depressed processing value or you can assign what types that values you can assign its value in the beginning and we have also some other and semantic types people and primitives so let's see how they behave so I have to the I have 2 programmes so the difference is that 2 1 uses locking and the the other 1 is not the 1 on the left does not use talking so we have shared memory so all processes have access to the same memory at all they all can immune you can read from it and the right to it in the same time so what you if you do that then they will act in their actually will be some something called race conditions so sometimes 2 or more processes may read the same value so let's say that the index to I have value tool and for processes read that and uh when they read that very had 1 to it so it's 3 and then they will 4 times right that 3 into the memory that's why we can achieve this so when you know when you're under the left program you will get different values depending on and what's going on in in your system but the and the answer will be run so for shared memory you always need to use some kind of synchronization and in this case I use talking so here we ensure that only 1 process can read shared memory at the same time and so on the 1 can write to it so with that you get good results but what's with the time you might say that the problem is too small or the dataset is too small but that won't be the case the case here is when you use locking and you have multiple processes in in fact get sequential execution because only 1 process can take something from memory make calculations and right back to it so your cold will either be slower or around and martinis at the same time and believe me that here it's really easy to spot and see and usually we don't have that simple uh problems and actually you know you can use something else so these shirts attack objects have their own blocks so you can use them there the output will be the same but you're not create additional locks and you need to really keep the number of paradoxes slow it's possible to not have to know what's going on the case so we also have managers in Python which are a hybrid between shared memory and message passing so managers are proxies through each charge processes connects data and when you created a manager it's poems new process which communicate through sockets the so actually if you create uh if you take multiprocessing manager it will create a new process and you can later give that to a children of that process where you can even use it for remote access because it's using a circuit yeah so I have for distributed memory the most commonly used tool it's pool not how many of you know and fused will not yet some of you so it's really simple and nice so just define how many processes you want a new map a certain function and a collection of words arguments and just sounds fine it's really high level and nice tool and some you can simply achieve speedup so but you need to remember to always close or terminator pool and later to to join and if you know if you're 1 of the kind that doesn't remember like me you can use it as a context managers and we also have something that looks like message more so we also have a pipes and cues so the basic difference is that the pipeline as on the 2 ends it's really fast because it's usually using operating system pipes and fuel that can have multiple producers and consumers but you will need to have in mind that behind the scenes there are pipes connecting all elements of the network the but um yes all so pull has some overlooked features because people usually use it like like this so they create a pool of numbers of processes and then they just map the map some function to some some some some input so what you can define his for example and max task tasks per child argument when sometimes feel processes growth and consume more and more memory and you want to all we start them once in a while so here can be defined how many should be executed per child until the signature they also can define a chunk size and I
didn't know that until yesterday I think because MAP usually maps 1 execution to 1 element so if you if you have a map of uh 4 processes and let's say at 12 in some uh 12 inputs for the default chunks Swan sold there will be a 12 round trips between the worker and the and the main process before the result will come back so you can be optimize with with the parameter so we can also define I map and I'm up on unordered the difference is that I not still waits until all uh process is finished and because when you come out you will get at least with the results when you call I you will get generator but you will still need to wait until everything's finished and when you use HyMap unordered you get what comes what finishes 1st so that's that's useful and also the there's an approach with a player saying this is what's actually going on behind the scenes or but this scotched to to use it because now maps are considered higher level and and better tools OK so we have different models a for part of programming levels of different models for distributed memory itself so we have something called worker based models so you can have a preferred model that June equine users so you might create your workers beforehand so you define that your application starts up with 4 processors were for threats and that's before you can have the water model where you define string execution how many workers you need for example you optimized to your dataset and how it how it how well it divides and uh multiprocessing pool is an example of that and we also have a hybrid approach so we can define the number of workers beforehand and later you can scale them dynamically which is useful when you're working on something like that the back-end server OK so when you're when you'll want to create an application and want to work in applications and let's say that you want to respond to some requests then you have basically 2 approaches you can either use reuse ports and to use other for the circuit and I won't really go into details but there's a really nice description on StackOverflow about uh but basically you can create as many processes or threads you want you can assign the same circuit to them so all of those workers can and what's what's coming from the circuit but you need to warm in in in this scenario you have to ensure locks and synchronization because if you're going to all read from all of those traits that you just get garbage so increase that that's a really neat way of doing this so you create a socket that's a TCP socket you respond that child processes and you can later uh adopt circuits for that for processes but that model and this is the approach which you should choose if you really want to tune your performance and if you really want to have access to some low-level stuff and if you don't you can take a different approach which is most common where you have just a single thread of the reading from from the circuit and this threat is responsible for ideal and later it delegates and the work tool to some other workers through a Q In the Q which I which I mentioned earlier so then you don't have any problems with synchronization and and stuff like that so up to now we've talked about a so called intron note communication so communication within a single uh a single CPU all were just 1 1 server but you can also run your to work on multiple machines so there is there are many libraries you probably have heard about and PI which a thing it's and the most commonly used slavery and to this day and I want comes still a scientist but there are some some other tools I personally like school maybe because it has a really nice slogan but um it uses 0 and q circuits for communication it's really similar to multiprocessing pool and 8 utilizes SSH connection for execution so you need to have SSH access tool to the machines you want to run your your application on yeah and later it connects to 2 of them since data and and Q 2 so can see that it's really really simple to use the OK so I've I I've encountered some traps and and some weird behavior over the years uh so I would like to share that with you so 1 possible trap is Hyper-Threading so the CPU's are often advertised as 16 hall in cost as 32 to cost 64 cost but how many physical cost you get usually half of them sold
hyperthreading works in this way so that you have CPU by plane and you have this call them slots in them so if you have slots to run 2 things at the same time 1 1 core then here to logical threats will run in Apollo but if you don't then they want so I had a problem I had that 12 core Intel Xeon machine and with 24 logical cost and when I ran my computation which it was a really uh really complex computation and um I'm sure that the result is not that caused by communication stuff like that they achieve these results safer that Intel uh launching a new tool for tuning and profiling Python so long as I think it might be interesting to to work with that also you
don't always want to own target 100 per cent utilization because if you have for a course you have you prepare for workers and then you have 10 per cent off each core not used in each secure epoch so what you want to do is to just add workers to use that but you won't gain anything and actually you will lose the someone wants to know why why do we lose time here because we should utilize that traditional spread 10 % and it should be faster yes exactly so we're switching contexts so that all processors are fighting for the for resources and switching them and copying them for for different course in this is really expensive operation so don't always target that 100 per cent also
and there's a funny thing uh in how pipes are implemented sold pipes pipes cannot that in the sewers pipes cannot send things both ways so if you defined by putting duplex you actually get the socket and that sometimes you get a socket and sometimes you get a pipe and if you take into consideration that they have different buffers and defined in the kernel that you will be you you might encounter a situation where sometimes you will be able to send some something and sometimes not so that's interesting and also you
have the usual topic which is that deadlocks so when 1 process has some resource the 2nd process has its own resources and they wait for for each other but they do not uh free their resources that they will wait forever and and do you know how to kill processes and threats in Python so there's a secure method so who used can matter OK so you couldn't use Q method because it does not exist because you cannot kill a threat is by design because you might end up in this situation where your throat holds when skilled other threats will never get it because it's never been free so that's why you need to use some different mechanics and when we are um
and threats uh there are some in there's a common misconception with Damon's so if you have 1 ritual or something similar in your throat it should be a and Damon's should not be joined once you set up a threat to us as they meant uh they should just run as long as they processes running and the only a clean way of stopping them and it's up to you so we also don't use global variables don't defined stopped equals false and then iterated and and this is it changes because you might never know what will happen and when those threats will be stopped so all the common pattern is to use events and to just send an event in 2 main threat and to wait for that event in in worker threats OK so and comes to Apollo and asynchronous uh we we finally reached the top pick and so we have basically 2 options the thread the adoption process option where we we can define the executors we can submit jobs to them and basically what you get is futures and also we can define them as context managers and you can have the running you can run that without starting on the IOP and just get futures and where you can use them with low of then you need to doctors and sometimes those conductors work and sometimes they don't seem to be really careful and also that keyword arguments are not allowed in executors so you might want to read that have to know why if you really need to keyword ke keyword arguments that versus partial also you can uh it can submit several jobs and wait for the mold for all of them to the finish so coming room to and why would you want that that so if you might want that for long-running tasks that might block the uh so you might want that if you have some quote which is incompatible with the with your eyes open and that will most certaintly look like requests we cannot use requests with uh with any kind that exists for now and also if you have some some Running blocking tasks that you want to run in parallel so what will you get 20 is that
had mood because running things asynchronously in trouble some and when you introduce all surrounding it the Apollo uh also from some solution really know that you knew that the
case with me around for a moment just before I finish so you all know this gentleman Peters and he she said that there should be 1 and preferably on the 1 of this way to do it so where have we gone wrong we have
currently for commonly used by loops we can 3 types of asynchronous calls so if there are some decisive pupil In the Croat let's let's think how to fix that because Python 3 was created in order to clean up some mass which accumulated over the years and I feel that now we're creating such mass again the OK so in summary by has a wide variety of options and comes to pile and asynchronous but programming we should really know your architecture when you use part of programming you should always test your code before entering parallel concurrent world so 1st sequential than than part of after you enter the concurrency world you should test it with fuzzing uh I didn't say anything about that but 2 countries to research that be aware of any incompatibilities between modules and I assure you that they do exist be sure when you highly being yet be sure when you should expect an awaitable objects in in synchronous programming to handle them properly and also in all those tools are for us and the mostly work and you can create in production code with it if you test it well so don't be scared to see new options and to boldly go where no man has gone before thank you FIL
you hopefully it will be this effect snakes and we had a holiday for questions that now we have a tremendous focuses the questions for me how of all that of any questions so actually I do it's something nice from Poland for people hold ask the best questions in the 1 you mentioned about deadlights I'm curious about text is it good at library also detect that in the program which can appear on I know that such solutions exist
but I but I don't remember that the names but you can mostly get that was fuzzing or just testing I some unexpected behaviors in Europe last this
light that you said that you mention today about the in goal but the BDTs between molecules maybe you can there's something more about their work but was a you experience with problems when incompatible and what think that you could say that again and then the the song yes be aware when Ingle but the BDTs between modules you user so what where you would have varying to the the and we
have this medium in complicated yet uncle but DVDs the OK sorry for
that and so that they can mentioned we have different i
loops so let's say that you want to use it has its own IU but you want to also use some process executor which uh does not runs really well and was that without adapters so what you might to do all these tool 1st about your program written in your 2 new programs to run on a single ideal and then around that those of the those executors and for example curio does not work with any other i a loop so we don't even get any of so don't even get and I think to to connect to them and yet for example in they don't you should not yield from an should yield from the core so there there's also something incompatibility and that's that's what I meant 14 dates minerals represented a few