Algorithmic Trading with Python

Video thumbnail (Frame 0) Video thumbnail (Frame 2420) Video thumbnail (Frame 3520) Video thumbnail (Frame 6167) Video thumbnail (Frame 8495) Video thumbnail (Frame 9449) Video thumbnail (Frame 11135) Video thumbnail (Frame 12946) Video thumbnail (Frame 16266) Video thumbnail (Frame 16919) Video thumbnail (Frame 18732) Video thumbnail (Frame 20837) Video thumbnail (Frame 22133) Video thumbnail (Frame 28414)
Video in TIB AV-Portal: Algorithmic Trading with Python

Formal Metadata

Algorithmic Trading with Python
Title of Series
Part Number
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
iztok kucan/Joris Peeters - Algorithmic Trading with Python This is a look behind the scenes at Winton Capital Management- one of Europe’s most successful systematic investment managers. The talk will mainly focus on how Python gives researchers fine-grained control over the data and trading systems, without requiring them to interact directly with the underlying, highly-optimised technology. ----- Have you ever wondered what technologies are used in a systematic trading system that utilises computer models and accounts for the majority of trading on the stock market? This is a look behind the scenes at Winton Capital Management- one of Europe’s most successful systematic investment managers. In this talk, we’ll run through an overview of Winton’s trading infrastructure, including data management, signal generation and execution of orders on global exchanges. The talk will mainly focus on how Python gives researchers fine-grained control over the data and trading systems, without requiring them to interact directly with the underlying, highly- optimised technology.
Algorithm Software developer Constructor (object-oriented programming) Projective plane Quantification Division (mathematics) Core dump Database Bit Data analysis System call Computer programming Disk read-and-write head Wave packet CAN bus Data management Information Data structure Game theory Computing platform
Statistical hypothesis testing Statistics Observational study Combinational logic 3 (number) Data analysis Mereology Theory Data management Writing Inference Spreadsheet Strategy game Different (Kate Ryan album) System programming Software framework Information Noise Process (computing) Office suite Extension (kinesiology) Computing platform Form (programming) Simulation Cellular automaton Constructor (object-oriented programming) Model theory System call Frame problem Particle system Computer animation Logic Telecommunication System programming Software framework Procedural programming Landau theory Simulation Resultant
Satellite Cross section (physics) Enterprise architecture Service (economics) Information management Transformation (genetics) Model theory Sheaf (mathematics) Time series Set (mathematics) Water vapor Data analysis Student's t-test Mereology Wave packet Data management Fluid statics Order (biology) Cross-correlation Insertion loss Internetworking Profil (magazine) Software Logic System programming Office suite Position operator Enterprise architecture Simulation Electric generator Decision theory Video tracking Data storage device Core dump Volume (thermodynamics) Line (geometry) Category of being Data management Computer animation Software testing Quicksort
Axiom of choice Dataflow Serial port Service (economics) Code Calculation Time series Insertion loss Event horizon Estimator Web service Strategy game Computer configuration Semiconductor memory Software Visualization (computer graphics) Core dump System programming Computing platform Simulation Graph (mathematics) Scaling (geometry) Dataflow Server (computing) Interface (computing) Computer simulation Data analysis Bit Position operator Event horizon Computer animation Visualization (computer graphics) Personal digital assistant Strategy game Right angle Simulation
Keyboard shortcut Graph (mathematics) Client (computing) Variable (mathematics) Different (Kate Ryan album) Object (grammar) Series (mathematics) Monster group Social class Scripting language Simulation Keyboard shortcut Computer simulation Bit Mereology Sequence 10 (number) Well-formed formula Series (mathematics) Website Summierbarkeit Energy level Simulation Spacetime Game controller Server (computing) Service (economics) Real number Virtual machine Time series Similarity (geometry) Causality Well-formed formula System programming Energy level Communications protocol Form (programming) Rapid Prototyping Graph (mathematics) Run time (program lifecycle phase) Weight Client (computing) Letterpress printing Computer animation Personal digital assistant Network socket Calculation Universe (mathematics) Social class Object (grammar)
Point (geometry) Rapid Prototyping Keyboard shortcut Game controller Code Graph (mathematics) Tournament (medieval) Direction (geometry) Wave packet Well-formed formula Object (grammar) Interpreter (computing) System programming Series (mathematics) Communications protocol Simulation Rapid Prototyping Graph (mathematics) Run time (program lifecycle phase) Fitness function Client (computing) Mereology Complete metric space Embedded system Computer animation Network socket Series (mathematics) Universe (mathematics) Interpreter (computing) Social class Energy level Object (grammar) Simulation
Implementation Multiplication sign File format Time series Insertion loss Data storage device Revision control Mathematics Strategy game Semiconductor memory Different (Kate Ryan album) Hash function Core dump Energy level Software testing Implementation Information security Library (computing) Shift operator File format Weight Data storage device Sound effect Generic programming Database Surface of revolution Formal language Frame problem Computer animation Series (mathematics) Revision control Arrow of time Pattern language Energy level Representation (politics) Data structure Resultant Writing Asynchronous Transfer Mode Library (computing)
Code View (database) Function (mathematics) Mereology Data management Information retrieval Web service Strategy game Bit rate Interpreter (computing) Visualization (computer graphics) Bus (computing) Data conversion Library (computing) Service (economics) Simulation File format Building Data storage device Bit Type theory Category of being Sparse matrix Data management MBus (SPARC) Series (mathematics) Telecommunication Interface (computing) output Pattern language Simulation Writing Trail Slide rule Functional (mathematics) Implementation Service (economics) Transformation (genetics) Time series Data storage device Mathematical analysis Code Theory Revision control Energy level System programming Communications protocol Rapid Prototyping Graph (mathematics) Dataflow Interface (computing) Weight State of matter Mathematical analysis Code Basis <Mathematik> Transformation (genetics) Software maintenance Frame problem Supersymmetry Computer animation Visualization (computer graphics) Interpreter (computing) Table (information) Library (computing) Singuläres Integral
Presentation of a group Information management Code Euler angles State of matter Multiplication sign Direction (geometry) Source code Combinational logic Sheaf (mathematics) Mereology Tracing (software) Public key certificate Subset Formal language Coefficient of determination Mathematics Component-based software engineering Different (Kate Ryan album) Mehrprozessorsystem Single-precision floating-point format Core dump Cuboid Extension (kinesiology) Social class Covering space Simulation Algorithm Channel capacity Feedback Moment (mathematics) Data storage device Electronic mailing list Sound effect Bit Price index Measurement Flow separation Entire function Connected space Category of being Message passing Telecommunication Order (biology) Right angle Quicksort Reading (process) Point (geometry) Slide rule Server (computing) Functional (mathematics) Service (economics) Sequel Open source Image resolution Virtual machine Time series Streaming media Mass Event horizon Theory Element (mathematics) Number Wave packet Revision control Frequency Goodness of fit Average Operator (mathematics) System programming Energy level Software testing Condition number Scaling (geometry) Information Weight Interface (computing) Total S.A. Database Limit (category theory) Personal digital assistant Logic Video game Speech synthesis Library (computing)
but now we have 2 speakers colcannon and Peters about algorithmic trading with Python very interesting thank you few and all this talk are the algorithmic trading with Python just to clarify something like training and mean buying selling financial instruments on financial exchanges of algorithmic I mean the region program computer program running kind algorithm that decides what to buy what to sell markets in with a capital we manage about 35 billion dollars using a platform provided constructive Python or large structure Python was also use Python for researching in data analysis on for for the future the talking followers we'll do with company of you will have a little bit overview into our research activities and the training pipeline itself and then you're going to give to quantitative details about how and where we aspire and nonexistent which I'm telephone call acknowledges went and yards have a very exciting new project data pipeline project that the heavy use of fight there if you've come across with the past you know this is called a quantifier a of a training set trading and outfits on a hedge fund wanting trade commodity trading advisory all those are valid but I think a lot of work which probably described as
a this is the companies use the scientific method to to conducted based on what we mean by scientific well the empirical study of what is the largest empiricism on hypothesis testing experiment construction and statistical inference in in how we derive strategies that upon but we have around 100 research which about what the company but typically with a background in academia academics accident and postdocs on these organizing teams of all of the activities the review so it's a fairly open activity how we arrived the signals on another course companies in engineering which again is a fairly empirical discipline itself geographically
but we from our UK companies 400 stuff from must forms of communication made in London some announced it but we expand globally for offices in Asia and 2 in the a lot of these officers are not just cells and cells of solvers of are actually growing so for example whenever a new data laughs the sentences go looking at generic data OK so this is a python constant so what about like the winter which is not about 20 years and then for the initial few years the the system so far simpler than what they are now not an effective around an excel spreadsheet of course the graduates lossless extensions of Excel spreadsheet and gradually those things were they taken on excel and formalize psychological the simulation framework and that was and remains of the call modeling tool and also execution for for adjacent but the founders the frame again flexibility we use Python to start combine these results in a more flexible way so for example if I want to do of delta theories and what you have here is of of using the same logic that would if I were to devote to this is an adult on the volatility but I want to combine different manner particles quite useful to that this is we engage them procedures are using positive manner he became very attractive for us to start writing strategies for past strategies in part themselves on and from that never really starts to the last 10 years we've adopted parts and for constructing a trading platform but also increasingly in data analysis and their use in research some societies
create history of service research invested was quite a strict distinction distinction of of what are is exploratory activity once training activities so we sort of which the research is looking at things that may lead to something all often will not but then again the recent research itself is conducted on 3 lines I would say Corey situations which is which research into signals on and on this school coordinated behaviors updated research which is research into data on the properties of the data and then in extended sense the rod data analytics like volatility profiles volume profile of correlation are from the data directly on and we're now as I said before whatever have a data lots of sections such system which accepts Mr. very datasets aspect datasets like a satellite imagery or know the deep dark corners of the Internet what was that was was signals that arise we transferred into the investor cross-section of the so much more rigorous exercise will have a quite static training pipeline on and again the key there is that you can do things their repeatable very reliable and secure manner which some sort of the Department's Office composed of roughly 4 stages of schooling data-management signal generation of water management and post of now Python is used a lot in research but also is now extensively in data management generation positive training impact
but we data management typically the things we do the training pipeline we obtain large sets of data are cleaned and transform them into the things we need all we use things like versioning to make sure that within repeatedly the data as it changes python underlies all the architecture of the duration of the pipeline we also used by Python extensively on so by the students so drive simulation of transducer transformation into an increasing so part also interface to all data storage and called time series on the nearest
and go into to that in more detail right so under a bit more detail about how we actually use Python's from low-level detail where exactly it's it's in the sense that the main reason we use really is because it presented quite a friendly face to research low-level code is always specifically to execution Estimation platform but it's not something you want to research into right so we expose all or quotes all the guys are typically in Python of your options the Pythons that main choice but it's not just for research because it's such a nice and programmatic interface user for monitoring typically to serve as a web service as well but indirectly signal generation the reason we chose Python is it's extremely well known it's very easy to learn if you don't know Python problem not too long before you do and it has a lot of just goes a lot support for data now this visualization so it's quite nice do as a researcher just to get all that batteries includes uh this is a
very large scale overview trained by pipeline as if you kind of core principles to it's the whole thing is event driven so if something happens which holds something else that happened in this case for example we get our data for example Bloomberg as soon as the data is there automatically because of the equities prices of futures prices once that's done automatically always strategies because and that kind of event driven flows it's really at the core of the winter technology these days and then we have as isn't that mentioned simulation sits at the right button there so it was ah when there is pretty much the graph it's real-time graphic just sits there all services stuff happening we have this simulation which is collected but in memory offline graph so essentially it's really we designed to do you have time series analysis Jakarta spin up a trading system that would take of 1 of his simulations you run them you can Terry than serialize and that's kind of the other main technology that we have so you a bit more detail about both the real-time graphical comets and then the kind of simulation strategies to testimonies suspect simulations which is written supposed loss the 1st
simulation is written entirely in C + + it's been going on for about 10 years now I think after right after we moved away from Excel pretty much that if you just ignore the left hand side it's kind similar in concept to what these days appears things like tends to flow so essentially it's kind of like a graph it's very well optimized i in our case from the site so you can just not be evident in everything you have to use it strongly typed data but as an example there over a graph it's quite a simple 1 2 datasets to data series feed into something like a form that can be the sum of these 2 series and then you calculate the thing and that's all running remotely on calculations of PD things can be be thousands tens of thousands of assets you don't run it on your local machine the calculation services but we expose on the left hand side the Python clients so any user any research this connects or monster support of any of these simulations connected and has full control over the remote nations that is actually an example there on the left it's real Python scripts so the 1st thing as it starts with session which is gonna cause 1 of these simulations to be constructed and launching server then subsystem time-series and those of the formula and the calculated that's the only thing I have to write in your full control over simulations this means researchers only know anything sequence lost anything they need really using the simulation to consider the concentrating system the constant universes all kinds of winning and essentially gives fairly high level control over anything they need to do a little bit about acknowledging that I go to deep here the bindings divided by the extremely light weight so they don't know anything about simulation perceived as soon as they launch assimilation they get everything they need from the simulation that populate applied clients with all the objects that the classes are dynamically generated the object responding to your space if you create a new new objects they're created both on the remote clients and the local clients essentially gives you full control locally as if you were doing it that is very friendly by invited every all the date of his return us from a series the difference or accounts 1 thing you can't do it
assimilation binding was you can control the graph so you can
limited to these things like formula or value series for university or particular training systems that technology is implemented in C + + but you can't really do this where it is if you have a complete outlandish trading system that you want try anyone a kind of plug into this graph if it doesn't fit this kind of former nowadays the kind of stocks and for that we designed embedded points
and so what you can actually do is in Python right 1 of these object that run directly in the simulation graph so from that on anybody can launch them remotely run your training systems and you don't have to write any C + + you can just contribute your Python code everybody can run its it wouldn't normally going to training but this is more I intended for rapid prototyping of research promotes built a trading system applied and tested in the simulation which means they came back to this in 1972 now that we have to write the plus because there you might of often have to wait like a month or 2 months for technology direction demands smaller really tournament so they can just build the thing running to test it's never happy with it and it is still implemented in C + + efforts the standard ideas that and rapid prototyping although some of it is actually to and acknowledging there unsurprisingly the steepest was acceptable hosts a interpreter have used was spited marshalling all the data it exposes and uh so even by API for performance yeah essentially the full control to his embedded Python for making your and your own Python trading system available in the simplest of back and it's extremely powerful that's an assimilation
as a sequential have this problem that we need to shift loss of time back and forth as an enormous amount of time series to be safely of hundreds of thousands of assets that we need to be able to very quickly lowered and writers to a database things like SQL way too slow because we do so much historical testing western mode all the data for 300 thousand securities from generic in memory or distributors and then write the results back to what we designed at a time when we started this was really a good as we build are versions and duplicate data storage we say I'm not gonna go into much detail but is column formats so as a she's super effective for storing lots and lots of time series a loss of columns very effectively the tight uh it's backed by mom would be that's kind of implementation detail anything that can store something from T to a binary blob would have worked uh and really the that's invisible data so 1 thing we don't want this once you've written your data frames if if you do it in Python all you want to get exactly that the different that we don't want the change if you've written something it can never change you always get it back exactly like that that that is really at the core of our strategies obviously if you should testing strategy you don't want to data to change underlying you what kind what reproducibility you wanna know exactly what you did you would be doing that forever the same way so decisively so it's really a revolution is that it actually opened up a lot of possibilities technology there
kind of the same pattern again so we tend to do something low level you really optimized weights supposed plus and every expose all kinds of high-level libraries to make it more accessible to users so the story here is back then 1 would be there's a C library that sits on top of it in a kind of the this corner store so that we can essentially story very effectively and every build very thin libraries on top of that C + + C sharp by years were building a JVM 1 as well as entities can be accessed by different kinds of technologies with that typically dissemination but a researcher might use the Python library and rather than having to deal with this kind of low level corner storage research and this could be a difference in their will get translated into serious and give it to them back to you as serious as well as as the differences the yeah
so small implementation details about how we don't this we use the function performed function interface and the nice thing is that it's such a friendly pipes interface give a data frame and you get different back you don't need to know about any kind of table formats of type conversions and all that kind stuff the
called transforms like a billion beginning wind is essentially a graph and simulation services sitting there waiting for something happened a reaction inputs that produce outputs the next thing is gonna listen to those outputs and again so this is what we call the common transform system this might serve as basis this waiting there on topics on the topic from the bus Kafka actually there's an example there's super simple we get from Bloomberg we write it for we announce that we've done that yeah the equities transformation gonna pick that up as right at the start and then as soon as that's done are track strategies this partaking in the notes strategies that might be 5 strategies waiting for it this price is the automatic of simultaneously distributed and soon as they're done we can start going to execution this is the next to last slide a little bit about technology which commentary on something kind of bringing it all of it together all the ratings are where we use Python having that threatens kind of low level and exposes Python as its external at the so all or even suppose Kafka use proved above Stratford communication we analyze the strongly typed in the kind of you can implement your schema and then are service that currently is in the shop it's a property of a service sector that essentially deals with them getting the code about translating brought about but the common transforms themselves or Python are caused by this sharper quite interpreters to anybody can write anything and become part of the graph that is wins and just by writing some fighting codes uh that Python interpreter might be a strategy that launches simulation that we use this in mind as I explained beginning that simulation can host your own trading system that you've written that would be an embedded Python simulation will read and write this data from the version store which is our are efficient way of storing time series and that then again use the Python library and use can be read by the patterns by the store libraries anybody can read of data that's been written by the solution everybody has access to its true Python store libraries that whilst always is quite complicated is lost acknowledges going around the theme is always relatively the same if low-level codes is really optimized tends to be written in C or C + + implementation details are quite properly theory can be wrote about can be cast but as a user you're only exposed to kind of well chosen API we've defined they're quite flexible the problematic because despite some good anything you want but it is antigen is accessible and by using by providing that as an interface it's still extremely performance and we find that is really good roughly as that's that it's all good but we think this works really well we're quite happy with the system biden throughout wins and if you're a researcher at the business you won't see anything else but fighting you just you don't even need to know that there's any C code under there is the primary interface really for data management and signal generation because it gives such finely control you don't really need anything else that this no need to go into SUSY cost plus you can uh and that's the technology doesn't need to go really fast but as a researcher you don't need it so you can do you can define all your own data transformations you can do with it whatever you want I can store data needed retrieve data you guarantees that will never change as a time series stop as discussed in backed by a very low level the sparse codes that implemented by technology and owned by technology and the main reason we're doing this is because of so grateful analysis visualization rapid prototyping maintainability any net because it's such a programmatic interface all the underlying code you can write web services you provide monitoring systems everybody can essentially circles it into them in Python which means enormous view on what's
actually going on in astray system they so focused and of course all I have to thank you thank you thank you want that OK people many questions OK so I just stopped from here thank you for your attention to the talk thinking in so you're using C and C + + code because during the high-frequency trading commodities leaves a legacy code or not the direction so we are in the low frequency trading his definition of legacy so even though it is low frequency we do continuous and historical back testing that means you know with my just trade 1 new data point you wanna be able to very quickly test simulation all the way from 1972 now so yeah that's the main reason has to go fast because we test the whole of history but we do trades over a period of months so common tools like by you and don't need to get requirements for and so on tools like Ponders's you don't meet that 2 requirements life for a huge effect testing at I think we find that it probably doesn't cover needs we kind of weird the traces that researchers companies are in Venice and so a number of them are going to trade and so things like the the trachea enjoyed those Roman banners and also into training I does actually have feedback back and we find that if things need to really fast and we find we need that kind of speech than perceived limitations still constantly faster to the extent that is worth doing hi and the open source your time series store and if you haven't 1 0 open source which 1 you time-series stolen from now we have an open source that's for no particular reason as tho there is actually uh initially it so inquiries that we start looking into open source I think this is actually on the list of potentially being open source there's nothing particularly training specific about is very generally applicable so yeah that might come up with more questions your slide suggests the Python 2 probably 2 . 7 is executive was that why not 3 and what what is the incremental cost of migrating to 3 there is a lot of poets aimed at the enormous legacy codebases invited to upgrading it because we have always the extensions right now is not trivial but it is being actively pursued as all the new code that regardless of developing will be part the membership gradually migrate all the stuff and the problem is there's not really an enormous business case right now it's a lot of efforts and we don't necessarily get a lot of it back at this very moment we get realize that especially as as the support is going to be dropped we will have to have moved to Python 3 so that's going be a main and obviously there's a lot of features would be considered very especially the multiprocessing and yeah that for me personally of we need a good business case however thanks for for the presentation coming which exchanges to train on and how many bytes of historical gotta do which changes the state of all the is more subset of the cultural 2013 exchanges of quantitative American equities Europeanization equities you should section of and fixed income as well as how much data we have so that they have started typically we just provide a billion numbers they a we have a petabyte class total capacity following so that's that's sort measure coming we need a definition of a sport that's what we have do we and thank you for a good dog and I have 2 questions so uh the reason we can't indication narrativization system uh there is some researchers can see only few machines and something then this so and now it's all doesn't world their API doesn't that we have our own property very alters agencies and basically token-based and there we have there is sequel server so that this sequel courts and backed by myself so we got the education there and every government database which is backed by certificates so it's difficult to 50 bits resolution and again in May 2nd question is the when a researcher was this data in a I guess you go goes to the Microsoft events and then get scrolls can operation sold on your data is going through HTTP and even those weights so fast because millions of events can be so research would actually go directly to the store so they make a direct connection to mobile they wouldn't necessarily have to be mediated they can directly considering to build like high-performance services in the middle like G PC-based or something but right now the the library that we exposed researchers that sits on top of all of this makes direct connection among was so that's why it's so fast yes so adult indication walks the light is a difficult so effectively that among the the best OK thank you more questions with just a fairly bad simulate questions that have cryptocurrencies or anything like that cluster right as you cross the reader and left it I guess it's something really thank you all questions have still sometimes I have a question so nobody OK what's the theory you why still thing and brutal but so what issue with this yet and there's no no other roles than just this 1 here at that is in the message was that see the message was at the top there we use Kafka tobacco or events so we have a box of side events between any consumer it kind of connects In any event it happens so we needed a box of mass was essentially a very strong after every puts put above some of the wire because the strongly typed is actually fairly very compact so we've been using a lot of data over its think of portable is actually really good combination so essentially capacitance reactive at the core of wind and it's all the events go to capture and everybody can much of can even example of such events so as to get a trade what what is this is a different scales 1 thing that can wanting to an event that can be announced is but room exists and I've actually downloaded remote data go find in the store but at a lower level actually use since every single piece of information across Kafka as well so that's before all the data we incessantly downwards streams or Kafka and then depending on I was interested they can be stored in my work and stored somewhere else it can be actually transformed begin run tests on its for all the data goes over the what about as an event itself well FIL thank you again with him I just wanted to know why they are into using any given drive and infrastructures such as that produced was something of this sleuthing severe effects solution is possible that we are actually investigating things like storm as far as I think all of them you have something to say that if you were a company is 20 years old so there's a lot articles on the right side of course in the like half become takes time to to migrate In this case migrant as well so something the new section of the and of course having 55 billion under management also means a lower making small mistake such investments just so that you can get 2nd technologies again not very justified so we don't we do like the adoption of new technology that will cost from high school I'm interested in how are you testing of technical systems like this goes I would argue that there are a thousand things that can go wrong yeah I mean it's a distributed system with a real-time distributed system so what's your approach to the reader when I say that initially gained so much by the readability of data that that you want know if you know that your data from the changed and you have raised conditions that this might need to write for this reads so immutability is definitely so 1 of the core principles and then everything is strictly event-driven strictly adapt that means that because everything is defined by the events you can write extremely test active the whole history can be reconstructed from the events of winter moderate it was solutions along every day for the entire history of example assimilation today we will compare the simulation of the previous appointment and will make sure that the evidence that the entire history of the nation sigh and incremental days that is also usually human verify still some human interaction because of the size of the of small and pollution yes so it just saves you time all relevant on the thank you are there was a slight was to sharply populations was false and altogether and the and the other 1 is probably how well those services which people don't have the the question was yeah exactly as a whole that communication is only in the components in different languages so a repeated once again the question is how the communication is done between the uh libraries in different languages accommodation between the library service in the company the between the libraries in the simplest class and vitamin C I don't know the details but all of them and you know it's not quite and our user see for a function face but essentially what we always aim for a fairly simple 389 an interface to the library which pass all the the logic and then the other libraries are built on top of that they tend to be fairly level and just through the mediation of marshalling of data so all the budget tends to be in the lowest layer and and all the other layers are just representing gating something that is useful for the language itself that answer this but OK no longer there is still room which we also have good better if you if you have more questions yeah maybe just the last version of the hello I'd like to ask you if you say some of safe some precomputed the say some signal so you take the source the dark and light compute something from from all of these stories are you apply to compute average in the retirement of yeah so that's what is the attitude in order to make sure can it's in your questions in order make sure that nothing has actually gone wrong at the time that no kind of about introduced we we run everything from history from the the history much too narrow yesterday we check that everything is exactly the same and only then we allowed the newly generated points to go to that it gives us an enormous amount of certainty that yet nothing is going a common so have problems like this you need to build up and they can change because to improve your algorithm on farm finds some of our and other things that can happen in very weak base like essentially so if we do it here but the change has to be in a controlled fashion so the only thing we want to avoid uncontrolled changed of course we if this any improvement that we will read baseline system thank you but OK so I think very nice talk very interesting thank you