Developing a real-time automated trading platform with Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 111 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21191 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2016111 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
Computing platformComa BerenicesDivision (mathematics)Computer programmingSearch engine (computing)Workstation <Musikinstrument>Meta elementOpen setService-oriented architectureIndependence (probability theory)MultiplicationThread (computing)Sample (statistics)Router (computing)Series (mathematics)Different (Kate Ryan album)Physical systemWave packetService-oriented architectureSoftware engineeringComputing platformImplementationStrategy gameGame controllerWindowIntegrated development environmentInternet service providerThread (computing)Meta elementTelecommunicationCircleNormal (geometry)Cross-platformBitMessage passingRouter (computing)Operator (mathematics)Workstation <Musikinstrument>Computing platformStatisticsIndependence (probability theory)InternetworkingData structureReal-time operating systemFunction (mathematics)Projective planeComputer programmingArchitectureMultiplication signView (database)Graphical user interfaceSource codePresentation of a groupArray data structureOperating systemGoodness of fitModal logicSoftware developerAutomationProgramming languageInformationComputer configurationMulti-agent systemSemiconductor memoryCASE <Informatik>Machine learningOffenes KommunikationssystemRaw image formatComputer scienceoutputRight angleRule of inferenceDivision (mathematics)Metropolitan area networkState of matterBridging (networking)Shared memoryGroup actionACIDArrow of timePrice indexMetadataBeta functionExecution unit1 (number)PlanningCopyright infringement
05:11
View (database)Series (mathematics)Computing platformService-oriented architectureRule of inferenceAlgorithmOrder (biology)AbstractionLevel (video gaming)Directed graphLinker (computing)EmulationVisualization (computer graphics)Real numberPressure volume diagramGraphics libraryElectronic visual displayAlgorithmRule of inferenceCuboidPhysical systemDialectCartesian coordinate systemLevel (video gaming)Message passingReal-time operating systemAbstractionSemiconductor memoryUser interfaceParticle systemParameter (computer programming)Transformation (function)Library (computing)Order (biology)Buffer solutionAssociative propertyPoint (geometry)View (database)Right angleInteractive televisionSeries (mathematics)Portable communications deviceData analysisMachine learningWave packetMultiplication signInformationOffenes KommunikationssystemPrice indexVideo game consoleGraphical user interfaceData structureService-oriented architectureHierarchyBranch (computer science)QuicksortElectronic visual displayPower seriesReal numberMoment (mathematics)WindowPattern languageCalculationNumberGraph (mathematics)Directed graphRouter (computing)Chemical equationTimestampVisualization (computer graphics)Raw image formatInterface (computing)Error messageLoginArchitectureZoom lensXML
10:21
WindowIndependence (probability theory)Different (Kate Ryan album)Web browserGraph (mathematics)2 (number)Computer configurationFrequencyCodeLine (geometry)CASE <Informatik>Single-precision floating-point formatZoom lensView (database)Computer animationDiagram
11:09
Price indexReal-time operating systemZoom lensPersonal area networkParameter (computer programming)AudiovisualisierungPlotterMenu (computing)Computer animation
11:51
Object (grammar)Local ringServer (computing)ImplementationArchitectureSocial classSeries (mathematics)Symbol tableSoftware testingPattern languageNetwork socketKeyboard shortcutVirtual machineProduct (business)Software frameworkAddress spaceVideo gameAliasingVideo game consoleMultiplication signMessage passingMereologyInformationRemote procedure callCASE <Informatik>InfinityEvent horizonPhysical systemLoop (music)Flow separationConfiguration spaceServer (computing)ImplementationNeuroinformatikCartesian coordinate systemLogistic distributionObject (grammar)System callBitState of matterProcess (computing)Array data structureArchitectureLambda calculusPosition operatorSimilarity (geometry)CodeProjective planeData storage deviceFunctional (mathematics)Queue (abstract data type)Correlation and dependenceComputer programmingComplex (psychology)Serial portInternetworkingInformation securityDefault (computer science)Scripting languageProxy serverLocal ringDatabase transactionStandard deviationGame controllerInterpreter (computing)Price indexSocial classOperating systemDecision theoryDataflowOcean currentGraphical user interfaceReal-time operating systemSoftware2 (number)Execution unitType theoryWebsiteThread (computing)Multi-agent systemReal numberCalculationRandomizationSampling (statistics)Different (Kate Ryan album)Open setTelecommunicationFile formatException handlingComputer fileControl flowSoftware developerClient (computing)Socket-SchnittstelleError messageConcurrency (computer science)Router (computing)Connectivity (graph theory)EmailService-oriented architectureOffenes KommunikationssystemLink (knot theory)Open sourceTwitterINTEGRALoutputSource codeClassical physicsTerm (mathematics)FrequencyData analysisWave packetStability theoryArithmetic progressionOrder (biology)Unit testingProblemorientierte ProgrammierspracheIntegrated development environmentLattice (order)Computing platformVisualization (computer graphics)Data streamAbstractionComplex system1 (number)Bit rateLevel (video gaming)Streaming mediaRootVolume (thermodynamics)Sign (mathematics)Mainframe computerArithmetic meanDialectRankingCrash (computing)Procedural programmingPower (physics)AverageFormal languageNumberParallel portForcing (mathematics)Digital electronicsAreaDivisorEndliche ModelltheorieOrbitInformation overloadRight angleLibrary catalogFocus (optics)Matching (graph theory)Medical imagingDigitizingDistanceMathematical analysisGroup actionMultiple Regression10 (number)Associative propertyPredictabilityTDMAQuicksortForm (programming)MultiplicationCuboidData managementEmbedded systemObject-oriented programmingGraph coloringAsymptotic analysisSummierbarkeitTransportation theory (mathematics)BlogNetwork topologyLecture/Conference
Transcript: English(auto-generated)
00:00
Okay, good morning everyone again So I'd like to introduce you our next speaker Miguel Sanchez de Leon peke Who's a software engineer at open system ass a medium company based here in Madrid and he's gonna be talking about automated trading Thank you. So welcome everybody to this presentation on Python for developing a real-time automated trading platform
00:29
I'm glad to see so many of you here today a Little bit about us This project has been developed in open system us Which is a company based in Madrid and in particular in the research and development department or division, which is led by Fernando, Monera a
00:48
Little bit about me as well My name is Miguel century on peke I'm an industrial engineer with backgrounds in electronics automatic control and computer science and I feel passion for programming data statistics and machine learning I
01:01
Programming with Python about two years ago exactly when I started working with open system As and since then I've been gradually falling a lot with this programming language So what's the motivation behind this project? If you go on the internet and search for currently available platforms for real-time automatic train You may find things like trade station meta trader ninja trader
01:23
And in fact many many others each of them with their own little differences. However, they all share some caveats So for example, they are all proprietary they are very heavily desktop oriented So many of them cannot even run if you don't have a desktop environment Do they provide Linux support? Well, no and most of them they don't they do not provide non windows support and the most important questions perhaps
01:45
Can I use Python to create my strategies and the answer here is a big no So what we wanted we wanted to have control over our own tool So we needed to have the source code to be able to modify it at will We wanted to have an optional graphical user interface
02:01
We wanted this tool to be multi platform so that we were able to write in at least in the main operating systems and We wanted it to be programming Python, of course, so is this possible? That's the question So allow me to introduce you to always markets OS markets is a broker independent platform for real-time automatic training with broker independent
02:22
I mean it should be able to work with any broker available out there as long as of course the broker provides you with a convenient API to interact with It is implemented over OS brain. We'll see later what that means Okay But for now, I think it is sufficient to know that OS brain is what it's called a multi-agent system in which different agents or actors
02:41
Run independently and communicate with each other using message passing. It is designed for real-time automated training So that means it must be fast, which it is and it must be also robust This is another view of System or architecture of OS markets as we can see the input of the system is just the raw market data coming from the broker
03:02
That means the updates on market data or market prices and the output of the system is of course the buy or sell Operations which are sent back to the broker We'll see What those little circles and theirs are the agents or the actors in the multi-agent system and the
03:24
the arrows in between represent the message passing the names inside the circles are just the roles that the agents are playing because Normally agents are specialized in a certain role that they play in the in the overall system So we'll start describing these agents we start from the left. The first one is called the feeder
03:43
I'm sorry The first one is called the feeder Its only purpose is to get market data from the broker That means this agent cannot be independent from the from the platform cannot be platform independent
04:01
So we must implement different feeders for different brokers. It is also multi-threaded agent. We'll see later what that means, but this particular agent useful needs to deal at least with One thread that is getting real-time market data updates and another thread that is able to Attend requests from the agents in the in the system for for example historical data
04:25
Just after on the right of feeder. We can find router router is Finally a broker independent provider. It's kind of like a feeder, but in this case is broker independent Its purpose is to store and distribute this market data among the agents in the system in OS markets
04:43
So the rest of the agents can subscribe to new updates But that come from router it is also able to update the market data that it has available in memory With all the information that is coming from the feeders and also is able to resample data that is necessary
05:01
Market data is implemented using numpy arrays and Perhaps the main market data structures are the ticks which are simply the ask and bid prices at a given time The bar which is usually an open high low close Prices during a time span and then and then the bar series which is simply a series of bars
05:25
Sorted of course by the time stamp As you may have noticed by bar series are updated in real time So in order to avoid full memory copies on each update what we do is Just use a view of a numpy array a base array or a buffer
05:41
Which is the one that we update with new bars So if a new bar comes we just update the base array and then we simply change the view Which is much faster than performing a full memory copy On the right of router we can find the brains The brains are the most commonly common agents in the system
06:01
Are able to subscribe to router in order to receive updates on market data and are able to subscribe also to other brains These agents run the actual algorithms for automated training so they are perhaps the most important agents in the system so they must be able also to send orders like buy or sell and
06:23
Well, of course today, I'm not talking about the algorithms that we'd have implemented in in open system us But I think it's important to know that thanks to the great ecosystem available out there for data analysis and machine learning in Python You can do almost anything you want with this brains
06:42
You can make use numpy of pandas scipy secret learn piano, whatever you want numba So at the end you can do anything that you can do with Python which usually means just anything One of the best things about this kind of architecture is that you can create some kind of abstraction
07:02
In this way just creating a hierarchy in which the lowest level brains are those that are Calculating using the raw market data and then as we go up in the layers, we start treating concepts more Which are more abstract or more? Like how a human trader works so we can start talking about
07:22
marketing text intention or market the structure or detecting some kind of patterns and The last agent that we have here in the overview is the one on the right which is the trader which is Similar to the feeder. It means it is broker dependent, of course, and it's only purpose is to handle orders
07:41
Okay, so buy sell orders coming from the brains and then it executes. It's these orders against the broker So just to make it clear. Here's an example Imagine that we have that system in which we don't have the blue brain in there and then at some point we spawn it We can do that in real time and then this brain requests some
08:03
Historical data to ruder ruder may not have this data available So it will send the request to fear Peter will get the data from the broker and then we'll send it back to ruder on From ruder back to the brain all in our sink in an asynchronous way. Okay, so from that point the brain May start
08:21
subscribing to a real-time updates from the from the router and then Performing some calculations in order to generate orders that will send to trader and also it may start publishing Some information about the things that we have calculated within this brain and all the brains may need This information on make use may may be able to use it
08:44
There are all the agents that we have mentioned such as logger to log error or warning messages The informer which we can use to publish information about the accounts balance or the open mind Which is kind of a console to control the agents that are running so we can kill some or spawn you once we want
09:04
We have also developed as I say a graphical user interface this graphical user interface is For real-time market data visualization and also for real-time visualization of indicators The indicators are just kind of mathematical transformation of the raw market data
09:22
This graphical user interface is really good in well integrated with OS markets. Okay, and we'll see later how we do how we did this In order to implement this graphical user interface. We have used QT perhaps Because it is the most widely used Also because it is really great. It has really great portability
09:43
But I think most importantly Is PI QT graph I could you graph is much less known than QT It's a great tool which is written in pure Python and it's a library for fast real-time display And the best thing about this library is that it implements all the user interface all the user interaction interactions
10:04
Sorry that we need so Panning or zooming in one axis or two axis is all implemented and working out of the box as well as updating in real-time the axis So here's an overview of the main window in which we can see a list of
10:25
The currently running brains in the network and from the main window We can create new tabs or new windows with different and independent charts Apart from that we can see how the chart is implemented. This is all PI QT graph
10:42
Okay, we can change from here the data that we are displaying from market so in this case, we have just changed the period from one seconds period to two seconds periods and We have the option to post automatic update of the view and then as you can see we can zoom in zoom out
11:01
We can see that everything works works pretty well and we haven't written all single line of code for that It's all PI QT graph We Also have this indicators menu in which we can select the indicators that we want to add to the chart We can add them as new plots as you can see so we can visualize indicators
11:21
Indicators below the market data and we can also add indicators that are plotted Over the market data such as for example this simple mobile moving average Of course, we are able to change the parameters of these indicators also in real time The graphics view gets updated
11:42
Of course You can remove the indicators and everything all the zooming and panning still works pretty well
12:01
so It is important okay to note that this graphical user interface as I said, it was optional So it is well integrated into always brain. How did we do this? Simply because all the charts are agents in this Multi-agent system. Okay. So when we are visualizing this data, it's simply because
12:22
The chart is subscribed to real-time updates from router and when we are visualizing those indicators It is also simply because we are subscribed to Brain that is published in this information to us So at the end all the calculations are performed in the multi-agent system. Okay in OS markets
12:43
So I have explained a little bit on how we have implemented the graphical user interface But I haven't explained how we implemented always markets or this multi-agent system. So This is always brain. Okay, and it's a general-purpose multi-agent system written in Python
13:01
This general-purpose multi-agent system implements independent agents that run And communicate with each other using message passing and in particular with always brain We have we were able to provide an easy configuration and deployment method Later how we did this
13:21
When I say general-purpose, I mean You can really use this for many of things. Okay, you can use it for not logistics logistics Transportation or even military and defense applications in our case. We also use it for distributed computing so we use these agents that are distributed among different machines and then
13:42
we distribute Some groups or random selected back test and then each of the workers perform An analysis and create a correlation matrix and calculate some kind of indicators and then we take into account to generate portfolios that are kind of optimized
14:03
So when I say that these agents are independent, the question is how I are they implemented, right? Processes or threats. Well, if you have met the Jill then you already know the answer is so of course processes And if you don't know the Jill then it's better for you because your life will be much simpler
14:23
I also said that we implemented a message passion. How did we do this? It was using the serum queue There are many reasons But the first one is just because it's a great project. It's so cool So really if you don't have a project that needs it, you should make one because it's really fun I totally recommend it
14:42
So right now we have an idea of what a basic agent is We know that it is a system process that runs independently It implements methods for easy binding and connecting with different patterns. These patterns are implemented in serum queue So we have the classic push pull or request reply or publish subscribe patterns
15:01
It activates on incoming message This mean use this means usually an agent is running on main thread main process, which is just waiting or pulling on Input sockets for a message to arrive and that's when we start executing some code in the agent Multisreading multiplying may be used as well within the agent and you know that to do
15:24
So what we do is to create in process sockets. That's also serum queue implementation To communicate with the main thread, okay, so we always try to avoid concurrency So we always use message passing even within the same agent
15:41
Configuration I said it was pretty simple So how we know that the agents are independent but demand they must know the addresses of all the agency We want to connect them. Okay, so at the end there may be many many sockets Perhaps creating a configuration file would be too tedious would be too complex We are not interested on that. And in fact for many agents or for most of them
16:04
Assigning a random address is perhaps simpler and more convenient. We just don't care which address this agent is binding to So if only we could it's just an idea have something like a name server Like an address book where we can register all these agents
16:21
Have something like another mind an entity that was able to Know the addresses that the agents are binded to you know The to send them to other agents that are going to connect to the first one So is this possible and here's where pyro 4 comes in which is another great piece of software and also
16:41
Not much well known. I think pyro 4 comes from Python remote objects It allows us to treat remote objects as local This is done with an object that is called a proxy and we can use this proxy to call methods That are implemented in the remote agent and then these calls are serialized and sent to the remote agent
17:04
Executed there and then the return is serialized back to us so we can treat them just as if we were Dealing with local objects, but everything is being executed remotely So it's great very very convenient and apart from that already has a name server implementation. So it's a win-win decision
17:25
So now we know what a real agent is it is a system process it runs a pyro multiplex a server The server actually serves nice an agent object Okay, just in pyro and then we have a main thread as I mentioned before that runs the main loop
17:44
This is what is called a one-way call in pyro So as conclusion we were able to implement a general-purpose multi-agent system with Python in which agents are independent and communicate with each other using message passing and
18:01
We have achieved an easy deployment and remote configuration process using pyro 4 So let's start with a little bit with some code samples just to understand how this works This is the most basic sample, which is a hello world and in this case as you can see In here, which we're just running a name server, okay a random name server and then in here
18:26
We're running an agent that we call example This is an alias and we tell them to register in this name server. And what do we have here? Okay, this variable is actually a proxy Okay, it is not the remote object this function will start a new process in the system
18:43
But that by default will bind to localhost and then with this With this proxy we can call methods that are implemented in the remote object So we can call for example the method log info which is just an information login and say hello world
19:01
Okay, when we do this this call log info hello world gets serialized it's sense It's been sent to the remote object the remote agent and the agent will execute it Okay, this is a second example, which is a simple one as well in which we are going to implement
19:23
Push-pull pattern. Okay. So in this case, we are doing just as before we're creating a name server And then we are creating two two agents a sender and a receiver we call them Alice and Bob and then What we do is to bind the sender
19:42
Using the push pattern which is again a serum cue communication pattern We give this socket an alias Okay, simply because it's easier to refer to it later that way and then what we got what we get in return Is the address the actual address where? The sender bind it because we didn't specify a port for example
20:04
So by default it binds to localhost and then it will select a random port because we just don't care So this call is executed in the remote agent, but then we get this variable here in this script Okay, and then what we do is again to serialize this call
20:21
We're going to tell the receiver to connect to this address So this will be serialized and we are going to serialize the address which is a local object And then we are going to serialize as well a function a function that will be used as a handler This handler is implemented in here and it's really simple It's just as before what we're going to do is to get this message the message that we received and simply lock
20:47
Some information and we will say received and the message So once we have done this We're going to start an infinite loop and then we're going to start sending message from the sender
21:00
So what would what would we get in return? Okay in the console if we executed this we would start seeing messages saying received Hello world received her war receives a hello world Okay, this would be another example in this case of in this case of
21:24
always markets in this we're going to Define a class Okay, that inherits from brain because we can also use object oriented programming if we like to so We're just telling this agent that on new bar. This is the behavior So if the close value right now is higher than before we're going to buy otherwise
21:45
Why it's going to sell and then we send the order. It's pretty simple and it's stupid and don't do it with your money of course and Here what we're going to do is to create a new under architecture this call is Will simplify the creation of all the common agents like the feeder
22:06
as well the router The trader they will all be created and they will all start working with Honda or against the on the broker So we just have to provide our account information and then we tell the system to start streaming real-time
22:21
updates on euro dollar Pair and then we can add a brain to the system in which we specify the class So we serialize again this class and this class will be used to create an object in the remote agent in this case in the remote brain and it will be served by pyro and
22:41
That's it Then we can just call this subscribe method which will make This brain the brain example that we just created subscribe to real-time updates from router on the euro dollar pair based on one minute period and Let's see it as a final conclusion
23:01
Python has been proved to be a great tool for implemented a real-time automatic train platform and one last thing I'm really happy To communicate that always brain has just been released You're the first to know in fact outside of the company so you can find this multi-agent system in
23:22
IPI and in GitHub if you want to check the source code And of course you can find some documentation. It's a work in progress because we didn't have much time So that's it if you want to contact us you can send an email to me there you have my email address or
23:42
There you have also some links if you want to visit our website or our blog or if you want to follow us on Twitter Also, we are hiring so if you like Python or if you like trading or if you don't like any of them But you still like data analysis or big data or data visualization or web?
24:00
integration or anything that you find in the world we do in our website then do not hesitate to send us your Your CV to our hh at open system as calm So that's it. Thank you very much. You have any questions comments or suggestions
24:24
Thanks, Miguel for this great dog. So we have the time for several questions and may I just ask you that we are recording everything So please wait for us to give you the microphone Hi, thank you for a talk really interesting. Um, just how does pyro for deal with?
24:47
Exceptions or errors like it's a traceback sent back to the caller or it's just silent in the agent that it's being It's not easy to send a traceback like the full traceback to serialize it So no, you just can get the message with the traceback
25:03
So you can use the traceback dot format X format exception and get that information if that is useful for you But of course you usually have agents that work well because you have tests that validate their functionality Without the need to run it remotely, right? So you you write the tests locally you validate the
25:25
Agent and then you run it remotely. But anyway, if it crashes yes, you get some information. Okay, but not everything of course Pyro uses by default. Well at least in always brain always brain uses by default Pickle for serialization and can use also deal if you are dealing with the proxies
25:45
Okay, but it's still that's that's not easy. I think Hello, and what is the Performance of this how many transaction can you do in a second or
26:02
Because Python it's considered to be slow. So you see Python that is the standard interpreter It's not we use C Python. Yeah, we use C Python because for us it has been proved to be Sufficiently fast for real time, of course that depends on your application our performance is limited to the performance of pi serum Q which is the
26:25
Python bindings for serum Q which is the message passing that is used between the agents I mean deal and pyro are kind of slower But we only use that for deployment and configuration after that all the message passing issues is done using
26:40
pickle for serialization and serum Q or pi serum Q for message passing so That will be our bottleneck. But anyway, the bottleneck is always With internet so against the broker It's always much slower to get the messages from the broker and then to send the messages to the broker through the internet like by That could take I don't know 50 milliseconds. But anyway for real time, it's more than enough
27:09
Hi, so first, thank you very much for the talk and I will have several questions namely Historically this kind of problems were handled with tools that are labeled as a CEP
27:24
Complex event processing and a lot of companies have invested a lot of money to solve that problem You have storm from Twitter, which was also open source. You have the lambda architecture from LinkedIn Samza spark and a
27:41
Gazillion tools and I'm just interested. Why did you feel that you have to invent something similar and which is like Very similar to the existing tools. That would be the first part of the question Well, I may disagree with that. They are very similar. This is completely written in Python
28:04
That might be an advantage. It is an advantage to us and we like that This project of course has evolved, okay, we didn't have this intention We didn't want to implement this at first right? We started simply using serum Q and separate processes, but at the end we find out that these tools
28:22
Could be glued together and work pretty well. So for example the use of pyro. I don't think you can Do things like that with those tools that you're mentioning and also while having a full Python implementation Yeah, that's a great answer The problems you mentioned are of course solved in a different way in different frameworks
28:44
But okay. Yeah So the other part would be I didn't notice that you would talk about the edge cases which are like very important in frameworks like this and Especially if you're playing with money So I'm talking about things like how are you guaranteeing that the events are processed exactly once and
29:08
You know If you're connected to several data streams How are you really sure that the events that are coming in are really ordered? Because it could happen that you get from Wanda
29:23
Sorry, I really what sorry I really Ordered okay one after another Could happen that you get something from Wanda at time T, but because of latency issues or whatever you get from Duca's copy or whatever exchange event which
29:42
Happened before that but later so traditionally people used something like Kafka Which solves your locking problems in distributed systems? Because of course traditionally you have a lot of smaller systems Which ones which wants to access the latest state of the system so you have the locking problems
30:03
That was a long question Anyway, yeah, I agree. Thank you Yeah, the thing is that with serum Q and that's what I like about this architecture. You are free to implement your Message passing so you mean how can you be sure that you only receive the message one? Well, it depends on your application is if that is important for you. You can implement manually those patterns for communications
30:25
For our case in fact, which is just publish subscribe because in real time we don't care Even if we don't receive one tick once which is something that doesn't happen, but we don't care We just have a flow of data And we want to process it as fast as possible if there is little wrong data or some missing data
30:45
We don't care. We just want to keep working. Okay, normally that would that won't affect your strategy So you're not afraid that you would make Based on different data the same decision twice because your workers would process the same event twice or
31:07
Three times and you would then basically go the same but as night capital or something like that Can you repeat the question? Sure, so you're saying that you're not afraid that based on one event
31:22
the workers would processes Process it multiple times and know that that won't happen Sorry if you understood that no that won't happen because we are only send Sending those that data once okay The worst that can happen is that we miss a packet
31:40
Okay a message once we don't care about that because usually you just have a tick So maybe your bar will be formed a little bit difference a little bit a little bit different But that's not really important When you change brokers the data that you get from those brokers is also different so It's not important if the data that you get is one percent different, which is which is not is
32:07
We don't care. Okay, so that's the good thing about this that we can implement Flexible not very strict message passing such as publish subscribe that works pretty well and pretty fast. We are more interested in
32:20
Speed than in for example getting all the exact data that brokers is providing us Because we don't trust that data in a way not that much
32:41
Okay, it's first. Thank you for this wonderful talk and my question is That in such multi-agent systems. We have another problem is for example, that's one of the processes May die unexpectedly and so how to monitor This overall system and
33:03
What is the reaction if for example some agent dies Yeah, of course monitoring is also implemented by the developer the developer so that depends on your application Well, we are monitoring our agents and our brains and one of the best things about this architecture
33:21
Is that exactly one brain for example dies crashes, which is something that should never happen and never happens Then you can kill it and you can respond it and everything works as expected and why is that and that it's again thanks to zero MQ because for example You can bind with sorry you can connect to a to an address that is not binded yet
33:46
So there's no problem. There's a client that is connected or subscribed to a Brain that is publishing information Then this brain can die We can kill it we can create a new one and it will start receiving the same information It won't break anything around so agents are completely independent
34:15
Thank you great talk NumPy arrays NumPy arrays why why did you choose those what features?
34:26
Were you looking for that? You couldn't find in in basic types. That's that's one point and second. I'd agree with the last two questions about Such a complex system. You've got an agent going down coming up
34:41
the discontinuity Worries me greatly about especially handling finance Sorry, can you repeat the question? Can you speak a little bit slower? Sorry, the the question was why NumPy arrays okay, and also Reassure me about
35:01
discontinuities nodes popping up popping down You've talked about pickles. You've talked about pickles over the internet. This is scary. Yeah Okay I'll answer the second one first. I didn't talk about pickle over the internet It is true that is this system is completely unsafe by default So it is meant to be executed in a control or restricted environment
35:24
For example in a in a cluster or just in your own computer if you want to communicate Between agents in remote machines and you have to pass over the internet Then it is up to you to implement the the security So you may want to use a VPN or an SSH tunnel, but of course, this is not secure
35:43
it's only secure if you have program it and For the first question why NumPy arrays? the answer is that all the ecosystem available out there for data analysis, not all but Most of the ecosystem seems to work well with NumPy arrays. So why wouldn't why wouldn't we use NumPy arrays?
36:07
okay, well Why wouldn't you because they're more expensive and I was wondering what features you're after but don't worry about it The the idea that the brain though doesn't use your own DSL, but actually is just pure Python code
36:21
I don't see what your product offers besides a collection of already existing frameworks for creating a large system I'm struggling to see Where the value is Can you ask again the question a little slower? I have difficulties for hearing you Sorry
36:44
You don't have a DSL domain-specific language Which abstracts away all the bad things that you could do as a developer when writing a brain, huh? So what does your products offer in terms of stability and predictability?
37:04
That the components will work together as a system because I don't see any yet No, what would you just implement test of? The independent agents and then of the whole ecosystem. That's it
37:26
I'm sorry that all the questions you you get are the kind why you are not using these on that technology But my question to you is about My question is about twisted which it is Python and I would expect to be good in this kind of a situations and
37:42
You are not using it. I guess you have a reason why so as I said, this has been evolving In the past months, so we didn't start with the idea of creating our own tool, but that just happened I mean we liked zero MQ very much I don't know if the tools that you are mentioning implement this flexible low-level
38:05
Communication patterns or not. And also I don't know if they implement some things like pyro I guess we could implement another tool using that That framework for example, and then using pyro as well for the configuration. Yeah, maybe maybe that works as well
38:27
Hi again, so I'm very interested. How are you thinking or how are you approaching? Testing this framework because as you already mentioned it's a composition of separate agents and
38:45
I mean, how are you approaching the testing and How are you testing? The whole system. Are you just using end-to-end tests? What kind of tests are there? Any tools you could recommend? So what's your experience with that? Because I think it's
39:04
Very a very different problem than what we are doing. I mean most of us here probably Each day I mean we know we are aware of the tests for the websites and things like that and
39:21
This kind of problem looks a bit different. So I would be interested in how are you testing this beast? we are testing this unit using unit test and pytest and That's it. Anyway, I'll be happy to talk to you and I'm sure you will explain me better tools for doing this
39:46
Hey one quick question In finance trading you your brain must know a current state for example all your open positions and
40:00
I would like to know how you save this state and how you replay what needed to know the state and if for example a Brain dies How you still know your open positions? Well, your open positions are usually anyway stored in the broker
40:22
But of course we store it locally as well that depends on your implementation. You can make trader store that information or you can make if you're going to implement some kind of We are handling of the orders you can implement it in the brain. Anyway, we've got as I mentioned I think before
40:41
other agents such as the informer which are publishing information about for example, the account state or The rates are which you can trade your accounts currency against other currencies So in this in this sense is completely flexible. I mean, it's it's up to you
41:06
Great. Thank you to all of you for asking so many interesting questions And please join me in thanking all the speakers from this morning again