Regulating the Securities Market with Django and Pandas

Video thumbnail (Frame 0) Video thumbnail (Frame 663) Video thumbnail (Frame 2220) Video thumbnail (Frame 4845) Video thumbnail (Frame 6987) Video thumbnail (Frame 9729) Video thumbnail (Frame 10528) Video thumbnail (Frame 13901) Video thumbnail (Frame 16071) Video thumbnail (Frame 18815) Video thumbnail (Frame 19947) Video thumbnail (Frame 21592) Video thumbnail (Frame 23597) Video thumbnail (Frame 24377) Video thumbnail (Frame 25664) Video thumbnail (Frame 27789) Video thumbnail (Frame 28808) Video thumbnail (Frame 30493) Video thumbnail (Frame 31672) Video thumbnail (Frame 32239) Video thumbnail (Frame 32920) Video thumbnail (Frame 33553) Video thumbnail (Frame 34072) Video thumbnail (Frame 34874) Video thumbnail (Frame 35940) Video thumbnail (Frame 36740)
Video in TIB AV-Portal: Regulating the Securities Market with Django and Pandas

Formal Metadata

Regulating the Securities Market with Django and Pandas
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Developing applications for civil service organizations can be uniquely challenging. The presentation discusses our experience with MASS a market surveillance and monitoring application developed for the TTSEC. MASS is based on Django and Pandas. We highlight not only the techinal aspects of the solution but also address the HR and organizational factors that impacted on the project
Computer animation
Computer animation Regulator gene Software developer Weight View (database) Projective plane Sheaf (mathematics) Combinational logic Cartesian coordinate system System call Computer architecture
Presentation of a group Regulator gene Causality Projective plane Shared memory Set (mathematics) Computing platform Physical system
Mobile Web Statistics Multiplication sign Decision theory Projective plane Execution unit Mathematical analysis Design by contract Set (mathematics) Analytic set Cartesian coordinate system Rule of inference Subset Number Process (computing) Computer animation Personal digital assistant Self-organization Video game Endliche Modelltheorie Physical system
Point (geometry) Slide rule Connectivity (graph theory) Stress (mechanics) Cartesian coordinate system Exception handling Computer architecture Physical system
Web crawler Beta function Multiplication sign Virtual machine Insertion loss Client (computing) Replication (computing) Internetworking Term (mathematics) Physical system Pairwise comparison Arm Graph (mathematics) Stress (mechanics) Fitness function Instance (computer science) Price index File Transfer Protocol Database normalization Process (computing) Computer animation Addressing mode Self-organization Lastteilung Data logger Quicksort Window
Atomic nucleus Functional (mathematics) Dialect Beta function Link (knot theory) State of matter View (database) Multiplication sign Projective plane Mathematical analysis Syntaxbaum Time series Set (mathematics) Bit Heegaard splitting Data management Computer animation Query language Calculation Order (biology) Volumenvisualisierung Physical system
Computer animation Strategy game Multiplication sign Energy level Endliche Modelltheorie Table (information)
Group action Dialect Standard deviation Statistics Motion capture Virtual machine Total S.A. Computer animation Query language Order (biology) Normal (geometry) Energy level Extension (kinesiology) Information security
Group action Key (cryptography) Link (knot theory) Cellular automaton Mögliche-Welten-Semantik Line (geometry) Client (computing) Number Frequency Computer animation Personal digital assistant Network topology Order (biology) Table (information) Information security Physical system
Online chat Computer animation Open source Integrated development environment Query language Multiplication sign Software developer Projective plane Quicksort Client (computing) Table (information) Inequality (mathematics)
Laptop Point (geometry) Statistics Context awareness Computer file Source code Workstation <Musikinstrument> Set (mathematics) Number Term (mathematics) Profil (magazine) Energy level Endliche Modelltheorie Physical system Stapeldatei Projective plane Maxima and minima Directory service Cartesian coordinate system Computer animation Pattern language Right angle Object (grammar) Thermal conductivity Window
Spreadsheet Wechselseitige Information Voting Regulator gene Different (Kate Ryan album) Execution unit Right angle Rule of inference Computer programming
Module (mathematics) Laptop Functional (mathematics) Wrapper (data mining) Multiplication sign Projective plane Sheaf (mathematics) Right angle Bit Quicksort Freeware
Subject indexing Digital photography Data management Computer animation Endliche Modelltheorie Frame problem Field (computer science)
Data management Process (computing) Computer animation Network topology Execution unit Time series Freeware Frame problem
Multiplication sign Data storage device Time series Endliche Modelltheorie Object (grammar) Frame problem
Subject indexing Time series Series (mathematics) Endliche Modelltheorie Data structure
Data management Computer animation Logic
Data management Computer animation Logic Mixed reality Line (geometry) Freeware Social class
Computer animation
at the the the in the the the the in and around the also then press because there was that and I will
1st small development company have the center that call Chris them
and my talk today describes the challenges that they face in building stock market regulation of surveillance and monitoring application and how he looking those challenges of using a combination of John 1 planned but I do want to look for the Securities and Exchange Commission after that and so my views use of my own and right so I talk is divided up into 4 sections of the 1st thing we do is talk about some background to the project and introduce salt solution architecture and then you look at a lot I thought or John were reusable out call jungle ponders and then we take a questions of course OK so in 28 11
of them try and the biggest stock market was forced to migrate to a new trade and black hole and this is a cause that the company that made a trading platform and had gone bankrupt some years ago when could you along the and Mantena platform could no longer were given support on on the black hole so the new platform and was created traders but did of provide a lot the regulatory hoax so that the regulator regulators needed for monitoring and and civilian and the stock market I have done some previous work with the Securities and Exchange Commission so as not so presenter proposal for a system that could bridge the gap the the requirements would be ready well structured way to things that he wanted was that the systems everywhere bees because they wanted to share of whatever system was developed with other regulators such as the as the center of Bonn Ministry of Finance and so on and the data we must all the system would would use would be complete and finalize the data which I never got the phone but what that meant was that I and understand it all proposal is ac cepted we would then will be key users to develop a set of more 2 requirements of the project will of course that is
that it could lead to 1 thing the
1st ingredient called it was a very slow procurement process I mean we expected that it's a government kind of bureaucratic organization and but 6 months between tender and the proposal and discussed in an acceptance signed up to the project that's a little long but it more actually more surprising was the fact that there's a lot of mobility 1 might think well being of the people work bureaucracies they're there for life but that's not really true desolate interdepartmental bility people and leave and so on so that the so what the users and that at the talent you'd and everyone left was that who by time we finish the project and we deliver the system the key users and who freely requirements in the 1st please there was born on the people wanted the left organization and 4 so for going on is that a lot of people replace and then you'll have to kind of statistical trading all technical about wrong but this kind of system 1 year in any case you have to try to
come explicating promotes system the people like a new set of people every 3 or 4 months which is at least there a number of problems you know you have long lead times decision making and underneath the requirements that we had come the would not well understood by the the the final users because if people who specifying the requirements in a longer there and people who say people reduces so why do we have feature X because that's a requirement why is that the requirements you have to go to XPD rational seldom project once again so I and of course it was a well we should you should use agile methods assume to deal with such problems but the in government contract and you find that it is a implicit waterfall model you know so that if he he you have to the requirements is out of this is a deliverable and once the requirements applicant please the the need to stay the effort until the end of the project even ability in no longer relevant along nobody understands why the the soul the I have so despite these challenges we were able to deliver a system to the end users and started rule in adult in the 2 what about 22 unit is called a mass market analysis analytics and surveillance system and it's so
pretty is a standard jumbo application with a couple exceptions but so this is the basic architecture components here the things that the private obviously read wide while use and on the and then you see that Red Hat Enterprise Linux 5 . 2 and for stress the point to will work in subsequent slides you explain why as here
the system that that grammatical and
so the set up the the 0 and maybe I went to the flies to quickly but do we have some really ancient down machines was but but are still out a legacy of but still consolidation exercise that was done in 2010 and again in government style organization is the idea here the whatever is done that has the speak and kind of creative 20 16 of obviously the bill of a modern John who application and because ran and the price index comes with at a python 2 . 5 we have to operate on tight on and and we have a Proc from process of my reading the system to post stress 9 . 3 so it would be if you know it in the background we we had we have to do so was when the at we originally hatchery but 1 was taken away from us because and xt feel some kind of loss of we we have to do with these 2 so so have we have our tool gone on and instances and we use the Internet I have the wrong robin load balancing and the PG full will is because we were using now and because use and pose graphs it . 3 or e . 1 when started is required for replication sorry replicated each of these between these 2 so it was so we get some kind of redundancy your arm and so when we when we finally migrate opposed as 9 . 3 we might get rid of the PG pool and replace it with something like bond so of the system itself
is very read centric and because the fit so you we have 1 place that ready upload data and it's not a real-time system because again the the users of the clients and once that we look at finer beta and so we in terms of the it so we we retrieve data from the trade in systems and so as long we we actually have a request um FTP or SIFT peas sound system of job that was up when they retrieves this log file which is an XML and monstrosity and we pass it these and and and and the and time that L XML because with it's really really horrible formatted windows size and and so outside of the trade in the data we also have a bunch of scrapie spiders that we use to get the data from various stock markets in the region because they have to use it for comparisons and that sort of thing and OK so what is as
use for in the system so we
we obviously we do a lot of so in statistical and time series that so that the quality of binders and then what what 1 of the 1 of the things that we found that we needed to do that because goes was an implicit in the requirements is that we need link trade borders and Buitelaar and rebuild the history of trade from when the order was originally nucleus so when it when it was changed and and and so forth so we react with the use of compound as an approach of split up that apply combined to reconstruct trade order books and calculate duration and 2 regions of the states I just and use of time as the system is the ability to do that what the analysis on the so we actually generate poverty was on flying by and this is rare as a lot of facility with the the then I actually have to you violence in all of and jungle views on because it can be easily rendered so and as data free could easily be rendered as Jesus on all CST or HTML or XML but how and and we have a project hola John Honda's which provides a manager and some other of functionality so so that you could easily render pandas beta freedoms from query sets so let's talk a little bit
doubled the normalization and and
passion so it would assist them as heavily statistically with other statistical packages obviously we're going to do that on the fly so we have a the the summary detail and the so tables all models storing the the summary the time these are populated when we actually look forward to the raw data and and we also when I extend this strategy so was so I have the level if you want the more expensive statistical calculations of
straighten out we not ready to make and that extensive use of actions I have said at the level of the dashboard and that however we we we intend to use and jungle cast machine to and do some query set passion and and it it would be the thing that is holding hold the data of the normalization and captures is the fact that the user requirement is that they must have room total flexibility all our the 2 regions and securities so there's no standard of queries so you can use you banner limited as to what you know what what you could pass because the 1 to putting any to so that they can get the statistics calculated but and we will get them to change their minds of order the Dutch requirements so
this is not this is you these are some screenshots for the mass whereby up so this is the gas you in some of the statistics that deleted this is
our activity filter which allows users to drill down from a traded and occur could click on it any trade unseen or the order history and behind the tree and knowledge you'd expect this this should be easy because clearly in any real system if you have a tree it would have a key link in trades the order but for some strange reason this trading system they would normally be treating trades and order so you actually have to use find capability of getting a 1 candidate groups where where this this this this trait could be linked to and then find the actual world trade so that at and if you can see the really green thing there so that's the 1 that is the the trade up above so I don't think that could have done easily well on this set an it's only about 4 5 lines in in parliament or use
and so this is equivalent book so in this case now we take in a trader and we are looking at history to about particular period that we have within our own we break and Dong by security and client and looking at a number of trades and you can drill down from this into the and to define a lot more of audio or what the traits and orders that meet up this this any cell in the table this is we use
Fondation by and of the it's it's some it works best and cruel and users love that because you know the time for some reason he has correct it's not to the developers heat and with users the even corporate environments so I so high taxes you sort of chart high chats and high stock they are that with this not open source and we use g querying beta tables for the interactive green granted the or any criticism of all that about the query data is not responsive but it's it's it's it's improved a lot
OK so that that's the traditional client it if you if you inequality that we was a UCI fighter new book with this project the idea came from 1 of our
directors who wanted to the give the ability of of the analyst of to perform ad-hoc statistical pattern lesions using the data that we had in system I sold out we we try and we did a number of experiments and so on and we came up with the best way to do it was that basically install so I'm kind of conduct typed on on the users of moves Windows workstations and put the the actual project source code and and we have as a minimum of sectors module and when he I installed actually points that the that the mid term to the to the model is that the 1 that's for was that the i prime on notebook level and we create a profile for each application that we want to use and the I I I quite early because this home start-up directory so you put us through the air that gives you that
sets the but they include application ancestor jungle um sentence more argues that any could imported on objects in the context of typewriter new book or sorry general objects in the context of my mind on new so then we gotta start up a new book uses the command and then the make it even easier we create a batch file and stick it on the desktop so the so could out and they believe it's kind in the right so as well the user's initiative reaction will you
wanted to in the program of the week right OK so yes yes but a few weeks later
the regulator the with the central bank of I wanted historical data for mutual funds and want to that in a certain way they wanted to put each is your what was the funds for that is you want 1 spreadsheet 1 rule book and a spread a sheet but each is through and that's like hundreds of the vote 40 something unit trust me and the issue was on hundreds of different funds and the human admit manually and so 1 of the analyst says and he wanted it the more you so why the analyst and set as I could you globin on a boat something that this thing could produce an excel spreadsheets and you do something about that so I sat down with the analyst and came up with
this the which basically did what we wanted right so when it when sort out the became into the right and I I put together some wrapper functions and and I added some documentation to the new book so this is the last section of this users I will put their use all the time I leave you with the modified it I just added the year of functions that I I B so they take somebody more complex stuff than I put it in the stock of 5 so what the right I know I know at all the little bits of like John rule and there's module and it it is
very is is them it's it's being used for the not in this project because we already use jungle in the traditional we what we try to use just go to the IPython notebook so it has 2 basic modules I will mark you and idea free market
at nite so this letter because jungle model the the this is
how you use the greedy every matter which is currently the only method in the I want you so it can adapt you could create a the the frame with all the you specific fields you can give you got it creates an index and you can use photos doesn't exclude the data managers based on
the past through my job by Pierre much and it's in the jungle 1 unit of which is a great my ecology and so what you do is you overwrite the forward manager with the jungle the data free manager and that this gives you access to tree matter
to a data frame 2 time series and to to that but the will be 2 data
frame method does exactly what we read I you know it to
time-series matter that supports 2 kinds of storage models of time series long and wide
so you why it is this basically up 1 each column in the all represents a column in the time series of time series is the the freed indexed by at the time of object In the long time
series model each um the did so in the structure of best 3 different series and is stored the last
I and and when you use it to time-series methods pose molten structures them as the the freeing with a time series index the talk
but we also have but as I
said before support but he ran the ran thing about it this as yeah you could use the but you rarely if the is to and some whole as the the the free monitor and you all us manager would the methods and and business logic that you need so when I started out when I
started my might use is look like this I had a mixing with all the business logic to create a a trade data free of so much you classes we huge and then I realized I could just solve class my
the the free manager and I could put all the logic so then I just have a single line a clue that that does you know what what I had before
we here and FIL and and you you you