Merken

Regulating the Securities Market with Django and Pandas

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
at the the the in the the the the in and around the also then press because there was that and I will
1st small development company have the center that call Chris them
and my talk today describes the challenges that they face in building stock market regulation of surveillance and monitoring application and how he looking those challenges of using a combination of John 1 planned but I do want to look for the Securities and Exchange Commission after that and so my views use of my own and right so I talk is divided up into 4 sections of the 1st thing we do is talk about some background to the project and introduce salt solution architecture and then you look at a lot I thought or John were reusable out call jungle ponders and then we take a questions of course OK so in 28 11
of them try and the biggest stock market was forced to migrate to a new trade and black hole and this is a cause that the company that made a trading platform and had gone bankrupt some years ago when could you along the and Mantena platform could no longer were given support on on the black hole so the new platform and was created traders but did of provide a lot the regulatory hoax so that the regulator regulators needed for monitoring and and civilian and the stock market I have done some previous work with the Securities and Exchange Commission so as not so presenter proposal for a system that could bridge the gap the the requirements would be ready well structured way to things that he wanted was that the systems everywhere bees because they wanted to share of whatever system was developed with other regulators such as the as the center of Bonn Ministry of Finance and so on and the data we must all the system would would use would be complete and finalize the data which I never got the phone but what that meant was that I and understand it all proposal is ac cepted we would then will be key users to develop a set of more 2 requirements of the project will of course that is
that it could lead to 1 thing the
1st ingredient called it was a very slow procurement process I mean we expected that it's a government kind of bureaucratic organization and but 6 months between tender and the proposal and discussed in an acceptance signed up to the project that's a little long but it more actually more surprising was the fact that there's a lot of mobility 1 might think well being of the people work bureaucracies they're there for life but that's not really true desolate interdepartmental bility people and leave and so on so that the so what the users and that at the talent you'd and everyone left was that who by time we finish the project and we deliver the system the key users and who freely requirements in the 1st please there was born on the people wanted the left organization and 4 so for going on is that a lot of people replace and then you'll have to kind of statistical trading all technical about wrong but this kind of system 1 year in any case you have to try to
come explicating promotes system the people like a new set of people every 3 or 4 months which is at least there a number of problems you know you have long lead times decision making and underneath the requirements that we had come the would not well understood by the the the final users because if people who specifying the requirements in a longer there and people who say people reduces so why do we have feature X because that's a requirement why is that the requirements you have to go to XPD rational seldom project once again so I and of course it was a well we should you should use agile methods assume to deal with such problems but the in government contract and you find that it is a implicit waterfall model you know so that if he he you have to the requirements is out of this is a deliverable and once the requirements applicant please the the need to stay the effort until the end of the project even ability in no longer relevant along nobody understands why the the soul the I have so despite these challenges we were able to deliver a system to the end users and started rule in adult in the 2 what about 22 unit is called a mass market analysis analytics and surveillance system and it's so
pretty is a standard jumbo application with a couple exceptions but so this is the basic architecture components here the things that the private obviously read wide while use and on the and then you see that Red Hat Enterprise Linux 5 . 2 and for stress the point to will work in subsequent slides you explain why as here
the system that that grammatical and
so the set up the the 0 and maybe I went to the flies to quickly but do we have some really ancient down machines was but but are still out a legacy of but still consolidation exercise that was done in 2010 and again in government style organization is the idea here the whatever is done that has the speak and kind of creative 20 16 of obviously the bill of a modern John who application and because ran and the price index comes with at a python 2 . 5 we have to operate on tight on and and we have a Proc from process of my reading the system to post stress 9 . 3 so it would be if you know it in the background we we had we have to do so was when the at we originally hatchery but 1 was taken away from us because and xt feel some kind of loss of we we have to do with these 2 so so have we have our tool gone on and instances and we use the Internet I have the wrong robin load balancing and the PG full will is because we were using now and because use and pose graphs it . 3 or e . 1 when started is required for replication sorry replicated each of these between these 2 so it was so we get some kind of redundancy your arm and so when we when we finally migrate opposed as 9 . 3 we might get rid of the PG pool and replace it with something like bond so of the system itself
is very read centric and because the fit so you we have 1 place that ready upload data and it's not a real-time system because again the the users of the clients and once that we look at finer beta and so we in terms of the it so we we retrieve data from the trade in systems and so as long we we actually have a request um FTP or SIFT peas sound system of job that was up when they retrieves this log file which is an XML and monstrosity and we pass it these and and and and the and time that L XML because with it's really really horrible formatted windows size and and so outside of the trade in the data we also have a bunch of scrapie spiders that we use to get the data from various stock markets in the region because they have to use it for comparisons and that sort of thing and OK so what is as
use for in the system so we
we obviously we do a lot of so in statistical and time series that so that the quality of binders and then what what 1 of the 1 of the things that we found that we needed to do that because goes was an implicit in the requirements is that we need link trade borders and Buitelaar and rebuild the history of trade from when the order was originally nucleus so when it when it was changed and and and so forth so we react with the use of compound as an approach of split up that apply combined to reconstruct trade order books and calculate duration and 2 regions of the states I just and use of time as the system is the ability to do that what the analysis on the so we actually generate poverty was on flying by and this is rare as a lot of facility with the the then I actually have to you violence in all of and jungle views on because it can be easily rendered so and as data free could easily be rendered as Jesus on all CST or HTML or XML but how and and we have a project hola John Honda's which provides a manager and some other of functionality so so that you could easily render pandas beta freedoms from query sets so let's talk a little bit
doubled the normalization and and
passion so it would assist them as heavily statistically with other statistical packages obviously we're going to do that on the fly so we have a the the summary detail and the so tables all models storing the the summary the time these are populated when we actually look forward to the raw data and and we also when I extend this strategy so was so I have the level if you want the more expensive statistical calculations of
straighten out we not ready to make and that extensive use of actions I have said at the level of the dashboard and that however we we we intend to use and jungle cast machine to and do some query set passion and and it it would be the thing that is holding hold the data of the normalization and captures is the fact that the user requirement is that they must have room total flexibility all our the 2 regions and securities so there's no standard of queries so you can use you banner limited as to what you know what what you could pass because the 1 to putting any to so that they can get the statistics calculated but and we will get them to change their minds of order the Dutch requirements so
this is not this is you these are some screenshots for the mass whereby up so this is the gas you in some of the statistics that deleted this is
our activity filter which allows users to drill down from a traded and occur could click on it any trade unseen or the order history and behind the tree and knowledge you'd expect this this should be easy because clearly in any real system if you have a tree it would have a key link in trades the order but for some strange reason this trading system they would normally be treating trades and order so you actually have to use find capability of getting a 1 candidate groups where where this this this this trait could be linked to and then find the actual world trade so that at and if you can see the really green thing there so that's the 1 that is the the trade up above so I don't think that could have done easily well on this set an it's only about 4 5 lines in in parliament or use
and so this is equivalent book so in this case now we take in a trader and we are looking at history to about particular period that we have within our own we break and Dong by security and client and looking at a number of trades and you can drill down from this into the and to define a lot more of audio or what the traits and orders that meet up this this any cell in the table this is we use
Fondation by and of the it's it's some it works best and cruel and users love that because you know the time for some reason he has correct it's not to the developers heat and with users the even corporate environments so I so high taxes you sort of chart high chats and high stock they are that with this not open source and we use g querying beta tables for the interactive green granted the or any criticism of all that about the query data is not responsive but it's it's it's it's improved a lot
OK so that that's the traditional client it if you if you inequality that we was a UCI fighter new book with this project the idea came from 1 of our
directors who wanted to the give the ability of of the analyst of to perform ad-hoc statistical pattern lesions using the data that we had in system I sold out we we try and we did a number of experiments and so on and we came up with the best way to do it was that basically install so I'm kind of conduct typed on on the users of moves Windows workstations and put the the actual project source code and and we have as a minimum of sectors module and when he I installed actually points that the that the mid term to the to the model is that the 1 that's for was that the i prime on notebook level and we create a profile for each application that we want to use and the I I I quite early because this home start-up directory so you put us through the air that gives you that
sets the but they include application ancestor jungle um sentence more argues that any could imported on objects in the context of typewriter new book or sorry general objects in the context of my mind on new so then we gotta start up a new book uses the command and then the make it even easier we create a batch file and stick it on the desktop so the so could out and they believe it's kind in the right so as well the user's initiative reaction will you
wanted to in the program of the week right OK so yes yes but a few weeks later
the regulator the with the central bank of I wanted historical data for mutual funds and want to that in a certain way they wanted to put each is your what was the funds for that is you want 1 spreadsheet 1 rule book and a spread a sheet but each is through and that's like hundreds of the vote 40 something unit trust me and the issue was on hundreds of different funds and the human admit manually and so 1 of the analyst says and he wanted it the more you so why the analyst and set as I could you globin on a boat something that this thing could produce an excel spreadsheets and you do something about that so I sat down with the analyst and came up with
this the which basically did what we wanted right so when it when sort out the became into the right and I I put together some wrapper functions and and I added some documentation to the new book so this is the last section of this users I will put their use all the time I leave you with the modified it I just added the year of functions that I I B so they take somebody more complex stuff than I put it in the stock of 5 so what the right I know I know at all the little bits of like John rule and there's module and it it is
very is is them it's it's being used for the not in this project because we already use jungle in the traditional we what we try to use just go to the IPython notebook so it has 2 basic modules I will mark you and idea free market
at nite so this letter because jungle model the the this is
how you use the greedy every matter which is currently the only method in the I want you so it can adapt you could create a the the frame with all the you specific fields you can give you got it creates an index and you can use photos doesn't exclude the data managers based on
the past through my job by Pierre much and it's in the jungle 1 unit of which is a great my ecology and so what you do is you overwrite the forward manager with the jungle the data free manager and that this gives you access to tree matter
to a data frame 2 time series and to to that but the will be 2 data
frame method does exactly what we read I you know it to
time-series matter that supports 2 kinds of storage models of time series long and wide
so you why it is this basically up 1 each column in the all represents a column in the time series of time series is the the freed indexed by at the time of object In the long time
series model each um the did so in the structure of best 3 different series and is stored the last
I and and when you use it to time-series methods pose molten structures them as the the freeing with a time series index the talk
but we also have but as I
said before support but he ran the ran thing about it this as yeah you could use the but you rarely if the is to and some whole as the the the free monitor and you all us manager would the methods and and business logic that you need so when I started out when I
started my might use is look like this I had a mixing with all the business logic to create a a trade data free of so much you classes we huge and then I realized I could just solve class my
the the free manager and I could put all the logic so then I just have a single line a clue that that does you know what what I had before
we here and FIL and and you you you
Systemprogrammierung
Loop
Imaginäre Zahl
Computeranimation
Architektur <Informatik>
Sichtenkonzept
Gewicht <Mathematik>
Schaltnetz
Systemaufruf
Kartesische Koordinaten
Gebäude <Mathematik>
Sichtenkonzept
Computeranimation
Font
Projektive Ebene
Garbentheorie
Computerarchitektur
Softwareentwickler
Regulator <Mathematik>
Webforum
Physikalischer Effekt
Gemeinsamer Speicher
Systemplattform
Physikalisches System
Kombinatorische Gruppentheorie
Migration <Informatik>
Systemplattform
Regulator <Mathematik>
Computeranimation
Physikalisches System
Menge
Projektive Ebene
Bridge <Kommunikationstechnik>
Regulator <Mathematik>
Prozess <Physik>
Selbst organisierendes System
Zahlenbereich
Kartesische Koordinaten
Analytische Menge
Computeranimation
Entscheidungstheorie
Physikalisches System
Informationsmodellierung
Selbst organisierendes System
Einheit <Mathematik>
Standardabweichung
Design by Contract
Analysis
Videospiel
Statistik
Prozess <Informatik>
Mobiles Internet
Datenmodell
Ruhmasse
Schlussregel
Physikalisches System
Entscheidungstheorie
Design by Contract
Teilmenge
Funktion <Mathematik>
Menge
Ruhmasse
Projektive Ebene
Rechenschieber
Dualitätstheorie
Architektur <Informatik>
Punkt
Speicherabzug
Ausnahmebehandlung
Zusammenhängender Graph
Kartesische Koordinaten
Physikalisches System
Computerarchitektur
Normalspannung
Computeranimation
Retrievalsprache
Server
Einfügungsdämpfung
Prozess <Physik>
Filetransferprotokoll
Selbst organisierendes System
Relationentheorie
Adressierung
Summengleichung
Ungerichteter Graph
Term
Analysis
Computeranimation
Lastteilung
Internetworking
Physikalisches System
Virtuelle Maschine
Client
Prozess <Informatik>
Datenreplikation
Bildschirmfenster
Indexberechnung
Strom <Mathematik>
Transaktionsverwaltung
Hardware
Distributionstheorie
Prozess <Informatik>
Datenlogger
Spider <Programm>
Betafunktion
Datenreplikation
Physikalisches System
Paarvergleich
Migration <Informatik>
Quick-Sort
Web log
Normalspannung
Instantiierung
Fitnessfunktion
Retrievalsprache
Bit
Wellenlehre
Pivot-Operation
Analysis
Rechenbuch
Computeranimation
Datenmanagement
Zeitreihenanalyse
Statistische Analyse
Volumenvisualisierung
Operations Research
Analysis
Tabelle <Informatik>
Lineares Funktional
Sichtenkonzept
Syntaxbaum
Betafunktion
Abfrage
Physikalisches System
Binder <Informatik>
Rechnen
Sichtenkonzept
Dialekt
Kern <Mathematik>
Menge
Reihe
Rahmenproblem
Menge
Komplex <Algebra>
Heegaard-Zerlegung
Projektive Ebene
Ordnung <Mathematik>
Aggregatzustand
Relationentheorie
Caching
Informationsmodellierung
Prozess <Informatik>
Renormierung
Strategisches Spiel
Modelltheorie
Computeranimation
Strategisches Spiel
Übergang
Tabelle <Informatik>
Caching
Retrievalsprache
Statistik
Total <Mathematik>
Wellenlehre
Computersicherheit
Gruppenoperation
Virtuelle Maschine
Ruhmasse
Abfrage
Dialekt
Computeranimation
Übergang
Spannweite <Stochastik>
Motion Capturing
Virtuelle Maschine
Ruhmasse
Maßerweiterung
Ordnung <Mathematik>
Normalvektor
Standardabweichung
Inklusion <Mathematik>
Tabelle <Informatik>
Computersicherheit
Pivot-Operation
Gruppenkeim
Zahlenbereich
Zellularer Automat
Mögliche Welt
Digitalfilter
Physikalisches System
Binder <Informatik>
Frequenz
Computeranimation
Netzwerktopologie
Client
Lemma <Logik>
Ordnung <Mathematik>
Simulation
Schlüsselverwaltung
Gerade
Normalvektor
Tabelle <Informatik>
Open Source
Abfrage
Quick-Sort
Computeranimation
Client
Ungleichung
Interaktives Fernsehen
Framework <Informatik>
Chatten <Kommunikation>
Notebook-Computer
Projektive Ebene
Softwareentwickler
Programmierumgebung
Benutzerführung
Tabelle <Informatik>
Schnittstelle
Punkt
Extrempunkt
Virtuelle Maschine
Zahlenbereich
Kartesische Koordinaten
Term
Computeranimation
Übergang
Open Source
Informationsmodellierung
Verzeichnisdienst
Notebook-Computer
Arbeitsplatzcomputer
Bildschirmfenster
Mustersprache
Speicherabzug
Skript <Programm>
Modul
Äquivalenzklasse
Statistik
Installation <Informatik>
Profil <Aerodynamik>
Physikalisches System
Quellcode
Elektronische Publikation
Kontextbezogenes System
Arbeitsplatzcomputer
Benutzerprofil
Objekt <Kategorie>
Menge
Rechter Winkel
Client
Ruhmasse
Notebook-Computer
Projektive Ebene
Modelltheorie
Stapelverarbeitung
Verzeichnisdienst
Wärmeleitfähigkeit
Abstimmung <Frequenz>
Subtraktion
Einheit <Mathematik>
Tabellenkalkulation
Rechter Winkel
Wechselseitige Information
Notebook-Computer
Schlussregel
Optimierung
Regulator <Mathematik>
Regulator <Mathematik>
Retrievalsprache
Bit
Stellenring
Freeware
Computeranimation
Spezifisches Volumen
Modul <Datentyp>
Notebook-Computer
Wrapper <Programmierung>
Modul
Lineares Funktional
Default
Indexberechnung
Speicher <Informatik>
Volumenvisualisierung
Modul
Quick-Sort
Menge
Rahmenproblem
Körper <Physik>
Funktion <Mathematik>
Rechter Winkel
Projektive Ebene
Garbentheorie
Notebook-Computer
Modelltheorie
Rahmenproblem
Indexberechnung
Digitalfilter
Computeranimation
Rahmenproblem
Informationsmodellierung
Datenfeld
Datenmanagement
Umwandlungsenthalpie
Körper <Physik>
PCMCIA
Automatische Indexierung
Digitale Photographie
Modelltheorie
Körpertheorie
Objekt <Kategorie>
Netzwerktopologie
Einheit <Mathematik>
Datenmanagement
Rahmenproblem
Zeitreihenanalyse
Freeware
Prozess <Informatik>
Modelltheorie
Computeranimation
Objekt <Kategorie>
Rahmenproblem
Schreib-Lese-Kopf
Datenmodell
Indexberechnung
Speicher <Informatik>
Digitalfilter
Frequenz
Computeranimation
Spezifisches Volumen
Objekt <Kategorie>
Reihe
Informationsmodellierung
Zeitreihenanalyse
Speicher <Informatik>
Modelltheorie
HMS <Fertigung>
Objekt <Kategorie>
Reihe
Informationsmodellierung
Zeitreihenanalyse
Schreib-Lese-Kopf
Automatische Indexierung
Stichprobe
Datenmodell
Reihe
Speicher <Informatik>
Digitalfilter
Modelltheorie
Datenstruktur
Computeranimation
Objekt <Kategorie>
Mathematische Logik
Datenmanagement
Abstraktionsebene
Modelltheorie
Sichtenkonzept
Mathematische Logik
Computeranimation
Objekt <Kategorie>
Freeware
Klasse <Mathematik>
Datenmodell
Abelsche Kategorie
Symboltabelle
Digitalfilter
Mathematische Logik
Sichtenkonzept
Computeranimation
Serviceorientierte Architektur
Datenmanagement
Login
Mixed Reality
Computersicherheit
Client
Warping
Gerade
Videokonferenz
Ereignishorizont
Computeranimation

Metadaten

Formale Metadaten

Titel Regulating the Securities Market with Django and Pandas
Serientitel DjangoCon US 2014
Teil 28
Anzahl der Teile 44
Autor Clarke, Christopher
Mitwirkende Confreaks, LLC
Lizenz CC-Namensnennung - Weitergabe unter gleichen Bedingungen 4.0 International:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben.
DOI 10.5446/32849
Herausgeber DjangoCon US
Erscheinungsjahr 2014
Sprache Englisch

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Developing applications for civil service organizations can be uniquely challenging. The presentation discusses our experience with MASS a market surveillance and monitoring application developed for the TTSEC. MASS is based on Django and Pandas. We highlight not only the techinal aspects of the solution but also address the HR and organizational factors that impacted on the project

Ähnliche Filme

Loading...