Security when nanoseconds count

Video thumbnail (Frame 0) Video thumbnail (Frame 1535) Video thumbnail (Frame 2761) Video thumbnail (Frame 3662) Video thumbnail (Frame 4545) Video thumbnail (Frame 6433) Video thumbnail (Frame 8764) Video thumbnail (Frame 11005) Video thumbnail (Frame 14683) Video thumbnail (Frame 16058) Video thumbnail (Frame 18382) Video thumbnail (Frame 19511) Video thumbnail (Frame 23546) Video thumbnail (Frame 25801) Video thumbnail (Frame 28068) Video thumbnail (Frame 29425) Video thumbnail (Frame 30277) Video thumbnail (Frame 32900) Video thumbnail (Frame 34818) Video thumbnail (Frame 36026) Video thumbnail (Frame 38264) Video thumbnail (Frame 40381) Video thumbnail (Frame 41325) Video thumbnail (Frame 42813) Video thumbnail (Frame 44822) Video thumbnail (Frame 45739) Video thumbnail (Frame 49426) Video thumbnail (Frame 50326) Video thumbnail (Frame 51865) Video thumbnail (Frame 53067) Video thumbnail (Frame 54847) Video thumbnail (Frame 56235) Video thumbnail (Frame 57596) Video thumbnail (Frame 58946) Video thumbnail (Frame 59845) Video thumbnail (Frame 61088) Video thumbnail (Frame 62585) Video thumbnail (Frame 63976) Video thumbnail (Frame 64848) Video thumbnail (Frame 65799)
Video in TIB AV-Portal: Security when nanoseconds count

Formal Metadata

Security when nanoseconds count
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
James "Myrcurial" Arlen - Security When Nano Seconds Count White Paper Here: There's a brave new frontier for IT Security - a place where "best practices" does not even contemplate the inclusion of a firewall in the network. This frontier is found in the most unlikely of places, where it is presumed that IT Security is a mature practice. Banks, Financial Institutions and Insurance Companies. High Speed Trading, High Frequency Trading, Low Latency Trading, Algorithmic Trading -- all words for electronic trades committed in microseconds without the intervention of humans. There are no firewalls, everything is custom and none of it is secure. It's SkyNet for Money and it's happening now. Speaker , CISA, is Principal at Push The Stack Consulting providing security consulting services to the utility and financial verticals. He has been involved with implementing a practical level of information security in Fortune 500, TSE 100, and major public-sector corporations for more than 15 years. James is also a contributing analyst with Securosis, founder of the think|haus hackerspace and has a recurring column on Liquidmatrix Security Digest. Best described as: "Infosec geek, hacker, social activist, author, speaker, and parent." His areas of interest include organizational change, social engineering, blinky lights and shiny things. Twitter: @myrcurial

Related Material

Video is accompanying material for the following resource
The following resource is accompanying material for the video
Open source Multiplication sign Firewall (computing) Expert system Sound effect Client (computing) Disk read-and-write head Number Process (computing) Different (Kate Ryan album) Vertex (graph theory) Utility software Information security output Information security
Programming language Building Digital electronics Total S.A. Mathematician Compilation album Twitter
Velocity Maxima and minima Distance Spacetime Neuroinformatik
Computer program System call 2 (number)
Satellite Message passing Multiplication sign Computer science Fiber bundle Compilation album Number 2 (number) Row (database) Formal language
Metre Vacuum Scaling (geometry) Decision support system Multiplication sign Plastikkarte Hecke operator Plastikkarte Distance Limit (category theory) Number Spacetime
Cue sports Real number Tape drive Process capability index Mereology Computer Neuroinformatik Wave packet Telecommunication Hacker (term) Average Charge carrier Alphabet (computer science) Vulnerability (computing) Social class Physical system Vulnerability (computing) Focus (optics) Multiplication Tape drive Software developer Plastikkarte Bit Software maintenance Type theory Subject indexing Arithmetic mean Software Telecommunication System programming Charge carrier Self-organization Quicksort Whiteboard Row (database)
Point (geometry) Purchasing Algorithm Chemical equation Multiplication sign Interactive television Flash memory Event horizon Number Neuroinformatik Formal language Mathematics Different (Kate Ryan album) Term (mathematics) Spacetime Information Scale (map) Scaling (geometry) Information Digitizing Moment (mathematics) Open source Interactive television Shared memory Personal digital assistant Self-organization Natural language Right angle Quicksort Resultant Spacetime
NP-hard Pulse (signal processing) Game controller Open source Multiplication sign Virtual machine Device driver Field programmable gate array Stack (abstract data type) 2 (number) Neuroinformatik Wave packet Architecture Optical disc drive Goodness of fit Very-high-bit-rate digital subscriber line Semiconductor memory Internetworking Core dump Computer hardware Cuboid Diagram God Physical system Computer architecture Scale (map) Information Software developer Interface (computing) PCI Express Plastikkarte Database transaction Prediction Cartesian coordinate system Category of being Type theory Software Right angle Quicksort Spacetime
Laptop Point (geometry) Server (computing) Multiplication sign Insertion loss Distance Mereology Automatic differentiation Predictability 2 (number) Number Neuroinformatik Roundness (object) Term (mathematics) Computer hardware Data conversion Communications protocol Router (computing) Position operator Dependent and independent variables Bit Line (geometry) Fehlererkennung 10 (number) Personal digital assistant Calculation Right angle Collision Quicksort Communications protocol
Satellite Predictability Metre Building Quicksort Local area network Metropolitan area network Fiber (mathematics) Distance 2 (number) Sound effect
Link (knot theory) Patch (Unix) Moment (mathematics) Connected space Neuroinformatik Intranet Virtual LAN Software Frame Relay Cuboid Diagram Graph drawing Right angle Fiber (mathematics) Physical system
Game controller Firewall (computing) Firewall (computing) Multiplication sign Decision theory Electronic mailing list Maxima and minima Process capability index Line (geometry) Canadian Light Source IP address Entire function Unified threat management Web 2.0 Data center Cuboid Quicksort Router (computing) Information security Position operator Physical system
Code Field programmable gate array Software bug Different (Kate Ryan album) Cuboid God Physical system MP3 Standard deviation Email Image resolution Interface (computing) Software developer Shared memory Code Database Software maintenance Entire function Type theory Process (computing) Software Telnet System programming Freeware Physical system Window
Game controller GUI widget Keyboard shortcut Set (mathematics) Solid geometry Process modeling Product (business) Goodness of fit Process (computing) Self-organization Right angle Endliche Modelltheorie Lie group Determinant
Wage labour Patch (Unix) Software developer Length Multiplication sign Software developer Patch (Unix) GUI widget Surface of revolution Perspective (visual) Number Product (business) Word Mathematics Arithmetic mean Different (Kate Ryan album) Video game Self-organization Right angle Communications protocol Communications protocol Annihilator (ring theory)
Web page Game controller Implementation Software developer Code Multiplication sign System administrator Sound effect Facebook Crash (computing) Causality Root Negative number Cuboid Physical system Execution unit Standard deviation Software developer Structural load System administrator Data storage device Sound effect Plastikkarte Computer network Bit Volume (thermodynamics) Entire function Hand fan Connected space Personal digital assistant Cube Video game Self-organization Negative number Musical ensemble Computer worm
Point (geometry) Code Prisoner's dilemma Web page Data storage device Database transaction Mereology Database transaction Optical disc drive Wave Message passing Object-oriented programming Personal digital assistant Network topology Program slicing System programming Self-organization Right angle Message passing Information security Computer forensics Physical system God
Point (geometry) Multitier architecture Decision tree learning Multiplication sign Flash memory Drop (liquid) Disk read-and-write head Neuroinformatik Frequency Crash (computing) Bit rate Average Queue (abstract data type) Lie group Hill differential equation Traffic reporting Error message Formal grammar God Twin prime Flash memory Point (geometry) Explosion Personal digital assistant Crash (computing) Quadrilateral Game theory
Email Group action Multiplication sign Firewall (computing) Sampling (statistics) Basis <Mathematik> Database transaction IP address Unified threat management Data mining Latent heat Process (computing) Software Software framework Software framework Pattern language Information security Information security
Web page Standard deviation Code Multiplication sign Flash memory Code Field programmable gate array Computer network Hecke operator Goodness of fit Software bug Crash (computing) Software Personal digital assistant Figurate number Information security Information security Position operator
Electric generator Process (computing) Self-organization Diagram Parameter (computer programming) Table (information)
Shooting method Function (mathematics) Expert system Computer network Information security Plastikkarte Information security Traffic reporting Product (business)
Decision theory Multiplication sign Real number Perspective (visual) Rule of inference Equivalence relation Neuroinformatik Number Mathematics Process (computing) Computer configuration Process (computing) Statement (computer science) Quicksort God
Rule of inference Trail Crash (computing) Dependent and independent variables Flash memory Traffic reporting Rule of inference
Open source Multiplication sign PCI Express Virtual machine Database transaction Connected space Derivation (linguistics) Core dump Authorization Window God Vulnerability (computing) Physical system
Link (knot theory) Multiplication sign Database transaction Twitter Pressure God
first things first disclaimer I'm employed and I'd like to stay employed one of the side effects being employed is that you end up doing a lot of research in your spare time all of the research has been done in my spare time this is all from publicly available sources you could discover this just as easily as I did if you know what you're looking for the voices in your head so
I've been information security for 15-plus years it's going to have to change to a different kind of number pretty soon and that scares me too I've done everything from firewall admit and log review all the way up to see is oval created financial so I've seen it all I've done it all and I'm not afraid to tell other people to do the crap jobs I've been doing for the last 15 plus years I've done a lot of stuff in utilities vertical you may have heard me talk about that crap I've done a lot of stuff in the financials vertical and I'm not an expert in anything remember somebody tells you they're an expert they're lying nanoseconds in researching
and thanks to the Twitter I came across this awesome clip of a rather incredible scientist and mathematician who gave us compilers and programming languages and much more exquisitely than I could possibly say is going to tell you about nanoseconds Admiral hopper they started
talking about circuits that acted in
nanoseconds billionths of a second I
didn't know what a billion was I don't think most of those men downtown or what a billion is either if you don't know what a billion is how on earth you know what a billionth is I've first in filmed from the one morning in total desperation I called over to the engineering building and I
said please cut off a nanosecond and send it over to me and I've brought you some today now what I wanted when I
asked for a nanosecond was I wanted a
piece of work which would represent the maximum distance but electricity could travel in a billionth of a second now of course it wouldn't really be through wire beyond space velocity of light so if we start with a velocity of light use your friendly computer you'll discover that a nanosecond is 11.8 inches long the maximum limiting distance that electricity can travel and a billionth of a second finally in about a week I
call back and said I need something to
compare this to could I please have a micro second I've only got one micro
second so i can't give you each one
here's a microsecond 984 feet and
sometimes think we ought to hang one over every programmers desk around the neck so they know what they're throwing away when they throw away microseconds now I hope you'll all get the nano seconds they're absolutely marvelous for
explaining two wives and husbands and children and admirals and generals
people like that and everyone to know
why it took so damn long to send a
message by a satellite and I had to
point out that between here and the
satellite there were a very large number of nanoseconds you can explain these
things it's really very helpful so be sure to get your nano seconds
for the record and for posterity if you haven't spent a significant amount of time studying Admiral hopper you're failing do it the woman was brilliant she gave us compilers she gave us programming languages her contributions to computer science are astronomical and her explanations of the mundane are brilliant and hilarious there's a clip of her going toe-to-toe with David Letterman that will knock you right off of your socks absolutely amazing stuff I had every intention of bringing some nanoseconds to give out but it turns out the Transportation Security Administration has a lot of problems with bundles of wire in luggage I have a lot of problems with pennies you can imagine what they think of pennies plus wire in the same bag so possibly for the
first time in the history of humanity money equals C the speed of light you need to be very aware of how far light can travel because it is for now the ultimate limiter the distance that light travels in a millisecond one one thousandth of a second is about 300 kilometers it's a 186 miles for the rest of you in a millionth of a second you get about 300 meters 328 yards 184 feet in a nanosecond you get about 30 centimeters or roughly a foot 11.8 inches these are the absolute finite distances this is as far as fast as you can possibly go in reality you go a whole heck of a lot slower than this because we're not talking about light moving in a vacuum at the at the best it's moving in air or optical fiber or its electrons moving through a conductor so those distances get quite a bit longer it's much more costly most people like she said don't have a good handle on what these kind of numbers are and the majority of this talk I mean nanoseconds is funny to talk about most of this talk we're actually gonna talk about microseconds we're not quite in the nanosecond space yet when you think of things that are really really fast in human scale you think of the blink of an eye well the blink of your eye 350 to 450 microseconds Trading is ten times faster than that so
before you ask this is a talk about money filthy lovely money and it's not about any of the other things on your buzzword bingo card for the second year
in a row I'm talking about finance at Def Con last year we talked about pci and we'll be talking about pci again this afternoon in this room it turns out that most of Def Con is offense it's how to be as offensive as you possibly can and also a little bit of defense but knowing sometimes that a vulnerability exists or class of vulnerabilities exist and I am going to talk about classes of vulnerabilities that helps to sort of focus attention and once attention is focused then research starts happening then development of solutions starts happening and sometimes crazy people like me will tell you about stuff that doesn't matter this year two years ago at Def Con my co-presenter Tiffany rat and I told you all about how you weren't gonna be able to keep you secrets outside of your brain and this year the Department of Justice has said give me your password or go to jail indeterminately you'll stay there until we decide to let you out so you had your to your warning last year I talked about bad things in the world of industrial control this year I think I'm the only talk that isn't about industrial control next year everybody's gonna be talking about money so most of us don't really
have a good idea of this whole training thing I mean you're you're used to you know when you're younger and your dad is watching the news on TV looking at the dow jones industrial average and the Nikkei index and nasdaq and you know alphabet soup will tell does that mean and facing fact we're hackers we don't have any money we don't have a legitimate money the idea of stock markets and of exchanges started back in about the 1200 s with commodity and debt training so this is you know pork bellies and orange juice just like Ferris Bueller in the 1500s we got inter market trading so we're trades would be executed for one organization on multiple markets in the 1600s with Dutch East India Company we got our first real equity trading where you didn't have to be a lord or up here you could own part of a company so this was the beginnings of corporations by the late 1800s or sorry early eighteen-hundreds this is a funny story everybody knows RFC 1149 IP by avian carrier yeah Reuters implemented that in the early 1800s they use carrier pigeons to go between akka and Brussels when you think about that for for just a minute you realize that patents and software are stupid because prior art is sometimes an awfully long time ago in the late 1800s the electronic ticker tapes aren't happening this is daddy warbucks kind of stuff you know the glass dome and the paper spews out and somebody has a snit this is telling you about what the prices were by the mid 1900s quotation system starting to come into active use these are things that looked an awful lot like a typewriter they had a little bit of electronic knowledge to them but what they were capable of doing was telling you the next price there were an inquiry type system by the late 1900s computers are maintaining all the records you know they used to maintain records on paper and with chalk boards and stuff like that if you're old enough you remember using little bits of paper to move money around at banks they called it deposit slips if you're from a modern country you've been using debit cards for the last 25 years if you're in the United States you start using them in another five checks really and get spelling wrong to its qu es and by the early 2000s computers are trading with each other and largely without human intervention humans are providing sort of stick handling Qaeda guidance but are not responsible anymore for individual trades some definitions that are really
important high-speed trading is committing trades on a scale faster than human interaction so remember the fastest humans can go is the blink of an eye there really isn't that much else about us that can go any faster and that's 350 to 450 microseconds trades are now in the very very low three digits and very very high two digits numbers of microseconds irith make trading is using math and at this point but half the room glazes over completely in the other half of the room starts giggling because they know that the speaker doesn't know enough about math based on the results of incoming information market data feeds and even English language or human language press releases computers are able to make trades executing literally in that flash moment I said change arbitrage this is a
funny term everybody like saying it because it's got that lovely arbitrage sounds very important very Wall Street it's the practice of taking advantage of the difference in price between two or more markets this is usually or sorry this is historically done in space where you'll have a single organization that trades on NASDAQ and the New York Stock Exchange and it has slightly different prices on the two so you would buy wholesale in itself retail you'd buy the low price and sell the high price in the other market everybody sort of naturally does this we're accustomed to buying wholesale and selling retail except if you're in farming in which case you buy retail and sell wholesale now we're talking about doing arbitrage in time so this is no longer about having to be in multiple places at once this is about being inserted interstitial II between two other events in time so from the time that you know that someone wants to purchase a large number of shares until the time that the purchase execute there's time there's a gap in there and that gap in there you can step in and say well I've got them for sale so it never goes out to the wider community because you're right there ready to sell so arbitrage is very interesting mostly because when you said I've got them ready to sell you don't have them yet you're going to buy them right away so that you've got them by the time they get around to saying oh yeah from you inside of hundreds of microseconds scary fast when markets
were new middle of last monium tray times were hours you know it's human beings they're arguing with each other about price and haggling everybody loves a good haggle right by the late 1800s we're down two minutes by the 1900's were down two seconds this is largely the trading technology that we're most familiar with because we've all watched Ferris Bueller's Day Off one too many times anybody do this thing yeah by the 2000s were in to hundreds of microseconds in the future I don't know tachyon pulse emissions or something future prediction kind of stuff because we're going to hit that wall of the speed of light and we're going to hit it kind of hard it's going to hurt a lot
the architecture of these systems is astonishingly simple when you look around the internet for architecture diagrams for high speed training what you'll see is that someone threw up in Visio and give you a chart this big and it looks like shit because they've included every damn thing they can think of we all have the blinky lights and shiny things disease these people have it in a way that we look at them and giggle because they're completely insane about this stuff in reality there's only four moving pieces you need to know about you've got an exchange of some type whether it's New York Stock Exchange whether it's a commodity market in Chicago whether it's a private exchange whether it's a know anything toronto stock exchange you have a trading engine that's tightly coupled to that and that the training engine is this odd special piece of hardware and i'll come back and talk to in a minute the trading engine gets its information from market data sources so this is telling you what's going on in other markets what's going on in the news what's going on in the market through tightly coupled to and there is a human who provides like I said the sort of stick handling control over what's going on that machine in the middle is is kind of an interesting box it's faster than most other computers you have it's one of the few times when you can go to a major supplier of rack mounted computer equipment and they will sell you something that has been overclocked and maxed out and liquid cooled and oh my god that's cool you know 12 to 24 cores 128 gigs of RAM is not unusual give it 10 gigabit InfiniBand PCIe interfaces between machines to cram that latency as low as possible the thing that you'll never see them use is the actual network interface on the box they'll always replace it with something faster you don't really realize quite how much latency is added by things like well trusted well-documented completely debugged ethernet drivers and IP networking stacks instead you should go out and have a couple of developers in their own little space designed for you a custom fpga card that interfaces directly with ethernet so that your application can build a trading in memory a fixed transaction in memory and pass it to the FPGA as it's being built and the FPGA will Ram that thing out on the wire as fast as possible you'll notice that those couple of developers don't have the kind of historical legitimacy that something like the bsd IP stack has think about that for a minute so how
fast is fast how fast is fast really matter need to keep hammering this point home because it's something that most people just can't grasp the first time around if you're executing trades in terms of seconds you have no position you will lose every single time you will lose your shirt in the market you're the kind of person who sits at home watching ads for itt tech sorry in milliseconds you're losing nearly every time and and when you think about it milliseconds you don't notice milliseconds right you take a brand new laptop brand new server and a brand new gigabit ethernet switch I tried this just last week at home side of have the number to prove it and send a ping to the server and get the response back half a millisecond it's pretty average so your home stuff is pretty slow right I mean you heard me say it before that half a millisecond is just way too much right sub-millisecond big players are regularly beating you because they've got better faster computers they're replacing them with asset time lines that aren't measured in years they're measured in weeks or maybe months so a new hardware goes in there as fast as possible because new hardware's faster than old hardware in the hundreds of microseconds ear a bit player you're winning based more on flip of a coin than on being able to actually win and if you're in the tens of microseconds you're almost always winning which is kind of a nice place to be almost as
important as being just plain fast just very very low latency is also being very very predictable so all those VoIP people out there that like to talk about you know crazy high jitter and how that completely wrecks the voice conversation and how people can't understand and be intelligible and they go all skype roboti and stuff in this this situation the jitter matters because it's been taken into account as part of the calculation you know what your round trip time to the exchange is and that round trip time has to stay the same if the round trip time becomes variable you've introduced a new variable in a calculation where you were expecting a constant this is going to cause you problems right you know the speed of light keeps changing it's really really hard to get your spaceship to go to work 9 we used to joke about this back in the olden days when I was a rotary switch jockey about what would happen if if the you start having packet loss and the packets were just sort of falling out of the front of the router and hitting the floor so you put a bit bucket underneath everybody remembers that horrible joke from the 90s in this case if you drop a packet you lose money so drop packets are the enemy you know Ethernet the idea that collisions can occur no collisions are not permitted to occur simply not allowed massively inefficient protocols like TCP that handshake are the enemy error correction who needs that crab proximity
relieves a lot of these issues right the closer you are the easier things are the less likely that weird happens you know when you're doing a an international ping think about the path that packets going to take theoretically it's going in an undersea cable but maybe you just got switch to satellite or maybe every other packet takes a satellite path instead of a wire path you'll lose all that predictability you got induced jitter that's just off the chart right so you move closer on be in the same city metropolitan area network kind of stuff well those are great you know dark fiber from building to building that's awesome remember how long a micro second is if you're more than a couple of buildings away that 984 feet is going to kick you right in the ass so you gotta be on the land you need to be close 300 meters is sort of the absolute top end that's a thousand feet in American and
remember that this latency costs money it's measurable you can look and you can say you know what we're always 10 microseconds late that means that we're losing thirty percent of our trades can you make that 10 microseconds go away we're going to start winning thirty percent more trades damn there's money there it's often a lot of money there so of course there's money if you got money to spend if you got money to make you're going to make it happen nobody freaked
out when I showed you that hyper simplified diagram nobody said there's stuff missing I've seen network diagrams before there were no boxes with flaming hair nobody caught that it's it's like it's morning you know what's the afternoon
right it's an oh crap kind of moment once it's pointed out to you and you go oh shit there's nothing actually protecting those links except that they're private how many people have private internet connections how many people have private frame relay how about private vlans how about private mpls ya need dumbasses there is no private but the marketeering jerks can sell to people who are uninformed who say to them well it's your private link it's your dark fiber that runs from 1 to 1 there is no patch panels in the way there are no switches that you're transiting through in a blind layer 2 ish system you know it's private it's all yours and nobody's ever been on a network with other participants right there's only two nodes on that network only two there's no one else there's no other customers no one else is connected to that exchange except you you are the only customer New York Stock Exchange it's such crap that it shouldn't be I mean people who use computers every day that have it's frustrating because
there's no firewalls like the simplest thing the most basic thing I want to control with whom I am communicating as tightly as possible I want to say that I will only communicate with this list of IP addresses and I will only communicate on these ports the bare minimum because guess what firewalls add latency really great firewalls awesome firewalls that are all you know utme they add a crap ton of latency what do we do about that well there's not really a whole lot we can do really we sort of suck it up and say well because we're security people we're okay with milliseconds of latency because nobody notices that when their web browsing and there's nothing in pci DSS that says you have to have a fast firewall you just have to have a firewall in a box in the corner of your data center providing a home to mice okay remember that latency stuff costs money and if you're not in a position where you're saving money you've gone from being an information security department that is a cost center to being an information security since that is a huge cost center so the risk is smaller than the cost which is smaller than the profit go away there are no firewalls let's go back further in time
anybody remember the 90s like six of us in here we used to use these things called ACLs access control lists we build them into our routers and stuff and we all had the Martian list everybody remember the Martian list how many things are on the Martian list anymore it was like four lines they don't use a CLS because they add latency any time you put an ACL in the router or switch has to receive the entire packet and only after it has received the entire packet doesn't make a decision latency most switches don't do cut through well acl's are on there are a couple that do now but they're rare few far between and risk is smaller than the cost which is small into profits so screw you know ACLs meaningful system
hardening well remember I told you about these crazy funky boxes that you would think are completely stripped I mean these are the race cars of this industry these are not your camry there are no airbags brakes are for the weak actually they look a lot like those kids who get their mom's civic and weld of coffee can under the tailpipe and put a type r sticker on because they all still have sold out exe and Calcutta XE if their windows boxes or send mail man pages of mp3 player for different databases that's all linux boxes all this custom interface crap is in there too right so remember our little team of two developers that wrote some custom fpga code that of course completely implements the ethernet standards and the IP standards and they did it all using john postals recommendation which is that you be very conservative what you send very liberal and what you accept we know they did a great job of writing this entirely bug free code not so much so you go to let me say well let's look at the code on your FPGAs not a secret you're not allowed to see that no sharing and then you've got all the usual complaints but maintainability i mean god once you take telnet exe off of a box then they can't see anything on the network all their troubleshooting is gone and these
specialized systems come from manufacturers that you've never heard of anybody in this room repre in a single name up there oh my crap is the entire trading technology industry in this room right now shit this is the only talk you came to Vegas for crap crap crap
everybody does good threat modeling right everybody has a good solid threat model for their organization they know exactly who they're bad guy is who they're trying to protect against me no I can get hired as a secretary anywhere I mean have you seen these legs we know what's missing out of our usual set of controls because we take for granted so much we take for granted that there's going to be changed control because of course everybody has changed control everybody uses ITIL right and not so much we take for granted that they've going to they're going to have done a great job of things like employee screening yeah I don't even do a good job of that in the government for crying out loud so you know how do we talk about this threat model how do we start to build who the potential bad guy is this is this is not simple stuff this is actually really hard so how do we do that determination well let's walk through a couple of things that I think
are threats like vendors are threat it's not that I don't love blinky lights and shiny things and it's not that I don't appreciate being taken out for dinner and drinks every now and then but you're trusting that the marketing slick is going to have exactly what you get on it anybody seen a lie on a marketing slick anybody bought a product that didn't have a feature they told you it had okay the other everybody hasn't put up their hand obviously hasn't bought anything this is a talk about capitalism you're also trusting that they haven't hired any bad guys so you know the shortcuts that you take in your organization's and they all take them too I think it's a
maybe I like to have faith that the vendors are doing the right thing I like to think that II Thicke's is a word that they know I'm probably completely kidding myself but imagine something very simple imagine that a vendor has a disgruntled customer yet all vendors have all disgruntled customers imagine that a vendor has a disgruntled customer that's making their life miserable imagine if one developer takes it upon themselves to screw with that customer they get custom patches only for them in which they've mucked around with the precision time protocol just enough that a micro second isn't a micro second anymore sometimes a little more a little less the end of the day they still the same number of microseconds they were just of different lengths kind of awesome isn't it I think developers
are a threat many developers in the crowd yeah I will tell you to your face that I love you but I'm lying remembered Admiral hopper said about wasting microseconds 984 feet of wire wrap it around your neck please and thank you in most algo trading the developer isn't developed the developers like a developer traitor there's someone who has more knowledge of what's going on from a market perspective than they have around me or traditional sdlc and best of all they don't do dev QA SI t staging all that crap they just put their changes right onto production and they do it multiple times a day nothing can go wrong nothing and remember how it's kind of odd when you're working for an organization you're giving up your IP right that's the whole you give me dollars and I give you brains you've become the means of production and the supply and also the labor it really breaks the whole notion in the industrial revolution sometimes people get a little attached to their IP anybody ever taken something that they made at work and taking a copy of at home anybody telling the truth it turns
out the developers are a bit of a problem they tend to take the crap home in this case we're talking about 32 megabytes of code anybody even remember megabytes yeah kind of like kilobytes I remember once upon a time I had less than a megabyte of storage in my entire life yeah 32 megabytes of code snuck it out over SSL connections went to jail for eight years and one month pedophile has a facebook fan page it's kind of awesome took them a while to catch them too I think there's another kind of
threat in the insider so in the financial world they talk about insiders they mean something very very specific we're not talking about that kind of insider we're talking about somebody who works in the organization my favorite to pick on here is traders who get smart every now and then start playing with their blinky lights and shiny things or administrators because remember all of these boxes still have in that dark IT department that nobody talks about that exists in the capital markets org they've all got their own administrators doing the same kind of in book that we've all done through the history of time it's just largely saying have you tried turning it off and back on again when these traders or administrators who have superior access to the system remember those comfy warm feelings about things that of course you know exist internal controls around segregation of duties and all that good stuff where everyone has root on the trading algo box I think I'm joking could you cause negative effects on other participants if you were an administrator on a box could you induce a couple of microseconds latency in someone else's connection remember those fancy FPGA boxes left PJ cards that probably don't have the entire standard on board if you send them a packet that makes them barf what kind of barf do they have you know if if we look at other places work seen this happen where we've seen half-assed implementations of Ethernet or IP will look to the world of industrial control and say a couple years ago if you have now and Bradley PLC and you sent it a ping with a payload simple ICMP packet with payload it would literally just stop processing didn't crash just stopped can you make somebody else's I'll go engine stop can you make the market I'll go market engine swap in one packet I think so or you've also got the disgruntled employee because you know every organization is completely free of that who says you know what I'm going to make sure that my employer never wins another trade I'm going to induce another 75 microseconds of latency I'm going to do it by adding a quarter of micro second every other day until it gets to 75 along the way I'm going to be very angry about pc load letter and the guy in the cube next to me who listens to music at a reasonable volume while collating turns out traders are a bit of
a problem this is my favorite favorite favorite case to wave in the face of people who say that printers are not part of information securities duties this dude took the code out on slices of trees and only a couple hundred of them that's really awesome they to do forensics on the printer to figure out what happened everybody knows that your printers have disk drives in them yeah sentenced to three years in prison plus two years of supervised released and at that point will likely be deported oops
I think the market is a threat I really do and it's an odd kind of technical threat in our world we've seen this kind of threat before it's an amplification kind of attack right you know once things start going sideways you can push them hard and make them go a whole lot more sideways if the market suffers from malformed messages where someone is sending the market malformed messages or worse the market is sending out malformed messages what's going to happen badness how would malformed messages happen shitty code push yeah there's a bunch of issues around transaction risk scrutiny and whether or not a transaction is permitted to go through and there's some compromised systems kind of problems that I think are very very real because you know every large organization with more than 30,000 employees has oh my god has never had a compromised system on the inside right there are no botnets running inside of fortune 50 fortune 100 companies it turns out that yeah the
market is actually kind of a huge threat in May of 2010 the Dow Jones Industrial Average plummeted 900 points in minutes this is the flash crash about three minutes there's a huge amount of documentation there's what I think is a great document from nanex I was told when I gave this talk of black head that the next report is a lie I don't know whether that guy had a tinfoil hat on under his toupee but the point is that this was a exactly that kind of amplification attack an algo started selling and it saw the price drop because supply increased on the market and so as the price was dropping it said oh my god I gotta sell and so it sold more and the price dropped more and went oh my god I gotta sell more and the price drop more and things went down hill from there hi guys i am so scared
about this next panel I met Ed Felton whom you should all know from Princeton did a really good teardown of it and thinks that it comes down to really kind of five points a bunch of weird quote requests went into the end the stock exchange computers normally these quotes are shoved in a queue but because of the high rate of requests the queue got backed up and there was an error the quote lists a price and a time in this case the price was determined when the quote went into the queue but it wasn't time-stamped until it left the queue and it was in the queue for more than a couple of microseconds and the price changed during that time period so what people thought was the price wasn't the price essentially and these got really confusing everything went crazy the market destabilized and the faster happened the weirder at God and remember we're talking crazy faster I mean this whole the whole issue happened inside of three minutes and those trades were all backed out so they essentially pulled a mulligan on the market I didn't know you could do that outside of stupid games of white men play so how do we trust in
this kind of framework I mean we've got these threats and these threats are kind of spooky scary nasty weird odd we don't really know what to do with them kind of threats we don't really trust our coworkers don't trust very many of mine they steal things from my desk like my stapler and there's no good way to monitor what's going on because this is happening so incredibly fast but the best monitors for this stuff that are available from the general IT market are sampling only they do not examine every packet the examine our sampling basis the more packets there are or the larger the packets are the less sampling they do the more inaccurate they become there are a couple of very very specific devices that are made for this market that do a much better job but you're in IT security not capital markets so you have got no idea who those companies are where those devices come from
traditional security is an absolute fail here just epic or a hundred thousand times too slow we come in and say well we're going to add a couple of milliseconds of latency to those transactions so that we can make you secure with our UTM device which will protect you from email based threats the only thing that's supposed to transit the network is fixed transactions on specific IPs and specific ports anything else that's moving on that network is illegal and shouldn't be there you should just forget drop it you don't need pattern behavioral detection and IT security is so god damn proud of itself that it's unwilling to listen it's not willing to learn it's unwilling to sit and be the student and say well shit teach me about this stuff because we're all focused on checkbox compliance that's so easy right oh do we have a firewall good I have 21 minutes left not five I know I started late so let's
answer these hard questions later okay the hard questions we're not in a position to answer right now because we're still way too busy figuring out where the heck are socks aren't pulling them up we're not going to be able to secure custom everything who possibly in this room besides maybe ten percent of you can go through FPGA code and figure out whether or not it's got a bug relative to a standard that spread over seven eight hundred pages worth of documentation yeah we've got to learn how to be fast enough and we're not there yet we don't even really understand this whole money thing and we need to make the case that security efforts are important because they reduce the chance of disaster we don't know for sure that the next time there's a flash crash that those trades will just be back down the market may just say caveat emptor suck it up I need you
to do something though pretty much anything it's time to party like it's 1999 and do some network security basics like those ACL things let's implement some shall we there isn't a whole lot in
the next generation juniper and cisco are coming to the table with some stuff that's that's pretty fast it's still not quite fast enough there's an argument but it's pretty fast we need to keep up and I don't know exactly how we're going
to do that but I am sure that we need to
do anything literally anything even just understand what what it is go and find the people in your organization that bar physio and get them to walk you through the diagrams follow the packet follow the money you follow the process start to understand exactly what's going on and make friends and influence people by some coffee it's it's kind of important the reason I'm so pissed off about this
is because this report came out just a couple of weeks ago that says IT security pros think that performance is more important than actual security we're shooting ourselves in the back while we're giving ourselves a self-congratulatory reach around and it's stupid senseless pointless dumbassery because we're all so damn sure that were experts in everything
challenge the vendors we want more than checkboxes you know they come back and they say well nobody's asking for it so you know because nobody's asking for
we're not doing it we're gonna give you raging a feature it?s instead we're going to change those LEDs from green to blue and that's going to make it better goddamn bullshit if you're a risk
process policy or GRC wonk what the hell are you doing at Def Con thank you work with the business folks you have equivalents in the business that aren't in IT that understand things from sort of a different perspective than you do and and their tolerances for risk are very different from yours you may still be suffering from the pallor ISM disease or you think that attaching an ipod to a computer is this oh my god hair on fire kind of threat they're not in that same kind of world they don't think Dogma is all that important they're ready to rewrite the rules every time something changes in the markets which is daily
you're never going to be able to change their minds about the cost of latency because it's real they've got the numbers to back it up they've got the profit you're not a profit Center your cost Center so work with them and learn to understand that just because we did it that way last year isn't a good reason to do it that way this year
compliance people meet your financial compliance people they know more about compliance than you ever will the SEC is
taking an active interest in this just last week they release something called the large trader reporting rule which is 13hd one you should read it it's really really interesting this is their entire response thus far to the flash crash and what they're saying is it would be nice if we had an audit trail yeah that's right all you need to be regulatory compliant is a better audit trail if you're in the trenches do some
fucking research we haven't seen original research at Def Con and God knows how long the next derivative piece of shit talk I am so fed up do the research please make the time if you don't have twenty percent or ten percent time from work you should be doing it at home find something and start picking at it and peel up the edge I only started really poking into this actively about six months ago I'm here you're not
understand your business partners build the POC lab so that you could show this stuff your POC lab is going to have to be weird Nesso Tarek it's not the POC lab that you're using right now for your little windows vulnerabilities and desktop bullshit you need 10 gigabit e you need infiniband you need pcie machine a machine you need machines that have 12 cores hundred twenty eight gigs of ram and stupid fast connections you need the open source fix transaction system so that you can build fixed transactions and send them out on the wire you don't understand this stuff and you try to talk with authority about it until you've played with it you don't get to talk about it it's kind of like teenagers and sex encourage the vendors
to get with the program please thank you oh my god vendors step up couple of them have and huge kudos for the couple that have
I'm in the next panel but I will do Q&A if you see me walking around I'll talk to anybody for anything yes you can
bring the gear up now oh you're waiting for I'm in the next talk I need the beer now I'm worried about those paper towels though yeah I always have my time thanks everybody really appreciate you coming out