Security when nanoseconds count
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 122 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40573 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DEF CON 1971 / 122
3
5
10
11
12
22
23
24
30
31
32
38
43
46
47
49
51
54
56
59
60
62
71
73
76
84
85
88
92
93
96
97
98
104
106
109
112
113
115
119
00:00
Information securityClient (computing)outputNumberDifferent (Kate Ryan album)Process (computing)Multiplication signCompilation albumDisk read-and-write headTwitterVertex (graph theory)Firewall (computing)Utility softwareOpen sourceMathematicianInformation securitySound effectExpert systemProgramming languageSystem administratorLogin
01:24
PlastikkarteDecision support systemVulnerability (computing)ComputerCharge carrierTape driveTelecommunicationSystem programmingSatelliteSpacetimeComputer program2 (number)Tape driveMessage passingScaling (geometry)Type theoryBitPhysical systemRow (database)PlastikkarteCompilation albumTelecommunicationNeuroinformatikMultiplication signVulnerability (computing)Cue sportsSoftware maintenanceCharge carrierNumberWhiteboardFiber bundleFormal languageLimit (category theory)VacuumHecke operatorDistanceComputer scienceMetreProcess capability indexQuicksortSoftware developerFocus (optics)Social classMaxima and minimaDigital electronicsAverageWave packetBuildingTotal S.A.Subject indexingReal numberMultiplicationSoftwareMereologyHacker (term)Alphabet (computer science)VelocitySelf-organizationTheory of everythingPeer-to-peer
09:34
Scale (map)Interactive televisionAlgorithmInformationOpen sourceSpacetimeChemical equationArchitecturePredictabilityCommunications protocolStack (abstract data type)Formal languageGame controllerQuicksortMoment (mathematics)Virtual machineOptical disc driveComputer hardwareData conversionSelf-organization10 (number)CalculationMereologyVery-high-bit-rate digital subscriber lineMultiplication signMathematicsCuboidTerm (mathematics)NeuroinformatikRoundness (object)CASE <Informatik>Interface (computing)Device driverGodPCI ExpressShared memoryNumberCartesian coordinate systemPoint (geometry)PurchasingPosition operatorSemiconductor memoryScaling (geometry)PredictionNatural languageInformationRight angleComputer architectureSpacetimePhysical systemAutomatic differentiationSoftwarePulse (signal processing)Dependent and independent variablesAlgorithmNP-hardInteractive televisionServer (computing)Category of beingPlastikkarteSoftware developerResultantDatabase transactionInternetworkingDigitizingDiagramCore dump2 (number)Different (Kate Ryan album)Flash memoryOpen sourceLine (geometry)LaptopField programmable gate arrayEvent horizonBitGoodness of fitType theoryWave packet
17:45
DistanceLocal area networkSound effectFirewall (computing)Physical systemCodeImage resolutionSystem programmingDistanceCommunications protocolPredictabilityPhysical systemLink (knot theory)CodeMetropolitan area networkNeuroinformatikField programmable gate arraySoftware developerRoundness (object)CalculationMultiplication signInformation securityCollisionElectronic mailing listCASE <Informatik>Game controllerSatelliteRouter (computing)QuicksortInsertion lossBitCanadian Light SourceLine (geometry)Fiber (mathematics)Building2 (number)Moment (mathematics)CuboidRight angleDecision theorySoftwareWindowType theoryMetreEntire functionDiagramGraph drawingFehlererkennungFirewall (computing)EmailDatabaseMP3Frame RelayIntranetData centerIP addressVirtual LANPatch (Unix)Connected spaceProcess (computing)Standard deviationFreewareSoftware bugMaxima and minimaInterface (computing)Shared memoryGodSoftware maintenanceUnified threat managementProcess capability indexWeb 2.0Position operatorTelnetFrame problemException handlingLogical constantWeb browserDrop (liquid)Error message
25:55
Process modelingGUI widgetPatch (Unix)Software developerCommunications protocolExecution unitSystem administratorComputer networkNegative numberSound effectWeb pageMessage passingDatabase transactionSystem programmingPoint (geometry)ExplosionFlash memoryCrash (computing)Decision tree learningQuadrilateralMultitier architectureTwin primeFormal grammarSoftware frameworkInformation securityCodeField programmable gate arrayGoodness of fitFunction (mathematics)PlastikkarteProduct (business)Process (computing)Statement (computer science)Decision theoryComputer configurationRule of inferencePressureTwitterLink (knot theory)CASE <Informatik>Network topologyComputer forensicsInformation securityGame controllerSoftwareSelf-organizationMultiplication signPoint (geometry)Process (computing)Structural loadOptical disc driveTable (information)Keyboard shortcutMathematicsLie groupProduct (business)CubeVector potentialMusical ensembleSoftware developerFlash memoryDatabase transactionArithmetic meanCrash (computing)Patch (Unix)WordVolume (thermodynamics)Parameter (computer programming)Annihilator (ring theory)GodRight angleNumberCommunications protocolPhysical systemTrailCodeDependent and independent variablesVideo gameTraffic reportingMessage passingData storage deviceBitDiagramFacebookHill differential equationSound effectNegative numberLengthSurface of revolutionSystem administratorDrop (liquid)Hand fanDerivation (linguistics)Perspective (visual)Thread (computing)Bit rateCuboidShooting methodNormal (geometry)Expert systemWeb pageWindowQueue (abstract data type)Virtual machineEquivalence relationNeuroinformatikVulnerability (computing)Rule of inferenceLatent heatQuicksortElectric generatorIP addressGoodness of fitComputer programAuthorizationConnected spaceRootSolid geometryPattern languageEndliche ModelltheorieData miningGame theoryBasis <Mathematik>FrequencyCausalityPosition operatorUnified threat managementHecke operatorSampling (statistics)WaveEntire functionGroup actionSet (mathematics)EmailOpen sourcePCI ExpressProgram slicingSoftware bugStandard deviationImplementationMereologyDeterminantFigurate numberDifferent (Kate Ryan album)2 (number)Surjective functionWage labourReal numberPlastikkarteComputer wormFirewall (computing)Core dumpPrisoner's dilemmaObject-oriented programmingDisk read-and-write headAverageSoftware frameworkError message
Transcript: English(auto-generated)
00:00
things first, disclaimer. I'm employed and I'd like to stay employed. One of the side effects of being employed is that you end up doing a lot of research in your spare time. All of the research has been done in my spare time. This is all from publicly available sources. You could discover this just as easily as I did if you know what
00:20
you're looking for. The voice is in your head. So I've been in information security for 15 plus years. It's gonna have to change to a different kind of number pretty soon and that scares me too. I've done everything from firewall admin and log review all the way up to CISO and publicly traded
00:41
financial. So I've seen it all, I've done it all and I'm not afraid to tell other people to do the crap jobs I've been doing for the last 15 plus years. I've done a lot of stuff in the utilities vertical. You may have heard me talk about that crap. I've done a lot of stuff in the financials vertical and I'm not an expert in anything. Remember, if somebody tells you they're an expert, they're
01:01
lying. Nanoseconds. In researching, and thanks to the Twitter, I came across this awesome clip of a rather incredible scientist and mathematician who gave us compilers and programming languages and much more exquisitely than I could possibly say is gonna tell you
01:21
about nanoseconds. Admiral Hopper. They started talking about circuits that acted in nanoseconds. Billions of a nanosecond. I didn't know what a billion was. I don't think most of those men downtown know what a billion is either. If you don't know what a billion is, how on
01:41
earth do you know what a billionth is? I fussed and fumed. Finally one morning in total desperation, I called over to the engineering building and I said, please cut off a nanosecond and send it over to me. And I've brought you some today. Now what I wanted when I asked where a nanosecond was, I wanted a piece of wire which
02:02
would represent the maximum distance that electricity could travel in a billionth of a second. Now, of course, it wouldn't really be through wire beyond space, velocity of light. So if you start with a velocity of light and use your friendly computer, you'll discover that a nanosecond is 11.8 inches long. The
02:21
maximum limiting distance that electricity can travel in a billionth of a second. Finally, at the end of about a week, I called back and said, I need something to compare this to. Could I please have a microsecond? I've only got one microsecond, so I can't give you each one. Here's a microsecond, 984 feet. I
02:44
sometimes think we ought to hang one over every programmer's desk or around their neck so they know what they're throwing away when they throw away microseconds. Now, I hope you'll all get your nanoseconds. They're absolutely marvelous for explaining to wives and husbands and children and
03:03
admirals and generals and people like that. An admiral wanted to know why it took so damn long to send a message via satellite. And I had to point out that between here and the satellite, there were a very large number of nanoseconds. You can
03:20
explain these things. It's really very helpful. So be sure to get your nanoseconds. For the record and for posterity, if you haven't spent a significant amount of time studying Admiral Hopper, you're failing. Do it. The
03:42
woman was brilliant. She gave us compilers. She gave us programming languages. Her contributions to computer science are astronomical and her explanations of the mundane are brilliant and hilarious. There's a clip of her going toe to toe with David Letterman that will knock you right off your socks. Absolutely amazing stuff. I
04:02
had every intention of bringing some nanoseconds to give out, but it turns out the Transportation Security Administration has a lot of problems with bundles of wire in luggage. They have a lot of problems with pennies. You can imagine what they think of pennies plus wire in the same bag. So possibly for the first time in the history of
04:21
humanity, money equals C, the speed of light. You need to be very aware of how far light can travel because it is, for now, the ultimate limiter. The distance that light travels in a millisecond, one one thousandth of a second, is about 300 kilometers. It's 186 miles for the
04:40
rest of you. In a millionth of a second, you get about 300 meters, 328 yards, 984 feet. In a nanosecond, you get about 30 centimeters or roughly a foot, 11.8 inches. These are the absolute finite distances. This is as far, as fast, as you can possibly go. In reality, you go a
05:03
whole heck of a lot slower than this because we're not talking about light moving in a vacuum. At the best, it's moving in air or optical fiber or it's electrons moving through a conductor. So those distances get quite a bit longer. It's much more costly. Most people, like she said, don't have
05:24
a good handle on what these kind of numbers are. And the majority of this talk, I mean, nanoseconds is funny to talk about. Most of this talk, we're actually going to talk about microseconds. We're not quite in the nanosecond space yet. When you think of things that are really, really fast in human scale, you think of the blink of an eye.
05:40
Well, the blink of your eye, 350 to 450 microseconds. Trading is 10 times faster than that. So before you ask, this is a talk about money. Filthy, lovely money. And it's not about any of
06:00
the other things on your buzzword bingo card. For the second year in a row, I'm talking about finance at Def Con. Last year we talked about PCI and we'll be talking about PCI again this afternoon in this room. It turns out that most of Def Con is offense. It's how to be as offensive as you possibly can. And also a little bit of defense.
06:25
But knowing sometimes that a vulnerability exists or a class of vulnerabilities exists, and I am going to talk about classes of vulnerabilities, that helps to sort of focus attention. And once attention is focused, then research starts happening, then development of solutions starts happening. And
06:41
sometimes crazy people like me will tell you about stuff that doesn't matter this year. Two years ago at Def Con, my co-presenter Tiffany Rad and I told you all about how you weren't going to be able to keep your secrets outside of your brain. And this year the Department of Justice has said, give me your password or go to jail
07:01
indeterminately. You'll stay there until we decide to let you out. So you had your two year warning. Last year I talked about bad things in the world of industrial control. This year I think I'm the only talk that isn't about industrial control. Next year everybody's going to be talking about money. So most of us don't really
07:23
have a good idea of this whole trading thing. I mean, you're used to, you know, when you were younger and your dad was watching the news on TV, looking at the Dow Jones Industrial Average and the Nikkei index and NASDAQ and, you know, alphabet soup, what the hell does that mean? And facing facts, we're hackers, we don't have any money. We don't have any legitimate
07:43
money. The idea of stock markets and exchanges started back in about the 1200s with commodity and debt trading. So this is, you know, pork bellies and orange juice, just like Ferris Bueller. In the 1500s we got inter market trading. So where trades would be executed for one organization on multiple markets. In
08:02
the 1600s with the Dutch East India company we got our first real equity trading where you didn't have to be a lord or a peer. You could own part of a company. So this was the beginnings of corporations. By the late 1800s, or sorry, early 1800s, this is a funny story, everybody knows RFC 1149? IP by avian
08:22
carrier. Reuters implemented that in the early 1800s. They used carrier pigeons to go between Aachen and Brussels. When you think about that for just a minute you realize that patents and software are stupid because prior art is sometimes an awfully long time ago. In the
08:41
late 1800s the electronic ticker tapes started happening. This is Daddy Warbucks kind of stuff, you know, the glass dome and the paper spews out and somebody has a snit. This is telling you about what the prices were. By the mid-1900s quotation systems started to come into active use. These are things that looked an awful lot like a typewriter. They had a little bit of electronic knowledge to them but what they
09:01
were capable of doing was telling you the next price. They were an inquiry type system. By the late 1900s computers are maintaining all the records. You know, they used to maintain records on paper and with chalkboards and stuff like that. If you're old enough you remember using little bits of paper to move money around at banks. They called it deposit slips. If you're from a modern country
09:20
you've been using debit cards for the last 25 years. If you're in the United States you'll start using them in another five. Checks? Really? And you spell them wrong too. It's Q-U-E-S. By the early 2000s computers are trading with each other and largely without human intervention.
09:40
Humans are providing sort of stick handling kind of guidance but are not responsible anymore for individual trades. Some definitions that are really important. High speed trading is committing trades on a scale faster than human interaction. So remember the fastest humans can go is the blink of an eye. There really isn't that much
10:00
else about us that can go any faster and that's 350 to 450 microseconds. Trades are now in the very very low three digits and very very high two digits numbers of microseconds. Algorithmic trading is using math and at this point about half the room glazes over completely and the other half of the room starts
10:20
giggling because they know the speaker doesn't know enough about math. Based on the results of incoming information market data feeds and even English language or human language press releases computers are able to make trades. Executing literally in that flash moment. I said
10:42
change. Arbitrage. This is a funny term everybody likes saying it because it's got that lovely arbitrage sounds very important very Wall Street. It's the practice of taking advantage of the difference in price between two or more markets. This is usually or sorry this is historically done in
11:01
space where you'll have a single organization that trades on Nasdaq and the New York Stock Exchange and it has slightly different prices on the two. So you would buy wholesale and sell retail. You'd buy the low price and sell at the high price in the other market. Everybody sort of naturally does this. We're accustomed to buying wholesale
11:21
and selling retail except if you're in farming in which case you buy retail and sell wholesale. Now we're talking about doing arbitrage in time. So this is no longer about having to be in multiple places at once this is about being inserted interstitially between two other events in time. So from the time that you know that someone wants
11:41
to purchase a large number of shares until the time that the purchase executes there's a gap in there and that gap in there you can step in and say well I've got them for sale. So it never goes out to the wider community because you're right there ready to sell. So arbitrage is very interesting mostly because when you said I've
12:01
got them ready to sell you don't have them yet. You're gonna buy them right away so that you've got them by the time they get around to saying oh yeah I'll buy from you inside of hundreds of microseconds. Scary fast. When markets were new, middle of last millennium trade times were ours. You know
12:23
it's human beings they're arguing with each other about price and haggling. Everybody loves a good haggle right? By the late 1800s we're down to minutes. By the 1900s we're down to seconds. This is largely the trading technology that we're most familiar with because we've all watched Ferris Bueller's day
12:41
off one too many times. Anybody do this thing? Yeah. By the 2000s we're into hundreds of microseconds. In the future I don't know tachyon pulse emissions or something future prediction kind of stuff because we're gonna hit that wall of the speed of light and we're gonna hit it kinda hard it's gonna hurt a lot. The
13:02
architecture of these systems is astonishingly simple. When you look around the internet for architecture diagrams for high speed trading what you'll see is that someone threw up Invisio and gave you a chart this big and it looks like shit. Because they've included every damn thing they can think of. We all have the
13:22
blinky lights and shiny things disease. These people have it in a way that we'd look at them and giggle because they're completely insane about this stuff. In reality there's only four moving pieces you know about. You've got an exchange of some type. Whether it's the New York Stock Exchange, whether it's a
13:40
commodity market in Chicago, whether it's a private exchange, whether it's a, well anything, Toronto Stock Exchange. You have a trading engine that's tightly coupled to that. The trading engine is this odd special piece of hardware and I'll come back and talk to it in a minute. The trading engine gets its information from market data sources. So this is telling
14:00
you what's going on in other markets, what's going on in the news, what's going on in the markets you're tightly coupled to. And there is a human who provides, like I said, the sort of stick handling control over what's going on. That machine in the middle is kind of an interesting box. It's faster than most other computers you have. It's one of the few times when you can go to a major supplier of
14:22
rack-mounted computer equipment and they will sell you something that has been overclocked and maxed out and liquid cooled and oh my god that's cool. 12 to 24 cores, 128 gigs of ram is not unusual. Gigabit, 10 gigabit, InfiniBand, PCIe
14:42
interfaces between machines to cram that latency as low as possible. The thing that you'll never see them use is the actual network interface on the box. Always replace it with something faster. You don't really realize quite how much latency is added by things like well-trusted, well-documented,
15:01
completely debugged Ethernet drivers and IP networking stacks. Instead, you should go out and have a couple of developers in their own little space design for you a custom FPGA card that interfaces directly with Ethernet so that your application can build a trading in memory, a fixed
15:22
transaction in memory and pass it to the FPGA as it's being built and the FPGA will ram that thing out on the wire as fast as possible. You'll notice that those couple of developers don't have the kind of historical legitimacy that something like the BSD IP stack
15:40
has. Think about that for a minute. How fast is fast? How fast does fast really matter? Need to keep hammering this point home because it's something that most people just can't grasp the first time around. If you're executing trades in terms of seconds, you have no position. You will lose every single time. You will lose your shirt in
16:00
the market. You're the kind of person who sits at home watching ads for ITT tech. Sorry. In milliseconds, you're losing nearly every time. When you think about it, milliseconds, you don't notice milliseconds, right? You take a brand new laptop, brand new server and a brand new gigabit
16:21
Ethernet switch. I tried this just last week at home so I'd have the number to prove it. Send a ping to the server and get the response back, half a millisecond. It's pretty average. So your home stuff is pretty slow, right? I mean, you heard me say before that half a millisecond is just way too much, right? Sub
16:42
millisecond, big players are regularly beating you because they've got better faster computers. They're replacing them with asset timelines that aren't measured in years. They're measured in weeks or maybe months. So new hardware goes in there as fast as possible because new hardware is faster than old hardware. In the hundreds of microseconds, you're a bit player. You're
17:03
winning based more on flip of a coin than on being able to actually win. And if you're in the tens of microseconds, you're almost always winning, which is kind of a nice place to be. Almost as important as being just playing fast, just very, very low latency is also being very, very predictable. So all
17:21
those Voight people out there that like to talk about, you know, crazy high jitter and how that completely wrecks the voice conversation and how people can't understand and be intelligible and they go all Skype robot-y and stuff. In this situation, the jitter matters because it's being taken into account as part of the calculation. You know what your
17:41
round trip time to the exchange is, and that round trip time has to stay the same. If the round trip time becomes variable, you've introduced a new variable in a calculation where you were expecting a constant. This is going to cause you problems, right? You know, if the speed of light keeps changing, it's really, really hard to get your spaceship to go to warp 9. We
18:03
used to joke about this back in the olden days when I was a router switchjockey about what would happen if you started having packet loss and the packets were just sort of falling out of the front of the router and hitting the floor. So you put a bit bucket underneath. Everybody remembers that horrible joke from the 90s. In this case, if you drop a packet, you're losing money. So drop
18:22
packets are the enemy. You know, Ethernet. The idea that collisions can occur, no. Collisions are not permitted to occur. It's simply not allowed. Massively inefficient protocols like TCP that handshake are the enemy. Error
18:40
correction? Who needs that crap? Proximity relieves a lot of these issues, right? The closer you are, the easier things are, the less likely that weird happens. You know, when you're doing an international ping, think about the path that packet's going to take. Theoretically, it's going in an undersea cable, but maybe you
19:00
just got switched to satellite, or maybe every other packet takes a satellite path instead of a wire path. You lose all that predictability. You've got induced jitter that's just off the chart, right? So you move closer. You want to be in the same city. Metropolitan area network kind of stuff. Well, those are great. You know, dark fiber from building to building. That's awesome. Remember how long a microsecond
19:21
is? If you're more than a couple of buildings away, that 984 feet is going to kick you right in the ass. So you got to be on the land. You need to be close 300 meters is sort of the absolute top end. That's a thousand feet in American. And remember that this latency costs money.
19:42
It's measurable. You can look and you can say, you know what? We're always 10 microseconds late. That means that we're losing 30% of our trades. Can you make that 10 microseconds go away? We're going to start winning 30% more trades? Damn,
20:02
there's money there. It's often a lot of money there. So, of course, there's money. If you got money to spend, if you got money to make, you're going to make it happen. Nobody freaked out when I showed you that hyper-simplified diagram. Nobody said there's stuff missing. I've seen network diagrams
20:20
before. There were no boxes with flaming hair. Nobody caught that. It's like it's morning. You know, it's the afternoon, right? It's an oh crap kind of moment. Once it's pointed out to you and you go oh shit. There's nothing actually protecting those links. Except
20:42
that they're private. How many people have private internet connections? How many people have private frame relay? How about private VLANs? How about private MPLS? Yeah, you
21:01
dumbasses. There is no private. But the marketeering jerks can sell to people who are uninformed who say to them well, it's your private link. It's your dark fiber that runs from one to one. There is no patch panels in the way. There are no switches that you're transiting through in a blind layer 2 system. You
21:21
know, it's private. It's all yours. And nobody's ever been on a network with other participants, right? There's only two nodes on that network. Only two. There's no one else. There's no other customers. No one else is connected to that exchange except you. You are
21:40
the only customer of the New York Stock Exchange. It's such crap that it shouldn't be. I mean, people who use computers every day that have, it's frustrating because there's no firewalls. Like the simplest
22:00
thing, the most basic thing I want to control with whom I am communicating as tightly as possible. I want to say that I will only communicate with this list of IP addresses and I will only communicate on these ports. The bare minimum. Because guess what? Firewalls add latency. Really great firewalls.
22:22
Awesome firewalls that are all, you know, UTME. They add a crap ton of latency. What do we do about that? Well, there's not really a whole lot we can do, really. We sort of suck it up and say, oh well. Because we're security people and we're okay with milliseconds of latency
22:40
because nobody notices that when they're web browsing. And there's nothing in PCI DSS that says you have to have a fast firewall, you just have to have a firewall in a box in the corner of your data center providing a home to mice. Remember that latency stuff costs money and if you're not in a position where you're saving money, you've gone
23:00
from being an information security department that is a cost center to being an information security center that is a huge cost center. So the risk is smaller than the cost, which is smaller than the profit. Go away. There are no firewalls. Let's go back further in time. Anybody remember the 90s? Like six of us in here. We used to use these
23:20
things called ACLs, access control lists. We'd build them into our routers and stuff. And we all had the Martian list. Everybody remember the Martian list? You know how many things are on the Martian list anymore? There's like four lines. They don't use ACLs because they add latency. Anytime you put an ACL in, the router or switch has to receive the entire packet. And only
23:42
after it has received the entire packet doesn't make a decision. Latency. Most switches don't do cut through while ACLs are on. There are a couple that do now, but they're rare, few, far between. And risk is smaller than the cost, which is smaller than the profit. So screw you,
24:00
no ACLs. Meaningful system hardening. Remember I told you about these crazy, funky boxes that you would think are completely stripped? I mean, these are the race cars of this industry. These are not your Camry. There are no airbags. Brakes are for the weak. Actually,
24:22
they look a lot like those kids who get their mom's Civic and weld a coffee can onto the tailpipe and put a type R sticker on. Because they all have Sol.exe and Calc.exe if they're Windows boxes or sendmail, man pages, MP3 player, four different databases. That's all Linux
24:41
boxes. All this custom interface crap is in there, too. Remember our little team of two developers that wrote some custom FPGA code that, of course, completely implements the Ethernet standards and the IP standards. And they did it all using John Postel's recommendation, which is that you be very
25:00
conservative in what you send, very liberal in what you accept. We know they did a great job of writing this entirely bug-free code. Not so much. So you go to them and you say, well, let's look at the code on your FPGAs. No, a secret. You're not allowed to see that. No sharing. And then you've got all the usual
25:20
complaints about maintainability. I mean, God, once you take Telnet.exe off of a box, then they can't see anything on the network. All their troubleshooting is gone. And these specialized systems come from manufacturers that you've never heard of. Anybody in this room recognize even a single name up there?
25:42
Oh, my crap. Is the entire trading technology industry in this room right now? Shit. This is the only talk you came to Vegas for. Crap, crap, crap, crap, crap. Everybody does good threat modeling, right? Everybody has a
26:00
good solid threat model for their organization. They know exactly who their bad guy is, who they are trying to protect against. Me? No. I can get hired as a secretary anywhere. I mean, have you seen these legs? We know what's missing out of our usual set of controls because we take for granted so much. We
26:20
take for granted that there's going to be change control. Because of course everybody has change control. Everybody uses ITIL, right? Not so much. We take for granted that they're going to have done a great job of things like employee screening. I didn't even do a good job of that in the government for crying out loud. So how do
26:41
we talk about this threat model? How do we start to build who the potential bad guy is? This is not simple stuff. This is actually really hard. So how do we do that determination? Well, let's walk through a couple of things that I think are threats. I think vendors are a threat. It's not that I don't love blink lights and shiny things and it's
27:01
not that I don't appreciate being taken out for dinner and drinks every now and then. But you're trusting that the marketing slick is going to have exactly what you get on it. Anybody seen a lie on a marketing slick? Anybody bought a product that didn't have a feature they told you it had? Okay. The other, everybody who hasn't put up their hand obviously
27:20
hasn't bought anything. This is a talk about capitalism. You're also trusting that they haven't hired any bad guys. So you know those shortcuts that you take in your organizations? They all take them too. I think it's a maybe. I like to have faith that the vendors are doing the right thing. I like to think that ethics is a word that
27:42
they know. I'm probably completely kidding myself. But imagine something very simple. Imagine that a vendor has a disgruntled customer. All vendors have all disgruntled customers. Imagine that a vendor has a disgruntled customer that's making their life miserable.
28:02
Imagine if one developer takes it upon themselves to screw with that customer. They get custom patches. Only for them. In which they've mucked around with the precision time protocol just enough that a microsecond isn't a microsecond anymore. Sometimes it's a little
28:20
more, sometimes it's a little less. At the end of the day, they still have the same number of microseconds. They were just of different lengths. Kind of awesome, isn't it? I think developers are a threat. Any developers in the crowd? I will
28:40
tell you to your face that I love you, but I'm lying. Remember what Admiral Hopper said about wasting microseconds? 984 feet of wire, wrap it around your neck. Please and thank you. In most algo trading, the developer isn't a developer. The developer is like a developer trader. There's someone who has more knowledge of what's going on from
29:00
a market perspective than they have around traditional SDLC. Best of all, they don't do dev, QA, SIT, staging, all that crap. They just put their changes right on to production and they do it multiple times a day. Nothing can go wrong. Nothing.
29:24
And remember how it's kind of odd. When you're working for an organization, you're giving up your IP. Right? That's the whole, you give me dollars and I give you brains. You've become the means of production and the supply and also the labor. It really breaks the whole notion of the industrial revolution. Sometimes
29:42
people get a little attached to their IP. Anybody ever taken something that they made at work and taken a copy of at home? Anybody telling the truth? It turns out that developers are a bit of a problem. They tend to take their crap home. In this case,
30:00
we're talking about 32 megabytes of code. Anybody even remember megabytes? Yeah, kind of like kilobytes. I remember once upon a time, I had less than a megabyte of storage in my entire life. Yeah. 32 megabytes of code, snuck it out over SSL connections, went to
30:22
jail for eight years and one month, paid a fine, has a Facebook fan page. It's kind of awesome. It took him a while to catch him too. I think there's another kind of thread in the insider. So in the financial world, they talk about insiders and they mean something very, very specific. We're not talking about that kind of insider. We're
30:40
talking about somebody who works in the organization. My favorite to pick on here is traders who get smart every now and then, start playing with their blinky lights and shiny things or administrators. Because remember all of these boxes still have in that dark IT department that nobody talks about that exists in the capital markets org.
31:00
They've all got their own administrators doing the same kind of admin work that we've all done through the history of time, which is largely saying have you tried turning them off and back on again. When these traders are administrators who have superior access to the system, you remember those comfy warm feelings about things that of course you know exist, internal controls around segregation of duties
31:20
and all that good stuff, where everyone has root on the trading algo box. I think I'm joking. Could you cause negative effects on other participants if you were an administrator on a box? Could you induce a couple of microseconds latency in someone else's connection? Remember those fancy FPGA boxes, FPGA cards,
31:42
but probably don't have the entire standard on board? If you send them a packet that makes them barf, what kind of barf do they have? If we look at other places where we've seen this happen, where we've seen half-assed implementations of Ethernet or IP, we'll look to the world of industrial control and
32:00
say a couple of years ago if you had an Allen Bradley PLC and you sent it a ping with a payload, a simple ICMP packet with payload, it would literally just stop processing. It didn't crash, it just stopped. Can you make somebody else's algo engine stop? Can you
32:20
make the market algo market engine stop in one packet? I think so. Or you've also got the disgruntled employee, cause you know every organization is completely free of that, who says you know what, I'm going to make sure that my employer never wins another trade. I'm going to induce another 75
32:41
microseconds of latency. I'm going to do it by adding a quarter of a microsecond every other day until it gets to 75. Along the way I'm going to be very angry about PC load letter and the guy in the cube next to me who listens to music at a reasonable volume while collating. Turns out traders are a bit of a problem. This is my
33:01
favorite, favorite, favorite case to wave in the face of people who say that printers are not part of information security's duties. This dude took the code out on slices of trees and only a couple hundred of them. That's really awesome. They had to do forensics on the
33:21
printer to figure out what happened. Everybody knows that your printers have disk drives in them, yeah? Sentenced to 3 years in prison, plus 2 years of supervised release and at that point will likely be deported. I think the market is a threat. I really do. And it's an odd kind of
33:40
technical threat. In our world we've seen this kind of threat before. It's an amplification kind of attack. Once things start going sideways you can push them hard and make them go a whole lot more sideways. If the market suffers from malformed messages where someone is sending the market malformed messages or worse the market is sending out
34:01
malformed messages. What's gonna happen? Badness. How would malformed messages happen? Shitty code push. Yeah. There's a bunch of issues around transaction risk scrutiny and whether or not a transaction is permitted to go through and there's some compromised systems kind of problems that I think are very very real.
34:21
Because you know every large organization with more than 30,000 employees has oh my god has never had a compromised system on the inside. Right. There are no botnets running inside of fortune 50, fortune 100 companies. It turns out that yeah the market is actually kind of a huge threat. In May of 2010 the
34:41
Dow Jones Industrial average plummeted 900 points in minutes. This is a flash crash of about 3 minutes. There's a huge amount of documentation. There's what I think is a great document from Nanex. I was told when I gave this talk at Black Hat that the Nanex report is a lie. I don't know whether that guy had a tinfoil hat on under his dupe. But the point
35:01
is that this was exactly that kind of amplification attack. An algo started selling and it saw the price drop because supply increased on the market and so as the price was dropping it said oh my god I got to sell and so it sold more and the price dropped more and it went oh my god I got to sell more and the price dropped more and things went down hill from
35:20
there. Hi guys. I'm so scared about this next panel I'm in. Ed Felton whom you should all know from Princeton did a really good tear down of it and thinks that it comes down to really kind of five points. A bunch of weird quote requests went into the stock exchange computers.
35:42
Normally these quotes are shoved in a queue but because of the high rate of request the queue got backed up. And there was an error. The quote lists a price and a time. In this case the price was determined when the quote went into the queue but it wasn't time stamped until it left the queue. And it was in the queue
36:01
for more than a couple of microseconds and the price changed during that time period. So what people thought was the price wasn't the price essentially. And these got really confusing. Everything went crazy. The market destabilized. And the faster it happened the weirder it got. And remember we're talking crazy
36:20
faster. I mean this whole the whole issue happened inside of three minutes. And those trades were all backed out. So they essentially pulled a mulligan on the market. I didn't know you could do that outside of stupid games old white men play. So how do we trust in this kind of framework? I mean we've got these threats and these threats are kind
36:41
of spooky scary nasty weird odd we don't really know what to do with them kind of threats. We we don't really trust our co-workers. I don't trust very many of mine. They steal things from my desk like my stapler. And there's no good way to monitor what's going on because this is happening so incredibly fast. But the best monitors for this
37:01
stuff that are available from the general IT market are sampling only. They do not examine every packet they examine on a sampling basis. The more packets there are or the larger the packets are the less sampling they do the more inaccurate they become. There are a couple of very very specific devices that are made for this market that do a much
37:22
better job. But you're in IT security not capital market. So you have got no idea who those companies are or where those devices come from. Um traditional security is an absolute fail here. Just epic. We're a hundred thousand times too slow. We come in and say well we're gonna add a couple of milliseconds of latency to those transactions so that we can make you secure
37:41
with our UTM device which will protect you from email based threats. The only thing that's supposed to transit the network is fixed transactions on specific IPs and specific ports. Anything else that's moving on that network is illegal and shouldn't be there. You should just frigging drop it. You don't need pattern behavioral detection. Um
38:02
and IT security is so god damn proud of itself that it's unwilling to listen. It's unwilling to learn. It's unwilling to sit and be the student and say well shit teach me about this stuff. Because we're all focused on checkbox compliance that's so easy. Alright. Oh do we have a firewall? Check. I
38:22
have twenty one minutes left not five. I know I started late. Um so let's answer these hard questions later. Okay the hard questions we're not in a position to answer right now because we're still way too busy figuring out where the heck our socks are and pulling them up. We're not going to be able to secure custom everything. Who possibly
38:40
in this room besides maybe ten percent of you can go through FPGA code and figure out whether or not it's got a bug relative to a standard that's spread over seven eight hundred pages worth of documentation. Yeah. Um we've got to learn how to be fast enough and we're not there yet. Um we don't even really understand this whole money thing. Um and we need to make
39:01
the case that security efforts are important because they reduce the chance of disaster. We don't know for sure that the next time there's a flash crash that those trades will just be backed out. The market may just say caveat emptor suck it up. Um I need you to
39:20
do something though pretty much anything. Um it's time to party like it's nineteen ninety nine and do some network security basics. Like those ACL things let's implement some shall we? Um there isn't a whole lot in the next generation. Juniper and Cisco are coming to the table with some stuff that's that's
39:40
pretty fast. Um it's still not quite fast enough there isn't an argument um but it's pretty fast. Um we need to keep up and I I don't know exactly how we're gonna do that. But I am sure that we need to do anything. Literally anything. Um
40:00
even just understand what what it is. Go and find the people in your organization that barf Visio and get them to walk you through the diagrams. Follow the packet follow the money follow the process start to understand exactly what's going on. Um and make friends and influence people. Buy some coffee. It's it's kind of important. The reason
40:20
I'm so pissed off about this is because this report came out just a couple of weeks ago that says IT security pros think that performance is more important than actual security. We're shooting ourselves in the back while we're giving ourselves a self congratulatory reach around. And it's stupid senseless pointless dumbassery
40:41
because we're also damn sure that we're experts in everything. Challenge the vendors we want more than check boxes. You know they come back and they say well nobody's asking for it so you know because nobody's asking for it we're not doing it. We're going to give you raging featuritis instead. Um we're going to change those LEDs from green
41:01
to blue and that's going to make it better. Damn bullshit. Um if you're a risk process policy or GRC wonk what the hell are you doing at DEFCON? Um thank you. Uh work with the business folks. Um you have equivalents in the
41:20
business that aren't in IT that understand things from sort of a different perspective than you do. Uh and and their tolerances for risk are very different from yours. You you may still be suffering from the pallorism disease where you think that attaching an iPod to a computer is this oh my god hair on fire kind of threat. Um they're not in that same kind of world. They don't think dogma is all that important. They're ready to rewrite the rules every time
41:40
something changes in the markets which is daily. Um you're never going to be able to change their minds about the cost of latency because it's real. They've got the numbers to back it up. They've got the profit. You're not a profit center. You're a cost center. Um so work with them and learn to understand that just because we did it that way last year isn't a
42:00
good reason to do it that way this year. Um compliance people meet your financial compliance people. They know more about compliance than you ever will. Um the SEC is taking an active interest in this. Um just last week they released something called uh the large trader reporting rule which is 13H-1. Uh you should read it.
42:20
It's really really interesting. This is their entire response thus far to the flash crash and what they're saying is it would be nice if we had an audit trail. Yep that's right all you need to be regulatory compliant is a better audit trail. If you're
42:41
in the trenches do some fucking research. We haven't seen original research at Def Con in god knows how long. The next derivative piece of shit talk I am so fed up. Do the research please. Make the time. If you don't have 20% or 10% time from work
43:00
you should be doing it at home. Find something and start picking at it and peel up the edge. I only started really poking into this actively about 6 months ago. I'm here you're not. Understand your business partners. Build the POC lab so that you can show this stuff. Your POC lab
43:22
is gonna have to be weird and esoteric. It's not the POC lab that you're using right now for your little Windows vulnerabilities and desktop bullshit. You need 10 gigabit E. You need InfiniBand. You need PCI E machine to machine. You need machines that have 12 cores, 128 gigs of RAM and stupid fast connections. You need the open source fix transaction system so
43:40
that you can build fixed transactions and send them out on the wire. You don't understand this stuff and you try to talk with authority about it until you've played with it you don't get to talk about it. It's kind of like teenagers and sex. Encourage the vendors to get with the program. Please thank you oh my god vendors step up. Couple of them have and huge
44:02
kudos for the couple that have. I'm in the next panel but I will do Q&A if you see me walking around I'll talk to anybody for anything. Yes you
44:20
can bring the beer up now. Oh you're waiting for I'm in the next talk. I need the beer now. I'm worried about those paper towels though. Yeah I always have my towel. Thanks everybody I really appreciate you coming out.
Recommendations
Series of 5 media