HVACking Understand the Delta Between Security and Reality
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 335 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/48366 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DEF CON 27101 / 335
8
9
12
14
32
38
41
58
60
61
72
75
83
87
92
96
108
115
128
132
143
152
158
159
191
193
218
230
268
271
273
276
278
295
310
320
321
335
00:00
Information securityVulnerability (computing)Computer forensicsMeta elementExploit (computer security)Process (computing)SharewareVulnerability (computing)Software testingInformation securityComputer animation
00:47
Control flowPhysical systemBuildingComputer programGame controllerGUI widgetTouch typingZugriffskontrolleBuildingTouch typingPhysical systemMultilaterationCommunications protocolGame controllerRouter (computing)Data managementBitPressureMoment (mathematics)Bus (computing)WeightComputer animation
01:57
Function (mathematics)Landau theoryoutputAnalogyDataflowCAN busBinary filePhysical systemState of matterMoment (mathematics)Function (mathematics)Module (mathematics)MultiplicationCAN busoutputInstallation art
02:41
Drum memoryVector graphicsState of matterService (economics)GUI widgetFirmwareSoftwareData modelScripting languageVulnerability (computing)Vector spaceCommunications protocolSoftwareDirection (geometry)Port scannerLevel (video gaming)Software developerMultiplication signComputer programEmailFirmwareRootPhysical systemSymbol tableJSON
03:36
Control flowComputer networkBuildingIPRevision controlUDP <Protokoll>Link (knot theory)Virtual realityData typeLengthMaxima and minimaService (economics)Axiom of choiceCommunications protocolMaxima and minimaBroadcasting (networking)TouchscreenSimilarity (geometry)Server (computing)Buffer overflowRouter (computing)CASE <Informatik>Attribute grammarModule (mathematics)Software testingReal numberFuzzy logicCore dumpComputer animation
04:34
Core dumpPhysical systemReading (process)BootingTerm (mathematics)Error messageMultiplication signVulnerability (computing)BitPhysical systemCore dumpTouch typingCrash (computing)TouchscreenSource codeJSONMeeting/Interview
05:27
Stack (abstract data type)Computer programIdentical particlesElectric currentThread (computing)InformationUltraviolet photoelectron spectroscopySoftwareSemiconductor memoryCrash (computing)Physical systemCore dumpSimilarity (geometry)Address spaceOpen sourceKey (cryptography)JSON
06:38
Address spaceBacktrackingCorrelation and dependenceAddress spacePhysical systemCrash (computing)Computer wormMultiplication signLine (geometry)Data storage deviceCodeSoftwareComputer animation
07:21
Functional (mathematics)Figurate numberSoftware bugAddress spaceCompilation albumVulnerability (computing)BitSymbol tablePoint (geometry)Wrapper (data mining)System call
08:04
SoftwareBuffer solutionOpen sourceCodeVariable (mathematics)Network socket2 (number)Software developerPoint (geometry)Computer animation
08:41
SubsetNumberExploit (computer security)Matching (graph theory)Line (geometry)Functional (mathematics)Buffer solutionSystem callResource allocationVulnerability (computing)Crash (computing)Buffer overflowSheaf (mathematics)Computer animation
09:31
Open sourceCondition numberAddress spaceSheaf (mathematics)Order (biology)Buffer overflowSemiconductor memoryVulnerability (computing)Exploit (computer security)Condition numberWritingComputer animation
10:17
BootingPhysical systemMessage passingError messageComputer architectureMathematical analysisFluid staticsType theoryServer (computing)Physical systemError messageGroup actionOrder (biology)BootingSymbol tableInstance (computer science)BitComputer animation
11:12
Binary filePatch (Unix)CodeBinary codePhysical systemPatch (Unix)Reverse engineeringLine (geometry)Right angleComputer animationMeeting/Interview
12:03
Serial portNetwork topologyDynamic random-access memoryWeb pageMach's principleGUI widgetLimit (category theory)Asynchronous Transfer ModeSoftwareLink (knot theory)Mountain passDuplex (telecommunications)Configuration spaceRootBinary filePhysical systemHecke operatorComputer hardwareBootingSoftware2 (number)Functional (mathematics)Combinational logicCodeTouchscreenComputer programSoftware developerSource codeJSON
13:17
Computer wormBroadcasting (networking)Data bufferControl flowPhysical systemCondition numberGame controllerCrash (computing)Broadcasting (networking)Semiconductor memoryComputer wormBuffer overflowUniform resource locatorBuffer solutionMathematical analysisComputer animation
14:09
Table (information)Control flowFunction (mathematics)Address spaceGame controllerFunctional (mathematics)Vulnerability (computing)Semiconductor memoryCartesian coordinate systemSheaf (mathematics)Run time (program lifecycle phase)Electronic mailing listCodeMathematicsTable (information)MultiplicationComputer animation
15:01
Chi-squared distributionRead-only memoryControl flowDefault (computer science)RootReverse engineeringGastropod shellVulnerability (computing)Functional (mathematics)Game controllerSemiconductor memoryReverse engineeringPhysical systemGastropod shellBuffer solutionScripting languageSoftware developerOrder (biology)Multiplication signCASE <Informatik>RootAddress spaceBranch (computer science)Data storage deviceBootingMemory managementCodeXMLComputer animation
17:26
Control flowRead-only memoryArmAsynchronous Transfer ModeThumbnailGraphical user interfaceMemory managementSemiconductor memoryLengthCoprocessorHydraulic jumpLevel (video gaming)WebcamAsynchronous Transfer ModeThumbnailDialectKey (cryptography)ArmString (computer science)Gastropod shellTotal S.A.Sheaf (mathematics)Internet der DingeJSON
19:15
Read-only memoryPhysical systemAddress spaceInclusion mapError messageRootLipschitz-StetigkeitGastropod shellPhysical constantGroup actionIP addressAddress spaceVirtual machineSemiconductor memoryNumberVideoconferencingInstance (computer science)SpacetimeGraphics tabletSharewareRootVulnerability (computing)Matching (graph theory)Reverse engineeringConfiguration spaceMessage passingInterface (computing)Physical systemLine (geometry)Letterpress printingString (computer science)ArmComputer animation
21:21
Physical systemCodeoutputState of matterRootGame controllerComputer hardwareTerm (mathematics)PhysicalismStrategy gameDatabaseTouchscreenPressureTouch typingComputer fileRaw image formatGoodness of fit
22:17
InternetworkingCommunications protocolSystem callMathematicsDisk read-and-write headExecution unitIntegrated development environmentVariable (mathematics)Object (grammar)Linker (computing)Point (geometry)Data structureDatabaseGame controllerFunctional (mathematics)outputBinary codeComputer programProcess (computing)CodeSpacetime1 (number)Object (grammar)Point (geometry)Dirac delta functionComputer hardwareLinker (computing)WritingModule (mathematics)Variable (mathematics)Symbol tableSemiconductor memoryIntegrated development environmentCAN busCategory of beingFunction (mathematics)MalwareRight angleSource codeComputer animation
24:00
Scripting languageVariable (mathematics)Integrated development environmentPoint (geometry)Object (grammar)Linker (computing)Read-only memorySpacetimeCodeComputer programStrategy gamePattern languageComputer programmingMilitary operationAerodynamicsStructural loadShared memoryNormal operatorPoint (geometry)Computer programNormal (geometry)Linker (computing)Line (geometry)Object (grammar)Structural loadOrder (biology)JSONComputer animation
24:38
Thread (computing)AerodynamicsObject (grammar)Structural loadLinker (computing)Computer programShared memoryComputer programmingMilitary operationBinary fileMalwareFunction (mathematics)MathematicsComputer programGame controllerWritingFunctional (mathematics)Order (biology)Thread (computing)outputDynamical systemStructural loadLinker (computing)Library (computing)Binary codeSymbol tableReal numberElectronic signatureRun time (program lifecycle phase)System callComputer animationDiagramProgram flowchart
25:40
Structural loadComputer networkComputer-generated imageryDatabaseFunction (mathematics)MalwareCAN busData structureThread (computing)Control flowComputer configurationMultiplicationComputer programFunctional (mathematics)Process (computing)Thread (computing)ConsistencyDiagramCorrespondence (mathematics)CoroutineoutputContent (media)Parameter (computer programming)MalwareOrder (biology)Game controllerSoftwareQuicksortLevel (video gaming)Condition numberShared memoryKey (cryptography)DiagramProgram flowchart
26:48
MalwareExecution unitDegree (graph theory)Computer hardwareData storage devicePay televisionString (computer science)outputFunction (mathematics)Data structureParameter (computer programming)InformationState of matterPoint (geometry)MalwareGame controllerEndliche Modelltheorie
27:58
Remote administrationMalwareoutputReal-time operating systemState of matterTelecommunicationKey (cryptography)Tracing (software)Interactive televisionComputer hardwareReal number
28:41
Data managementBroadcasting (networking)Data transmissionComputer networkVector spaceEmailReal numberInternetworkingKey (cryptography)Online helpSoftwareNatural numberMultiplication signDefault (computer science)NumberPay televisionRemote procedure callVector spaceBitBroadcasting (networking)Different (Kate Ryan album)FirmwareOpen sourcePhysical systemVulnerability (computing)Slide ruleDiagramGroup actionProcess (computing)Data transmissionMalwareSet (mathematics)Computer animation
31:17
Touch typingIndependence (probability theory)Ocean currentType theoryComponent-based software engineeringWater vaporSharewareReal numberState of matterServer (computing)Boss CorporationPhysical systemDesign by contractDifferent (Kate Ryan album)Order (biology)Sound effectExecution unitFunctional (mathematics)Independence (probability theory)Video gameForm (programming)
32:51
RootDirectory serviceScripting languageReverse engineeringMotion captureGastropod shellPasswordFehlerbaumBinary fileDialectControl flowExploit (computer security)MalwareRight anglePay televisionScripting languageVulnerability (computing)Game controllerHacker (term)BootingOrder (biology)MereologyMedical imagingPhysical systemTouchscreenFunction (mathematics)Functional (mathematics)Computer animation
33:36
AnalogyFunction (mathematics)Variable (mathematics)Binary fileBefehlsprozessorWhiteboardoutputPosition operatorTouchscreenHand fanUniform boundedness principleLevel (video gaming)Point (geometry)Set (mathematics)Lie groupData typeAmsterdam Ordnance DatumMalwareSatelliteTotal S.A.Open setElectronic mailing listRight angleMultiplication signFunction (mathematics)MultiplicationDegree (graph theory)Physical systemAnalogyoutputGreatest elementPoint (geometry)AreaHand fanTouchscreenSound effectPay televisionDifferent (Kate Ryan album)Independence (probability theory)Data centerComponent-based software engineeringType theoryCuboidBinary codeSharewareFrequencyComputer animation
36:09
InternetworkingControl flowComputer networkBuildingPhysical systemVariety (linguistics)Different (Kate Ryan album)Moment (mathematics)BitTouchscreenPressureRadical (chemistry)Dependent and independent variablesData storage deviceSoftwareOrder (biology)AreaInternetworkingPoint (geometry)Game controllerSensitivity analysisStreaming mediaState of matterCASE <Informatik>OvalReal number
38:37
Patch (Unix)MereologySound effectNumberStandard deviationPresentation of a groupPatch (Unix)Type theoryOrder (biology)Moment (mathematics)Information securityVulnerability (computing)Computer animation
Transcript: English(auto-generated)
00:00
Thanks for coming and uh please join me in welcoming Douglas McKee and Mark Barraza. Thank you and welcome. We're really excited to be here. We're glad you could come see our talk today. We're gonna be talking about a zero day uh that we discovered. We're gonna walk through our vulnerability research process, our exploitation process and post
00:23
exploitation and hopefully have some really cool demos for you uh at the end. So first off, who are the two crazy idiots standing in front of you? Uh my name's Douglas McKee. I'm a security researcher for McAfee's advanced threat research team. I've got a little over 8 years of experience in vuln research and pen testing. And I'm Mark Barraza. I have a little over 8 months of experience in vulnerability research and
00:44
pen testing. That's great. So first off, most importantly, what are we here to talk about today? We're gonna talk about an ICS system produced by a company called Delta Controls. They refer to it as their Intellibus Manager. And what this is, is it's
01:00
called a building controller. It's used to manage systems in a commercial building. Things like HVAC, lighting, pressure rooms, etcetera. And important thing to note is that this device uses a networking protocol called BACnet. That's largely how it manages these devices. BACnet stands for Building Automation Control Network. And I also
01:22
wanna highlight that this is not specific to Delta systems. This is an industry wide used protocol. It also has a slight, a little bit of router capability built into it, which I'll touch on here a little bit later. And before I get, go any further, I often get the question, how do you pick your targets or why did you pick this target? Uh,
01:41
generally at ATR we try to pick targets that can have a very high impact as if they were compromised. Obviously there's a lot of critical systems that, that fall into that. And this happened to be one that we were able to attain. And also something of interest is the fact that it uses the BACnet protocol, therefore it's network accessible. So let me take a moment to explain to you how this works in the real world before we jump
02:00
into hacking this device. So what you see in front of you is we have a wiring diagram of one of these devices, uh, potentially for an HVAC system. So you have the Delta controller, which talks over a CAN bus to something called modules. At the highest level, these modules take inputs to use, and uses it to control outputs. So each device is programmed differently from the installer, and depending on the state of
02:24
those inputs, it changes those outputs. So in the wiring diagram in front of you, assuming it's an HVAC system, depending on the temperature, it'll do something like turn on, on or off a boiler, turn on and off a pump, etc, etc. And this is just one example, it's used across industries for multiple things, which we'll get back to here momentarily. So understanding this, how do we start looking for bugs, or where do
02:45
we start looking for vulnerability research? Well as I said, one of the most important things for us from attack vector is we like the fact that it's network accessible. So in front of you, you can see, uh, an nmap scan of the fact that it uses the BACnet protocol. So now we know what direction we want to attack it from, we're gonna do a network
03:01
attack on this, hopefully. Well what else do we need? It'd be really useful to have the software. Well, lucky for us, the developers left, uh, UART headers on the board, which was completely unauthenticated root access to the system. This made it simple for us to pull off the firmware. So if we, if we look at that firmware for a moment, a couple important things to note, is one, there was no ASLR, and NX was enabled. And
03:25
also, it was quickly able to see that symbols were left into the program. This will become extremely useful and I will reference this multiple times throughout this presentation. So we want to use a network attack vector, so what does that mean? Well we're gonna start with fuzzing, like everybody else. So, we decide to use a
03:41
commercial fuzzer called DefenseX. Uh, this, this fuzzer has a built-in module for a BACnet server. We weren't quite using a BACnet server, but because of the router protocol, the routerness of the device, it had a lot of similar attributes. And one of that is, the BACnet protocol uses broadcast traffic pretty frequently. So we use this fuzzer as a starting point, and started to fuzz the device. This is one of those packets that you
04:03
see on the screen, that the fuzzer chose to use as a test case. Couple things I'll highlight real quickly. Uh, besides the fact that it's a broadcast packet, the fuzzer here is sending about 8,000 bytes in what's called the BLVC layer, the BACnet layer of the packet. You can even see that Wireshark is telling you that this layer is really only
04:21
supposed to be a maximum of 1476 bytes. I think it's pretty easy to see here that the fuzzer is trying to look for a buffer overflow attack. So we fire up DefenseX, we run this test case, and after, after a few minutes, we see this on the UART screen. Quite simply, what you see is that it's a core dumping and the system's rebooting. Now I'm a
04:40
little bit of a pessimist when it comes to vulnerability research, so I never actually believe I found something the first time. And so I run the DefenseX uh, at least 2 or 3 more times. And for your amusement, I took a picture of Mark's face, uh, once we were running the fuzzer, because this is pretty much what it looked like. And he got excited because we had a reproducible crash. Ironically though, we, I went back to that packet
05:02
that I showed you on the previous screen and I sent just that packet to the device. Interestingly enough, the device did not crash. Well, I sent that packet twice and the device still did not crash. Through a lot of trial and error, I actually discovered that it took 97 packets to crash this device. And I'll touch on this here momentarily. But after discovering it took 97 packets, I was able to do it repeatedly. So that
05:23
meant it was time to investigate further. Well, as you saw, the system could produce a core dump. So let's take a look at that core dump as a first place to start to try to figure out where to move on. Obviously we have a seg fault here, that makes a lot of sense, the device crashed. But what can we learn about that looking at the registers? Well, let's start with the R0 register. For those of you that haven't fallen asleep
05:42
yet, you might notice some similarity in what's in the R0 register to the packet of which DefenseX decided to send to the system. There's a lot of 22s or ASCII quotation marks. That's interesting, we know that that's network data. If we keep thinking about network data and we look forward at R3 and R8, R3 has the value 81 in it. For those of you that were smart enough not to read the BACnet RFC, you wouldn't know that
06:04
81 is the first byte uh set in all BACnet packets. This indicates the BLVC layer. Again, this indicates that there's some networking data in the system. And if we take a look at R8 and we account for the Indians inside the computer, we know that this is 47808 or the source or destination port. So once again we have some clues that we have
06:22
networking data where the crash is occurring. Obviously the next and most important question is where in the system is it crashing? Uh and we can look at the backtrace in the R- LR register to give us hints on that. So let's just do that. Let's go to the exact addresses in memory where the crash is occurring. Well, lucky for us, we find ourselves inside memcpy. And not only do we find ourselves in inside of memcpy, but
06:45
the exact line of code which it's crashed on, it is trying to take the data in R3 and store it at the address located in R0. Well this fully uh explains why the system crashed. R0 has a invalid address in it that came from our payload. And it's attempting to store the first byte of the networking payload into that address. So this is
07:05
obviously extremely interesting. But I don't know for you, the last time I opened up any commercial Linux application, memcpy was called more than once. So this is not enough. Sometimes memcpy is called hundreds of times. So what is the exact code path that triggered this, that triggered this crash? Well we can use the LR register to help
07:22
us figure that out. In ARM, the LR register points to the instruction that's going to be executed after a function call. So to actually know what function was called, we need to look at one instruction before the address being pointed to by the LR register. But in doing so, we can see a function called scnetmove. Thank you again for leaving the
07:40
symbols in, it was extremely useful. If we dig into this scnetmove function, it's actually just a wrapper from memcpy. And this becomes a place to start as far as understanding why this vulnerability exists. And if we continue down this path and we use the ID compilers and a little bit of, a little bit of intuition, we can actually very
08:00
easily statically analyze this bug and figure out why it's there. So, I've renamed some variables here to help this be a little simpler to see. But what we have is we have a buffer which I've named source. And that buffer is being used to read data off the wire through a socket function. That buffer is hard coded to be 1732 bytes, both in
08:21
size and what it is to read off the network. And then after it's reading that off the network, it stores that in a variable and I've renamed that size. So, I wanna point out for a second that this in itself, there is no development flaw here. It's statically defined, it's ensuring that it doesn't read more than 1732 bytes, this is perfectly fine code. However, if we continue to look at this code, we'll see a second buffer
08:44
which I've named destination, uh, which is allocated using the Linux packet alloc function to 1476 bytes. Again, if you didn't fall asleep in the beginning of the presentation, that number might mean something to you. That is the number that is the, the acceptable amount of bytes which is expected in a BACnet packet. However, it doesn't match the 1732 from the
09:03
other buffer. Those experienced with vulnerability research and exploitation probably see where I'm going with this. There's a further call down the line where that SCnet move function is called and it uses both those buffers but it hasn't validated that it's the same size and hence we have a buffer overflow. But now I'll step back and remind you that I
09:21
said just sending that large packet did not crash the device. In fact, sending 2 of the large packets didn't crash the device, it took 97. So, what's that all about? Well, the interesting thing is the large packet triggers a buffer overflow, it overflows a section of memory where an address is being stored for a future memcpy. So, what we've
09:40
overwritten is that address, hence why it was stored in R, uh, R0 in the beginning. So, in order to exercise this vulnerability, it took 97 more packets before it tried to access that address. And what's, so what we have is we have the initial packet which overwrites a section of memory because of the buffer overflow and 97 packets later, it takes that
10:01
data and tries to write it to that address. And we have a typical write what where condition. This is really promising for us as vulnerability researchers because this means that exploitation is highly likely. Well, now that we've analyzed this statically and we think that we have something worth going forward on, we'd really like the ability to dynamic, dynamically debug the system. Anyone that has done this type of work before,
10:24
static analysis is great, but dynamic analysis really makes it a lot simpler. So, as expected, we compiled GDB server for the correct architecture of our device, plopped it on there, and, and started to debug, except for we instantly had a crash. Uh, and as you can see, this has been heavily redacted at the request of the vendor, but what you
10:43
have going on here is there is a, a, a watchdog error which occurs because we've triggered a watchdog timer and the system reboots. For those not familiar, in ICS systems, watchdog timers are extremely common. They're used so that if the system stalls in any way, because these are critical systems, it takes some action in order to ensure that the
11:01
system doesn't stay down. In this instance, the system reboots. However, it's a problem if you're trying to dynamically debug the system. Well, because the error messages were left in and because symbols were left in, it was actually very easy to track down this line of code. As you can see here, there are, I think it's about 3, 3 counters which are being decremented by 5. So, when the counters hit 0, it triggers the
11:23
watchdog timer, and it trips, and the device reboots. Well, this is not a hard problem for us to fix as reverse engineers. We will just simply binary patch this, uh, we take the 5, we change it to 0, and now as far as the system is concerned, it, it still
11:41
executes that line of code and decrements the value, but it decrements it by 0. As you can see on the right hand side of the screen, even IDA now removes that line. It's not that it's not there, it's just ineffective. So, thinking that we're, you know, pretty smart, we reload this back on the system, and we start debugging, and of course it starts to work, until exactly 3 minutes later. And 3 minutes later, the
12:03
system reboots itself again. Now, at this point, Mark's pretty agitated, because he thought he already fixed the problem. So, now we're, we're looking and trying to figure out why the heck did the system reboot. Well, uh, unfortunately, we forgot that in ICS systems, there's generally a hardware watchdog timer as well. And this hardware
12:23
watchdog timer was set in boot, as you can see on the UART screen here, to 180 seconds or 3 minutes. So, how do we get past this? Well, the hardware watchdog timer, that means the software on the system has to kick that watchdog timer. Well,
12:41
in this instance, the developers, for all of their functions, exported the, the functions of the SO files, which means that all we had to do was use their code to kick the watchdog. So, we wrote a small C program, we, uh, put, uh, compiled the program for the device to execute those functions, and then set it to run on boot up, and so when the
13:03
system boot up, it called our watchdog kicker, and it always kicked the dog. A combination of this hardware watchdog fix and the binary patch, and we were able to effectively remove the watchdog as a requirement for debugging. And we were on our way. Now, that was a short tangent, but I thought it was worthwhile. So, let's get back to the job at hand. We want to write an exploit for this system. So, where
13:23
are we at? Well, we have a broadcast packet with a payload of 1732 bytes. By sending that packet, we can trigger a buffer overflow. That buffer overflow by itself does not crash the system. In fact, it takes another 97 packets to cause the crash. But, when we cause the crash, it manifests itself into a write what where condition. We know
13:44
from our initial analysis that there is no ASLR enabled on the system, but we do know that NX or DEP is enabled. So, the next step is we want execution control. And if we have a write what where condition, this question simply becomes, what do we want to write where? Well, considering that NX is enabled, we want to find a location in memory
14:05
that is writable that would allow us to gain some control of execution. Lucky for us, there's something called the global offset table. This is, uh, native to Linux. And it's a table, uh, created at run time. Therefore, it is usually writable. And simply
14:20
speaking, it has a list of function names and addresses. When a function is called, it looks up the address for that function and then it jumps to that section of memory. Well, since we have a write what where and we can write anywhere we want, hypothetically speaking, we could take something like malloc, change the address of malloc, and then when malloc was called, we would be able to jump to our section of code. This would
14:42
gain us control of execution. But, we also need to know what function do we want to overwrite. Now we know where we want to overwrite, but what do we want to overwrite? This is a, uh, multi-threaded application. And so we really want to make sure that we gain execution control as soon as possible after the, the vulnerability is triggered. If we go
15:01
back to where we saw the vulnerability exist in the scnet move function, we can see approximately 20 to 30 in- assembly instructions later, we end up with a function called scdecode backnet UDP. This becomes the ideal place for us to gain control of execution as fast as possible from when the vulnerability is triggered. With all that background, the next simple thing to do was to simple, to write our own code to gain, to, to
15:24
execute. And for that, I'm going to turn this over to Mark. Thanks Doug. Alright, well, before we could finish pwning this device, it's important that we step back and look at the resources we already had available. So, to begin, if
15:40
you're paying attention, haven't fallen asleep, you know that we have execution control in the system using the GOT overwrite that Doug mentioned previously. Additionally, the developers were kind enough to leave Netcat installed on the device. This is your granddaddy's kind with the dash e flag, so you knew this was gonna be good. Uh, lastly, we still control some memory on the heap from our initial buffer overflow,
16:01
although a lot of it does get clobbered by the time we gain execution control. Okay, so that's all well and good. The next question we have to ask ourselves is, what do we want? A simple question, but as Mr. Gosling can attest to, sometimes you just gotta ask. Well, we decided to start pretty simple. We wanted root access, obviously, but ideally we also wanted persistence on the device across reboots. So, the last questions
16:23
we had to ask ourselves were, how do we get this, but more importantly, how do we get it easily? You'll soon see that we still had a very long road ahead of us, so we wanted something quick. Well, by looking back at what we already had available to us, uh, a straightforward way to get root access would be to use the Netcat already installed on
16:41
the device to obtain a root shell, or a reverse shell. We could accomplish this hypothetically by putting some shellcode into that heap memory we control, and this shellcode would essentially, uh, store a Netcat command for the reverse shell into R0 register, and then branch to the address of the system function in libc. Uh, the system function, as you may or may not know, uh, executes the command stored in its first and
17:03
only parameter, in this case a Netcat reverse shell command stored in R0. This ultimately comes together to create what is commonly referred to as a return to libc style attack, and although this would not grant us persistence on its own, ultimately we could use our reverse shell in order to alter some of the startup scripts, and that would, that
17:22
would grant us persistence on the device. So that's all well and good, but if we actually take a look at the heap memory we control, we'll see that it's actually not so easy. So the, the memory we still have in met, uh, that we control on the heap is the 22's bytes you see in front of you. And particularly, we have, the longest
17:40
continuous chunk that we control is only 32 bytes long, highlighted there in red. Now why is this a problem? Well, the reason being is that somehow in this 32 byte section, we had to fit all of our shellcode instructions and the command string for Netcat, which on its own is 24 bytes. Using some very complex arithmetic, this leaves us with a grand total
18:01
of two ARM instructions, which is simply not enough to do what we want. Now I know some smartasses out there are probably thinking to themselves, just use thumb mode you dingus, those instructions are half the length. But, although this is a great idea, through trial and error, we discover that thumb mode is actually not enabled on the processor on the device. So that's out. I wanna be, I'm gonna level with you guys
18:22
here. At this point, we were stuck. I couldn't sleep, I couldn't eat, I even considered switching over to an IOT webcam. Things were looking very, very dark. But suddenly I was struck with some motivation. I thought to myself, and I could, if I could live off of chili chow, cup of noodles, registered trademark, for four years, then this
18:42
should be child's play. And then I got thinking some more. Child's play, children playing, children play hopscotch, memory hopscotch. That was the key. Now what the hell am I talking about? Well, when we started looking, digging in further, we found that we don't
19:00
have to just use the 32 continuous bytes, we could use any section of memory we control, as long as it's at least 8 bytes long. This grants us one shell instruction, and a jump to the next region of memory we control, or hop, if you will. So, if we look back at the heap memory we control, you'll see two such regions that qualify. This effectively doubles the number of shellcode instructions we have from 2 to 4, which is a
19:23
great news. But we weren't out of the woods just quite yet. Unfortunately, ARM, ARM is actually very, uh, cheap with its immediate values to be, to be able to do as an understatement. In fact, it limits them to only a single byte each with optional zero padding left or right. This meant that if we wanted to load the address of system into a
19:43
register, it would take all 4 of our instructions, and that wasn't gonna fly. Well, thankfully, we were able to use the address already located at R4, because it was pretty close to the address of system, so with a simple offset, we could get there. In the end, this resulted in shellcode that met our tight space, memory space requirements, and granted us a reverse shell. But instead of explaining in detail how cool
20:04
this all is, how about I just show you instead. So, here we have a demo video of the reverse shell in action. On the right half of the screen, I hope you can read it, is the UR interface of the device. Now, the first thing we're gonna do is we're gonna run IF config to find out that it's IP address. So, you can see there it ends in 7.14.
20:25
Next, we're gonna print out the first few lines of the pro, er, of the device's startup script. This is just to prove the, that we are, in fact, in the delta device, and you can see the copyright message there. Uh, next, we're gonna show that there's no instances of netcat already running on the device. As you can see, we'll refer to this later.
20:42
Moving over to the attack machine on the left, we start up a netcat lesson there in port 9, this will capture a reverse shell. And on the same attack machine, we launch our exploit script, which leverages the vulnerability and launches the netcat command you see there. So, once it's finished executing, we move back to the UR interface,
21:01
and show that there is, in fact, an instance of netcat now running that wasn't before. And it has the same command string. So far, so good. Okay, moving back to the netcat lesson there, which should have connected, we run IF config again, and the IP address matched. Imagine that. And, just to confirm, we do, in fact, have root access. Okay,
21:21
that's great. Mission accomplished, right? We got root access to the system, everything's good. Raw. Like a tenured physics professor, we don't know when to quit. I mean, after all, we wanted IO control on the device. We're more interested in the HVAC systems, the access control, the pressure rooms that are being managed with this device. And
21:42
root access does not grant us that on its own. We need to keep digging. Our next strategy was to look at how the device, uh, natively handles controlling the IO hardware, and maybe get some of that code to work for us instead. So, the first approach we tried is to look at the database files located on the device's file system. Now,
22:01
something that immediately jumped out at us here is that I had no idea what the hell I was looking at. This was too complicated. Well, the next thing we tried is to look at what happens in terms of, uh, packet sent, when we try to change the IO state from the touch screen. And maybe we could replay some of those packets. Now, those of you paying very close attention might notice that the structure of the packet is actually very
22:22
similar to the database in that I still have no idea what the hell I'm looking at. This was still too complicated. Well, the last thing we tried is hooking to the IO control functions natively. And this turned out to be just right. Now, as Doug mentioned before, symbols being left and proved invaluable here because functions of names like
22:42
CANIL write output immediately jumped out at us because we knew that the IO modules were controlled through the CAN bus. Furthermore, through reverse engineering of these IO control functions, we discovered that there was different ones dedicated to controlling each category of IO hardware and there were 6 such categories. You got your inputs, you got your outputs, and you got variables and this can, these can all be either
23:04
analog or binary. Now, our first initial naive approach was to try and execute these functions from, uh, external C code like we did with the watchdog kicking code. But unfortunately, the handles to this, to these hardware devices were trapped inside the program space allocated for the delta's existing programming. And they simply
23:24
couldn't be accessed from an external code. Somehow, we had to figure out how to insert our malicious code, whatever ended up being, inside the program space of the existing programming. Well, the solution ended up being something called LD preload. And for those that don't know, LD preload is a Linux environment variable. And its job is that
23:43
any shared objects this variable points to are loaded first by the dynamic linker when a binary is executed. Now, this is not a new or novel idea. This is actually a pretty common approach for inserting code into an existing program's memory space, which is exactly what we wanted to do. So, in our case, we, uh, used our persistence on the
24:02
device to alter one of the startup scripts, have this variable point to our malware, and then get it, uh, inside the existing programming. And that's all well and good. But how does this work mechanically? Well, let's begin by looking at the normal operation of the device. So, in normal operation, the first thing that occurs is that the
24:20
delta programming will execute. Pretty straight forward. Once the dynamic linker starts doing its job, it loads the shared objects in the following order. First, the dactetra binary, which is the entry point. Then, all the delta proprietary shared objects. And finally, all the other miscellaneous libraries, like libc. Now, once the programming is, is actually running, you have an IO polling thread that consistently
24:42
runs in the background, and this is what calls those IO control functions I mentioned earlier. So, in this case, we're gonna look at an example where it calls the function can IO write output in order to flip a relay. Okay? Now, how does our use of LD preload change all this? Well, as before, the delta programming will execute, but now the
25:01
dynamic linker ro-loads our malicious binaries first, before any of the other shared objects. This becomes important when that background thread is running, because now it ro- it loads the uh, it calls the function with the same signature in our malicious binary instead of the real one, because those symbols are loaded first. Better yet, through the use of a C function called dlsim, we're actually get- able to get a handle to
25:24
the real function at run time. Which means we can sort of do a catch and release approach, where we intercept the calls to these IO control functions, mess with them however we want, and pass it back to the real deal with the program being none the wiser. Okay? So, this is all well and good. We had malware, we knew how to insert
25:42
it, but the question is, where the hell do we put it? Well, here are our options. Now, this diagram is a very high level overview of the startup routine for the delta programming. And through some reversing, we discovered that our best approach would be to use the function highlighted in yellow. Can IO init? The reason why is because as Doug
26:02
mentioned, or maybe not mentioned, I don't remember, is uh, that this programming is all vastly multi-threaded. So, in order to avoid race conditions or inconsistencies, we wanted a function that was called early in boot, called by a single thread and only called once. And this function met all those criteria. Now, what does it look like once the malware is inserted here? Well, this is what it looks like. When this
26:24
function is called, our malware spins up a thread that runs in the background, and it listens for attack, er, listens for commands set by the attacker over the network. Based on the content of these commands, the malware intercepts the corresponding IO control function in order to control the device the attacker wants to uh, influence. Now, I've
26:41
sort of been doing a lot of hand waving about how we actually altered these parameters in order to get it to do its job. So, why don't I dig into that a little bit? Now, although there's a whole array of IO control functions, they did share one key commonality that we were able to take advantage of. And that's the fact that the first parameter passed to them is a data structure that essentially just describes the hardware
27:01
being managed or being controlled. So, in this case, you see an output of it here through GDB and highlighted in red is the device's ID, the first 4 bytes. This is unique to each device. At a 12 byte offset highlighted in blue is the current state of the device, which is actually the most useful. In this case, we have uh, 75 degrees Fahrenheit as a floating point value. And then, at an uh, at an arbitrary offset, you
27:25
can also find a descriptive string of the device, which is also extremely useful because this is how we know this device in particular is responsible for monitoring room temperature. Using this information is actually pretty straightforward to just alter the state stored in these data structures, pass it along and then we have control of IO.
27:42
Ok, so now mission accomplished, right? We control the IO, we got root access. Well, the model is still no. The reason being is that our model here at ATR, unofficially, is to overdo everything. And in that spirit, we decided to pimp out our malware of all kinds of premium features. First of which, you get automatic discovery of all the IO
28:01
devices. Sit back and relax as the malware does all the recon for you. Next, you actually get remote control of the malware through an interactive TCP session. My therapist told me that the most important key to a healthy relationship is 2 way communication. Now, am I saying this malware will fix your marriage? Yes, that's exactly
28:21
what I'm saying. And next, you can see the state of all the hardware in real time. So you can see exactly how much damage you're causing. Who said happiness couldn't be quantified? And last, but certainly not least, with a single command, you can revert the device to its original, unhacked state, wiping it of any traces that the malware was ever there. And, if you call in the next 30 minutes, you can get all this and
28:43
more for just 30,000 easy payments of 19.95. Taxes and shipping not included. And if you're still not convinced, be sure to check us both out at ICS Village where you can see this bad boy in action. Now, we were getting pretty happy with our malware. Our baby
29:00
was looking pretty good. But we're still missing one key premium feature. And that's a remote attack vector. Now this looked pretty grim at first because after all, we were using broadcast traffic and that typically doesn't travel over the internet. But thankfully, a certain BACnet technology came to save the day. And that is BBMD, or BACnet IP Broadcast Management Device. A real tongue twister, but suffice it to say, it
29:25
allows for the transmission of BACnet broadcast traffic, exactly the kind we're using, imagine that, over, uh, the internet between different networks. So on the diagram on the right, you can see two such BACnet networks, each having a BBMD. The way this works is
29:41
that, uh, broadcast traffic intended for a foreign network is first sent to the source network's BBMD. This sends it to the destination network's BBMD and its job is to rebroadcast on the destination network. That's a bit of an aside, but how does this help us? Well, through testing, we found out that using this technology allows for our
30:01
exploit to work entirely remotely. Now let me repeat that. At the time, the slide was created, any EBMGR connected to the internet of its default network settings could be pwned 100% remotely using this exploit. This is all pretty scary stuff. But what's the actual impact? What can someone do with this? Well for that, I'm gonna hand
30:21
it back to Doug. Thanks. So, as Mark said, we confirmed that this would work 100% over the internet. Now I know everyone in the room will agree with me that no one would ever connect these systems to the internet. That would be completely unreasonable. Oh, that's
30:44
right. Shodan tells us a little bit differently. At, at the time that we created this slide, there was approximately 500 of these exact devices running firmware that was vulnerable to this connected worldwide. Because of the nature of this vulnerability and where it's found in, in the code, if we extrapolate out that other delta devices likely
31:03
have the same vulnerability, this number balloons up to 16 to 7 hundred teen, uh, I'm saying 17 hundred to 17 hundred devices, uh, give or take about 10% for known honeypots. So, as Mark said, in the spirit of overdoing things, we actually decided to build a fully functional HVAC unit controlled by a delta system, uh, in order to demonstrate the
31:24
effectiveness of this type of attack. So, if you'll watch on the video, as we move to the back of this demo unit, you'll see all the components you would see in a real working HVAC system. You'll have valves, pumps, fans, etcetera, and this was actually created by a contract company that actually puts these systems into current data centers. So
31:44
this is as accurate as we could possibly make it. The only thing that's not 100% accurate is we use cold water instead of a compressor, because my boss would not pay $8,000 for a compressor. On this side, we have an enclosed data, uh, miniature enclosed data center, again with all the normal HVAC components, a raised floor, and a server which
32:02
generates heat. For a moment, I want to draw your attention to something that was just on the screen, the independent temperature sensor. The reason this is important is it's the only component that you would not find in real life, and it's there to, we always know the true temperature in the data center, regardless of what the delta says. Then on top of the system, we've added some lights that just show the state of different
32:22
devices, and to indicate an alarm is active. Once again, the boss got annoyed at having a siren, so he forced us to turn it into a light. So, but this alarm would simply indicate whether an alarm's been triggered. This could be in the form of, uh, an audible siren, a light, or even email notifications. This just indicates whether an alarm has been sent or not. So, if this is the system, an HVAC system that is
32:45
cooling a data center, what would happen if we applied the research we did to this system? Let's take a look. Here on the, on the left hand side of the screen, you can see an attacker running the exploit script. In the top right is a picture in delta controller. The attacker leverages the vulnerability to download the premium
33:04
malware, which Mark was talking about, and a few other extra pictures, and the device reboots in order to implement the LD preload strategy, which Mark discussed. Now, when the device comes back up, we've been a little creative with the images to show that the device has been hacked, and we will also log into the system. I want to
33:21
emphasize that logging into the system is not part of the attack, we are simply demonstrating what the screen would look like and to follow along. It is not needed for the trigger, anything of the attack. So, something the attacker might want to do is control outputs on the device. This is, as I said, the device's function is to turn outputs on and off. So, what would it look like if the attacker started to modify
33:43
these outputs using the premium malware? You can see in the bottom right hand corner that the device will react accordingly. In the top right hand corner, the delta screen, you might take notice that it is not updating. This is because the attacker has the ability to choose to change or not change the screen. If they want the user to know that
34:02
something is changing, they can change it. If they don't, it won't, providing this attacker with a lot of different attack scenarios. Now, on these type of devices, there are multiple types of outputs. There are analog and binary outputs. What you just saw was binary outputs. We either turn it on or off. Analog outputs require
34:21
specific value, a floating point value, like Mark was mentioning, of room temperature earlier. So, here what we're gonna see the attacker do is modify one of these values and this time the attacker is going to choose to modify the delta screen as well, just proving that we can do it either way. So, we have the fan speed and we have the valve and you'll notice that once again that the components on the actual demo rig react
34:42
as you would expect. So, and also, as a premium feature, we built in a reset command, which sets everything back to its initial state, now, with the attacker still having persistence on the box. So, controlling outputs is kind of fun. We can think of scenarios that we may be able to influence by controlling each individual output. But if
35:05
you remember from the beginning of my presentation, the devices are designed to take data as input and react. So, what if the attacker was to modify the data the system was ingesting or the inputs? As you can see here, the attacker is going to change what the system is reading for temperature to third from uh, the real temperature to 30 and 40
35:24
degrees respectively. Now, what happens in your house if the HVAC system thinks it's 30 degrees in the room? Well, it's not gonna kick on, it's gonna turn off, right? And you can monitor our independent temperature system on the bottom right hand corner of the screen and the temperature in the data center is starting to rise. And there's no
35:40
alarms being triggered because the system thinks it's 30 degrees. Why would it trigger an alarm? There's nothing going, nothing that's going on wrong. So, this is just one small scenario of one purpose of these devices in HVAC system in a data center of the impact we might have. I don't know if anyone here manages a data center, but what would happen
36:00
if your data center went unattended with no HVAC for a long period of time? I cannot imagine that it would be a positive effect on the system. So, let's look at some of the other scenarios that are possible with control of an industrial control system like this. One of the interesting things about systems like this is they're used in a very wide variety of industries because they're used in commercial buildings. You're not gonna find
36:24
one at home, but you will find them in the different industries listed here. I'm gonna take a moment to talk about just a couple. I'm gonna look at healthcare, government and education. Excuse me. Healthcare is an interesting new case because it's usually used for something called pressure rooms. For those that don't understand what that means, in a
36:45
different rooms are kept at different pressure points in order to prevent disease from spreading. And it's not much, it's just a little bit different here and a little bit different there in order to protect things like the OR or quarantine areas. And if attacker was to get into one of these systems, which is, this is what it's used for,
37:01
and to just set all the pressures to the same value and chose not to update the screen and not to allow any alarms to send, nobody would notice, it's not something you're gonna feel going in a room that the pressure is different. All of a sudden those diseases that were contained by one of these systems could easily spread throughout the hospital. Might sound slightly dramatic, but it's definitely possible in this scenario. If
37:23
we move on to a different situation and think about government, through OSINT we were actually able to pull up documents openly available on the internet to see that a- these systems are used for access control in some state government buildings. I don't think I have to explain too in depth what the impact would be if an attacker was able to open
37:40
and close doors as they chose in certain, uh, pretty sensitive areas of buildings. And lastly, I want to talk about education just because I think it- it's fun when people ask me questions, why would I care if the- you changed the egg vac in my kid's room? Well, you probably don't care. But one of the other avenues of attack on these systems is they're an entry point into the network. If I have 100% control of one of these
38:03
devices and it's not isolated, which again is nothing we ever see in the real world, if- but if they're not isolated from other networks, this now becomes a point that we can leverage in order to get onto other systems. I don't know, maybe stream cameras from different rooms out to the attacker's terminal screen or other things like that, all of a sudden
38:22
you start to become slightly more concerned. So this is just an example of- of some uses in- in the industry. And because of this, and because that it's a true RCE over the internet, it was assigned a CVSS store of 9.8. Per McAfee's responsible disclosure policy, uh, once we understood what we have, we reached out to the vendor. This was in
38:42
late 2018. We assigned the CVE number and we started working very closely with the vendor. The vendor was extremely responsive and ends of June of 2019, they have released a patch which fully mitigates this vulnerability. Uh, ATR has also tested the patch and can confirm that it is in fact effective. I'd like to take just a moment to
39:02
highlight this because I think sometimes there's a problem in our industry with security researchers and vendors. Uh, Delta was extremely cooperative, it was wonderful to work with, and we were actually able to come up with a solution to help fix this problem. And I think this really needs to become the gold standard in the industry for security research. We need to do this type of research to find these type of bugs, and the
39:23
vendors need to be receptive, just like Delta was, in order for us to be able to fix these problems before the- before malicious attacks can be created. I hope we've demonstrated that the impact of these attacks can be pretty severe, and if they're not attended to, we can actually have a problem. So thank you Delta for your cooperation.
39:47
So with that said, that concludes the technical part of our presentation. I thank you very much all for coming. Uh, if you have any questions, as long as the goons allow us to stay up here, we'll answer them, or please see us in the back, I believe that's where they're doing questions. So thank you.