Modchips of the State
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 165 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39212 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
35C3 Refreshing Memories146 / 165
2
5
6
7
8
9
10
11
12
13
15
16
17
22
26
29
30
31
33
37
38
39
40
44
45
48
49
53
54
55
57
59
60
62
65
66
69
70
72
73
74
77
80
82
83
84
85
86
87
89
92
94
100
104
105
106
107
108
111
113
114
115
116
117
119
121
122
123
124
127
132
133
136
139
141
143
144
145
146
148
149
150
154
155
156
157
158
159
160
161
162
163
164
165
00:00
Inheritance (object-oriented programming)Proof theoryComputerMikrocomputerLevel (video gaming)Video gameFunctional (mathematics)Roundness (object)Computer hardwareMusical ensembleBeam compassGroup actionLecture/Conference
01:01
Sigma-algebraComputer hardwareMusical ensembleMultiplication signSource codeLecture/ConferenceComputer animation
01:32
Gamma functionBootingSystem programmingHorizonServer (computing)Sigma-algebraFirmwareSoftwareComputer hardwareLaptopDisk read-and-write headBootingMereologyPhysicalismProjective planeServer (computing)ConcentricComputer animationLecture/Conference
02:09
Hacker (term)Zoom lensSystem on a chipServer (computing)Component-based software engineeringFirmwareComputer hardwareMedical imagingWhiteboardEmailMereologyBus (computing)Information securityLecture/ConferenceComputer animation
02:40
Hash functionMemory managementChainComa BerenicesAddress spaceProcess (computing)Information securityChainFirmwareVirtual machineMoment (mathematics)QuicksortComputer animation
03:10
Hacker (term)Computer hardwareElectronic program guideMemory managementKey (cryptography)Multiplication signRootVirtual machineMereologyVolume (thermodynamics)Condition numberInternet service providerPCI ExpressHoaxCore dumpFlash memoryPower (physics)Physical systemWhiteboardData managementSoftware testingSemiconductor memoryWebsiteSoftwareExtreme programmingGame controllerServer (computing)Functional (mathematics)Serial portBus (computing)FirmwareAcoustic couplerComputer animation
05:00
PeripheralComputer programInformation securityData recoveryDisk read-and-write headInformation securityComputer animationLecture/Conference
05:34
WindowMotherboardServer (computing)Information securityQuery languageExplosionPurchasingWhiteboardInheritance (object-oriented programming)FirmwareData centerSet (mathematics)Computer animation
06:07
PermianElement (mathematics)Computer hardwareInformation securityElectric currentSet (mathematics)Operator (mathematics)Source codeText editorInformation securityAmenable groupComputer animation
07:01
Computer hardwareMotherboardBlogInformation securityRow (database)Identity managementComputer networkServer (computing)Inheritance (object-oriented programming)Design by contractChainElement (mathematics)Process (computing)Dean numberControl flowComputer hardwareChainFactory (trading post)Element (mathematics)Digital photographyProcess (computing)Inheritance (object-oriented programming)Computer animationSource code
08:08
Order (biology)Computer fileDrill commandsAuditory maskingProcess (computing)MathematicsTracing (software)Computer animation
08:38
MultiplicationFactory (trading post)Tracing (software)Physical systemConnected spaceSoftware testingProcess (computing)MathematicsProduct (business)Level (video gaming)
09:30
WebsiteLevel (video gaming)Function (mathematics)Data integrityProduct (business)Component-based software engineeringProcess (computing)Functional (mathematics)Visual systemOpticsRegular graphMotherboardDesign by contractMultiplication signMereologyDesign by contractWebsiteProduct (business)Line (geometry)Computer animation
09:58
SurfaceMereologyComputerBinary fileConnectivity (graph theory)Entropie <Informationstheorie>
10:27
Process (computing)Connectivity (graph theory)Product (business)Perspective (visual)CASE <Informatik>Constructor (object-oriented programming)MereologyInformation securityFunctional (mathematics)Different (Kate Ryan album)
11:18
Computer wormLipschitz-StetigkeitProcess (computing)Data integrityFunction (mathematics)WebsiteLevel (video gaming)Product (business)Component-based software engineeringVisual systemOpticsFunctional (mathematics)MotherboardRegular graphDesign by contractMiniDiscZoom lensDigital electronicsWhiteboardMotherboardSoftware bugDigital photographyConnectivity (graph theory)QuicksortComputer hardwareFactory (trading post)ChainInformation securityWhiteboardComputer animation
12:17
ChainBlogInformation securityExtension (kinesiology)Group actionQuicksortMathematical analysisType theoryChainWhiteboardInheritance (object-oriented programming)Condition numberConnectivity (graph theory)Computer animation
12:53
Computer hardwareMotherboardProcess (computing)Surjective functionComponent-based software engineeringAuthorizationFunctional (mathematics)WhiteboardFunction (mathematics)Kolmogorov complexityComplex (psychology)Installation artMereologyInformation securityProcess (computing)Computer animation
13:21
Inclusion mapOperations researchNon-standard analysisComputer-generated imageryWorkstation <Musikinstrument>GodProduct (business)Mach's principleDynamic random-access memoryFirewall (computing)User interfaceExecution unitOperator (mathematics)Information securityComputerIntercept theoremComputer hardwareInstallation artLibrary catalogServer (computing)Quicksort1 (number)Computer animation
14:05
Ideal (ethics)Covering spaceExecution unitWhiteboardComputer hardwareComponent-based software engineeringFunction (mathematics)Kolmogorov complexityMotherboardSurjective functionAuthorizationFunctional (mathematics)Process (computing)Complex (psychology)MultiplicationPhysical systemChainGoodness of fitFactory (trading post)MotherboardCovering spaceComputer animation
14:37
Component-based software engineeringSurjective functionFunctional (mathematics)MotherboardAuthorizationComputer hardwareWhiteboardKolmogorov complexityProcess (computing)Complex (psychology)Installation artData managementGame controllerPhysical systemFunction (mathematics)SoftwareFirmwareComplete metric spaceComputer hardwareComputer animation
15:08
Personal identification numberSymbol tableComputer hardwareFunction (mathematics)outputModel theorySerial portMultiplicationRead-only memoryBitFlash memoryConnectivity (graph theory)Function (mathematics)BefehlsprozessorInternetworkingFlash memorySerial portSeries (mathematics)Physical systemBitWhiteboardData miningPersonal identification numberComputer animation
15:51
Commodore VIC-20Data miningBitWhiteboardArmSpacetimeMoore's lawFitness functionBefehlsprozessor
16:22
Function (mathematics)outputSerial portMaizeWritingEmoticonAsynchronous Transfer ModeComputer hardwareBitMultiplicationSymbol tablePersonal identification numberFlash memoryEmailPhysical systemPower (physics)Data transmissionPersonal identification numberFunction (mathematics)outputAddress spaceSelectivity (electronic)Flash memoryComputer animation
17:06
Personal identification numberFlash memoryMultiplicationSerial portBitEmoticonoutputAsynchronous Transfer ModePower (physics)Function (mathematics)BitLimit (category theory)MathematicsPersonal identification numberComputer animation
17:37
Streaming mediaEstimationContent (media)Data streamGroup actionIncidence algebraEstimatorBitFlash memoryContent (media)Noise (electronics)CASE <Informatik>Computer animationDiagram
18:10
Kernel (computing)Hash functionMIDIPartition (number theory)Interior (topology)Kernel (computing)SpacetimeRootNoise (electronics)1 (number)BootingFile systemMathematicsMereologyBitAuditory maskingFlash memoryEmailSource code
19:02
Simultaneous localization and mappingSoftware protection dongleFunction (mathematics)Meta elementSyntaxbaumWindowClique-widthBuildingSoftware-defined radioComa BerenicesVideo game consoleComputer hardwareDemo (music)SoftwareComputer fileGastropod shellInheritance (object-oriented programming)Kernel (computing)WhiteboardConfiguration spaceDevice driverSpacetimeVideoconferencingPartition (number theory)Process (computing)Right angleFile systemScripting languageFlash memoryMusical ensembleSource code
20:55
Video game consoleSyntaxbaumGastropod shellOctahedronDemo (music)Computer hardwarePoint (geometry)Crash (computing)Revision controlVirtual machineMultiplication signRootGastropod shellPasswordBootingBitBus (computing)Source codeComputer animation
21:42
QuiltServer (computing)MotherboardSerial portMultitier architectureVideo game consoleCodeFigurate numberComputer programmingFlash memoryQuicksortComputer animation
22:13
Reading (process)1 (number)WebsiteBootingVirtual machineData streamLogikanalysatorFlash memoryLogicBus (computing)Task (computing)
22:55
FlagSequenceLengthPermianRegulärer Ausdruck <Textverarbeitung>Frame problemInternetworkingCommunications protocolSource codeStreaming mediaPrice indexMaxima and minimaFirmwarePhysical systemBackdoor (computing)Computer hardwareRootRead-only memoryBootingHypercubePhysical systemSemiconductor memoryPhysicalismPoint cloudCASE <Informatik>FirmwareSource codeComputer animation
23:30
Hill differential equationMemory managementQuarkLaptopPhysical systemPolar coordinate systemFirmwareQuarkInheritance (object-oriented programming)Computer animationProgram flowchart
24:03
FirmwareCodePhysical systemBitComputer hardwareDisk read-and-write headComputer animation
24:32
Field programmable gate arrayMiniDiscNP-hardRAIDDivision (mathematics)ComputerPower (physics)Data storage deviceBefehlsprozessorService (economics)Computing platformFirmwarePhysical systemNational Institute of Standards and TechnologyModule (mathematics)Computing platformRootGroup actionCore dumpMinimal surfaceComputer hardwareVideo GenieComputer animation
25:12
Extension (kinesiology)SoftwareTime zoneInformation securityGoogolKerberos <Kryptologie>Integrated development environmentCloud computingServer (computing)Right angleFirmwareAnnihilator (ring theory)Physical systemRootComputer hardwareOpen setStandard deviationComputer animation
26:03
WhiteboardInheritance (object-oriented programming)Computer hardwareOpen setMachine visionMotherboardServer (computing)Computer fileComputer animationEngineering drawing
26:31
Whiteboard1 (number)Menu (computing)FirmwareGoogolExploit (computer security)Information securityBootingConnectivity (graph theory)1 (number)Physical systemFirmwareOpen setBootingProjective planeEngineering drawingLecture/ConferenceComputer animation
27:04
BootingOpen setBuildingRootBootingSinc functionFirmwareOpen sourceSemiconductor memoryFormal languageVulnerability (computing)Source codeCollaborationismPhysical systemJSONXMLUMLLecture/Conference
27:38
Computer hardwareRootCodeVirtual machineGame controllerRootComputer hardwareOpen set
28:10
Game controllerFirmwareState of matterComputerPressureStandard deviationProjective planeBootingMusical ensembleLecture/ConferenceComputer animation
28:48
NumberLine (geometry)Internetworking3 (number)Lecture/Conference
29:31
Physical systemFirmwareOpen sourceBootingNumberBefehlsprozessorHand fanPower (physics)Programming paradigmComputer architectureLecture/Conference
30:23
Series (mathematics)Serial portWellenwiderstand <Strömungsmechanik>2 (number)Power (physics)MathematicsDuality (mathematics)MotherboardPersonal identification numberoutputSet (mathematics)Function (mathematics)Exploit (computer security)Different (Kate Ryan album)MereologyCASE <Informatik>Lecture/Conference
32:01
Parameter (computer programming)Product (business)MathematicsWhiteboardBitComputer fileFactory (trading post)Latent heatMereologyRandomizationConnectivity (graph theory)FamilyCASE <Informatik>Lecture/Conference
33:35
NumberMotherboardConnectivity (graph theory)WhiteboardError messageParallel portWeightLecture/Conference
34:12
System on a chipHill differential equationHacker (term)Trigonometric functionsSoftware bugPosition operatorWhiteboardPersonal identification numberFunction (mathematics)Flash memoryCircleShape (magazine)Digital photographySource codeComputer animation
34:55
CodeSerial portElectronic signatureFirmwareVideo game consoleSign (mathematics)NumberProcess (computing)File systemKernel (computing)RootBootingQuicksortPlastikkarteLecture/Conference
36:14
Semiconductor memoryCartesian closed categoryScalable Coherent InterfaceRoundness (object)Musical ensembleLecture/ConferenceDiagram
Transcript: English(auto-generated)
00:21
So, Trammel Hudson, who is standing here, he's taking things apart. Don't worry, not live on stage, but he will give us a proof of concept and some details and functionalities about hardware implants. So the same things that we heard from Bloomberg article talking about Apple and
00:43
super microcomputers with implants that were implanted into those computers. And I'm really excited to see this in action. Please give a warm round of applause to Trammel Hudson.
01:08
Before we begin talking about hardware implants, just two quick disclaimers. The first from my employer, Two Sigma Investments, as it says around chocolate bars, this is not investment advice. And secondly, I don't actually know what the story is behind the super micro story.
01:25
No one outside of Bloomberg and their sources do. But I have spent a lot of time thinking about hardware implants, starting with the Thunder Strike firmware attack against MacBooks, as well as the Thunder Strike 2, where we were able to get software
01:42
to write into the firmware on the MacBooks. I've also been thinking a lot about how to defend against hardware implants with things like the heads firmware for slightly more secure laptops. And also as part of my co-lead on the Linux boot project, we're thinking about how to protect servers from physical and software attacks.
02:07
So with all of this concentrated thinking about firmware and hardware attacks, I was really excited when I saw the Bloomberg story back in October. But what really intrigued me was the animated image that they had at the header
02:24
that highlighted one small part of the board as where the implant was. But what I found really interesting is that is exactly where I would install a hardware implant as they described on the spy bus. A lot of other people in the hardware and firmware security community
02:44
thought it sounded plausible. Other people pointed out that supply chain attacks come up periodically, and they are definitely a concern. Some people thought the attack, as described, was entirely implausible. And in general, we sort of had a Whiskey Tango Foxtrot moment
03:05
as everybody scrambled to figure out what's going on inside their machines. So let's step back very quickly and review what the key claims that Bloomberg alleged happened. First, they said that Amazon's testers found a tiny microchip
03:23
that wasn't part of the board's original design, that had been disguised to look like a signal conditioning coupler, and that these illicit chips were connected to the baseboard management controller, or the BMC, which gave them access to machines that were turned off.
03:43
That might sound kind of extreme, but that's actually what the role of the BMC is. That in most servers, the BMC is running any time the machine is hooked up to power, and it's connected to the power supplies so that it can turn the machine on and turn it off.
04:01
Frequently, you want to be able to do this over a network, so it has its own dedicated LAN port, but it can also share the LAN port with the main system. Serial over LAN is a really useful way to debug the systems, so it provides that functionality. It can also provide fake USB volumes to allow you to do unattended OS installation.
04:27
A lot of sites also want remote KVM, so it has VGA. But that VGA support means that it's on the PCIe bus, and because it's on PCIe, it can do DMA into main memory. It also is typically muxed into the SPI flash for the host firmware,
04:44
which allows it to modify it. And on some systems, it's even connected to the TPM, which allows it to circumvent the core root of trust. So with all of this capability inside this chip, it's really unfortunate that they are really not well put together.
05:06
The head of Azure Security says they have no protection against attacks, there's no ability to detect if an attack has happened, and there's no ability to recover from an attack. So having a hardware implant on the BMC is a really big concern.
05:26
The other claim in the article is that it affected 30 different companies, including Apple. And Bloomberg alleges that Apple found malicious chips independently on their Supermicro boards and went to the FBI about it,
05:43
and that they then severed ties with Supermicro. This particular claim was interesting because it corroborated a story that had shown up back in early 2017, that Apple had removed Supermicro from their data centers.
06:01
Apple denied that there was a firmware issue, but it's interesting that perhaps these two were related. The third set of claims is that on some of these implants, they were actually put between the layers on the PCB. And then the most explosive claim is that this was done by
06:22
operatives from the Chinese People's Liberation Army. With a story with this many claims and this significant of allegations, we'd hope that it'd be really well sourced.
06:41
And for a normal story, 17 independent sources that Bloomberg editors agreed to grant anonymity to, including six national security, two people inside of AWS, and three senior insiders at Apple, seems like pretty solid sourcing. Except as soon as this article was published, everyone denied it.
07:03
The Director of National Intelligence said they'd seen no evidence of this. Amazon said that they've never found any issues of modified hardware, nor have they been engaged with the government over it. Apple was even more blunt. CEO Tim Cook said,
07:21
this did not happen. There is no truth to this. And Supermicro wrote a fairly lengthy letter about what they do to protect their supply chain and why they think this attack did not happen. And it is worth going through to look at some of the things that they say that they do to protect their supply chain.
07:43
They point out that if there's any unauthorized physical alterations during the manufacturing process, other design elements would not match, and those things would be detected. To sort of understand how circuit boards are made,
08:01
I recently visited a PCB factory in Guangzhou. This is not a Supermicro factory. This is just holiday photos. So in order to add new vias, they would have to modify the drill files, which would then get electroplated. If they had to add new traces,
08:20
they would have to be able to subvert the masking and etching process. And any changes to either the drills or the etching on individual layers would be caught by the optical inspection that's done on those bare circuit boards. Additionally, the allegation that things were inserted between circuit boards
08:44
would require that the lamination process be subverted and the implant somehow aligned into the system. If that implant changes any of the connectivity, the flying probe testers would pick it up, or the bed-of-nails testers,
09:01
which checks all of the connectivity of all the traces to make sure that there are no shorts and to make sure that everything that is supposed to be connected is electrically conductive. So it would be very difficult to circumvent the production process at this stage,
09:21
and it also would be very difficult to contain because the PCB factory doesn't know which customers are going to receive those circuit boards. Supermicro also points out that during the assembly process, when the parts are installed, they have their employees on site the whole time.
09:42
On my same holiday trip, I also visited some PCB assembly companies and spoke with companies that are doing contract manufacturing. And they said that they also send their employees to the production line to observe the pick-and-place machines and the reflow
10:01
and the rest of the surface mount assembly. Their big concern is that if they don't have someone there, the parts that are fed into the pick-and-place will be replaced with either counterfeits or with salvaged parts. I visited the electronics market in Washan Bay
10:21
where there are people desoldering e-waste and then sorting the parts into bins and selling these salvaged components by the kilo. And for a few extra from NIMBY, they'll put them on reels for you so that you can save a few pennies on your production process.
10:40
The other concern that these companies have is not just salvaged parts, but straight-up counterfeits, especially for things that cost more than a few dollars each. The Arduino community was hit a few years ago with a bunch of counterfeit FTDI chips where the internal construction was entirely different.
11:02
In this case, it caused reliability issues, but you can imagine from a security perspective, this is really worrisome that parts that look identical might have completely different functionality inside of them. Supermicro also mentions that they x-ray their mainboards
11:24
to look for anomalies. And I wasn't able to take any photos inside a factory that was doing x-rays, but in this Wikipedia photo, we can clearly see active components like this SOIC chip are different from things like the SMD resistors and capacitors.
11:45
So if an attacker were trying to subvert the supply chain by putting a disguised component, it could be detected at this step. Another interesting thing in this photo are these inductors that are encased in DIP packages.
12:01
This is really common in a lot of Ethernet boards, and occasionally people have thought they had some sort of hardware implant when they found inductors in their Ethernet jacks. But it's fairly common, and it shows it pretty clearly on the x-ray. Some other security researchers like Sofia De Antoine
12:23
did an extensive teardown of supermicro boards, including x-ray analysis. And her group found a few oddities, but they didn't find anything malicious. There were no smoking guns. They just appeared to be sort of supply chain-type things.
12:41
You can read her blog post for more details about where they found things that shouldn't have been there, but turned out to be just actual signal conditioning components. So in supermicro's letter,
13:01
they keep reinforcing the manufacturing process, that is, the assembly process. It's during the manufacturing process. And I agree with them. It would be very difficult to circumvent their security in a reasonable way in that part of the process. But that's not the only place this could happen.
13:22
We know that national security agencies intercept shipments of computer hardware and then have their tailored access operations open the computers, install hardware implants, reseal them, and then have them continue on their way in shipment.
13:42
The NSA even has a catalog of hardware implants like this JTAG implant, Ethernet jacks with embedded computers in them, as well as firmware-specific ones that target servers, SMM, and then some that can do data exfiltration via RF.
14:04
So that sort of tailored access operation is really ideal for this supply chain attack because it allows them to contain the exploit to a single customer. It allows them fairly good concealment as well as good cover that, if it's discovered,
14:21
it's really hard to attribute where things went wrong. Unlike if you find something inside your motherboard between the layers, you know that had to have happened at the factory. So Supermicro also claimed
14:40
that this was technically implausible, that it was highly unlikely that unauthorized hardware would function properly because a third party would lack complete knowledge of the design. I think that's inaccurate,
15:02
both because we know the NSA does it and also because I've done it. Really, all that you need to know is that these are common components. These flash chips show up on all the boards. You can search the internet for the data sheet and find exactly how it's wired
15:23
into the rest of the system. And the only thing that we need to know to communicate to the BMC is the serial output pin from this component. So the BMC Flash is connected over to the BMC CPU
15:41
via the serial output and it goes through a small series resistor and that is where my implant goes in. Mine's a little bit larger than that resistor. It clips onto the board and it has a small FPGA that hangs offside. But it's completely plausible
16:02
to fit it into something that small. In fact, a modern ARM M0 fits in the space of two transistors from a 6502 from a few years ago. The Moore's Law means we can pack an amazing amount of CPU into a very, very small amount of space.
16:24
So that 0603 resistor could fit around 100 Cortex-M0s which would be plenty powerful for this system. The problem is we only have those two pins.
16:40
So ordinarily on the SPY Flash you need at least six pins but we don't have power and ground so we have to passively power this through the data signal that's passing through it. We don't have the chip select pin so we have to guess when this chip is being talked to. We don't have the data input pin
17:02
so we don't know what addresses are being read or what commands are being sent. We have to reconstruct it from the data output pin. And we also don't have a clock pin so we have to figure out how to synchronize to that clock. Lastly, we don't have the ability
17:21
to make arbitrary data changes. All we can do is disconnect the pin from the BMC so we can only turn one bit into zero bits. We can't go the other way around. So with these limitations we can still do some pretty interesting things. Recovering the clock is actually pretty easy.
17:40
We can look at the data stream and find the shortest bit transitions from 010 or 101 to estimate what the clock is which allows us to then reconstruct that data stream being sent to the BMC. If we look at the flash contents we can see that a lot of it is fairly random noise
18:04
but a lot of it is all white which in this case would mean that it's all one bits. If we look at the way the flash is organized we can see there's the U-Boot bootloader and that's executable that's kind of difficult to make useful changes in.
18:22
The kernel and the root file system are both compressed so that they look effectively like random noise. But the NVRAM region is a JFFS2 file system and this file system is three megs
18:40
it's mostly empty and all that empty space is FF which is all ones so this is plenty of ones for us to work on. Additionally it has fairly nice headers that we can match on so when we see these magic bit masks we know when we've entered different parts of the file system.
19:03
Given that we can now reconstruct the clock we can figure out where we are in the file system this hardware implant can start to inject new data into what was the empty space. The short file that we put in here is a small shell script
19:22
and it is one of the network configuration scripts. This is where I'm going to try a live demo and I hope this works. We're running in QEMU since I didn't bring a super micro board and what we have on the left
19:43
is the hardware implant console and then on the right we have the serial console from the BMC. So we can see it has loaded the kernel and then in a second we should see a bunch of traffic.
20:01
So the implant is active it has replaced the data when that NVRAM file system was mounted. The BMC is now continuing on doing its setup it's going to load a bunch of device drivers for the video it pauses here for some reason
20:21
that I haven't diagnosed because that's not my job and eventually it's going to configure the networks and it does that by running that shell script off of the NVRAM partition. It starts the KVM stuff
20:42
brings up some things all right! Okay, so luckily we got to that point without me having to fake the demo.
21:01
In the hardware it's really flaky my version works about one and eight times but it doesn't typically cause a crash so that's actually good for concealment because it becomes now much harder to determine which machines are affected. In QEMU because it's emulating it's a little more reliable but still it's only two out of three.
21:23
If we let the BMC boot a little bit further it actually prints out this message and if you hit enter it drops you to a shell with no password and you can then just run commands as root on the BMC and that's a lot easier than all this stuff with the spy bus if you wanted to build a hardware implant against it.
21:43
I don't know where the serial port is on the Supermicro but on a different tier one server main board I was able to program with the oscilloscope and locate the serial console for the BMC figure out it's 115 kilobaud and it has the same code that you hit enter
22:01
and you can run commands there. So that's a much easier way to do it. A big question a lot of people have is how would we actually detect this sort of flash implant? A lot of high assurance sites replace all of their ROMs with ones that they flash themselves
22:20
but that doesn't get rid of the implant because it's outside of the ROM chip. Likewise reading the ROM chip doesn't show anything because it's not in the ROM itself it's outside of it. Even hooking up a logic analyzer to the bus and watching as the machine boots and seeing the data stream coming out of the flash
22:42
won't actually reveal the implant because you would have to put the logic probes on the BGA pads on the BMC itself and that's a much harder task. Some people think oh well we can see the weird network traffic when the BMC tries to exfiltrate the data
23:01
but that's only one way for the BMC to affect things. There was a great talk a few years ago at Defcon from Intel ATR where they showed how something that can control the system firmware can back door hypervisors and then they gave a use case where a unprivileged guest on a cloud system
23:22
could read all of the rest of physical memory so it could see all of the other guest memory. So what do we do? You know the big problem is the BMC has way too many privileges. It's connected to pretty much everything in the system
23:41
but the BMC is not our only concern. As White Pork said, our PCs are just a bunch of embedded devices in a trench coat and they all have firmware. In fact pretty much everything on your system more complex than a resistor probably has firmware and if you have one of those super micro implants
24:01
maybe even your resistors have firmware as well. I've found that the firmware and things like the power supplies can be used to gain code execution on the BMC. You know it's really interesting how tightly connected all of our systems are and as Joe Fitz pointed out in his Black Hat EU talk
24:23
these are not multi-million dollar attacks that these are five euro bits of hardware that we now have to really be worried about. I really like the guidelines that NIST has published that suggest that we think about our systems more in this holistic manner
24:42
although they end up rooting pretty much everything into the TPM as the trusted platform module for doing this attestation and I think we as a community need to do more to use the TPMs. They're actually a really good tool for securing our systems but they are also potentially subject
25:01
to their own hardware implants. The NCC group TPM Genie is able to subvert the core root of trust by interposing on the TPM. So a lot of folks are proposing we should move to other trusted execution environments like SGX or TrustZone and I think these have a lot of promise
25:22
especially for trusted cloud computing. There's a lot of innovation in hardware roots of trust going on right now between the Google Titan which initially was for their servers and is now showing up in all of their Chromebooks. The Microsoft Cerberus chip which again is the Azure system.
25:42
They're actually publishing their firmware and the ASIC design so that people can have a little more faith in it and they hope it will become an open standard. And companies like Apple have also gone their own way with the T2 and the T2 is a really amazing chip for securing systems but it does so at the expense of user freedom.
26:04
And that gets in the way of what I think the real way that we need to solve this problem is we need to get rid of a lot of these secrets. Counter to what the super micro CEO said having a secret motherboard design does not make you more secure.
26:22
Things like the open compute hardware I think is a good vision for how we can move forward. That when you buy an open compute server it comes with full schematics and Gerber files so that motivated customers can verify that the systems that they're buying are the ones that they think they're buying.
26:41
That all of the components are what they think they should be. I think the firmware also needs more openness. Ron Minick at Google is my co-lead on Linux Boot Project and we think that Linux in the firmware is a way forward to get a more secure, more flexible, and more resilient system.
27:03
We're working with a spin-off project called MicroBMC that is using the Linux Boot tools to build BMC firmware. And this is open source. It's reproducibly built. It can work with roots of trust for attestation. It's written in a memory safe language
27:22
since it's a Google collaboration, it's in Go. And more importantly, we've thrown away all of the legacy features that have been a source of a lot of security vulnerabilities in these systems. So did it happen? I don't know. Is it technically possible?
27:42
I think so. I hope I've convinced all of you that this is definitely a technical possibility that we need to be concerned about. And I hope that the way forward through hardware roots of trust with attestation and more importantly, with open hardware so that we know that the machines we're running
28:01
are running code that we know, the code that we built that we understand, and that we can actually have a good chance of being able to take control back of them. If you're interested in more discussion on this and also in open firmware, there's an assembly here in this hall
28:21
that has a bunch of folks working on Core Boot and Linux Boot and a lot of these projects where you can help contribute and you can help also pressure vendors to make this a standard and a way forward for a more secure computing. So thank you all for coming.
28:41
And I really enjoyed the chance to show off my mantra for the state. Great talk. Thank you very much, Trammell.
29:01
We have 10 minutes for questions. So please line up with the microphones if you have questions. And we also have the signalation probably with questions from the internet. So any questions?
29:25
Microphone number three. Yes, I was going to ask what's your opinion on the Talos systems, the open PPC-based ones? So the question is about the Talos Power9-based systems.
29:40
Power9 is a really interesting architecture. It is using an open firmware very similar to Linux Boot called a Petiboot that moves Linux into the bootloader. I'm a big fan. There's a lot of folks in the open source community who are very excited about it.
30:01
I'm hoping that there would be more Power9 systems coming out. I'm also very excited about the RISC-V systems. I think having open source CPUs is a real way that we can have more assurance that our systems are what we think they are.
30:22
Thank you. Microphone number two, please. I was wondering, if you just have a scope probe over this serial, because it's just a serial resistor which you were replacing, if you just put two scope probes on there and measure the voltage over it, in your situation, you would see a voltage change there once in a while.
30:41
Yes, yes, yes. While in a normal case, you would see actually quite constant current. If you lower the input impedance of the BMC chip, you might already have fixed part of the attack because the output sourcing current of your exploits is probably limited
31:02
due to the limited supply. Do you see a way to actually get more power into your setup, maybe using other power sources, other than the two pins, or maybe there's some way of...
31:21
So the question is, would there be a way to do more arbitrary changes through redesigning the implant? One of the goals was to fit with only those two pins so that a single piece on the motherboard could be replaced. With a dual probe soldering iron, you can pop it out and stick a new one down
31:40
in a matter of seconds. So yes, if you have more pins where you can get more power from, you can do much more interesting things. But that would require a different set of changes to the motherboard.
32:01
Thank you. Microphone 1, please. So a lot of the arguments that these implants were not feasible by a separate microwave will also show the pictures from the FAB that you have to change the etching and the optical inspection, and so on and so on. But how probable would you rate the fact
32:23
that some actor just intercepted the manufacturing files and added that component already in the file? Because then all the optical inspection and that would all say, well, that matches what was sent to us, but that is not necessarily what that micro sent to the FAB.
32:42
So the question is, could someone have modified all of the manufacturing files that went to the factory? And that's absolutely a possibility. But that's also very likely that that would be detected by Supermicro itself. That in a lot of cases, you don't necessarily want to trust the company that is making the product
33:02
to also test it. You probably want to have a separate company that does random spot checks to verify that the boards are actually being produced to the specification that you desire. So it's certainly possible. And I really don't want to speculate
33:21
as to the accuracy of that part of the story. But yeah, it would require quite a bit more changes and also would be much more likely to be detected in the spot check. Great, microphone number two, please. Yeah, so for a lot of motherboards,
33:42
there are also quite a few components not populated, some of which are on, which you could consider sensitive nets. Wouldn't that make it, yeah, exactly this. Wouldn't that make it very easy to just pop something on there
34:00
in parallel with one of the components and not have it be detected because the board isn't modified. There is a component there. You have no way of telling whether it had to be populated or not. Supermicro puts a lot of extra pads on the board. In this one, particular one, they have both 8-pin and 16-pin flash chip pads
34:25
that are just in parallel together. So depending on which chip is cheaper that day of the week or who knows what, they will populate one or the other. So that's why in this particular photo, having the position of that circle
34:45
on the data output pin is very, very interesting. Question answered. Okay, so one more question, microphone number two, please. How far can a signing of Fyme
35:00
be a solution to this problem? Signing firmware solves a lot of the issues. It does, however, typically not all of the firmware is signed. Specifically, U-Boot is probably going to be signed in a modern BMC.
35:20
The kernel and maybe the root file system might be signed. But the NVRAM file system in this BMC is designed to be user modifiable. So it can't be signed by the manufacturer. So this sort of attack would work against a signed BMC just as well. Also, the hit enter to get a serial console attack
35:45
circumvents any signing. There are things on the host firmware on the x86 like boot card that do a really good job of making it harder to get code execution during the boot process.
36:00
But there have been several CVEs where it has been implemented poorly. So even though signatures, the firmware is signed, people have still managed to get code execution during that process. Great, thank you, Trammel Hudson. Again, a warm round of applause. Thank you very much.