Pentesting the Smart Grid
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 122 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40585 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DEF CON 1960 / 122
3
5
10
11
12
22
23
24
30
31
32
38
43
46
47
49
51
54
56
59
60
62
71
73
76
84
85
88
92
93
96
97
98
104
106
109
112
113
115
119
00:00
CollaborationismFunction (mathematics)Information securityArchitecturePenetrationstestService (economics)Electronic mailing listStandard deviationWechselseitige InformationPlastikkarteGroup actionLocal GroupSelf-organizationExplosionCybersexGradientAmicable numbersHypermediaMessage passingSystem programmingMetreRepresentation (politics)Computer hardwareAsynchronous Transfer ModeProduct (business)Data modelACIDPower (physics)Utility softwareOperator (mathematics)InformationDiagramInformation securityType theoryFood energyProcess (computing)Right angleDifferent (Kate Ryan album)Game controllerRevision controlContext awarenessView (database)Line (geometry)Multiplication signTelecommunicationElement (mathematics)InternetworkingComputer architecturePresentation of a groupDomain name1 (number)Service (economics)DataflowInternet service providerMoment <Mathematik>Self-organizationFlow separationDistribution (mathematics)Workstation <Musikinstrument>Point (geometry)Neighbourhood (graph theory)Slide ruleTurbo-CodeGroup actionElectric generatorNumberPhysical system2 (number)Content (media)Term (mathematics)Arithmetic meanEvent horizonArrow of timeAdditionOffice suiteMereologyAreaCuboidPole (complex analysis)Field (computer science)State of matterGreatest elementBitSpecial unitary groupWordDrop (liquid)Standard deviationSoftware testingPoint cloudGoodness of fitWebsiteSet (mathematics)PlastikkarteGreen's functionData management
08:16
Distribution (mathematics)Electric generatorInformation securityDistribution (mathematics)Traffic reportingServer (computing)Field (computer science)Link (knot theory)Endliche ModelltheorieInformationNational Institute of Standards and TechnologyType theoryTelecommunicationLengthCuboidWeb pageCommunications protocolEncryptionComputer architectureMereologyPower (physics)Goodness of fitLine (geometry)Volume (thermodynamics)Level (video gaming)Game controllerUtility softwareSheaf (mathematics)Right angleDomain nameIntegrated development environmentState of matterSelf-organizationDifferent (Kate Ryan album)SineMatrix (mathematics)WaveCausalitySingle-precision floating-point formatTransport Layer SecurityBitStreaming mediaDialectCoordinate systemOperator (mathematics)Band matrixLimit (category theory)DiagramAverageMathematicsMetre2 (number)MeasurementNumberFlow separationReading (process)ElectronvoltControl flowStrategy gameMultiplication signReal numberTrailPoint (geometry)FamilyPhase angleSerial portDecision theoryAreaSerial communicationFront and back endsConnectivity (graph theory)CASE <Informatik>Module (mathematics)SynchronizationInterface (computing)Workstation <Musikinstrument>Logic gatePhysical systemPhysical lawGroup actionMaxima and minimaDistanceCircleModal logicComputer animation
16:32
Wechselseitige InformationDiagramComputer networkPhysical systemTelecommunicationFrequencyMetre1 (number)Interface (computing)Communications protocolNeighbourhood (graph theory)Firewall (computing)Field (computer science)Different (Kate Ryan album)Drop (liquid)Type theoryGame controllerPower (physics)InformationNumberPoint (geometry)BitLine (geometry)Disk read-and-write headReading (process)In-System-ProgrammierungConnected spaceRouter (computing)Raw image formatMereologyAreaTable (information)DivisorPolygon meshLink (knot theory)Band matrixDistanceTowerCASE <Informatik>Single-precision floating-point formatRight angleOperator (mathematics)Combinational logicPerimeterServer (computing)Standard deviationGeneric programmingUtility softwareLatent heatDiagramImplementationWhiteboardComputer fileConfiguration spacePattern languageAlgorithmData streamWeb serviceTerm (mathematics)Vermaschtes NetzModule (mathematics)Front and back endsModemPunched cardAssembly languageBroadcasting (networking)Endliche ModelltheorieProduct (business)SoftwareSpectrum (functional analysis)View (database)Wireless LANDirection (geometry)NP-hardGoogolMultiplication signData management2 (number)TwitterOnline help
24:48
Wechselseitige InformationDiagramChemical equationExecution unitComputer networkUtility softwareSurfaceComputer hardwareServer (computing)Workstation <Musikinstrument>InternetworkingFunction (mathematics)Web browserSystem programmingMalwareSoftwareInformation securityBoundary value problemMechanism designClient (computing)Type theoryInterface (computing)Data managementWebsitePhysical systemWeb portalInformation securityGame controllerInterface (computing)Power (physics)Different (Kate Ryan album)Execution unitUtility softwareComputer wormMetreRight angleComputer hardwareEndliche ModelltheorieIncidence algebraType theoryCombinational logicMultiplication signProcess (computing)TelecommunicationComputer architectureSoftwareDependent and independent variablesWeb serviceAngle of attackBitUser interfaceVermaschtes NetzBuffer overflowFront and back endsBlock (periodic table)1 (number)KreisprozessLastprofilSoftware testingAdditionDisk read-and-write headOperator (mathematics)Vector spaceField (computer science)outputServer (computing)Vulnerability (computing)Workstation <Musikinstrument>Self-organizationPolygon meshProof theoryService (economics)InternetworkingPhysical systemModule (mathematics)Neighbourhood (graph theory)Slide ruleWeb 2.0Discounts and allowancesMereologyInformationRemote procedure callStability theoryPerspective (visual)CodeGoodness of fitCommunications protocolCross-site scriptingCycle (graph theory)SurfaceClient (computing)Radical (chemistry)Line (geometry)Serial portDirection (geometry)AreaConnected spaceElectric generatorDiagramProduct (business)Computer programmingSingle-precision floating-point formatTable (information)Message passingFormal languageOrder (biology)Instance (computer science)Link (knot theory)Data structureFitness functionException handlingComputer fontStandard deviationWebsiteCommutatorFunctional (mathematics)
33:04
Server (computing)Information securityCommunications protocolStandard deviationPerspective (visual)Source codeHacker (term)Sparse matrixSoftwareComputer networkVisualization (computer graphics)Mathematical analysisScripting languageCryptographyWireless LANSystem programmingAsynchronous Transfer ModeEncryptionData recoveryInjektivitätFrequencyAlgorithmStack (abstract data type)HistogramInclusion mapRAIDStream cipherData integrityBlockchiffreAdvanced Encryption StandardCollisionRotationDerivation (linguistics)Äquivalenzprinzip <Physik>ImplementationPhysical systemData modelInformation technology consultingExpert systemComputer hardwareClient (computing)MetrePole (complex analysis)Vulnerability (computing)FirmwarePerimeterReflektor <Informatik>Mechanism designProxy serverWellenwiderstand <Strömungsmechanik>Level (video gaming)outputMeasurementFood energyUtility softwareDependent and independent variablesPersonal digital assistantComponent-based software engineeringVibrationDigital electronicsPower (physics)WhiteboardEEPROMContent (media)Key (cryptography)BootingOperations researchProcess (computing)Euclidean vectorGame controllerFlash memoryPoint (geometry)Computer hardwareComputer architectureRight angleRepresentation (politics)Cycle (graph theory)MetreBus (computing)Entropie <Informationstheorie>Communications protocolEmailMicroprocessorAsynchronous Transfer ModeMaxima and minimaType theoryProduct (business)Reverse engineeringComputer wormWeb 2.0HistogramSound effectDifferent (Kate Ryan album)Key (cryptography)TelecommunicationWireless LANMultiplicationProxy serverGreatest elementGraph (mathematics)NumberAsymmetryVulnerability (computing)Computer configurationInformationEncryptionDisk read-and-write headPerimeterCryptographyCuboidGame controllerInformation securityHydraulic jumpChemical equationOrder (biology)Serial communicationFrequencySchlüsselverteilungFlash memoryPublic-key cryptographySymmetric-key algorithmCASE <Informatik>PasswordRoundness (object)Multiplication signPower (physics)Field (computer science)Dot productData storage devicePhysical systemNeighbourhood (graph theory)Interface (computing)SoftwareAlgorithmPolygon meshMechanism designMereologySoftware testingNeuroinformatikSingle-precision floating-point formatFood energyException handlingPhysicalismIdentifiabilityWhiteboardTerm (mathematics)EEPROMTelepräsenzWordEndliche ModelltheorieRemote procedure callDistribution (mathematics)Student's t-testPairwise comparisonMedianComputer animation
41:20
EEPROMCore dumpComputer networkData recoveryRadiusKey (cryptography)Symmetric matrixBinary fileFlash memoryRandom numberEncryptionStrategy gamePhysical systemSystem on a chipLimit (category theory)Group actionDistribution (mathematics)Uniqueness quantificationMetreProof theoryComputer hardwareMechanism designData storage deviceMinimal surfaceImplementationFirmwareRotationInformation securityElectric currentPlastikkarteWechselseitige InformationLocal GroupSoftware testingInformationSurfacePersonal identification numberDifferent (Kate Ryan album)Connected spaceFlash memorySymmetric-key algorithmComputer hardwareUtility softwareKey (cryptography)EEPROMProduct (business)Content (media)InformationGroup actionCore dumpCombinational logicGoodness of fitFitness functionSpacetimeRight anglePlastikkarteNational Institute of Standards and TechnologyTraffic reportingMathematical analysisInformation securitySource codeService (economics)Communications protocolSemiconductor memoryBinary codeAsymmetryTouchscreenSystem on a chipComputer configurationSlide ruleNeuroinformatikTelecommunicationWebsiteBitMicroprocessorUniform resource locatorModule (mathematics)Bus (computing)Hydraulic jumpCompilation albumFirmwarePublic-key cryptographyBlu-ray DiscMotion captureMultiplication signDomain nameLengthMultilaterationGreatest elementElectric generatorComputer animationLecture/Conference
Transcript: English(auto-generated)
00:00
Today, we have on behalf of or from Utilisec, Mr. Justin Cyril, who has spoke at a number of different conferences, including Black Hat and Def Con. Let's give him a big hand and welcome him today. All right. How's everybody doing today? Good.
00:20
All right. Good to hear. Good to hear. Okay. So at Black Hat, I did a four-hour presentation, a four-hour workshop on this material. You guys get the condensed version of it, but I still have a lot of slides. So we are going to go through the slides. We're not going to hit all the points on the slides, but we are going to have them uploaded to the Def Con site, so you do have availability to all the content. Okay. So first off, I am a managing partner for an organization called Utilisec, and we specialize
00:44
in security services for electric utility companies. We provide penetration testing services, secure architecture design, and other types of services, including trying to represent the utilities and their interest in a lot of the standards that are out there, trying to build in security and some of the standards.
01:03
In fact, some of the groups that we have worked in, we currently lead and facilitate one of our managing partners leads and facilitates the NERC-CIPs, that's Joe Bucciero. We also lead many of the different groups that the electric utilities have put together
01:21
to try to build security or try to generate industry awareness in security issues that we have inside of the smart grid. So the purpose of this talk, a lot of times we hear a lot of different talks, Black Hat, Def Con, SchmooCon, TorCon, of different types of things in the smart grid. We have a lot of good research going on, a lot of good researchers that are doing
01:42
a lot of good in this industry. We also have a lot of talks that are smart grid talks that are simply there to try to generate a lot of the hype. So the purpose of this talk is really to try to give everybody a very clear picture of what the smart grid is, give you an idea of some of the issues that we're dealing with, some of the different attacks that we're seeing and some of the different attacks that we perform as penetration testers on the smart grid, and more importantly,
02:03
show you that the smart grid is more than just SCADA. The smart grid is more than just smart meters. There's a lot more out there to do. And a lot of the people inside of this audience that are security professionals, your skill sets are very applicable in many areas of the smart grid. You just may not know where to look and where to try to generate that business.
02:24
And ultimately, one of the ultimate goals of this talk is to really try to generate more awareness, more interest in the security community to try to get more researchers and more people in this field because the smart grid is not perfect. Name a single vertical industry that is secure. We need to get more security. We need more people in this. We need more expertise.
02:41
We need more technical expertise in particular. So really, that's some of the goals. So the first half of the presentation, we're going to be focusing on trying to let you guys understand a little bit more about the smart grid architecture. And then the second half, we're going to be talking about some of the different penetration attacks and the defenses that we're recommending and working with vendors and utilities to try to address these issues and mitigate them.
03:03
Okay, so first off, what is the smart grid? The smart grid, anytime you hear the term smart grid, this is something very similar to hearing the word internet. It is a marketing term. It can mean anything and everything you possibly ever want. Ultimately, what the goal is of the smart grid is to try to take our existing infrastructure and add additional intelligence to it, add capabilities where in the past,
03:25
we had to have people sitting in the control rooms looking at different sensors coming back and having them sit and toggle the different switches, toggle the different controls to cause reactions in the grid. With this smart grid, we're trying to add more infrastructure to be able to give us a better view of what's really happening in the grid,
03:42
a better view of what's happening at the homes of each one of you consuming power instead of being able to see once a month how much consumption you have, be able to see within a 15-minute interval of how much energy you're consuming. Hopefully, this is going to be something to benefit the rest of the community as well. I don't know about you, but I personally want to know exactly how much power I'm using
04:00
in 15-minute intervals in my house because I can do a lot of really cool things for my own self for that. Of course, attackers can do a lot of cool things with that as well, but ultimately, that's the goal. We try to do the same exact thing with the technologies out in the substations themselves. If you look at this diagram, this diagram goes through and shows you the different elements or different major domains in the smart grid.
04:22
We see the ones across the top. We have the markets. We have the operations and the service providers. These are a lot of the organizations and the companies that are kind of the glue holding the different processes together. Then, if you look on the very bottom, we see that dotted yellow line across those four clouds on the bottom. This is the dotted yellow line represents the energy that's flowing
04:42
from the bulk generation plants to our homes. All the blue lines are communication lines. We have a lot of different types of communications between these different entities and these different domains. This is basically the same exact diagram. This is showing you just more information and more of the devices.
05:00
These are the actual systems that you're going to see inside a lot of the utility companies that are out there in their back offices. This is all color-coded. If you look at the yellow part, this is the operations. This is what most of your electric utility companies are doing. Each one of those boxes are a lot of the main control centers. These are some of the major systems that they have to control your power and to monitor your power.
05:22
If we look in the... Actually, let me go back just one slide and show you one thing. Bulk generation on the very bottom. Bulk generation, I think that's pretty obvious. These are going to be the power plants, nuclear power plants, the coal plants that are generating the power for us. That power flows over to a group of organizations called transmission operators.
05:40
These transmission operators are what take this power from the bulk generators down to the distribution operators, which are more the companies that we think of as electric utility companies because they're the ones that we're buying our power from and that power flows back into our home. The transmission operators and distribution operators, for the most part, they're very distinct. There are several different organizations
06:00
and utility companies that act as bulk. When we look at this diagram, we see the transmission operators in the upper left-hand corner. These are the transmission field devices, the devices that are on those big giant steel poles with the big power lines that we see crossing state borders. My wife and I are falconers and we always call these the steles
06:21
because when we're looking for falcons to trap, some of your bigger falcons like the jeers and your peregrine falcons like to sit up there in the morning and catch the sun as it's rising. So that's how we find a lot of those birds. So those are the transmission operators and their field devices out in the field. The upper right-hand corner,
06:40
that's the distribution field devices. These are the devices that are put out in the field to control the power that's ultimately flowing down into each one of our houses. That's primarily done through what we call substations. Those are those big things that have the big fences around them, the security cameras, all the barbed wire. So those are substations. The transmission operator will take this bulk power
07:01
and they'll drop it down to the distribution operators usually in one of these large substations and then from there the power is distributed out to the smaller substations that are closer to our neighborhoods and each of our neighborhoods then are connected back to those connected back to those those substations to be able to pull the power. We have different types of devices like feeder switches
07:22
that allow the utility company to control which substation one single neighborhood is connected to. So if you ever have a certain circumstance where your power goes out and the power is only out for five to seven seconds we're going to have feeder switches and relays that are going to automatically be connecting us back over
07:41
to another, a different power source and so that's why you drop out just for a couple seconds and you come right back on because they've had some automated events switch you back over to a different power source to try to avoid the power issues that you're experiencing. Now if you look on the very bottom the things that are probably more interesting to a lot of you are going to be the devices in our home and the lines and the communications between our home
08:01
and the electric utility companies. That's the bottom right hand corner with all the green. So these are the smart meters that are inside of our inside of our homes and the other devices that we ourselves bring some of our home automation devices those that have electric vehicles. How many people have electric vehicles in this room? Out of curiosity. Now that we can actually start buying them in quantity so we have one or two.
08:23
So now we can start buying them we're going to start seeing more of them. Now look at this diagram this kind of gives you the overall of the different components what I'm going to do is I'm going to remove the labels and remove the nice pretty fluffy clouds and show you the communication links between each one of these devices. Okay, this is what we lovingly call the spaghetti diagram.
08:41
Okay, this is the diagram that I created for, I should say Darren Hyfield one of my partners and myself. The two of us created this diagram for NIST in an interagency report that we released last summer. So if you go back and check out this NIST report you can see the reference on the very bottom for those that are interested in seeing it 7628.
09:01
This diagram, this document is about, I don't know 700 pages in length comes up in three different volumes. There's a lot of good information if you're interested in learning more about some of the issues, some of the more details some of the concerns some of the security architectures and security controls that we're recommending at a high level, realize that this is high level this doesn't go into a great amount of detail.
09:23
That's a great document to be able to get that information from. Okay, now what I'm going to do is you see this overall architecture and the communication links I'm going to point out in different areas that we hear all the buzzwords. Everybody has heard buzzwords like SCADA, right? Everybody's heard buzzwords smart grid or excuse me, smart meters. I'll show you exactly where each one of those different areas are.
09:42
So first off, the SCADA. When we talk about SCADA these are the types of devices that allow us to read sensor information from the field and be able to then make decisions on sensor information and send control signals back out to the field to cause reactions and changes in our real world environment. So you can see those blue sections
10:02
in the upper right hand corner that are circled those are going to be the sensors in the different devices IEDs and other types of devices yes, IEDs does have another meaning inside of the smart grid and RTUs, the devices that are controlling a lot of the devices out there and the central brains that usually collect a lot of the sensor information and send it back.
10:21
We're using SCADA protocols to send those back to what we call our back end SCADA systems. So here that one single yellow box that's circled is our distribution SCADA. In the smart grid we have several different protocols that we use, some of the earlier protocols like Modbus, serial communications across basic serial lines. Then as history went forward we started taking that
10:41
serial communication, packetizing it and throwing it in TCPIP streams or UDP streams, sending it back across higher bandwidth lines back to the organization. And some of the newer protocols DNP3 is probably one of the most commonly used SCADA protocol in the electric sector here in the United States. A lot of these protocols have very, very limited
11:02
security and it's something we're trying to address we're trying to build new protocols to replace them. DNP3 being the most popular one, it's only been about two years ago that we even had even encryption capabilities inside of DNP3. And we are looking at replacing DNP3 with additional protocols that have much stronger security models. So that's distribution. Transmission SCADA is very
11:22
similar as well. Of course they have their own field devices out there. There's usually their own transmission SCADA server that's controlling a lot of those signals. You can see that there are some communication links between the distribution and the transmission devices. And then we have last but not least the bulk generation.
11:42
Generation plants, nuclear power plants all those guys, yes they have more infrastructure than what we show with this one single box. The reason why it's only one box is that was for the most part out of scope other than the lines of communication between those bulk generation and the utilities themselves. It was out of scope for the work that we were doing for NIST. So that's why you only see that one box. But realize that there's
12:01
a lot more technology and a lot more devices and communication links inside of the bulk generation. The next buzzword we want to talk about is the electric vehicles or the PEVs. A lot of us that are bringing these back to our house, right now they're fairly simple in their communications modules. For the most part when you get a PEV you are going to be
12:21
talking to your electric utility company and they'll usually either issue you a second separate electric meter for your house that's specific towards your electric vehicle because each one of these electric vehicles on average consume about the same amount of power to charge its batteries as your whole entire house uses. So there's a lot of power going on
12:41
there. Each utilities are trying to find different ways and different strategies to try to deal with additional load inside of our infrastructure. So part of that is with separate meters. Sometimes they'll have you just plug into your normal links inside of your house but then you usually end up getting charged more. Most utilities will give you a price break
13:02
by having the separate meter in there. Right now, like I said, there's little to no communications. There's a lot of work going on with communications to be able to allow your PEV to communicate back to the electric utility vehicle. It'll be interesting to see whatever happens with this but part of one of the initiatives and some of the vendors that are out there
13:21
are trying to get to the point where the electric vehicle can self-identify itself and when you plug power in or plug your vehicle into either your employer or your neighbor's house or some family member's house, a lot of people in the industry have this idea that they want to be able to charge track where your vehicle goes and it still gets paid on your bill no matter if you're plugging into your neighbor's house
13:42
or your family's house or somewhere else. Of course, this becomes a security nightmare trying to tie this together especially when we start mixing up different electric utility companies. For most of us in the states, it's fairly easy. Each regional or each city or each town is tied to one single electric utility vendor. It's not quite as easy a case with those people down in that little state called
14:01
Texas. They're a little bit different beast for most of us in the electric sector. Okay, the next area. Synchrophasers. Synchrophasers are another technology. Now, for those of the people in the room, who understands the difference between a digital multimeter and an oscilloscope? Raise your hand. Okay. So, right now, if you want to think about it,
14:22
the way that in general, once again, this is oversimplification, but in general, the way that the electric utilities right now are controlling and measuring the power inside of the grid is more or less with a whole bunch of really smart digital multimeters that are in all the different substations that are taking readings on average about
14:41
every two seconds to find out what's going on, how much power is being used, how much voltage is there, and all the other pertinent measurements that they need. They realize that while that's necessary, while that's good, they have some good information, it's not quite as finite as they need. And so they're trying to employ additional technologies to give them something that's more visibility like you would see with an
15:01
oscilloscope. Instead of just getting a digital number telling you what the power is, they're putting these devices out that will more or less let them recreate that sine wave or what the power is really doing and how it's really flowing. They actually call these phasors and they do phase angles to try to figure that out. These measurements, these synchrophasors are making readings of the power
15:22
at minimal 30 times per second. At maximum, a lot of the vendors are doing up to about 120 and there are some discussions about pushing it all the way up to 240 times per second down in the future. But right now, I'd say probably 30 to 60 seconds, or 30 to 60 times per second are what most
15:41
of the utilities that are deploying these synchrophasors are trying to get to. To give them a little bit better idea on what the power looks like at one end, at one state to the other state across their whole domain. Because in general, that sine wave should be nearly identical across the whole entire grid. The United States has actually broken into three separate grids. We have the east, we have the west,
16:01
and we have Texas. Like I said, they're kind of their own entity and their own beast and they have a little bit different legal and political issues to deal with as well when it comes to power. So that's what the synchrophasors are. The synchrophasors are primarily being used by your transmission operators, the people that are actually pushing
16:20
the power long distances. This information is also being sent back up to what we call regional coordinators. Regional coordinators are entities in the United States that try to work with the transmission operators and help the transmission operators make sure that the power is balanced out and make sure that we have a nice stable grid.
16:40
So there's about 15 different regional coordinators inside of the United States that help try to manage this power. The next buzzword we usually hear about are the smart meters themselves. While these are definitely fun devices to play around on, this is something that is a relatively minor issue that we have compared to a lot of the other issues that we're facing with the smart
17:03
grid. So down here you can see that the smart meter in that bottom right hand corner, the one right in the middle, that's going to be an interface that is usually deployed on the meter itself. It's not necessarily its own device in most circumstances for residential people. When we get into larger deployments
17:20
for corporations and more so even in industrial, there's going to be a separate interface to be able to control and manage all the different meter readings that are deployed out there. So this is where the meter is. There's lines of communications between these smart meters that are being deployed back to the electric utility company to what we call a head end that's in the back of the electric utility company.
17:42
With this infrastructure, traditionally for most of the people inside of this room, most of you do not have the new smart meters on the sides of your house. If you're interested to find out what type of meters you have on the side of your house, just simply go out there, look to see what the manufacturer is, look to see what the model is. It's usually very visible right on the front of the meter. Punch that into Google
18:02
and you can get spec sheets on any of these meters to tell you information about what capabilities it supports. A lot of you will be able to go to the meters and you'll find that information. You'll also see this little acronym that's ERT on the face of the meter. What ERT is is this is a protocol that probably the majority of us inside of this room
18:22
have inside of our meters. This is not the new smart meter technology. This is a one-way broadcast protocol that the meters will go through in protocol in about every two minutes what their data consumption is. One way there's no way to be able to use this communication protocol to talk back to the meter. It's broadcast out using a
18:42
900 megahertz protocol that does frequency hopping. Well, it doesn't even do frequency hopping. It randomly chooses, when it comes up to its time, its allotment every minute of two seconds. It randomly chooses one of 40 channels to broadcast that information on. So that way it tries to avoid stepping on top of anybody else inside of the neighborhood.
19:02
That's what we have with the meter readers. They used to have to come up and read the meter themselves. Now they can just do drive-bys because they're collecting these ERT signals that are being sent back out. There are other protocols besides ERT. There's also some that transmit and communicate over the power lines themselves. But the ERT is probably the most commonly deployed
19:21
precursor or semi-intelligent meter that are being deployed right now. In fact, we call these meters AMR meters, these ones with the one-direction communications that are out there. As opposed to AMI meters, AMI meaning the more intelligent ones that have bi-directional communications. The AMI meters, the new smart meters, their deployment is relatively small.
19:42
Depending on the research that you see, you'll see that deployments anywhere from 10 to 25% across the US. Once again, there's a lot of debate on the exact number of those smart meters out there and exactly what's the difference between some of the smart meters and what are not some of the smart meters. So to dig a little bit deeper into
20:01
the smart meters themselves, and that generates a lot of interest. In general, if we take all the detailed information out and abstract this to a generic architectural view, this is true for most of the field devices in general that we deploy in the smart grid. Just differences of protocols in the exact terms of the devices. But when we look at the AMI meters themselves
20:21
and these smart meters, we see on that far left-hand side of this diagram the electric utility company themselves and all their back-end systems that's reading this information. The very first device that's part of the AMI network, these servers that are purchased with the meters, are going to be these head ends. These head ends usually are protected by some type of a firewall or some type of perimeter that
20:41
they set up before we get out to the field devices. Then of course we'll hit some routers and the routers will then put it back out, for the most part out to the ISPs and the telco companies that are connecting to them. Most of the meter data is going across cellular connections.
21:00
So you can see that the different links we have there, the good majority is through cellular connections out to these meters. We do have some proprietary third party offered RF towers. We also had in certain circumstances, especially in the case of industrial customers, we'll have leased lines out to each one of these. These communications for the meters go down then,
21:20
these control signals are when they tell you to shut the power off at your house, will go down to what we call the drop point or the aggregator for the network. These aggregators are devices that are deployed out in the field either as a pull top device up on top of a pull or as a meter, kind of a more intelligent meter that's placed on
21:40
the side of one house inside of your neighborhood. You can tell when you have these take out points or these aggregator meters when they're on the side of somebody's house because in general these will usually be sticking out a few inches further than all the other ones inside of the neighborhood because they have to make room for that additional communications. Namely, they have to make room for the cellular communication module that's inside
22:00
of the device. Beyond that, in some deployments and some of the vendors that have these smart meter products that are being sold, they will have, some of them will be deployed with cellular modems in every single one of the meters, but I would say that the vast majority of the vendors that are selling here in the United States instead will have a meshing technology set up so that all the meters inside of the neighborhood
22:22
will set up a communications mesh to get that data and assemble that data back to the take out point or the aggregator to push that back up to the utility company itself. Of those devices that have this mesh network, it's rather interesting. Every single one of the vendors that are out there, while they may
22:42
be using meshing technology, and almost every single one of them is using a 900 MHz frequency to do that meshing technology, every single one of the meshing technologies and frequency hopping patterns of those 900 MHz spectrum communications is different. Generally what the vendors are doing is they're choosing chips from TI and the other different types of communication
23:01
chips that are out there that are generic off-the-shelf commodity chips for their boards. They're taking these. They are taking the configuration file that dictates the frequency hopping. They'll build their own frequency hopping algorithm and the frequency hopping patterns based on a number of different factors. By changing the frequency hopping, they can change the amount of bandwidth and the amount
23:21
of distance they can get out of some of the devices. They'll tweak that to get the magic numbers that that vendor's interested in, and then on top of that, they'll build the meshing protocol, and every single meter vendor out there that has a mesh network has their own proprietary mesh network. There's no shared technology between the meshing technologies. Above that meshing technology,
23:42
we'll see a combination of vendors, I would say probably about 50% of them are using a standard protocol for meter communication called C1222. Each one of them, of those 50% that are using C1222, even their implementations are different. While they meet the specification
24:01
to the letter, they don't necessarily meet it to the point where it's interoperable with anybody else because of a number of different factors. The other 50% of the vendors out there for communications, they're going to be using a TCPIP connection, either IPv4 or IPv6 depending on the vendor, and then pushing either their own proprietary protocol across it, either
24:21
raw C1219 tables, which is C1219 tables or how each of the meters actually store their data inside of the meters themselves. Or they are going to be using some standard protocols. They might be using some web services across it. There's not many vendors out there doing web services or XML data streams and exchanges on those meters, but there's a couple that are doing that here in the United States.
24:41
Of this area of the mesh, when we think mesh, wireless mesh at 900 MHz, most of us automatically think Zigbee who lives in the field. As I realized, as I wanted to clarify, I did mention that every single vendor is using their own proprietary one. That means that nobody is using mesh for this meter-to-meter network neighborhood. Every single one uses a proprietary.
25:01
The place where most of these devices are using Zigbee, which most of them do have Zigbee modules in them, are between the meter down to the devices inside of our house. That's where the Zigbee communications come into play. If you can get these slides later, you can go through and see some of the different protocols and dig a little bit more into it. If we look at the
25:21
payloads that are being sent across, to give you a high-level idea of what happens in the type of communications that pass back and forth between these devices, I have gone through and listed out some of the different payloads. Notice that the first block of payloads primarily are communications between the meters themselves and the head end. The one below it, the three lines below it, at the very bottom of the slide,
25:40
these are pass-through communications. These are going to be the communications that come from the Zigbee the Zigbee network inside of our home area network, the devices inside of our house, passing back to the utility company. These will usually either be tunneled across the connection in some situations, or they will simply be data inside of some of the C1219 tables that are
26:02
being passed with whatever communication protocols are there. The head end will simply pull out that input from that table, pass it back on to whatever devices need it on the back end. For the most part, a lot of the utility companies aren't doing a lot of those pass-through communications yet. It's there for future use. Some of the goals that they have is with demand response programs.
26:22
This is where a utility would give you a discount to the power that you're paying in order to allow them to have some limited control of some of your high consumption devices in your house, like your AC unit, during times of peak load. For instance, I live in Salt Lake City, Utah. A lot of people up there, the electric utility
26:41
company, Rocky Mountain Power, give this discount if you have this device called a CoolKeeper to put on your house between your AC unit and the control unit in your house. What this does is, in the middle of July when it's the highest temperatures, when it becomes a threat to the power stability inside of Salt Lake, because we're having a bit of a power
27:01
issue up there over the last couple of years, they can go through and they can power cycle for small intervals the AC units inside of the neighborhoods. They'll actually do this in a coordinated fashion. They'll take approximately one-fifth to one-eighth of the AC units inside of the neighborhood. They'll cycle them down for five minutes or seven minutes, and then they'll go ahead and let them come back up, and then they'll cycle down
27:21
the next fifth for that amount of time. So that way they're at least trying to decrease some of the load inside of the grid. They've had a lot of success with doing this. As customers, we appreciate that to some degree, except for on the hottest days when they actually start turning the cycling on. But otherwise, these are some of the different types of programs they have. Expect to see additional demand response
27:43
programs here in the future. On the meter communications themselves, we have some basics. Of course, we have the consumption data that's coming back to the head ends themselves. We have control signals to be able to turn off power at the house. Part of the reason why they have these communications in there is when people move out, it's always
28:02
been a big problem to have people going in and squatting inside of the houses and getting free power. So now when you call up and say, hey, I'm moving. Can you go ahead and remove from the bill? They can immediately just shut down power at your house. When you move in, you can call up and say, hey, can you enable power? They can do that while you're right on the phone with them. That one piece of functionality, that remote disconnect, is probably the greatest
28:21
threat from a security perspective of these meters. I would say the other major threat from these meters themselves is realizing these meters are collecting information and sending them back to a controlling server on the back end. And any time we have data passing input data from a meter back to a controlling server on the back end, there isn't a chance for an attack there. I've never seen any good proof of concept code
28:42
to be able to attack any of these head ends, but that still is a concern in my book and something that I think personally deserves a little more research that are out there. Seeing if you can take the few input fields that those head ends are accepting from the meters and seeing if you can get a buffer overflow or some other type of attack to gain control of that head end. Because ultimately that would give you the most control over these meters themselves.
29:01
There are other attacks out there with vulnerabilities. If you can find the right combination of vulnerabilities, you might be able to get in and try to control some of the other meters remotely by building your own devices to communicate on these mesh networks. Or if you repurpose one of the meters and try to make it so it communicates on it. If you can get the right combination of vulnerabilities, you can do
29:21
things and attack the meters remotely. The good thing is most of these meter manufacturers are on their third and fourth generation products right now. If you go back and look at the first generation products, it was a huge nightmare and very, very much a possibility to be able to perform these types of attacks. With the current models right now with these third and fourth generation devices that are currently being deployed, it's
29:42
a much harder attack surface to be able to find that right combination of vulnerabilities to attack that infrastructure. Most of them are doing a fairly decent job now with their security infrastructure and security architectures. Okay, so attacks and defenses. That gives you a little bit of an idea of what the architecture looks like. So let's talk a little bit more about some of the attacks that we perform as penetration testers
30:02
and some of the defenses. Okay, so this is an oversimplified chart. Realize this is not what a utility company really looks like. This is highly oversimplified. This just goes through to show that we have client-side attacks, server-side attacks, network attacks, just like any other industry. We do have that additional vector of network attacks in the field devices and the hardware attacks on these devices as well.
30:22
So really fast, client-side attacks, it is a threat. Honestly, out of all the attacks that are out there inside of the smart grid, all the attacks that we can do to the meters on our homes, all the attacks we can do to the big iron products in the transmission operator substations that are out there, at the end of the day, these are the things that personally keep me up at night.
30:41
The things that I'm most worried about. Because electric utility companies are just like any other organization out there. They have internet links. They have clients that are surfing the web. And while they're, for the most part, they're control center technicians and control center operators, they don't have direct internet access on their workstation. They do have connectivity to services and different devices inside of the
31:01
organization. So if an attacker comes in through the front door through a client-side attack, gets a presence inside of the organization, it's only a matter of time before he can get his way and work his way to the right workstations and the right subnets to be able to gain access to some of these servers. So that's what keeps me up at night. I personally think the day when we see the biggest
31:20
incident from attackers that affect us from the smart grid, I personally think that the front doors, the launch of this attack is going to be coming through a client-side attacks. Server-side attacks, nothing different here. Same vulnerabilities. Realize that all these control servers that I showed you, the ones that are controlling all the communications for the SCADA networks and for the AMI meters,
31:41
this is commodity operating systems on commodity hardware. So the things that everybody in this room does for penetration testing of devices, your knowledge is directly applicable here for the server-side attacks. And just like any other industry, we have a lot of the controlling interfaces are moving from fat client-type control interfaces and serial and terminal interfaces.
32:01
They're moving to web-based interfaces as well. So things like cross-site request forgery immediately become problematic. And for those of you that do web pen testing, you know cross-site request forgery is out there in probably 95 to 98% of every single web interface you ever touch. So it's been a huge issue and this is something that's a very valid attack angle for us.
32:21
Network attacks. In the network attacks, when we're looking at the field devices, inside the organization is just like anything else. Most of the protocols, most of those devices that you saw in the earlier architecture diagrams that have communications, most of those are using web services and other types of very common protocols that we're used to dealing with.
32:41
When we get to the field devices, we get a lot of proprietary one-off protocols, especially when we get to the substations, because for a lot of the devices that were deployed in the substations in the last 20 to 30 years, a lot of them were custom built for each one of those utilities. And so they're very
33:01
customized proprietary languages. Even when we have standard languages, like I told you C1222 for the meters, each of the vendors are doing customizations of each one of those protocols as well. So when we look at network attacks, for the most part, when we do penetration tests, if we have enough time and enough budget, we will go through and try to reverse engineering the proprietary protocols that are out there. But a lot of times,
33:21
like it always is the case when we do pen testing, there's always limited budgets to work with. And so we always want to at least do a very, very simple check of the network communication protocol for some basics. Some of the basics that we're going to be checking for is going through and checking for the cryptography they're using, checking for encryption, just trying to see if they actually have encryption enabled at all.
33:42
Other thing we often see is for the wireless communications is a misconception that frequency hopping is one of their security controls. Frequency hopping is not a security control with the right equipment and enough time, you can go back and you can backtrack and trace what that frequency hopping algorithm is and bypass that. It's more of a method for obscuring anything. And ultimately,
34:01
the reason why we do frequency hopping isn't for security, it's ultimately to be able to provide a higher quality of communications. We also see problems with communications with cryptography because a lot of the protocols that we're used to dealing with cryptographic protocols in normal IT simply are too heavy-handed and
34:21
the embedded devices that we're deploying don't have that capability, so we have very limited capabilities for some of these devices. So a lot of times we're going to be messing with asymmetrical encryption, AES encryption, because it's a lot easier to have these embedded devices use it. We do have some asymmetrical protocols out there, but there's very few
34:40
asymmetrical options that we have at our disposal. So when we're dealing with these protocols, one of the first things we're interested in is how they do their key updates and key distribution, because that's one of the biggest weaknesses in Achilles Heels to symmetrical encryption. When we get a proprietary protocol that we've never messed with before or never seen before, the first thing we're going to do is
35:01
we're going to try to determine whether it's an encrypted protocol or not, and if it is encrypted, try to determine how encrypted it is. So here's some charts just showing you different ways to determine. You capture the packets, you strip out any of the header information, you take just the payload, we do a we build a histogram checking the entropy of the data inside of the payloads themselves. The graph on the chart,
35:23
the graph on top is going to be this is the prime model, this is the one that we want to see when we want to see encryption, something that's very very evenly distributed out. The one on the bottom is still encrypted but it's not as good as in its encryption mechanism because we don't have quite the even number of balance. And if you ever
35:42
see when you're doing this histogram in a comparison, if you ever see a large jump right in the very middle and right back down, usually it means you've come across an ASCII-based protocol because you hit the big jump right in all the middle of the ASCII characters and very very few representation anywhere outside of those common ASCII characters.
36:00
So, another issue we have is is in the web world, we have products vendors that are selling web-based products that say yes, we're secure because we use SSL and TLS. We have the same exact problem we have inside of the embedded world we have vendors coming up and saying yes, we're secure because we use 8 yes. So, one of the first questions we're always
36:21
going to be asking is what types of cycle modes are you using for your AES and trying to start digging down into the architecture there. Of the hardware attacks, the hardware attacks does represent not just the meters themselves, this can be performed by any of the pulltop devices like the aggregation points or the feeder switches. This also takes
36:40
effect of the RTUs and the different devices in the substations themselves. Of those hardware attacks, most of these are susceptible to different types of physical attacks themselves either just getting in and trying to get information back out or making some modifications in the hardware themselves. A lot of the vendors that are out there have minimal capabilities
37:01
or minimal controls around trying to detect when people are tampering with these devices but for the most part they're fairly weak. When we are trying to attack each one of these hardwares the primary purpose for us to perform a penetration test on one single piece of hardware isn't to compromise that hardware. Just like everybody in this room knows, when you have physical access to the computer we assume everything on
37:21
that computer is compromised with some minor exceptions and some edge cases. Same thing with the meters themselves and any of the embedded hardware devices deployed to the field. We get our hands on these devices not to show that we can attack these devices and get information off these devices. We attack these devices to try to identify vulnerabilities and try to get information to enable us
37:42
to attack the other devices inside of the infrastructure. Some of the things I'm going to be looking for are the cryptography keys, the asymmetrical or asymmetrical cryptography keys that are stored on these devices because with access to this information I can go through and I can attack the other devices inside the infrastructure either directly, let's say I can get the infrared password because all the meters have that little infrared interface, that's the little round
38:02
thing with the two little dots on the front of your meter. It's an infrared interface. If you get the infrared interface, of course I can go through and just very quickly launch attacks at each one of the other meters inside of the neighborhood. But what I'm more interested in is getting the keys for the wireless mesh communication protocols because if I can get the asymmetrical or asymmetrical keys for that and I can find a vulnerability in the way
38:20
that they've been implemented, I might be able to launch other types of attacks like impersonating another meter or impersonating the head end and sending control signals down to the other meters from my device. The other thing we have is with energy theft of these meters. This is something we've had forever. It's not a new
38:40
thing. Ever since they've had power, people have been stealing power from the meters themselves. There's a lot of information. Any of these terms, if you just punch them into Google, you'll be able to get some details information about how people have done this in the past and some of the attacks. This is basic stuff, simply trying to steal power. That's not all that interesting to us, but it is kind of fun to talk about it, especially when you see really cool pictures like this of people stealing power, piggy-backing multiple meters together,
39:02
ripping the meter out, just getting in and actually hard-wiring communications back to one single meter, or usually outside of that one single meter so it's not being read at all. We also have physical bypass. A lot of these devices are being tried to protect with different controls like locked cases and lock boxes on them and fences
39:21
and perimeters and security cameras and all these things. Of course, locks, we can pick the locks. Fences, we can climb the fences. Cameras, well, number one, if they have cameras. Number two, if they have cameras, do they actually monitor them? If they do monitor them, how long does it take to roll something out? If you get a good attack and you can actually replicate this attack and make it very
39:40
easy to go through and launch, you can go to each of the different substations, launch your little attack, get back out within a 10-15 minute minute, at least give you some remote presence, some wireless remote presence inside of that substation to be able to go through and continue your attack down the road. So these are things that we're trying to combat, some of the things that we're trying to address
40:00
inside of the smart grid. When we're working with the hardware itself, it tells you we're looking for keys in the hardware. We want to go into these keys and try to extract these keys out. Two methods that we use for extracting these keys out is either going through on these hardware devices and identifying where the data is stored at rest, so the EEPROMs, the RAM chips, the flash memory inside the chips, the on-board storage of
40:22
the system on the chips and the microprocessors, trying to gain access to those and dump the data back off of them. Another thing that we'll do is we'll go in and identify communications between the microprocessor and other keys like the other chips on the board, like the wireless board, and we'll jump in and we'll sniff the bus communications off those devices,
40:40
trying to find those key exchanges. This is something that we'll occasionally find is in order to set up the encryption, if a lot of the encryption is being done by the microprocessor and the RF chip itself is just a very, very dumb chip building the medium for it to communicate on, we might not be able to get any data off because the data is already becoming encrypted, but if that RF chip
41:01
has more intelligence and is doing the encryption itself, quite often the data is being passed plain text in a serial link across that bus to the chip and quite often that key is also being sent clear text over that chip as well because the key is part, usually stored on the microprocessor itself. So if we can capture that key back off we can get access to the cryptography keys for the communications channel. So these are
41:21
just some pictures of going in, showing you how we jump in on some of the different devices, connect different wires to it. Using syringe is that one thing on the left hand side, that's a big syringe that we use. It makes it very easy to use a syringe to go in and get some of the small surface mounted pins that are on the devices themselves. Occasionally we have to go through and make some minor modifications to try to get chips out of the way from
41:41
interfering with our communications and try to stop the attacks that we're doing. We avoid this when possible, it's not always possible. Here's just a little screenshot of us going through and dumping out the contents of EPROM usually using either the I2C or the SPI communication buses on these chips themselves to basically bypass the microprocessor to get our own hardware and trade it
42:01
into that EPROM or the flash module and dump the contents out. Here's an example of bus snooping, identify the bus, jump in between the two chips that we're trying to get the captured information from and capturing that information so we can go back and do analysis later. Once we capture that information we have to identify the information that we want out of the side of this dump because a lot of times these dumps are very
42:21
very cryptic. Two different ways to do this if we have symmetrical encryption keys we do attacks very similar to the attacks that they did with the blu-ray discs where we systematically go through and take the exact length of the key that we know that they're using and we systematically step through the memory dump until we find a key that successfully decrypts whatever traffic we're
42:42
trying to decrypt. If you see here in the very bottom this is a combination of Travis Goodfette's Goodfette tool and Josh Wright's Killer Bee protocol that we do this attack in when we're dealing with ZigBee. So dump the memory contents off of a ZigBee chip or grab it straight out of RAM. We then use that binary dump to step through it until we can successfully
43:02
decrypt a ZigBee packet. Asymmetrical is even easier. Asymmetrical keys are randomly pseudo-randomly generated. So all we do is we go through and do an entropy analysis of the dump we have and as long as the rest of the data on the device is not encrypted you can very easily decide exactly where that asymmetrical key is. So you see that
43:22
one spike right in the middle that was the asymmetrical key inside of this dump. So they're fairly easy to identify. Once we go through and we have this information once again the goal is to try to leverage this for other purposes. Some of the defenses we're recommending for the utilities is try to use this on a chip as one of the best defenses they have as well as
43:42
try to limit the cryptographic keys that they're deploying out to the other devices. Another item that we're going to be using or looking for besides the key itself is the firmware. Because with the keys while we can get the keys is all great, fine and dandy if we can get the firmware itself the firmware gives us capability
44:01
and greater insight to these devices and you can go through and use these or binary binary decompilation and go through and analyze the binary itself or do source code review on the device. Those are two different options if you have a copy of the source code. If you are going to be doing binary analysis on the flash it does become a little more problematic than most of you do in binary analysis
44:21
on common everyday computers because each one of these embedded devices has different microprocessors with different instruction sets so you quite often will have to go through and build decompilers to try to simply gain access to the instructions in the first place. So that's a huge obstacle that we come across and for those that like writing decompilers we could definitely use a lot more of them
44:41
out there from the embedded chip space so great huge opportunity for you to get into. Other than that for conclusion yes the smart grid is out there the smart grid does have security issues out there no industry is perfect but at the end of the day I think we're moving forward and we have a lot of people looking at these devices and we can always use more. For those that are running security shops out there and providing security
45:01
services realize that there's a huge potential inside of the smart grid working with the vendors of the smart grid as well as the utility companies that are buying these products. Your skills are directly applicable in many of the different areas and for those that want to learn new skills there's a lot of new skills out there you can pick up with some of the hardware hacking techniques and some of the proprietary
45:21
networking protocols that are out there. So a lot of good information out there. If you want more information you can go through and check out a couple of different resources throughout the slide deck I try to give resources wherever I possibly could. If you want more information about some of the attacks specific my previous employer in guardians they do have on their website an attack methodology that goes a little bit more into
45:41
detail about some of the hardware attacks that are out there. That's a great source. Of course I did mention the NIST inter-agency report that was released last summer you can go through and grab that the URL is up there. And then also if you go to smart gridopedia it's a great place to get information and specifically on smart gridopedia there is ASAP SG one of the work groups that I've been working on
46:01
for the last two years. All the products that we create and all the documents that we create to try to help the utilities secure their infrastructure we post publicly on this website. You can gain access to any of these and get more information about any of the specific domains that I've spoken about today. Other than that thank you very much I will go ahead and take questions in the Q&A but thank you.
46:21
It's been good.