Election Cybersecurity Progress Report
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 165 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39287 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
35C3 Refreshing Memories88 / 165
2
5
6
7
8
9
10
11
12
13
15
16
17
22
26
29
30
31
33
37
38
39
40
44
45
48
49
53
54
55
57
59
60
62
65
66
69
70
72
73
74
77
80
82
83
84
85
86
87
89
92
94
100
104
105
106
107
108
111
113
114
115
116
117
119
121
122
123
124
127
132
133
136
139
141
143
144
145
146
148
149
150
154
155
156
157
158
159
160
161
162
163
164
165
00:00
ExplosionComputer animationLecture/Conference
00:57
Meeting/InterviewLecture/Conference
01:44
VotingNumberState of matterPhysical systemClosed setOrder (biology)Point (geometry)Lecture/ConferenceMeeting/InterviewComputer animation
03:09
State of matterVotingNumberResultantMathematicsTotal S.A.Standard deviationComputer animationLecture/ConferenceMeeting/Interview
04:01
InformationVirtual machineNeuroinformatikResultantComputer clusterInformation securityLecture/ConferenceMeeting/Interview
04:52
Beer steinState of matterClosed setNeuroinformatikVotingRight angleResultantExpert systemVirtual machineTrailLecture/Conference
06:56
Operations researchNon-standard analysisProcess (computing)Form (programming)Hacker (term)CybersexVotingConfidence intervalOrder (biology)Process (computing)Lecture/ConferenceJSONXMLComputer animation
08:17
VotingHacker (term)LeakServer (computing)Denial-of-service attackOrder (biology)Source codeInformationServer (computing)ResultantEmailDenial-of-service attackLeakLecture/ConferenceComputer animation
09:28
LeakMessage passingImage registrationSystem programmingEmailSimilarity (geometry)Message passingGroup actionHacker (term)EmailProcess (computing)InformationEntire functionHand fanComputer animationLecture/Conference
10:56
PasswordGoogolMathematicsLeakMessage passingImage registrationSystem programmingInformation securityLink (knot theory)AuthenticationProgrammable read-only memoryPasswordEmailStaff (military)Multiplication signXMLLecture/ConferenceComputer animation
11:54
LeakMessage passingAddressing modeWebsiteImage registrationSystem programmingDivision (mathematics)EmailMessage passingInformationForm (programming)TelecommunicationReading (process)Artificial neural networkComputer animationLecture/Conference
12:48
State of matterInjektivitätImage registrationLeakMessage passingSystem programmingPhysical systemCategory of beingInjektivitätSequelImage registrationVotingGroup actionState of matterRow (database)Multiplication signComputer animationLecture/Conference
13:56
VotingImage registrationPhishingProduct (business)Service (economics)Non-standard analysisInformationEmailMilitary operationWordVisualBASIC ScriptPower (physics)Gastropod shellInstance (computer science)SoftwareComputer hardwareCybersexOperations researchNeuroinformatikInformationSoftwareLocal ringSeries (mathematics)ComputerIntercept theoremPrisoner's dilemmaComputer animationLecture/Conference
14:50
Object (grammar)ComputerVotingExecution unitTowerBuildingHacker (term)Tablet computerWhiteboardSoftwareImage registrationPhysical systemEmailLocal ringInformationImage registrationEmailVotingPhysical systemLocal ringSet (mathematics)CASE <Informatik>TowerExecution unitWeb pageComputer animationLecture/Conference
16:25
VotingInflection pointExplosionIntelHypermediaOperations researchBusiness reportingArithmetic progressionOperator (mathematics)HypermediaLecture/ConferenceJSONXMLComputer animation
17:22
EstimationPort scannerSign (mathematics)IntelVotingVirtual machineSocial softwareOperations researchSet (mathematics)HypermediaBusiness reportingExtension (kinesiology)InternetworkingPrice indexVirtual machinePattern languageLecture/ConferenceComputer animation
18:11
IntelVotingVirtual machineUsabilitySocial softwareOperations researchMassCapability Maturity ModelBit rateVirtual machineResultantVotingError messageUsabilityLecture/ConferenceComputer animation
18:58
Twin primeUsabilityIntelVotingVirtual machineSocial softwareOperations researchNumbering schemeWhiteboardNumberVotingNormal (geometry)Greatest elementNumberOperator (mathematics)Computer animation
19:44
Image registrationVotingMaxima and minimaState of matterMultiplication signDomain nameState of matterEmbedded systemMultiplicationPosition operatorBusiness reportingHacker (term)Image registrationMassNumberVotingChaos (cosmogony)Group actionLecture/ConferenceXMLComputer animation
21:03
Moving averageClosed setDivision (mathematics)Image registrationVotingVulnerability (computing)CountingLecture/ConferenceXML
21:54
Hacker (term)VotingVotingVulnerability (computing)Set (mathematics)Virtual machineExtreme programmingMereologyLecture/Conference
22:46
Read-only memoryPlastikkarteComputer fileVirtual machineSurface of revolutionCausalityVotingMemory cardRule of inferenceCountingLecture/ConferenceComputer animationMeeting/Interview
23:45
SoftwareData bufferBuffer overflowMachine codeComputer programmingPlastikkarteVirtual machineMechanism designTape driveFormal languageMalwareTelecommunicationMultiplication signNumberVotingDifferent (Kate Ryan album)Semiconductor memoryVotingSoftwareMachine codeCountingProgramming languageMalwareVirtual machineProcess (computing)Business reportingMechanism designBuffer overflowMemory cardRight angleRow (database)ResultantLecture/Conference
24:41
Error messageVulnerability (computing)LogicComputer-generated imageryBootingMachine codePublic key certificatePasswordInterpreter (computing)Memory cardVotingExploit (computer security)Data bufferAddress spaceComputer fileVirtual machinePlastikkarteInformation securityKey (cryptography)Physical systemSmart cardMenu (computing)Local ringOperations researchReflektor <Informatik>Content (media)Read-only memoryMalwareDatabaseBlogAuthenticationCommunications protocolFunction (mathematics)System administratorUbiquitous computingSource codeMereologyState of matterGame controllerResultantMemory cardMalwareWeb pageMultiplication signIndependence (probability theory)State of matterMereologyGroup actionContent (media)Virtual machineBusiness reportingAuthorizationVulnerability (computing)Flow separationTable (information)Lecture/ConferenceComputer animation
25:28
State of matterPrinciple of localityScale (map)Kolmogorov complexityWindowVotingOrder (biology)Revision controlPhysical systemLine (geometry)Source codeBusiness reportingSoftwareFirmwareBuffer overflowState of matterSoftware testingVirtual machineRule of inferenceLecture/ConferenceComputer animation
26:26
Sample (statistics)Smith chartDuality (mathematics)Drum memoryMaxima and minimaTouchscreenRead-only memoryComputerEndliche ModelltheorieOpticsDirected setTelecommunicationVotingVotingMereologyWeb pageLocal ringState of matterCoefficient of determinationNumberScaling (geometry)Virtual machineElektronische WahlTouchscreenLecture/ConferenceComputer animationTable
27:32
VotingMalwarePlastikkarteIndependence (probability theory)Information securityVirtual machineMach's principlePurchasingVotingRow (database)Semiconductor memoryTouchscreenNeuroinformatikVirtual machineBelegleserCASE <Informatik>Vulnerability (computing)Lecture/ConferenceComputer animationJSONXML
28:18
Hacker (term)VotingInternetworkingResultantPhysical systemState of matterVirtual machineType theoryVulnerability (computing)Lecture/ConferenceComputer animation
29:10
Hacker (term)VotingState of matterInternetworkingMathematics1 (number)Figurate numberResultantVulnerability (computing)VotingInternetworkingRight angleVirtual machineModemCellular automatonLecture/ConferenceComputer animation
29:59
Read-only memoryPlastikkarteData managementComputer programmingComputerStaff (military)Service (economics)System programmingVirtual machineSingle-precision floating-point formatVotingState of matterMalwareWorkstation <Musikinstrument>Computer programmingWindowMemory cardCuboidLecture/ConferenceComputer animation
31:05
Computer programmingData managementComputerEmailDigital photographyImage resolutionWebsiteCASE <Informatik>Directory serviceEmailProcess (computing)Address spaceSoftwarePhysical systemVirtual machineComputer programmingMalwareState of matterLecture/ConferenceComputer animation
31:52
MathematicsHacker (term)VotingState of matterComputerData managementMalwareVotingComputer scienceElektronische WahlExpert systemInformation securityCASE <Informatik>Physical systemSatelliteCompass (drafting)Computer animationLecture/Conference
32:39
Memory cardVotingBelegleserPhysical systemVotingRow (database)TelecommunicationResultantEngineering drawingLecture/ConferenceComputer animation
33:32
Constraint (mathematics)StatisticsRow (database)Computer scienceProcess (computing)Confidence intervalNeuroinformatikState of matterMarginal distributionResultantComputer animationLecture/Conference
34:47
MalwareVotingHacker (term)MathematicsState of matterData managementComputerEmailPlastikkarteRead-only memoryComputer iconState of matterVotingMalwareData managementVirtual machineRow (database)Physical system1 (number)Computer animationLecture/Conference
35:32
Information securityVotingMathematicsState of matterBusiness reportingVotingPhysical systemModal logicInformation securityComputerImage registrationAuthorizationMultiplication signJSONXMLLecture/ConferenceComputer animation
37:13
Virtual machineMultiplication signVotingSound effectPlanningLevel (video gaming)State of matterLecture/Conference
38:03
Local ringOperator (mathematics)ResultantSelf-organizationOrder (biology)State of matterPlanningMultiplication signLecture/Conference
39:01
ImplementationPhysical systemState of matterVotingAddress spaceSystem programmingCybersexOrder (biology)State of matterPhysical systemMereologyImage registrationStandard deviationComputer animationLecture/ConferenceMeeting/Interview
39:51
GradientPapierloses BüroInformation securityMalwareDigital signalComputer-generated imageryState of matterVirtual machineArithmetic progressionSampling (statistics)TouchscreenSpring (hydrology)Port scannerVotingDigital photographyResultantInformation securitySocial classStudent's t-testMalwareComputer animationLecture/Conference
41:25
GradientPapierloses BüroState of matterMultiplication signVirtual machineVotingArithmetic progressionRow (database)Rule of inferencePapierloses BüroComputer animationLecture/Conference
42:14
GradientVotingMultilaterationState of matterSet (mathematics)Standard deviationLecture/Conference
43:12
GradientPapierloses BüroVotingImage registrationVirtual machinePhysical systemHacker (term)Computer programmingDisk read-and-write headModemSoftwareLaptopImage registrationTrailVotingComputer animationLecture/Conference
44:33
Web pageVotingState of matterImage registrationPhysical systemSample (statistics)WhiteboardOffice suiteReading (process)Server (computing)Entire functionUniform resource locatorSeries (mathematics)Vulnerability (computing)VotingImage registrationComputer fileRow (database)Sampling (statistics)Uniform resource locatorServer (computing)SequenceNumberPhysical systemInformation securityState of matterProcess (computing)Lecture/ConferenceComputer animation
45:45
VotingWeb pageState of matterReading (process)Server (computing)Physical systemEntire functionOffice suiteImage registrationUniform resource locatorBinary fileHacker (term)Information securityOffice suiteHacker (term)VotingPhysical systemState of matterDisk read-and-write headComputer animationLecture/Conference
46:46
RoundingInformation securityInformationLevel (video gaming)Message passingDifferent (Kate Ryan album)Sign (mathematics)Information securityArithmetic progressionCybersexTerm (mathematics)NumberInformationInternet service providerState of matterStandard deviationComputer animationLecture/Conference
48:38
Proof theoryState of matterTrailData recoverySoftware testingInformation securityVotingRow (database)Information securityRow (database)Image registrationPhysical systemAnnihilator (ring theory)Dependent and independent variablesVotingState of matterResultantModal logicComputer animationLecture/Conference
49:47
VotingBuildingLocal ringGroup actionData integritySimilarity (geometry)Digital signalSign (mathematics)Hacker (term)VotingNeuroinformatikDeterminantResultantHacker (term)Similarity (geometry)Information securityCategory of beingOnline helpFlow separationGroup actionMultiplication signAuthorizationE-learningPhysical systemComputer animationLecture/Conference
51:17
ExplosionExecution unitNumberLecture/Conference
52:03
ExplosionSampling (statistics)RandomizationOrder (biology)Process (computing)Lecture/ConferenceMeeting/InterviewJSONXMLUML
52:58
Vulnerability (computing)Software bugNumberLecture/ConferenceMeeting/Interview
54:04
1 (number)CASE <Informatik>Web pageInformationUniform resource locatorMultiplication signResultantTrailCatastrophismState of matterSelf-organizationMoment (mathematics)Physical systemLecture/Conference
55:37
Instance (computer science)NumberAttribute grammarBasis <Mathematik>Business reportingBranch (computer science)Line (geometry)Reading (process)TelecommunicationWeightInformationModal logicInformation securityLecture/Conference
57:21
VotingSelf-organizationVirtual machineGroup actionInformation securitySoftwareStatisticsResultantTrailSoftware developerOpen sourceAreaLecture/ConferenceMeeting/Interview
58:48
Computer wormExplosionSemiconductor memoryCartesian closed categoryComputer clusterVirtual machineVotingDefault (computer science)Lecture/ConferenceDiagram
Transcript: English(auto-generated)
00:18
Mr. Haldeman, Professor of Computer Science at the University of Michigan, famous for
00:27
inventing things like Let's Encrypt, finding the S-more, but wait, there's more.
00:42
Logjam, I love buzzword bingo, and Z-map, and now he's going to talk about American elections. Thank you. All right, thank you so much. It's fantastic to be back at Congress this year.
01:02
Two years ago, I was here with Matt Bernhard, one of my PhD students, and we gave an update about what happened during the 2016 presidential election. Today, a lot has changed and a lot remains the same.
01:20
I'm here to let you know what we've learned about what happened in the 2016 election and what we still need to do to make sure elections in the U.S. and around the world are well-protected. So, a quick flashback. On November 8th, 2016, Donald Trump became President of the United States by beating
01:43
some other person. Now, history quickly forgets the losers in presidential elections, and it really doesn't matter who Donald Trump beat, because today, for better or for worse, he is the president. But how close was the election?
02:02
President Trump likes to talk about how he won by a landslide. But actually, he was the fifth person in American history to win the presidency while losing the popular vote. In fact, his opponent received three million more votes in the election than President
02:23
Trump did. How can that happen? Well, we have this crazy system called the Electoral College, and in the Electoral College, each state has a certain number of points, and Donald Trump ended up getting more of those points.
02:41
But if we want to ask how close was the election really, well, that depends on the way each state allocates its electoral votes, and most are winner-take-all. So we might ask, how many votes would, say, an attacker have had to change in the smallest
03:02
number of states in order to change the election result, in order to, say, make it a tie instead of a win for President Trump? And it turns out that if you look at the three closest states, they could be flipped with a very, very small number of votes changing. And changing just any two of these three states would have been enough to reverse the outcome
03:24
of the presidential election. If we look at the next few closest states, they also have very small margins, and any three of these six states would have sufficed to change the election result. In total, just changing 27,500 votes from Donald Trump to Donald Trump's opponent would
03:49
have changed the outcome of the U.S. presidential election. There were 137 million votes in total. That's a change of just .02 percent. That is a very close electoral result by even contemporary American standards.
04:05
And that's why the possibilities of computer hacking, voting machine manipulation, information warfare that actually did take place, some of them, in 2016, not only have the possibility
04:22
to have affected the 2016 election result, but stand to have the possibility to affect future election results as well. And that's why election security is so important right now. But if we go back to 2016, when I was speaking here two years ago, the main thing
04:43
I was talking about were recounts in three states, Wisconsin, Michigan, and Pennsylvania, that I and other election security advocates had a big role in orchestrating. Well, we realized after 2016 that this was a close and unexpected election result, but
05:01
no one was going to go back and check the physical evidence of the votes, the actual paper ballots, in any states that really mattered, to make sure that the computer election results we had been told about were right. Well, when I and others pointed this out to the public, it resulted in an overwhelming
05:21
show of support, and one of the third-party presidential candidates, Jill Stein, stepped in and had the legal standing to demand recounts in states where she stood for election, even though she had no chance of winning, and she raised, through small donations from the public, more than $7 million to fund efforts to go back and count and check the
05:45
votes to make sure things were right. Unfortunately, a recount after an American election is a politically-fraud process, and in all three states, we found opposition from the apparent winner of the election,
06:01
we found challenges in the courts, and only one of those states, Wisconsin, ended up recounting all of its ballots and found no evidence of fraud. In Michigan, the recounts were halted after only a few days, with less than half of the votes counted, after a court challenge by the Republicans.
06:24
Again, no evidence of fraud in the votes that were recounted. And in Pennsylvania, unfortunately, like many states, most of the state had no paper trail at all. There was nothing to recount, just digital records and machines. The courts denied the Stein campaign the right to have independent experts examine
06:45
the machines, and in very few of the places in the rest of the state, the small amount that did have paper actually did complete a recount. But still, there was no evidence of fraud. So in all, there is no evidence that hacking of voting machines,
07:04
hacking of actual vote counts, changed the outcome of the 2016 election. But there is abundant evidence that cyber attacks of other forms had a major influence on the election, certainly could have a huge influence on future elections.
07:22
And that's what I'm going to talk about today. So first, looking back at 2016, in the two years since I was last here, we have learned a lot more about what really took place during the 2016 election. Starting just January of 2017, when the US intelligence community,
07:45
the CIA, NSA, and other three letter agencies, who often in this community we don't trust, still came out and released a joint assessment in which they rated with very high confidence.
08:01
The conclusion that attackers linked to Russia were ordered by Russian President Vladimir Putin to interfere with the American election in order to weaken Clinton, boost Donald Trump, and discredit the electoral process as a whole. They called it a significant escalation of long-standing Russian efforts
08:22
to undermine the US-led liberal democratic order. So where's the evidence that this actually happened? And what actually happened? According to not only the intelligence reports, but other information from other sources we can use to see whether it's credible.
08:43
Well, what happened in the US actually looks a lot like something that happened in 2014 in Ukraine, where according to other published reports, attackers linked to Russia engaged in a multi-pronged attack to try to undermine the presidential election there.
09:02
They released targeted leaks of emails linked to the presidential campaign. They attacked the election commission servers in order to cause them to initially post the wrong presidential winner. And this was apparently detected and narrowly averted only hours before the winner was to be announced.
09:22
And they orchestrated DDoS attacks to try to delay the election results. In the US in 2016, we saw a similar multi-pronged attack of targeted political leaks, trolling and message amplification on social media, and attacks against election infrastructure.
09:42
So the targeted political leaks, you've probably heard about some of this. You have emails stolen from the Democratic National Committee through a hacking campaign that involved two different Russian-linked military groups hacking into the DNC servers, installing
10:03
customized malware, and exfiltrating thousands of emails that were then published by WikiLeaks. Later, John Podesta, Clinton's campaign chairman, also had his personal Gmail compromised. And Podesta's emails were similarly published by WikiLeaks.
10:23
Whatever you think about WikiLeaks and government transparency, and I myself, I'm a huge fan of transparency. There's clearly something subversive and manipulative about just one side being targeted, and being targeted by other foreign nations, and
10:41
having its dirty laundry aired for the world to see. This is subverting the entire notion of transparency, turning our need for true information about politicians against us, and manipulating the entire process. John Podesta, since his emails were all leaked to the public, well,
11:00
we can go and see the phishing attack email that got his password, and here it is. So this mail sent to John Podesta claims to be from Gmail, saying that someone has tried to sign in with his password, and he urgently needs to change it by clicking here. Well, he did click there, and Russia got his password.
11:22
We also see his staff talking about this email, and one of his staffers recognized that this was a phishing attempt. And emailed urgently telling John Podesta to change his password immediately, but he typoed. In dashing out this email, he wrote that this is a legitimate email.
11:41
He subsequently claimed every time he's talked about it, he meant to write illegitimate, not legitimate. Well, the rest is history. A couple of extra letters might have changed a lot. So beyond the email leaks, we've seen an orchestrated campaign on social media,
12:00
through trolls and false identities, to try to manipulate people's opinions, to try to create political divisions between people, to try to amplify certain discordant messages. That could be a whole talk in itself, and I'm not going to go deep into the trolling and message amplification, but
12:21
it's a subject that is an ongoing form of attack, that again, turns our tools of communication against us. People need to know whether the information they're reading is really what other people they know and are like them think, or whether it's being generated by bots, by attacks.
12:41
This kind of artificial amplification and manipulation of messaging turns us against each other. Finally, and the category of attacks that I want to talk about most today, because I think they're the most relevant for our community, are attacks against election infrastructure itself.
13:01
The increasingly computerized systems that we use to run elections, not just in the US, but in countries around the world. There were attacks against voter registration systems in states across the country, organized by the same Russian groups. There were attacks against companies that make technology used in polling places.
13:24
In all, the intelligence assessment is that up to 21 states had their voter registration systems probed. Now, of course, how can you go back in time and know for sure that others were not probed, were not compromised? That's very difficult, even if you're, say, the NSA and
13:41
are watching everyone's network traffic. However, we know that in multiple states, the attackers got in through sequel injection through other attacks, and were able to steal hundreds of thousands of voters' registration records.
14:01
More information came out later in 2017 through leaked information from NSA. So this woman, Reality Winner, an NSA contractor, leaked to the Intercept a series of intelligence assessments that showed the Russian attacks went even farther, that they executed
14:25
attempts to break into the computer systems of at least one election computer software vendor, and then after breaking into their systems, started trying to fish their way into the computers of local election administrators, the people who actually run the technology on election day.
14:41
For sharing this information with us, Reality Winner is currently serving a five-year prison sentence for violating the Espionage Act. But the information that she leaked has since been corroborated. In July of this year, prosecutors in the special counsel's office,
15:04
this is the Robert Mueller investigation of Russian interference and collusion, indicted a set of GRU officers, Russian military officers, in conjunction with the voter registration system attacks, the theft of email from the Democrats, and
15:22
the attempts to indict local election officials. If you're interested in this stuff, I highly recommend you read this indictment. It's about 20 pages of very detailed information, asserting to, apparently detailing exactly who these people were,
15:41
where they worked, what they did, step by step. Now it's scary to think that we might have such detailed information about crimes that took place in the past. It doesn't say how we learned. For instance, that this certain officer, Anatoly Kovalev, was working for
16:03
Unit 74455 of the GRU at 22 Kirova Street building, the tower. And quite how he pulled off each step in the attack that's asserted here. But as the Mueller indictments advance, as the special prosecutor's case comes together, we're likely to learn a lot more.
16:25
And what's to come in 2018, as the Mueller investigation winds down, I think we're going to learn a lot more about quite who ordered what, about who in the United States was involved, and about whether the attacks went even further than we have so far discovered.
16:48
So that's 2016, and what we've learned about 2016. But I'm here today to give you a progress report on 2018. So what happened during the 2018 election?
17:04
Well, we saw several things during the November election this year. According to intelligence, once again, we have allegations of continued social media influence operations. This time, allegedly linked to not only Russia, but China and Iran.
17:21
Now, I think it's very difficult to independently comment and establish on whether these allegations are true, or even to understand the full extent of the social media involvement. Because it's just a small set of large internet companies that have the raw data that we need to analyze.
17:40
However, the best reports we have are these assessments from the intelligence community that the social media influence is ongoing. We also saw sporadic breakdowns of voting machines. Now, patterns of breakdowns of voting machines could be the indication of an attack, but in 2018, all of them seem to have perfectly natural explanations.
18:02
In New York City, for instance, many optical scan machines broke down and jammed and caused long lines, but apparently it was because it was raining. And that causes the paper to swell a little bit, these machines to misfeed, and so on. So this is probably just natural failure.
18:21
We also had unfortunate human error. For not the first time, an election in Florida potentially had the result changed because of very bad usability design in just the layout of the ballot. So in Broward County, Florida, 3.7% fewer voters cast a vote
18:43
at all in the US Senate race than the race for governor. This was potentially enough, because of the demographics of Broward, to change the outcome of the Florida Senate race. Here's why, here's the ballot. So this is the race for governor, which most voters filled out,
19:01
as you would expect. Right down there underneath that long column of instructions is the US Senator race. So you imagine this ballot, it's much larger than a normal piece of paper. The bottom of that is hanging off your desk as you're filling it in. I can see how 3.7% of voters might have completely missed that race in
19:22
the first column. Finally, we had old fashioned political fraud. In North Carolina, a race for the House of Representatives was decided by only about 900 votes. But it's come out since then that operatives working for the Republican candidate allegedly stole or
19:41
manipulated a large number of absentee ballots. The candidate there hasn't been certified yet, likely won't be seated on time. There's multiple investigations going on into exactly what happened. But it goes to show you that political fraud is a reality. And even outside the domain of computers, it continues to this day.
20:04
Now if you can imagine an election can be changed by just a few people working on the ground, going around collecting people's mail-in ballots and promising to return them for them. Well, imagine what nation state attackers could do to a vulnerable and highly computerized online infrastructure.
20:23
But on the whole, 2018 was, well, eerily quiet. But if we go back to 2016, so the US Senate Intelligence Committee, a bipartisan group controlled by Republicans in the Senate,
20:42
issued its report earlier this year about 2016. They pointed out that they found that in a number of the states where Russia attacked the registration systems, the Russian hackers were in a position to, at a minimum, alter or destroy the voter registration data. Which, if undetected, would have caused massive chaos on election day
21:03
when people showed up to vote and were told that they weren't on the election rolls. But those attackers chose not to pull the trigger. And I think that's exactly what happened in 2018. It was quiet, not because we've adequately secured our election systems, but
21:20
because our adversaries this year chose not to pull the trigger. They're waiting for the bigger prize in 2020 when we're likely to once again have a close and divisive presidential contest. So, what do I worry about?
21:40
What I worry about most is not the last war, registration systems, all of that, but the bigger prize, the 2020 election, and the vulnerabilities in the way that we cast and count votes in the US. Now, I testified about this in 2017 to the Senate Intelligence Committee, and that's actually not me, that's former FBI Director Comey.
22:05
But two weeks later, I was sitting in the same chair with far fewer TV cameras and testified that the real lesson of 2016 is that the threats are real and that the attackers will be back. And this is the picture I painted.
22:22
So, US voting machines have their own extreme set of vulnerabilities. I was going to bring one of these machines, an AccuVote TSX, with me here today. This machine is still used in many parts of the US, but my machine has been in Germany for about a week, and FedEx doesn't know where it is.
22:44
So, if it shows up, I'll have it somewhere for people to play with. But my advice is if you have to ship something urgent to Germany, don't send it via FedEx. What I would have shown you, though, is a mock election on this machine. And the mock election I always like to do to keep it from getting too political
23:03
is between George Washington, the father of the country, and Benedict Arnold, the traitor of the American Revolution. And of course, everyone likes to vote for George Washington, but these machines are so vulnerable. So I would have shown you an attack whereby I can compromise this machine and
23:24
cause it to report the wrong election outcome without having any direct physical access to the voting machines. Instead, all an attacker needs to do is be able to infect these memory cards that election officials use before every election to program the machine with the design of the ballot, that is, the races, the candidates,
23:43
the rules for counting. If an attacker can infect the memory card, there are a whole host of different ways that the attacker can compromise the machine and install malware on the voting machine itself. There's an unauthenticated software update mechanism that can replace
24:01
the election software. There are buffer overflows in the code that's used to read the ballot design and process it. Why there's even an interpreted programming language that's used to generate the reports of who won. So you can just replace the honest counting software with dishonest counting software right on the memory card.
24:21
And that's what will get executed and determine the election results. Any of these ways would be sufficient. So when the machine counts the votes at the end of the election, it prints out a little cash register receipt that becomes the official record of the result. That's controlled by the interpreted programming language on the memory card.
24:42
And on my machine, no matter who you vote for, Benedict Arnold is going to win. And that's because the malware I install via the memory card is in complete control of the election results. And there are more problems than that. So these voting machines, like the AccuVote TSX,
25:00
have been studied by academic researchers, by independent researchers, by groups commissioned by secretaries of state in various states around the country. And every time the same machine is studied again, groups find new vulnerabilities. This is part of the table of contents from a report I helped author ten years ago about the AccuVote TSX.
25:22
And you can see, just this is one page of several pages of vulnerabilities in this single machine. These things are so poorly designed, they're so complex. Each of the voting systems has on the order of a million lines of source code. And that's on top of, in this case, on top of an old and
25:41
unsupported version of Windows CE. There's no way that these things could possibly be secure. But the AccuVote TSX is still used in 18 states. In many of these states, it's still used with software that predates that 2007 report I just showed you.
26:00
We've had known buffer overflows and other problems in this firmware for more than ten years. And some states still have not updated the software. That's how bad it is. But it's not just that one machine. So in the US, every state gets to pick its own election technology. There are no federal rules that require states to do any particular kind of
26:25
technology or testing. And you might ask, especially from the European perspective, well, why don't we just count votes by hand like a civilized country? Well, here's part of the answer. This is one example of a ballot from one part of the country.
26:42
And it's eight pages long. We insist on voting for not only the federal races, but the state and local races, and even city races. The joke is even for dog catcher. And this complexity, well, the counting ballots by hand scales linearly with the number of questions.
27:02
And our ballots by tradition are just too complicated to efficiently count manually. So we turn to computers. And about half the country, well, really they're two different styles of voting machines that we use. Some of them are optical scanners, where the voter fills in a piece of paper and
27:23
it gets scanned in by a computer. The rest are touchscreen machines and others that we call DREs, direct recording electronic. On these machines, voters cast a vote on the screen. It gets recorded in electronic memory. Some of them will also generate a printout of each vote, but
27:41
that's relatively rare. In many cases, the only record of the vote is in a computer memory. So in study after study, these machines have been examined. And in every case, for both the optical scanners and the DREs, where a machine has been tested by qualified people, well,
28:02
it's been found to have vulnerabilities that would allow an attacker to install vote-stealing malware and change the electronic results, every single case. So how hard would it be to go from hacking these individual machines to,
28:20
say, changing the results of a presidential election? Unfortunately, much easier than we might think. There'd be three challenges to doing this in a way that would likely be invisible. The first challenge is that the machines are, well, many different types. They're diverse, they're decentralized. Each state system is independent.
28:42
And thank goodness, because that means that we don't have just a single place you can hack into to change results nationwide. Unfortunately, because of our electoral college system, this diversity of technology can turn into a weakness in very close elections.
29:01
So remember, I said that just any three of six states, for instance, in 2016, would have been sufficient to flip the outcome of the presidential election. Well, before an election, an attacker can scan all the states, figure out which ones are most weakly protected. And if they can find enough weakly protected ones to strike in,
29:23
that could be sufficient to change the national results. So the attacker gets to pick and choose, because our diversity of technology also means a diversity of strength and weakness. The second challenge is that, as election officials often point out, the voting machines aren't connected to the Internet.
29:42
Or at least they're not supposed to be. It turns out that some of them are, because they upload their results over a 4G cellular modem right after election results are complete. But let's just suppose they're not connected to the Internet. All right, turns out that's still not enough to protect us.
30:03
So as I said before, every election, every single voting machine in the country has to be programmed with the ballot design. And that ballot programming is created by election officials on a computer workstation somewhere, usually an old Windows PC. Those computer workstations can sometimes service an entire county,
30:23
sometimes an entire state. Sometimes they're controlled by independent external contractors that can perform work across multiple states. And if an attacker can infiltrate one of those systems, they can spread vote stealing malware on the memory cards to election,
30:40
to voting machines across the whole region. So how hard would it be to break into one of these systems? Well, in Michigan, my state, in 2016, about three-quarters of counties outsourced this programming to just three small businesses. These are 10, 20 person companies operating in strip malls and so forth. The same companies that the jurisdictions buy their ballot boxes and
31:04
I voted stickers from. Here's the website of one of them. You can see it doesn't have HTTPS. Has lots of nice high resolution photos of their warehouse, in case you want to burglarize it. And probably most interestingly to an attacker, they have this nice employee directory with everyone's name, photograph,
31:23
job title, and email address. So if I wanted to break into elections in Michigan, I might start by, say, forging an email from Larry the President there to sue his administrative assistant and say, I urgently need you to open this file. After she does, of course, it installs my malware on their network.
31:43
I'm in, I'm one step away from the election programming system and spreading malware to machines across a quarter of the state. All right, there's one more challenge. And that's that today, more than 70% of US votes are recorded on a piece of paper.
32:01
And this is great, this is much more than ten years ago. Because officials have been listening to computer scientists and security experts who have been warning about the dangers of fully electronic voting. Now paper might seem like a step backwards, but it's actually a pretty high tech way of thinking.
32:21
In any kind of critical system, if we can afford to have a physical fail safe in case of technology problems, it's a good idea to do that. This is why if you fly on a commercial aircraft, well it has a very fancy satellite guided navigation system. But also by law, there's a magnetic compass in the cockpit.
32:40
It's also why in your car, well you probably want to have a mechanical linkage between the brake pedal and the brakes, just in case, well you know. So paper can be a very sophisticated defense. It's relatively slow and expensive to tally, but it's something that's verified by the voter.
33:02
And that can't be changed later in a cyber attack. Meanwhile, we also get an electronic record from systems like optical scanners that's fast and cheap to tally, but unverified. As long as we make sure that these records agree, well then changing the election result would require you to change the electronic record through
33:21
a high tech attack, and the paper records through a low tech attack, and in a way that agrees, and that would require a truly extraordinary conspiracy. And to check that the paper is right, well we have high tech approaches to that too. You don't have to count all of it. In fact, over the last ten years, computer scientists and
33:43
statisticians have developed very sophisticated ways of just spot checking the paper record to make sure that it's right. And these are called risk limiting audits. A risk limiting audit is a statistical process in which you can count randomly selected ballots until you establish with high confidence that hand counting all of
34:02
them would determine the same winner. There are many ways to do this, but they all turn out to be, or many of them turn out to be, incredibly efficient. In a typical state with a fairly wide margin of victory, just spot checking a handful of ballots might be enough to establish with high
34:22
confidence that the winner really did win by a landslide. Of course, if the election result is a tie, logically you do have to look at all the ballots to establish that it is indeed a tie. So the amount of work you have to do depends on how close the election was. But in all cases, you can find an efficient approach to determining,
34:42
without trusting the computer systems, that the paper really does reflect the true winner. Unfortunately, well, most states don't do risk limiting audits. In fact, most states don't look at enough paper at all to determine that the winner of a close election was genuine.
35:02
So hacking a national election would probably be easier than most of us thought. You can use pre-election polls and scanning to determine which states to target. Hack into the election management systems in the most weekly protected ones. Then infect voting machines with malware to change, say, a few percent of the vote.
35:23
The paper records might catch the fraud, but you can rely on the fact that most states will throw it away without looking at enough of it to determine who actually won. And that's the sorry situation that unfortunately in 2018, we are still in.
35:40
So since 2016, however, there has been a change in mindset. Increasingly, election officials have been listening to the scientific community when we say you need a paper trail, and they're starting to think that that is correct. Almost all states that don't have paper trails today
36:00
at least have people strongly advocating for replacing the equipment that's there. And most other states, well, they at least have people starting to look into the security and testing the security of other election related computer systems like their voter registration systems to make sure that they're shored up.
36:21
Now you don't have to take it from me that paper ballots and post election audits are the way to go to secure our election systems. Just this fall, the National Academies of Science, Engineering, and Medicine, the authority on scientific advice to government, released a report with their highest level of advice,
36:41
a consensus report, urging the adoption of paper and risk limiting audits. Pointing out that this is a pragmatic, robust, and necessary defense for elections. This report was written in conjunction with election officials, people with experience administering elections.
37:00
And it just goes to show you that at least the election officials who have taken the time to understand the threat are waking up and starting to pay attention to the path to a solution. The problem is that that solution will take time to implement. And if we look at which states still don't have a paper trail,
37:22
turns out that there are 14 where some or all votes still aren't recorded on paper. And it's going to take between $130 and $420 million, according to credible estimates, to replace all the machines still in those states. Some of them, like Pennsylvania, are working to do that now.
37:41
But in other states, there still are no plans in effect to get rid of the vulnerable machines. If we look at the national map for post election audits though, the picture is a lot worse. And this is what concerns me most. Although many states in 2018 did small pilots of risk limiting audits,
38:03
the majority of states still do not conduct audits that can rigorously guarantee the electronic results of an election. And many still have no plans to do so in time for 2020. Because risk limiting audits are so efficient,
38:21
the cost for auditing nationwide is ridiculously small. It would cost, according to my estimates, less than $25 million a year to audit every federal race nationally, potentially a lot less than that. But it requires organization on the ground. And unfortunately in our system, operations on the ground
38:41
are conducted by about 13,000 local jurisdictions on election day. We need national leadership. We need much more dispersed expertise in order to get these protections in place. Because if you don't actually look at the paper, you might as well not have it in the first place.
39:03
So this year did see some movement in Congress. In the spring, as part of the omnibus appropriations process, Congress gave the states $380 million in emergency election funding in order to start working to secure their registration systems and polling places.
39:22
This was great in that it was money available immediately. And if you've been paying attention, getting Congress to do much of anything these days is pretty hard. On the other hand, the money came with very limited oversight with no standards about how that money should be used, and isn't even enough to eliminate
39:43
all of the paperless machines because of the way it's spread out amongst the states. But it's an important first step. We can look at a few of the states to see how they're doing. And I pick these as a representative sample of the diversity of progress.
40:01
In Maryland, for instance, which until 2016 used AccuVote touchscreen machines vulnerable to all of those problems I talked about. Finally replaced the machines with paper ballots. That's a huge step forward. Unfortunately, Maryland, instead of auditing them by having people
40:22
look at the ballots, decided it would be more efficient to audit them by having people look at digital scans of the ballots from the voting machines. As I think everyone in this room probably realizes, but maybe some in a broader audience would not. It's pretty easy to manipulate digital photographs.
40:44
In fact, I have work from students in an undergraduate security class I taught this term, who implemented a machine learning algorithm. That can take scans of ballots and just automatically change the marked results to produce whatever outcome you want.
41:02
And we'll have more on that in a publication this spring. But unfortunately, these audits are security theater. They might catch human error, but they're not going to catch a sophisticated attacker who has the ability to manipulate how the machines are reading the ballots, can be easily fooled by malware.
41:21
So I give Maryland on the whole maybe a C. Pennsylvania, another state that just two years ago during the recounts was practically a laughing stock of the country for its lack of paper records of votes and its Byzantine rules about recounting them. Well, today is making really good progress.
41:42
The state recently committed to replacing all of its paperless machines with paper ballots in time for the 2020 election. And it's committed to implementing robust post-election audits by 2022. Unfortunately, 2022 is going to be too late to secure the 2020 presidential
42:02
election, and this just emphasizes the need to get moving more quickly. There are also questions about whether the auditing regime they implement will be truly statistically rigorous. There are a lot of details to get right. But on the whole, Pennsylvania has made so much progress.
42:21
I think out of sympathy, I can give them a B. All right, now let's look at a top performer. This is the state of Colorado. Colorado has become a leader in election security, because not only does it have paper ballots statewide, largely vote by mail, which has its own problems, but that's a subject for later.
42:43
But Colorado also was the first state in the country to implement these statistically robust risk limiting audits statewide, and has been doing it since 2017. They've got both of these critical protections in place. And yes, they actually do choose the random seed for sampling the ballots during the risk limiting audit by rolling a set of ten
43:03
sided dice. So that's a great way to do it in a public ceremony. So Colorado gets an A. They're very well protected by these standards. Then there's Georgia. So Georgia, in 2018, voted statewide with the AccuVote TSX voting machine,
43:23
the one that FedEx has that I've hacked. They haven't updated the software in their AccuVote TSX machines since 2005. And they claim that the machines and their election programming systems are air-gapped.
43:40
But during a court hearing about this earlier this fall, their head of elections described that their system was air-gapped. Yes, it's perfectly secure, it's air-gapped. The only way you can get into it is through the bank of modems attached to it. It's air-gapped except the bank of modems. Also, it turns out he programs it by moving a USB stick back and
44:03
forth from his personal laptop. Georgia also, of course, doesn't have robust audits because, well, meaningful post-election audits would require a paper trail. And none of those machines have paper. This alone would be enough to give Georgia an F.
44:22
Except there's one more thing, their voter registration system also was shown in 2018 to have some problems. So you're not gonna believe this story. One more story. So in Georgia, they do online voter registrations through a website.
44:40
And in 2018, just a few days before the election, the Georgia Democratic Party learned from someone working for them, from a volunteer, about a series of vulnerabilities in this voter registration system. Well, it turned out that you could read and manipulate anyone's voter
45:01
registration records just by changing a sequential ID number in a particular URL. There was another URL for viewing a sample ballot that if you just changed the path of the file it pointed to, you could read any file in the server's file system. Well, these are pretty bad problems, right?
45:20
Even though Georgia apparently had gone through the process of having a security assessment of its registration system performed and didn't catch these. Well, so the Democrats, less than five days before the election, learned of these problems and disclosed them to the Secretary of State's office, which is responsible for running the election system.
45:42
There's Secretary of State Brian Kemp, who also, it turned out, was candidate for governor in a very close race. So not only was he running the election system, but he was the candidate in the most important race in the state, where the polls were projecting that the election was going to be a dead heat.
46:03
So an hour after receiving the security disclosure, Secretary Kemp's office put out a press release with this headline. That after a failed hacking attempt, they're launching an investigation into the Georgia Democratic Party, and they've called the FBI on the Democrats.
46:24
So Brian Kemp won the election and is now the governor-elect of Georgia. So this guy who did so well handling the security of the voting system while he was Secretary of State is now the head political officer of
46:41
the state of Georgia. I think Georgia's F just might stick with them through 2020. So- Thank you. So there is hope, though. I want to end on a message of hope, because despite this, with all of these
47:01
different levels of rigor and of readiness across the different states, I believe we need more national leadership, national standards, and national resources thrown into securing elections. And a bill to do just these things made a lot of progress in the Senate during the past term.
47:21
This is a bill called the Secure Elections Act that was introduced by Senators Lankford, Republican of Oklahoma, and Klobuchar, Democrat of Minnesota. And it ended up gathering a large number of bipartisan sponsors split evenly between Republicans and Democrats. It would have required states to adopt paper, to adopt strong audits,
47:44
and to adopt stronger information sharing practices to let each other and the federal government know if they saw signs of people trying to break in. This bill made it a long way, but unfortunately got stuck in the committee after some opposition from the White House,
48:03
just days before it was going to be marked up and hopefully then made its way to the floor. But this shows that bipartisan cooperation is possible, even in this Congress. And that there are a lot of serious people who now realize that election cyber security is a matter of national security and defense.
48:22
I think in the next Congress, there's a good possibility that we will see effective legislation to provide national standards and leadership for elections. But it's a question of threading a political needle and getting Congress to act. So to defend our elections, we don't need rocket science.
48:43
We need simple steps like applying security best practices and expertise to secure registration servers, adopting a paper record of every vote, and applying simple post-election audit techniques to make sure the paper record is right. If we do these things, well, we'll have a much more robust and
49:04
evidence-based election system that can detect and recover from attack attempts. Unfortunately today, our dialogue about elections isn't based on evidence. It's largely based on faith, on faith in the democratic process, on faith in the people and the technology that's responsible.
49:24
But I think voters deserve better. Voters deserve, if they're reasonably skeptical, to have it proven to them that the election result was right. And that is possible with simple and practical technology that we have today. All it's going to take is national leadership to make sure that all states,
49:42
even states like Georgia, adopt the necessary protection soon. So what can you do? Well, as a hacker or a computer scientist, you can work with your election officials to help explain the technology, the threats, and the defenses. You can work to explain the threats to the public,
50:02
because we all need to understand, just as a matter of modern civics, how elections can be attacked and defended. You can work to build better ways to use technology to make voting on paper easier and more efficient. Well, technology can help voting in a lot of ways. Just, we shouldn't trust it as the only way in which votes are counted and
50:23
results are determined. And as a citizen, well, you can demand that election authorities implement paper in risk limiting audits. Get involved through activist groups to help campaign for protections like this. And especially, please urge the US Congress to pass legislation like
50:42
the Secure Elections Act and similar bills to make sure that election systems across our country achieve these security properties. You can learn more from an online course I have for free on Coursera called Securing Digital Democracy that provides several weeks worth of material
51:01
about the history and the technology of election defenses. But we've got to get going. It's only been two years, believe it or not, since Donald Trump became president. And it's only about 22 months until the next presidential election. It's time to get moving. Thank you.
51:31
Thank you very much. What I got from this talk is it takes 27,400 people. So we have to scale up Congress.
51:43
We're going to do a Q&A. And I think we'll just start with mic number two because I can see that one. What if someone targets the- If not, we need mic number two live.
52:16
Try again. Okay, great.
52:25
Yes, so definitely you need to have secure randomness in whatever auditing method you're doing if it's going to be via statistical sampling. That's one reason why the auditing techniques that Colorado practices, they actually have a public ceremony in which officials throw dice in front of
52:44
TV cameras in order to pick the random seed. And, but a lot of thought has to go into designing that process well so that it's not only truly random, but also something that people can know and believe is truly random. Thank you. Okay, mic number six.
53:03
Thank you so much for the talk. You spoke about how in Georgia, the disclosure of vulnerabilities was punished almost. Is there any talk or movement towards having something like bug bounties for election systems?
53:22
Yes, in fact there's another bill that was introduced in Congress that would do just that and establish a kind of bug bounty program. I'm not sure that that idea yet has a lot of legs, but I think it would help. I think right now though, we don't really need all that much more incentive for
53:43
people to want to try to help secure democracy. A lot of people, including I'm sure a lot of people in this room, would gladly volunteer to do so. We need a way of organizing that effort and making sure that people can discover and report problems without fear of having it turn into some political weapon
54:00
to be used against them. Mic number one. Yeah, hey, thanks for the talk. Like the case in Georgia doesn't sound that terrible because like in Lithuania a couple of years ago we had this issue where you just didn't need to change the URL. You just did have to refresh the page and here you go.
54:21
You have the information about different citizens. My question is like, what if the paper trail leads to the knowledge that the election was rigged in some particular area? Like two years after the election or like one year after the election,
54:41
what happens then? Does it change anything? A year or so after an election would be a great catastrophe if we only learned then that the political leaders were not legitimately elected. We don't really have any precedent for that. That's why the recommendation and what some states like Colorado are starting to do
55:03
is they're implementing stronger audits is to make sure the audits are completed as soon as possible, ideally before the election result is certified. I recently came out with a paper with Philip Stark and Ron Rivest that gives an audit system that you can start doing even the moment polls close on election night and
55:23
perhaps have in a not so close election a full complete audit by the time results are announced on election night. So it's possible to do it quickly with sufficient organization. Okay, mic number eight.
55:40
Hi. I'm curious about the attribution of the attacks. Is there possibly any instance at which you would be not sure that it was Russia that performed the attacks or maybe it was China?
56:00
So how do you know that it is exactly Russia or China or India? Well, so all we have to go by really is the assertions of our intelligence agencies in the U.S. and in some cases like for the Democratic National Committee breaches the assertions of private security firms that were involved in the
56:22
investigations. And I agree with you, attribution in general is a darn hard problem. But if you're willing to accept the credibility of the intelligence reports and read between the lines just a little bit, it looks like the basis for their attribution is largely not technical but based on intercepted communication of
56:45
people who were involved in organizing the attacks in Russia. And I think more information about that is likely to come out as the Mueller investigations proceed. So I mean there's some necessary grain of salt and you can see what incentive people might have to try to trump up, so to speak,
57:03
the involvement of Russia. But you can also see in the current political climate why at least the executive branch would have a reason to try to tone down allegations of Russia's involvement. So you'll have to interpret the weight of the evidence as you will.
57:22
Okay, the last question from the internet. We're running out of time, sorry. Has any organization or group unveiled a voting machine designed to address all of the security issues that you have brought up here? Is there a solution to the problem? I'm sorry, could you repeat the beginning of that question? Has any group or organization unveiled a voting machine that is designed to
57:43
address all of those security issues that you have brought up? Okay, so there are efforts to develop voting machines that are based on open source software, that are based on better validated software. Ben Adida, a researcher in this area who has done a lot of great work,
58:05
is one person who's recently launched an effort to do that, although there are others. And I think that will help. But at the end of the day, I think however well designed the software in our voting machines is, that can raise the bar for attacks,
58:20
but it's never going to be enough to also be able to convince skeptical voters that everything is okay because, well, among other things, how do you know that that software is really what's running in the machines that are counting your votes? So there's a lot we can do to make voting machines better. At the end of the day, they're also going to have to have that paper trail
58:43
and those statistical audits so that everyone can believe the results. Thank you very much. That concludes the talk. Thank you. I think you'll be around for a few more answers on the Congress,
59:05
so everybody who's here can ask questions in person. And hopefully tomorrow there'll be a default voting machine somewhere around here for everyone to hack themselves. Thank you again. Let's hack that thing.