Hacking Driverless Vehicles

Video thumbnail (Frame 0) Video thumbnail (Frame 1179) Video thumbnail (Frame 2161) Video thumbnail (Frame 3516) Video thumbnail (Frame 6426) Video thumbnail (Frame 7881) Video thumbnail (Frame 9548) Video thumbnail (Frame 10756) Video thumbnail (Frame 12451) Video thumbnail (Frame 13736) Video thumbnail (Frame 15096) Video thumbnail (Frame 16871) Video thumbnail (Frame 19986) Video thumbnail (Frame 23236) Video thumbnail (Frame 25443) Video thumbnail (Frame 28481) Video thumbnail (Frame 29441) Video thumbnail (Frame 31896) Video thumbnail (Frame 33009) Video thumbnail (Frame 34668) Video thumbnail (Frame 36226) Video thumbnail (Frame 37331) Video thumbnail (Frame 38273) Video thumbnail (Frame 39396) Video thumbnail (Frame 40663) Video thumbnail (Frame 41606) Video thumbnail (Frame 43456) Video thumbnail (Frame 44586) Video thumbnail (Frame 45823) Video thumbnail (Frame 47954) Video thumbnail (Frame 49358) Video thumbnail (Frame 50311) Video thumbnail (Frame 51661) Video thumbnail (Frame 52708) Video thumbnail (Frame 53793) Video thumbnail (Frame 54759) Video thumbnail (Frame 55788) Video thumbnail (Frame 56821) Video thumbnail (Frame 59071) Video thumbnail (Frame 60943) Video thumbnail (Frame 63053) Video thumbnail (Frame 64278) Video thumbnail (Frame 65376) Video thumbnail (Frame 66828) Video thumbnail (Frame 68131) Video thumbnail (Frame 69213) Video thumbnail (Frame 70518) Video thumbnail (Frame 71469) Video thumbnail (Frame 72786)
Video in TIB AV-Portal: Hacking Driverless Vehicles

Formal Metadata

Title
Hacking Driverless Vehicles
Title of Series
Author
Zoz
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2013
Language
English

Content Metadata

Subject Area
Abstract
Are driverless vehicles ripe for the hacking? Autonomous and unmanned systems are already patrolling our skies and oceans and being tested on our streets and highways. All trends indicate these systems are at an inflection point that will show them rapidly becoming commonplace. It is therefore a salient time for a discussion of the capabilities and potential vulnerabilities of these systems. This session will be an informative and light-hearted look at the current state of civil driverless vehicles and what hackers or miscreants might do to mess with them. Topics covered will include common sensors, decision profiles and their potential failure modes that could be exploited. With this talk Zoz aims to both inspire unmanned vehicle fans to think about robustness to adversarial and malicious scenarios, and to give the paranoid false hope of resisting the robot revolution. He will also present details of how students can get involved in the ultimate sports events for robot hacking, the autonomous vehicle competitions. Zoz is a robotics interface designer and rapid prototyping specialist. He is a co-founder of Cannytrophic Design in Boston and CTO of BlueSky in San Francisco. As co-host of the Discovery Channel show 'Prototype This!' he pioneered urban pizza delivery with robotic vehicles, including the first autonomous crossing of an active highway bridge in the USA, and airborne delivery of life preservers at sea from an autonomous aircraft. He also hosts the annual AUVSI Foundation student autonomous robot competitions such as Roboboat and Robosub.

Related Material

Prototype Process (computing) Demo (music) Very-high-bit-rate digital subscriber line Robotics Autonomic computing Video game Iteration Physical system Spacetime
Beat (acoustics) Student's t-distribution Device driver Bit Student's t-test Distance Prototype Word Software Bridging (networking) Robotics Universe (mathematics) System programming Autonomic computing Energy level Local ring Spacetime
Information overload Multiplication sign Decision theory Device driver Water vapor Food energy Surface of revolution Cartesian coordinate system Surface of revolution Hand fan Carry (arithmetic) Robotics Energy conversion efficiency Operator (mathematics) Data conversion Physical system
Point (geometry) Slide rule Presentation of a group Mathematical analysis Surface of revolution Subset Process (computing) Robotics Hacker (term) Different (Kate Ryan album) Internetworking System programming Autonomic computing Right angle Data conversion Quicksort Information security Vulnerability (computing)
Game controller Direction (geometry) Device driver Cartesian coordinate system Exploit (computer security) Field (computer science) Number Arithmetic mean Whiteboard Robotics Personal digital assistant System programming Speech synthesis Whiteboard Information security Metropolitan area network Physical system Vulnerability (computing) Spacetime
Domain name Curve Mathematics Potenz <Mathematik> Different (Kate Ryan album) Operator (mathematics) Right angle Nichtlineares Gleichungssystem Information security
Mapping Transportation theory (mathematics) Right angle Transportation theory (mathematics)
Domain name Group action Logistic distribution Civil engineering 1 (number) Information privacy Cartesian coordinate system Sphere Field (computer science) Entire function Subset Robotics Right angle Transportation theory (mathematics) Physical system
Prototype Crash (computing) Execution unit Videoconferencing Oscillation Spacetime 2 (number)
Expected value Impulse response Uniformer Raum Different (Kate Ryan album) Software cracking Boundary value problem Right angle Exploit (computer security) Oscillation Physical system Systems engineering 2 (number)
Area Curve Mapping Angle Multiplication sign Videoconferencing Routing
State observer Functional (mathematics) Game controller Greatest element State of matter Mereology Computer programming Neuroinformatik Bit rate Hacker (term) Robotics Hierarchy Logic Energy level Office suite Data structure Traffic reporting Control system Exception handling Stability theory Task (computing) Logical constant Information Key (cryptography) Inheritance (object-oriented programming) Decision theory Counting Software maintenance Exploit (computer security) Nernstscher Wärmesatz Hierarchy Process (computing) Estimation Logic Right angle Energy level Collision Cycle (graph theory) Data structure Task (computing) Local ring ARPANET
Dynamical system Group action System call State of matter Multiplication sign Decision theory Fluid statics Mathematics Bit rate Single-precision floating-point format Logic Stability theory Control system Physical system Mapping Simultaneous localization and mapping Principal ideal domain Complete metric space Order (biology) Chain output Data structure Task (computing) Navigation Spacetime Point (geometry) Principal ideal domain Virtual machine Programmschleife Robotics Hierarchy Energy level Condition number Task (computing) Form (programming) Shift operator Graph (mathematics) Weight Punktschätzung State of matter Planning Plastikkarte Software maintenance Sphere Logic Gravitation Finite-state machine Collision Routing Local ring
Point (geometry) Axiom of choice State observer Complex (psychology) Group action Greatest element State of matter View (database) Robot Direction (geometry) Decision theory Multiplication sign Compass (drafting) Function (mathematics) Login Software bug Wave Doppler-Effekt Mechanism design Term (mathematics) Robotics Logic Codierung <Programmierung> Logic gate Condition number Task (computing) Vulnerability (computing) Graph (mathematics) Compass (drafting) Machine vision Digitizing State of matter Inertialsystem Exploit (computer security) Vector potential Wave Data management Radio-frequency identification Linearization output Condition number Right angle Arithmetic progression Data structure Local ring Resultant Maß <Mathematik>
Addition Noise (electronics) Suite (music) Source code Model theory Execution unit Compass (drafting) Digital signal Bit rate Distance Timestamp Wave Estimator Velocity Personal digital assistant Robotics Single-precision floating-point format Noise Position operator Tunis Pressure
Domain name Axiom of choice Inference Asynchronous Transfer Mode Information Robotics Multiplication sign Information retrieval Asynchronous Transfer Mode
Satellite Point (geometry) Noise (electronics) Trail Group action Dot product Hoax Electric generator Function (mathematics) Data transmission Number Power (physics) Frequency Order (biology) Videoconferencing Website Position operator Physical system
Model theory Electronic visual display Control flow Physical system
Filter <Stochastik> Existential quantification Login
Vapor barrier Robotics Multiplication sign Automation Right angle Measurement Value-added network
Addition Mapping Information Surface Reflection (mathematics) Orientation (vector space) Range (statistics) Function (mathematics) Distance Robotics Point cloud Configuration space Right angle Collision Quicksort
Frequency Robotics Range (statistics) Horizon Right angle
Multiplication Matter wave Reflection (mathematics) Electronic mailing list Water vapor Limit (category theory) Angle Different (Kate Ryan album) Robotics Internetworking Hash function Video game Routing Spacetime
Existential quantification Hoax Mapping Quantum state Information Tesselation Reflection (mathematics) Machine vision Horizon Line (geometry) Laser Graph coloring Message passing Robotics Konturfindung Videoconferencing Right angle Pattern language Object (grammar)
Medical imaging Overhead (computing) Sign (mathematics) Sine Robotics Fuzzy logic Collision Cartesian coordinate system Pressure Form (programming)
Robotics Model theory Model theory Compass (drafting) System programming 1 (number) Total S.A. Inertialsystem Function (mathematics) Distance Maß <Mathematik> Physical system
Rotation Information Compass (drafting) Model theory Compass (drafting) Physicalism Counting Distance Mereology Field (computer science) Vector potential Cumulant Goodness of fit Component-based software engineering Error message Robotics System programming Right angle Whiteboard Codierung <Programmierung> Error message Physical system
Point (geometry) Predictability Causality Bridging (networking) Surface Codierung <Programmierung> Diameter
Robot Particle system Robotics Multiplication sign Vapor Vapor Right angle Bit Codierung <Programmierung> Laser Absolute value Local ring
Mapping Simultaneous localization and mapping Robotics Videoconferencing Right angle Mereology
Point (geometry) Fluid statics Pattern recognition Sign (mathematics) Mapping Robotics Structural load Network topology Single-precision floating-point format Machine vision Real-time operating system Vulnerability (computing)
Point (geometry) Robot Dynamical system Meta element Mapping State of matter Multiplication sign Reflection (mathematics) Device driver Grass (card game) Data transmission Graph coloring Rule of inference Data transmission Fluid statics Bit rate Robotics Network topology Right angle Routing Condition number
Rule of inference Standard deviation Simulation Logical constant Mapping Information Real number Cellular automaton Real-time operating system Mereology Data transmission Rule of inference Exploit (computer security) Subset Independence (probability theory) Software Robotics Videoconferencing Software testing Right angle Intercept theorem Local ring Local ring Form (programming)
Transportation theory (mathematics) Multiplication sign View (database) Device driver Spektrum <Mathematik> Revision control Causality Robotics Logic Data structure Vulnerability (computing) Modem Mapping Block (periodic table) Compass (drafting) Physicalism Exploit (computer security) Personal digital assistant Logic Order (biology) Configuration space Right angle Whiteboard Data structure Asynchronous Transfer Mode Geometry
Robot Hoax Mapping Robotics State of matter Artificial neural network Real number Collision Task (computing) Force Task (computing)
Noise (electronics) Standard deviation Overhead (computing) Mapping Reflection (mathematics) Materialization (paranormal) Sign (mathematics) Arithmetic mean Term (mathematics) Robotics Aerodynamics Collision Local ring
Slide rule Envelope (mathematics) Robotics Robot Student's t-distribution Order (biology) Autonomic computing Right angle Hacker (term) Mereology
Beat (acoustics) Multitier architecture Dot product Mapping INTEGRAL Workstation <Musikinstrument> Computer network Water vapor Image registration Perturbation theory Canonical ensemble Cryptography Code Symbol table Serviceroboter Uniform resource locator Software Bit rate Robotics Flag Right angle Information security Task (computing) Task (computing)
Rule of inference Pattern recognition Freeware Remote administration Multiplication sign Data recovery Rule of inference Band matrix Data management Shooting method Hacker (term) Object (grammar) Telecommunication Object (grammar)
NP-hard Rule of inference Student's t-distribution Robot Planning Limit (category theory) Limit (category theory) Rule of inference Dimensional analysis Hacker (term) Robotics Hausdorff dimension Right angle Hacker (term)
Robotics Surface of revolution
I'm Czar's this is hacking driverless vehicles quick stealth intro my academic background is autonomous robots but I'm probably best known for a little cable TV show I did with kingpin called
prototype this we did some autonomous robot stuff tried to hack together some cool demos basically so for example we did a UAV that would protect unguarded
beaches so if you'd press a button if you're out in the ocean in trouble it would fly out and drop you a life preserver sometimes it also dropped on the ground itself we learned a lot about unmanned air systems on that on that episode so and a lot about iteration in the design process we also turned our
hand to driverless ground vehicles for one of the most pressing technological challenges in the ground space which is maintaining the American lead in high-speed pizza-delivery so here's the
solution for local delivery sharing routing space with pedestrians and here's the long distance method
operating within the shared automobile network so this shot right here this was the first-ever autonomous crossing of u.s. highway bridge can be hard to convince human delivery drivers to make this trip so screw them let's let the robots do it since prototype this one of
the things I've been doing is hosting autonomous vehicle competitions for rubber nation and the OVS a foundation these are student competitions university level and below ground vehicle competition to air vehicle competitions one for boats and one for submarines and I'll say a little bit like about more about this at the end but I want to say a few words at the
start here about the motivation for this talk I'm a huge fan of unmanned vehicles I love robots I think the other future and they are definitely coming because there are so many advantages energy efficiency not having to carry a human driver not having to carry food and water bathrooms or go out of its way to acquire them same with time efficiency not having to deal with fatigue boredom taking rest stops operator changeovers and so on and all the new applications that are going to be enabled where it wasn't practical to do it with a human driver so the revolution is coming you
can't stop it even if you want to you it's here but but like everything else that humans have ever made these systems are going to be hacked so I want to stop that conversation now when I start talking about it before the systems are too entrenched for us to go back on decisions so what I'm definitely not
trying to do here is spread FUD fear uncertainty and doubt all right so it's not going to be some kind of alarmist
anti robot propaganda you like that one how about this one
that's not it at all it's not every presentation where I get to make to Hitler jokes on the same same slide I couldn't resist hope it's not too soon but I think that
the you know this revolution is coming and I really got a sense of this recently this is footage I shot a year ago at suas this is a vehicle called fire scout it's a autonomous unmanned robot helicopter and it's doing a take-off run there and it just looks like it's sitting still on the ground right it's so stable and you look at that and you think yeah that's that's how to do it let's let the robot drive so I'm not trying to interfere with this process at all this is my point DEFCON is not a security conference per se right it's a hacking conference a hacking convention and to me that's hacking in the old-school sense right figuring out how things work what's wrong with them if there's stuff wrong learn how to improve on them so this talk is about getting people excited about contributing to that conversation the analysis discussion design and the exemplar eventual acceptance of driverless vehicles right so you can think of this as a recruitment talk it's sort of like general Alexander's NSA keynote last year the big difference of course being that I don't care I don't want to see what you FAP to on the Internet all right you can you can keep that to yourself so we're going to talk about vulnerabilities and interfering with driverless vehicles but it's out of love it's kind of tough love right it's kind of like when you teach people to swim the Australian Way by putting in the pool and throwing in a crocodile this this is true this is how we all learn it
speaking of managing vulnerabilities right so there's the fire scout it's very stable but notice the people operating that robot setting it off a hiding behind the stock cut down here it just in case anything goes wrong that's a security mindset so when I talk about exploits and countermeasures I want you guys to think about counter countermeasures right stuff that I say well okay here's how we design around that here's how we'd fix that problem increase the robustness of the system
all right so I'm man system space is kind of wide means that basically is no human driver or pilot on board but it
doesn't necessary really mean that you don't have someone off board controlling it or supervising it for an unmanned system not actually I'll Thomas you could have supervised the autonomy you might also have a safety pilot on board or it might be carrying passengers so there might be people on there but they just don't have a direct control role of course the military early adopters most of this field has been dominated by military spending and military applications for a number of reasons that I'm sure that you can guess many of them and a pretty
significant amount of the uptake has been the airspace so for example this is Global Hawk if you look at Global Hawk
flight hours it's looking pretty much like an exponential curve right and that's because it works well there's a lot of simplifications that apply to the airspace but they really want to push these changes down to the other domains so 12 years ago Congress insisted that one-third of all operation operational ground vehicles were to be unmanned by 2015 now Congress can put up the money and say whatever the hell it likes it doesn't mean they're going to get it right and we're pretty clearly not going to make that 2015 deadline but the will is there but I'm not going to talk a lot about military vehicles in this talk because the capabilities are largely classified so it's very speculative they already have a really active interest in resistance to adversarial engagements right so that you already think a lot about the security stuff there's a different cost equation they have the highest quality sensors and of course hopefully most of us will never encounter one unless Edward Snowden is in the audience but hopefully we will not encounter one of these here's just a
quick example of a military ugv a driverless ground vehicle specifically designed with threats in mind right you can see a lot of sensors design it looking for people it's got weapons on board you just have to get close enough to this thing to press one of these kill switches it's it's it's kind of like putting an unshielded reactor exhaust port on a Death Star right it's just going to leave that presumably they'll remember to remove those in the final version but
let's start thinking more about what where these things are going to show up in our backyards so we've got transportation nothing bad about about these guys by the way we love these guys their friends not singling them out but they're doing a pioneering stuff in transportation oceanography terrestrial mapping
filmmaking is big right if you go to go see the film filmmaking conferences you see UAVs everywhere some of the weird esoteric ones you wouldn't necessarily think of like powerline inspection with UAVs and of course logistics like the pizza delivery lots lots more and there
are two main priorities that the industry advocate advocacy group has for unmanned systems in the civil sphere precision agriculture actually a lot of
combine harvesters now are practically mobile robots right because they can do optimized paths over the fields and so on and secondly self-driving cars this is big right that wide applicability over the entire country roadblocks that I've taken the civil domain shared infrastructure right they have to share this stuff with humans it's much tougher if the robots have to interoperate with humans and hand-in-hand with that goes the acceptance that people have like do they trust the safety and robustness so you don't have to just demonstrate that but you've got to convince the public give them a perception of it also privacy is important so now that we're talking about safety and robustness the fun stuff here is failure let's let's take a look at that so here's a couple
of classic failures first of all a UAV
failure this is the RQ 3 darkstar there's a surviving example in the Smithsonian Air and Space in Washington DC it was supposed to cost 10 million dollars per unit ultimately right units 11 to 20 and 1994 dollars so the first couple were very expensive the first prototype failed on its second flight test on its second takeoff actually now I've seen the video of this crash with my own eyes but I was not able to obtain it for you guys I apologize for that so you just got have to imagine what happens here he said he's a non crash iq3 takeoff so just extrapolate from that and I'm at if it didn't take off like that it's coming down the runway and it starts to wobble up and down and oscillate and these oscillations get more and more and eventually it pitches nose up and comes down hard and it's a huge fireball and it's millions of dollars down the drain there's the this
is the quote from the Journal of unmanned systems engineering about what happened here's what as I was as it was explained to me by my researchers involved this is what happened they had modeled the takeoff run with a flight control system on an asphalt runway the second flight takeoff was on a prefabricated concrete runway and so those gaps those cracks between the prefabricated concrete panels will underdamped and they set up this oscillation that eventually caused the failure of the vehicle right so just those small impulses to the inertial sensors so the moral of that story is that the expectations of the designers are critical even even a seemingly trivial detail like the runway composition can mean the difference between success and failure of these systems and so if there's going to be exploitation there's a good chance that it's going to happen at these cracks between the boundaries of the designers expectations here's a second example
this was the favorite vehicle to win the first DARPA Grand Challenge back in 2004
this is the desert race from Los Angeles to Las Vegas fully autonomous this vehicle was done by a CMU off sheet called offshoot called Red Team Racing and it got a few miles seven miles or
something like that before it took a hairpin turn wrong and ran off the side of the road its engine caught fire and it was all over well we wait for that video to catch up let's see if we can
play both of them at the same time Hey look at that there it is failed what went wrong apart from the fact that these legends don't like running on weird angles they had a huge team this team and they had extensively mapped the course beforehand even though they only got the final route two hours beforehand they knew every Road in the area - very precise like that people walking that course with GPS receivers and one of the people on that team told me that their map was so good they could have just about made it on map data alone and their big problem was they paid too much attention into their other senses if they just ignored the laser rangefinder on that curve they would have got through just fine so the moral of the story there is
the robot faces a constant battle of deciding what it knows best what information is reliable and what isn't so correct correctly estimating the state that you can't observe that hidden state is the key to all decision-making under uncertainty so hacks and exploits have some of the best chances of succeeding if they subvert or undermine that state estimation process so let's take a look
at some of these logic structures at a higher level just like with humans we can think about behavioral logic of a robot in a hierarchical fashion so at the bottom we have the control loops stability maintenance this stuff run typically runs independently at a high cycle rate it might run on a completely different computer from the other stuff and so sometimes I'll see a robot that's completely crashed it's not doing a damn thing but it's maintaining perfect stability in the water or in the air it's just hanging out there above that you might see something that collision avoidance right that's preservation of the robot it's kind of like you know Asimov's third law I think taking precedence over everything except low-level control then above that navigation and localization all right so that's part of the mission it might even be all of the mission if it's just a navigation mission and then above that high level mission task planners reason is stuff like that so what can we take away from that arrangement well first of all there's an implicit dependency right so if you attack at lower level in the hierarchy you can defeat everything above it if the robot can't maintain functionality there then it can't definitely count at the higher levels so I like to think of it you know for those people who have office jobs no one thinks about filling out their TPS report while they're actually being kicked in the balls okay you can go back to work and try that and see if it's true but secondly more engineering effort might have been spent programming or guaranteeing robustness at those lower levels stability super important but getting lost once in a while you might be able to recover from that so the lower layers might be juicier attack targets but they also might be better defended and harder to find bugs in
so a couple of examples that I mentioned before from how to take this just looking at the way things were arranged the life-saving drone had a autopilot that did all the stability maintenance has low-level control loops for all of the basic air worthiness the tunable for the different environmental conditions you might expect to encounter nothing in the way of collision avoidance this is missing from just about all UAVs and that's one of the things that needs to come being implemented and designed in order for that shared space arrangement to happen and there be shared space approval of UAVs navigational localization is GPS based that has waypoints and this involves also control loops PID loops controlling how the aircraft approaches those waypoints and what it weather when it discovers when it decides whether it's hit won or not or whether it should go back around for another try and then at the top we had our kind of bombing run planner that would actually set up a temporary Waypoint they give it the best possible approach path so it could do that input impact point estimation so the system of course fully vulnerable to collision because there was no effort to not be and the high level logic depends on one single sensor the GPS so that single point of failure is a big vulnerability and they're actually really common in the robot sphere local pizza delivery has to do has to have all kinds of control loops for stability maintenance because it's balancing on two wheels and it's got to do weight shifting for when the pizza gets removed and the center of gravity changes lots and lots of collision avoidance that's pretty much almost everything that the system does the main strength is dealing with those dynamic obstacles they're not on the map at the high level navigation by route route planning from a map that's pre generated using simultaneous localization and mapping so that's where the discrimination between static and dynamic obstacles happens and then the high level task is again a simple one it's just to dispense the correct pizza when the correct credit card gets given to it so this kind of system is vulnerable to redirection trapping and map confusion attacks rate all those things that that attack where the robot thinks it is and of course you can
always try and get the pizza if you didn't pay for it you know so there's the the mission stuff so now that we're thinking about that logic hierarchy let's look more generally at the upper end of the hierarchy in what kind of logics going on there this the that usually some form of state machine represents the mission and what eventualities that the designers envisaged so the robot considers itself to be in some state and then each decision point can either stay in that state or transition to a new state and eventually you get a directed graph of all these states they define what logic the robot runs at any given time so these states may correspond to tasks and the transitions may be tasks completions or they might be contact switches by caused by things like priority shifts they might just be simple timeouts anything like that and as represented here they may into the selves contain subordinate states and reasoner's and planners and so on and so forth people watching you know if this looks kind of like a Markov chain you know there's a good reason for that these feature a lot in robot control systems and the thing to be aware of though with with these state machines is that the machine isn't the state machine is not necessarily deterministic right just because we think we're in a state doesn't mean that we're actually there and so there's this hidden state that the robot has to the mirror can't necessarily observe and has to figure out and that's where things get tricky so to put some put some like labels on this stuff and not have it be
totally abstract here's the Robo sub mission because it's of chosen this because it's kind of simple kind of linear and it's easily broken down into these various mission states so first the sub has to navigate through a start gate and this is often done open-loop right you just point it in the right direction and drive it for a certain amount of time but then you have to start making decisions the subs got to start looking for a buoy and then trying to touch it so it's got to decide you know can it see the buoy has it touched it yet should we try again so you got some choices and then you've got to start looking for a path on the bottom of the pool and you've got more choices once you find it all the different subtasks you might want to do the obstacle course identifying and these targets and dropping markers on them finding the torpedo targets and firing torpedoes through them and then it's an underwater manipulation task so you've got to determine the data that in terms of finding it and then also whether you've managed to complete that task we've managed to make any progress on it you can time out from all those tasks right so a lot of these a vision gated but at a certain point you can transition from anywhere in the graph so the final thing because it's it uses a completely different sensor the hydrophone like a localization task so you've got to find this pinger and go and retrieve a package so looking at this from the point of view of second-guessing the designers where are the vulnerabilities and potential exploits oh there's the package they may be in the state estimation right what does the robot think it's trying to do versus where actually is it the transition between states can we spoof them or prevent them from occurring or are there bugs in the state's themselves just unexpected conditions or results and here's a key thing when rope when designers watch the robot in action they don't necessarily know even though they programmed the whole thing why it's doing what it's doing they can only kind of guess from that output right because there's so much complexity going into a single output so until you see the logs you don't necessarily know so the would be exploiter also has to put themselves in the mind of the designer and think about what what might they have been thinking about and what might they have got wrong but I don't want to talk about
attacks that would work on any vehicle even human driven vehicles because there's no point in that like digging a big pit and perfectly camouflaging it so I want to talk about relevant physical attacks so attacks on the robots input mechanisms so the input mechanisms are the sensors they can be active or passive which there's the important distinction between that we will cover as we go and some common examples are GPS of course for terrestrial vehicles laser rangefinder cameras millimeter wave radar is there at the back a digital compass and inertial management measurement unit are the wheel encoders and then for the specialty vehicles like the subs we have things like Doppler
velocity loggers scanning sonar pressure transducers for the air and subsurface like barometric altimeters for example in addition to all of this there's the map and we'll definitely talk about that too so sensors don't give a perfect
picture of the world just the best guess and you've got plenty of sources of uncertainty we all know about noise of
course it's a constant battle associated with noises drift latency and update rate coming to play so when we were doing the life-saving drone we had one Hertz update read on the GPS so that meant we know with an unknown time stamp on that so that meant that the vehicle could be anywhere within like a 70 meter distance when we got that position estimation so getting a participant position estimation that could be not valid anymore by the to the tune of up to 70 meters that's hard to do a five meter precision bombing run and you have to model these uncertainties under various assumptions so you need to know what the underlying noise models of the sensors are so a lot of people don't use GPS because they don't they can't get from individual units what the noise model of the GPS is so they can't develop a noise model that's super reliable you may think that fusing sensors together might be more useful than a single sensor and that's true in many cases using sensors and registering it together can be a lot more useful than taking the separate sensors but what do you do when the sensors disagree so which one do you trust and how much so the robot the robustness of the robot in the end may come down to how smart is it at discounting one single bad or spoofed sensor even though it might have a whole suite of sensors onboard so let's look
at sensor attacks two basic kinds denial
basically preventing the sensor from recovering any useful data and then spoofing causing the sensor to retrieve information that's specifically incorrect that the attacker wants it to retrieve and then you've got a basic attack mode choice here you can either directly attack the sensors so give them instantaneous bad data or you can try and mess with the aggregated sensor data they are accumulating over time so they lead the robot into long-term poor inferences so I'm going to quickly go through most of the common sensors here and leave the specialized domain sensors for another time GPS we know is a major
reference for vehicles that have access to it in the atmosphere denial is straight-up jamming you can buy a GPS
jammer from a number of sketchy Chinese websites you can also find plans for
them online if you want to build your own and it's basically throwing a big bucket of RF noise at the GPS frequencies alright these frequencies the transmissions a week from the satellite so you just over you can also spoof them you can you need to generate fake GPS satellite signals at a higher power than the satellites themselves in order to override the receiver and this has been demonstrated
with UAVs this is a group from UT Austin
demonstrating taking over a UAV using GPS so the attacker on the Left is broadcasting a GPS signal that is of a higher power and first is aligning to the GPS signal that the UAV is really receiving and those three dots are matched filter trackers on that output that's finding that peak and then you start to move your signal off and you take the tracking points with you and so now convincing the UAV that it's in a position that it's not really in and it's flight control system is trying to correct for that and moving it somewhere else so here's an example here's some video of doing it to a real UAV that
going to take control over over it and convince it via GPS that it is moving upwards at a certain speed and the flight control system is going to try and account for that and move the helicopter down and you can see if the safety pilot doesn't take over here you could drive it straight into the ground
this technique was claimed to be used by the Iranians when they brought down in our q7 surveillance UAV there's a
picture of it captured on display in Tehran
they said they spoofed it into the ground this is widely believed to be a model because the original crashed but seems unlikely to me because the military systems use encrypted GPS and the military don't rely on GPS for UAVs because there are jamming and less likely spoofing for military GPS but we do know that for civilian systems it's easily jammed due to that weak amplitude of the real signals and easily spoofed I
mean so in the civilian realm the GPS is used as a primary sensor unlike the military you'll take the GPS and maybe resolve the last couple of meters with your laser rangefinders so it's important to point out here that those sensors all have filters on them right so that the GPS will drag the vehicle slowly from ensure incorrect correct trajectory other than just snapping it off its path so here's some grand challenge flus from
2005 again without access to the logs it's hard to know exactly what's happening here but it looks like because there's nothing on the road it looks like we're seeing some GPS drift on off the road that's then being corrected by the laser rangefinders so that that's typical of what you might see there's something's relying on GPS here's
another DARPA Grand Challenge example looks like a GPS guided run that's drifting off the road here who knew that a van could drive over a jersey barrier something maybe for the next DEFCON Cannonball Run to keep in mind okay next
up laser rangefinder or lidar it's basically a scanning laser rangefinder
originally sent a sensor for industrial automation but then the robot guys got hold of them like oh this is awesome right so we're going to start using these outside so mechanically scanned by a rotating mirror and they essentially measure time of flight right so it's an active sensor it depends on the return
signal coming back these are primarily used for collision avoidance and map making I don't know why I do these bullets because I always forget to advance them they return a point cloud of reflected distances within the laser range so you can denial due denial on them by actively overpowering them or by preventing a return signal with dust smoke mist that kind of stuff you can also spoof them by manipulating the surface absorbance of the same things that they're looking for so basically manipulating absorbance and reflectivity to give the receiver specifically incorrect information about what it's looking at ladar often a 2d sensor so it's highly
orientation dependent depending on how its mounted right so it's often mounted in a push broom configuration looking down at the ground in front what that means is you're looking for obstacles really nicely but if the ground slopes up it can just look like a brick wall so you can see here that some output this is from pizza pizza delivery the on the left is the lidar output as the robot comes along and you can see that basically you know you get these ranges stopping when you see an obstacle and if you look over there to the you know as it as it sort of turns off to the right where the street slopes up you can see that range just drop away to nothing it kind of looks obstacle like in addition to that if it's shooting down over the top of the low obstacle or particularly a discontinuity like a curb then it can miss it entirely right so it can can fall in a ditch if the if the ditches orientated right active emission sensor
so it only returns that active signal back to the receiver so no return means that it assumes that nothing is there think about that over the horizon and out of range returns no signal so most of the world returns no data so what that means is the things that absorbing the laser frequencies look exactly like nothing and things that the transparent in the laser frequencies also look like nothing so if you were to paint an absorbent tunnel on a wall right it's
just like wily coyote the robot would not see that similarly if you were to
make obstacles out of glass it sees right through them now of course glass is transparent but also reflective so there's a limit to this it might miss a bottle but it's probably going to see a cowboy some new ally does also have this multi echo suppression so that if you have glass that's really close to them it's definitely designed to ignore them so it doesn't get confused by glass over the sensor it's all about what gets returned to the sensor so reflective things confuse it for example a puddle on the road it's very reflective and they can make faraway things like obstacles look near if the angles match up right so this is something that the robot actually has to deal with or if there's nothing to be reflected the signal goes out into space and that lack of return of the puddle makes it look like a big hole in the road not just water in puddles even fresh asphalt can not give a return and look like a big big hole on the road one of the DARPA Grand Challenge vehicles I went once say which one ran into a brand new black SUV just plowed into it because it was so shiny and reflective and recently washed that it looked like nothing so even a new car can be a problem that's what the millimeter wave radar is for so you can definitely use reflective surfaces to make things look like a ditch the vehicle won't consider safe to drive over and will have to take a different route people out there bad
guys out there have have good reason to try and use these kinds of techniques this is just something for fun I found this in my travels on the Internet this is a documents that were from al Qaeda in the Arabian Peninsula that were captured with an al Qaeda offshoot in Timbuktu in Mali and you have to trust me what it says unless you read Arabic but this is item 2 it's referring to a Russian made rock I'll GPS jammer this is all this is a document on how to avoid UAVs and drone strikes and then item 3 advises them to place reflective plates on their vehicles to reflect off the laser designator just to make the missile have to miss slightly right that could be the difference in life or death of course for this to work you need a material that was reflective at the laser wavelength so who knows whether these guys can make it work on that but it's on the list of technics laser
reflectance is also a feature the road motion elite mostly gives a decent return unless it's that's fresh asphalt but the white lines are quite reflective so they look like gaps in the road so they actually do this use this to do Road line detection so a fun consequence of this is that you could make fake road markings in a way that's invisible to the human like black on black but the robot is going to see them perfectly so you could paint some black on black swarthy lines for example the human does not know why the robot is swerving all over the road well you can even you know try to be a little more kind about this and leave the lines as they are but do your black on black has hidden messages for the humans back at base when they go and look at the map cameras
are used as well but not as much as you might think because vision is really hard specialized object detection sometimes sometimes stereos used to get a depth map but it's noisy often what people will do is colorize the lidar data with the cameras right so you're registering your laser rays fighter data to this color information from your camera and why isn't this video going come on come on all right here we go so this is stop a grand challenge stuff from o-5 this is Stanley the way that Stanley drove so fast is that it used its lasers to get an idea of what was road and what wasn't road and then it used the camera information to match colors and say all right everything that looks like Road in front of me I'm going to extrapolate that based on color out to the horizon and that's where I can drive and I'm just going to compute my path based based on that that was a really nice technique of course cameras are easily dazzled and subject to the blinding attacks just like we always talked about with anti-surveillance stuff you can also do spoofing with it right because just like camouflage works for the Marquand human eyeball you can use camouflage techniques you can mess with those color assumptions right so if something's saying well all right this stuff in front of me that looks like Road and it's that color everything that's color that that color is Road well you just make your obstacles out of Road colored stuff repeating patterns and tessellations confuse the hell out of stereo cameras because they don't know what matches up with what next sensor
millimeter wave radar this is being used a lot to vehicle applications to get do that obstacle avoidance stuff you've probably all seen millimeter wave radar in one form or another because this is the stuff at the airport that shows off
your junk to the TSA primarily usually as I said for collision avoidance looking for things that work reflect the radar well like signs and other other vehicles lower resolution than the lidar pressures kind of fuzzy images and at least in kind of a weird world where everything's a mirror right so lots lots of stuff is very reflective so you can't use it a lot for fine decision-making like any radar you can confuse it with chafing all right so spitting out things that reflect the radar signal it also gets a big return from things like sine so an overhead sign the robot might be programmed to ignore that as an obstacle because it's getting such a huge return from it but if there happens to be a dynamic obstacle underneath that it might miss it ??mu encompass stands for
inertial measurement unit so basically
you're integrating the output of accelerometers and gyros this is the primary navigation sensor for a lot of systems because they can be very very robust and because they can be very resistant to any kind of spoofing or attack you can get everything from high fidelity models to these hobbyist ones that are often pretty noisy a commercial aircraft IMU alright or a commercial robot IMU like a Boeing triple7 IMU has a cumulative error of about 1.1 percent of the distance travel total distance travel so they're used on
a lot of these like Arctic you UVs and stuff like that because what it means is you travel 300 kilometers and when you get to the destination you pop up and get a GPS fix your cumulative distance error is about 300 meters so that's really easy to deal with very difficult to interfere with because they're all on board they're fully encapsulated you're just recording what the robot feels which is why the military systems depend on them however especially the compasses very susceptible to magnetic field so potential for physical attacks with magnets
another part of doing this dead reckoning for ground vehicles is wheel odometry so basically encoders on the
wheels and giving your rotation information that you can integrate up they're actually a really key component because they give you good speed information relative to the ground and they let you know when you're stopped for sure right it's like one of the only sensors that can let you know for sure that you stopped which you might not necessarily be able do especially if you're in a tunnel right you can't use a GPS so they're so important that
actually what happened when we did our
pizza delivery we had to we had to got
get interrupted at the end because it took this turn tight and it scraped off
the wheel encoder and that was bad enough coming off the bridge that we had to rescue the vehicle at this point
there it is hanging off the side very sad so there are some things you could look at to increase the wheel odometry uncertainty or just trying to remove it so audiometry drift by changing the wheel diameter slightly for example got physical access to the vehicle slippery surfaces Mike cause drift in the odometry and then of course when you remove them potentially unpredictable but unpredictable behavior or stoppage
so now you know we've talked about all these physical attacks we can do sounding bit like a James Bond car package right we've got our GPS jammer
to knock out the absolute localization we've got smoke dust or vapor ejectives to confuse the lidar the IR lasers perform particularly bad in mist we got chaff dispensers for the millimeter wave radar could be sparse enough that a human wouldn't see anything but these fine metal particles make the robot just stop suddenly all the time glass caltrops if you're James Bond you have to have caltrops and of course an oil slick to prevent the encoders from telling when the cars really stopped so nice if you've got a James Bond budget for your countermeasures and of course an Aston Martin to put them on but there's
another really important thing to talk about besides all these sensor attacks the map the old-school mobile robots went into the world pretty much knowing nothing about it and just using the census lots of emphasis on doing slam and using their sensors to build up the map as they went along but but that data that aggregated map data is now so cheap and ubiquitous but there's a huge emphasis on pre acquired mapped map data right think about what for example what what Google does is another huge part of their business right mapping so the map is so comprehensive that it's often treated like the ground truth here's here's an example the video of mapping
with sensor data so this is the kind of map that we created for the Treasure Island pizza delivery so the map is so comprehensive that it's treated as the ground truth right it's it's really powerful because it reduces the recognition load on the robot in real time the robot can instantly map its sensor data to static features such as traffic lights trees vegetation even speed control and traffic signs stuff like that speed bumps but reliance on one single thing even a big thing like this can also be a weakness a single point of failure so there are potentially all kinds of things that we can do we can make use of you they're relying too much on this map
traffic lights for example this is how some vehicles locate traffic lights 100% robustly locating a traffic light is hard right they could be anywhere that you can see you got to do vision to find them and you've got to have 100% you can't just have your robot go around occasionally blowing through a red light because you know the robot you know it doesn't have to get completely right now it's got to do that because you're on the same Road as humans if it's just robots on the road maybe you can deal with that kind of thing but the human is expecting that if the lights red that guy is going to stop but if you've got a map of every single traffic light and it's registered to your GPS and so from everywhere you see it you can you know
exactly where to look to see that
traffic light then detecting the color of the traffic light is trivial you know where it is you just have to look for that blob but now you've got a potential schism between the humans assumptions and the robots assumptions because the human assumes that the robot can see the traffic light under any conditions but the robot assumes that that traffic light is exactly where the map says it is so for some reason that traffic light gets moved shifted around or altered human drivers have no problem with that they're going to see it fine it's going to be fine but the robot isn't going to recognize the new state vegetation
detection is another example so let's say the robot has some kind of rules for determine what's determining what's vegetation and what kind of vegetation it is so you might have some blob of vegetation you've got your colorized lidar looking with your laser rangefinder in your camera looking for green stuff or you might have some kind of transmission classifier rates how much of the laser is coming back because the vegetation is not 100 percent reflective and you know that if it's on the ground then it's grass and you can drive over it and you've got all your trees mapped out they're all in the map so you know where those all are and you treat them as static obstacles so it's no big deal but since the last time you drive that you drove that route the trees have grown and the foliage is overhanging the road and suddenly it's now spotted by the robot as dynamic obstacles here's this vegetation right in front of me and the robots going crazy and stopping everywhere even though it's this light vegetation that could it could just drive right through so the MIT the meta point here is that the rules for guessing what
things are that the human designers have come up with are often pretty brittle they represent the best efforts that the designers have been able to do the design acceptable tests but you have this when you have this great thing that looks like the truth in the form of the map you come to depend on it and the pennants on the map they may exacerbate that brittleness of these discrimination rules in the way that opens the door for exploitation one more thing about the
map is you've got to have constant updates this video is an example in simulation of what robot vehicles could do at an intersection if they have completely reliable real-time local information right there's no need to stop they just button go through the intersection totally terrifying right so you can do that if you if you have that at my update map update but if you have a local map then you can't do that kind of stuff and you're vulnerable to these unexpected real world features things that pop up in the real world that aren't on the map and if you have a remote map then you're vulnerable to all those attacks on the network right so you can do denial you can jam your 4g map updates and also spoofing you can man in the middle the map data as it comes through all the cellular intercept techniques that we are familiar with from other parts of DEFCON so looking at
some of the general vulnerabilities here let's talk about the overarching logic structures again and how we might like craft and exploit so we want to maximize the uncertainty facing the vehicle in order to cause mission failure so some of the maneuvers that a vehicle needs to do when it can only do on board sensing more uncertain and therefore more fragile than others simply because because of the geometry so one example is a right turn on red you've got uncoming traffic coming from the left and the view could be blocked by other vehicles so the same problem that a human driver has so the robot is going to necessarily be more cautious here and this provides an opportunity to trick it so we might want to force the robot to require manual assistance right so I might be unable to continue without supervision we might want to confuse or annoy the occupants so they abandoned robot vehicle transportation even regularly dropping the vehicle back into manual mode might do that inconveniencing the other road users right so if the robot stops and blocks traffic you've got robot road rage so getting back to these fragile maneuvers like the right turn on red if you can make it too uncertain so that it sits there and blocks traffic or if it benches out at the wrong time and gets t-boned then that's the that's the vulnerability to be exploited here now
if you have physical access to the vehicle you can do these kind of physical attacks on the logic kind of like a 21st century version of slashing the tires obviously highly dependent on the configuration and mission but if you have the abilities for example to get near the compass and stick a device on the other millimeter-wave that has a strong electromagnet that's also got a 4G modem you could figure out where on the map the vehicle is that it's doing a fragile maneuver and mess with its compass at just the right time an obvious style of attack is
redirecting the robot somewhere away from where it's um where it's supposed to go or even trapping it in a spot that it can't get out of so an attack on the collision and navigation collision avoidance of navigation layers force it to postpone its high-level tasks so you
can have obstacles that move and you can force the robots to stop if you can put obstacles around it so that it can't get out or you can tell it you know you can have moving obstacles that guide the robot off the path somewhere you want to take it you can even have obstacle swarms of other robots artificials traffic lights is another one the robots depending on the map so you can't put on a fake traffic light but if you can use the real traffic light and modify it so the robot thinks it's in a different state then the human would figure out right away but the robot has to stop another general task of mission fail
attack is clobbering this is a term from the cruise missile world basically you make the robot run into something so like a piece of terrain or something like that so you're submerging it's collision
avoidance ultimately to incapacitate the vehicle perhaps so you might want to completely crash into something or you might want to scrape off sensors you can do this by doing subtle deviations from the map changing things on the map especially near those fragile maneuvers or by changing things post mapping you can do it by imitating light vegetation so it thinks it can go through it but it can't simulating obstacles at speeds so the robot has to stop suddenly or swerve and perhaps around into something else the skies in entrance walls like the fake tunnel so putting reflective honors or materials within the localization noise so it goes too close to one side or overhanging piece that scrapes off the top sensors or obstacles as I mentioned underneath big radar reflectors so that the robot is programmed to that the robot is normally programmed to ignore like big overhead traffic signs so now that I said all of these things that you could potentially do to me a robot mean and nasty thinks the
driverless vehicles I want to reiterate
driverless vehicles are cool don't do any of these things I'm saying this don't hassle the hoff I mean don't hacksaw the bots but instead if you're into autonomous
vehicles and getting involved in the future of Transportation and all these other things why not get involved in the hard challenge of actually making them work right screwing them up is easy
getting them right is the cool part so for any students here I'd like to close out by just mentioning some stuff about the autonomous robot competitions especially the three that I am involved with suas Robo boat and robosub in that order on the on the slide I want more Def Con people involved in these competitions because Def Con people like to push the envelope and so here is just
a quick run-through of the tasks that are done and what you guys might be interested in so suas Waypoint navigation search for in identifying secret symbols on the ground and connecting to a narrow beam Wi-Fi network and downloading the secret codes right so this is Def Con right here secret codes Ward Wharf laying also coming soon hopefully package dropping right so legit excuses to rate bombing runs for UAVs cool challenging ask cool challenges in SE suas involve visual map making registering with GPS because you've got to report locations as well as codes if you're into panorama stitching automatic target ID not a lot of teams are doing this yet so there's like a lots of opportunity to put together a sophisticated entry Robo boat is
actually one of the most difficult competitions because the challenge stations we got things like channel navigation directing water cannons and dots onto a target identifying thermally hot items on ground stations disabling water sprays deploying a rover and retrieving a package a team this year had a boat that launched a quadcopter to retrieve this package capture the flag from another boat so this is this is all Def Con stuff right camera and lidar sensor integration this year a team had a lighter they couldn't afford a lighter off the shelf so they hacked one out of a robotic vacuum cleaner in Reverse that's that's what we need more of discrimination between vegetation and water and detecting when the robot is stuck up on things right people that just haven't been good at that but that's where we need people with a security mindset to think about it
robosub many ways the big one because underwater is the poster child for autonomy just because you don't have the communications bandwidth to remote control even if you wanted to so we got 3d navigation target recognition underwater shooting torpedoes and dropping markers manipulating objects and package recovery with a sonar ping er all without GPS I'm just flying through this because I know I'm slightly running over time one big thing again I think like we need the hacker mindset here for is like all these things that people don't think of before they go into the water like thermal management but the thing that I most want people to
be involved in with this is because I think that the rules need to be hacked right they're there to have loopholes
found in them that's what the people in this room do this UAV is the ScanEagle it doesn't need a runway right planes need runways he'll know let's just fly it into a cable and catch it so this kind of stuff that is what I want to see non-traditional vehicles experimental power supplies there's dimension limits there's all these rules like that in the competition but they are there apply apply at the start who's to say you can't change your dimensions while you're doing it and and hack things that way swarms of vehicles right let's get Voltron on this stuff so I think that this is the ultimate hacker
sport right it's technologically awesome it's bloody hard and there are loopholes to be exploited so I hope that people here in the audience who are students and have eligibility to be in this will check them out there's a going to be a big Daddy Robo boat next year in Singapore called robot X they'll give you a 50,000 boat and 25,000 for sensors if your team is selected to compete in it all right
that's it at the start of this talk just before the goons drag me off I want one more propaganda poster because I had these hats anti robot propaganda posters at the start the way when I was delivering my disclaimer about I didn't want to spread fear uncertainty and doubt about the robot revolution so here's a motivational propaganda poster about how one day we might live the ultimate Austin Powers style dream
letting the robot take care of the driving while we get down to business in the backseat I hope you'll get involved in making that come true thank you
Feedback