CRYPTO AND PRIVACY VILLAGE - JARVIS never saw it coming: Hacking machine learning (ML) in speech, text and face recognition – and frankly, everywhere else

Video thumbnail (Frame 0) Video thumbnail (Frame 888) Video thumbnail (Frame 2498) Video thumbnail (Frame 3444) Video thumbnail (Frame 4746) Video thumbnail (Frame 6939) Video thumbnail (Frame 10274) Video thumbnail (Frame 11360) Video thumbnail (Frame 12635) Video thumbnail (Frame 14355) Video thumbnail (Frame 15253) Video thumbnail (Frame 16888) Video thumbnail (Frame 19131) Video thumbnail (Frame 21320) Video thumbnail (Frame 22463) Video thumbnail (Frame 25283) Video thumbnail (Frame 27706) Video thumbnail (Frame 28778) Video thumbnail (Frame 30911) Video thumbnail (Frame 31805) Video thumbnail (Frame 39484) Video thumbnail (Frame 41295) Video thumbnail (Frame 43178) Video thumbnail (Frame 45458) Video thumbnail (Frame 47583) Video thumbnail (Frame 48784) Video thumbnail (Frame 50052) Video thumbnail (Frame 52814) Video thumbnail (Frame 56413) Video thumbnail (Frame 58578) Video thumbnail (Frame 60789) Video thumbnail (Frame 62205) Video thumbnail (Frame 64008) Video thumbnail (Frame 64946) Video thumbnail (Frame 67605)
Video in TIB AV-Portal: CRYPTO AND PRIVACY VILLAGE - JARVIS never saw it coming: Hacking machine learning (ML) in speech, text and face recognition – and frankly, everywhere else

Formal Metadata

CRYPTO AND PRIVACY VILLAGE - JARVIS never saw it coming: Hacking machine learning (ML) in speech, text and face recognition – and frankly, everywhere else
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Pattern recognition Smoothed Particle Hydrodynamics Intel Pattern recognition Machine learning Information Speech synthesis Musical ensemble Normal (geometry) Virtual machine
Covering space Presentation of a group Intel Turing test Proper map State of matter Projective plane Goodness of fit Endliche Modelltheorie Quicksort Resultant Local ring Physical system
Point (geometry) Slide rule Greatest element Context awareness Information Link (knot theory) Building Field (computer science) Right angle Data conversion Information security Metropolitan area network Physical system
CAN bus Type theory Demo (music) Multiplication sign Image resolution 1 (number) Virtual machine Videoconferencing Bit Quicksort Physical system Formal language
Ocean current Context awareness Building State of matter Decision theory Computer-generated imagery Virtual machine Artificial intelligence Order of magnitude Metadata Formal language Medical imaging Coefficient of determination Machine learning Googol Computer-assisted translation Physical system Machine learning Observational study Information Radical (chemistry) Digital photography Order (biology) output Right angle Information security
Machine learning Context awareness Machine learning Artificial neural network System programming Virtual machine Energy level Data conversion Mereology Perspective (visual) Physical system Physical system
Goodness of fit Graph (mathematics) Information Software Weight Virtual machine output Endliche Modelltheorie Function (mathematics) Arc (geometry) Physical system Connected space
Predictability Graph (mathematics) Information Confidence interval Weight Multiplication sign Virtual machine Function (mathematics) Neuroinformatik Data model Medical imaging Coefficient of determination Software Matrix (mathematics) Representation (politics) output Endliche Modelltheorie Computer-assisted translation Resultant Physical system
Complex (psychology) Functional (mathematics) Divisor Weight Virtual machine Parameter (computer programming) Heat transfer Disk read-and-write head Mereology Neuroinformatik Data model Iteration Different (Kate Ryan album) Matrix (mathematics) Representation (politics) Software framework Process (computing) Endliche Modelltheorie Category of being Physical system Graph (mathematics) Information Artificial neural network Weight Line (geometry) Demoscene Category of being Software Nonlinear system Übertragungsfunktion Function (mathematics) Network topology Physical system
Greatest element Virtual machine Perspective (visual) Dimensional analysis Product (business) Medical imaging Mathematics Latent heat Machine learning Semiconductor memory Vector space Matrix (mathematics) Representation (politics) Endliche Modelltheorie Codierung <Programmierung> Sampling (music) Data buffer Predictability Binary code Sound effect Bit Linear algebra Line (geometry) Product (business) Arithmetic mean Vector space Point cloud output Quicksort Matrix (mathematics) Row (database)
Functional (mathematics) Confidence interval Multiplication sign Computer-generated imagery Set (mathematics) Function (mathematics) Raster graphics Mereology Neuroinformatik Wave packet Intermediate language Medical imaging Latent heat Coefficient of determination Mechanism design Pattern matching Different (Kate Ryan album) String (computer science) Matrix (mathematics) Representation (politics) Computer-assisted translation Descriptive statistics Physical system Predictability Multiplication Algorithm Information Mapping Weight Sampling (statistics) Digital object identifier Arithmetic mean Calculation Order (biology) Inference output Quicksort
Point (geometry) Dataflow Functional (mathematics) Virtual machine Set (mathematics) Machine code Revision control Data model BLACK MAGIC Matrix (mathematics) Representation (politics) Software framework Data structure Endliche Modelltheorie Reverse engineering Hydraulic jump Physical system Operations research Presentation of a group Algorithm Scaling (geometry) Information Mapping Binary code Machine code Mathematics Logic Function (mathematics) Inference output Quicksort Collision Data structure
Spezielle orthogonale Gruppe Observational study Divisor Algorithm Decision theory Virtual machine Function (mathematics) Machine code Wave packet Data model Medical imaging Goodness of fit Different (Kate Ryan album) Hierarchy Software Matrix (mathematics) Software framework Process (computing) Endliche Modelltheorie Physical system Predictability Machine learning Information Mapping Interactive television Machine code Regulärer Ausdruck <Textverarbeitung> Software Video game Matrix (mathematics) Physical system
Goodness of fit Information Personal digital assistant Rule of inference Wave packet Physical system
Area Slide rule Asynchronous Transfer Mode Presentation of a group Focus (optics) Algorithm Algorithm Building Surface Virtual machine Client (computing) Density of states Data model Latent heat Mathematics Inference Linearization Endliche Modelltheorie Object (grammar) Physical system
Parsing Multiplication sign File format 1 (number) Function (mathematics) Open set Raster graphics Stack (abstract data type) Mereology Machine code Bookmark (World Wide Web) Software bug Neuroinformatik Data management Medical imaging Matrix (mathematics) Videoconferencing Software framework Physical system Vulnerability (computing) Graphics tablet Parsing Algorithm Mapping File format Software developer Moment (mathematics) Bit Data management Process (computing) Prediction Chain Buffer solution Software framework output Trail Functional (mathematics) Software developer Patch (Unix) Virtual machine Control flow Focus (optics) Machine vision Field (computer science) 2 (number) Chain Latent heat Crash (computing) Energy level Representation (politics) output Traffic reporting Multiplication Patch (Unix) Projective plane Machine code Exploit (computer security) Mathematics Word Video game Matrix (mathematics) Library (computing)
Demo (music) Virtual machine Machine code Neuroinformatik Data model Medical imaging Crash (computing) Machine learning Semiconductor memory Endliche Modelltheorie Vulnerability (computing) Physical system Predictability Service (economics) Demo (music) Software developer Denial-of-service attack Machine code Density of states Leak Personal digital assistant Crash (computing) Phase transition output Remote procedure call
Touchscreen Link (knot theory) Multiplication sign Bit Raster graphics Mereology 2 (number) Medical imaging Crash (computing) Befehlsprozessor Semiconductor memory Personal digital assistant Videoconferencing Physical system Library (computing)
Service (economics) Service (economics) Host Identity Protocol Demo (music) Moment (mathematics) Virtual machine Similarity (geometry) Bit Machine code Machine code Causality Read-only memory Semiconductor memory Videoconferencing Gastropod shell Point cloud output Remote procedure call Musical ensemble Physical system Library (computing)
Moment (mathematics) Virtual machine Bit Function (mathematics) 2 (number) Data model Mathematics Ring (mathematics) Videoconferencing Endliche Modelltheorie Physical system Physical system Library (computing)
Data model Demo (music) Algorithm File system Bit Endliche Modelltheorie Physical system Physical system
Greatest element Service (economics) Open source Multiplication sign Virtual machine Set (mathematics) Black box Wave packet Data model Latent heat Sign (mathematics) Cloning Cuboid Damping Information Endliche Modelltheorie Reverse engineering Physical system Domain name Machine learning Dependent and independent variables Pattern recognition Software developer Weight Projective plane Bit Database Similarity (geometry) Arithmetic mean Übertragungsfunktion Network topology Order (biology) output Point cloud Hill differential equation Right angle Resultant Cloning
Ocean current Virtual machine Set (mathematics) Function (mathematics) Public key certificate Computer programming Order of magnitude Machine vision Wave packet Number Medical imaging Coefficient of determination Machine learning Matrix (mathematics) Energy level Endliche Modelltheorie output Computer-assisted translation Backdoor (computing) Physical system Social class Enterprise architecture Email Information Mapping Entire function Word Software Prediction output Collision Physical system Resultant Spacetime Reverse engineering
Rifling Context awareness Turtle graphics Observational study Real number Multiplication sign Zoom lens Virtual machine Maxima and minima Turtle graphics Machine vision Medical imaging Coefficient of determination Mathematics Prediction Gastropod shell output Object (grammar) output Physical system Physical system
Computer virus Presentation of a group Serial port Service (economics) Virtual machine Set (mathematics) Function (mathematics) Philips CD-i Perspective (visual) Number Data model Frequency Different (Kate Ryan album) Series (mathematics) Sampling (music) Physical system Dependent and independent variables Inheritance (object-oriented programming) Sampling (statistics) Volume (thermodynamics) Total S.A. Machine code Digital object identifier Antivirus software Sample (statistics) Speech synthesis output
Information Virtual machine Sampling (statistics) Set (mathematics) Database Information privacy Mereology Information security Information privacy Physical system
Point (geometry) Game controller Presentation of a group Observational study Transportation theory (mathematics) Source code Virtual machine Information privacy Inversion (music) Wave packet Data model Medical imaging Different (Kate Ryan album) Personal digital assistant Software framework Information Endliche Modelltheorie Information security Physical system Authentication Machine learning Inheritance (object-oriented programming) Projective plane Connected space CAN bus Inversion (music) Process (computing) Personal digital assistant Information retrieval Network topology Order (biology) output Website Endliche Modelltheorie Right angle Key (cryptography)
Slide rule Machine code Machine code
next up is gai and Ezra with Jarvis never saw it coming hacking machine learning in speech text and face recognition and everywhere else thank you very much [Applause] [Music] first of all thank you for having us and we'll do our best to share the information that we have with you it's a bit awkward because I have a clicker in one hand and making the other so I can't really gesture so I'll do my best
first and foremost everybody is required to read and sign off of this legal disclaimer everybody did excellent ok
second disclaimer even more important we really haven't harmed any system here mmm sort of we'll cover that in a couple of states to introduce us I'm guy and
this is my good friend Ezra we're both cofounders and participants in the besides Tel Aviv and various community activities as well even the local DEF CON chapter in Israel and we are working together for almost a year now on a couple of very exciting projects some of those results want to share with you today
so the first thing that I want to mention is that nothing that everyone anyone is doing is doing by themself so it throughout the slides are switch hands throughout the slides you will be able to see short links at the bottom at the bottom right this is the reference if you want to dig deeper or to find more information and to find credits to whoever was behind this but the basic point is nobody's working alone we're all working on top of others people work and credit should be given work at credit is due so how did we get here
last year we had a couple of pretty good conversation around the Deaf man to a couple of people and the idea came about well everybody is talking about the latest buzzword and that is AI and how can we actually do something with AI systems our AI system even secure and through that conversation we came to the realization that nobody really thought about what security of AI system really means in that context everybody was doing lots of AI work everybody is running forward pretty fast but secure none of those system against attacks what kind of attacks are even possible is something that is kind of an unexplored field so we are going to discuss those
kinds of attacks a bit maybe about what kind of mitigations how you go method methodologically about constructing such attacks and what types or what the landscape looks like and what we feel is going to be the most important ones in the future to come whenever you see something highlighted in yellow just like here that means that this is something you should be paying extra close attention to we are not going to release zero days today we are not going to do a very surprising disclosure so it might be very surprising to you but it's not like nothing that anybody has ever conceived before we are going to do a couple of pretty nifty things in our demos like breaking machine learning systems live world sort of lives in a live video that was recorded last week and I want to start with a story about
the horse so a show of hands who ever heard about the horse named Hans or about 10% so I'll go ahead with the story so clever Hans was a very clever horse 1903 just the turn of the last century he went on a tour in Austria around the Austria and Germany and the reason that he went on this tour because it was very clever he could count up to five you could do simple arithmetic two plus three one plus four he could spell in German which is amazing because he could spell in German and he is a horse does that absolutely mind-boggling how could someone do that and it was very uncommon for horses to spell and to do solve arithmetic problem at the time it's pretty common today but as psychologist was dispatched to try to understand how can that horse actually do this and that psychologist and the work that he's done we now know as the double-blind test and what he found out is that the horse is a pretty clever it is a very clever horse but if couldn't spell and he couldn't really do arithmetic but it could be read the body language of his handler to make sure that he knew when he got the right answer so for example if the his handler would ask how much is one plus four he would pop his hoof one two three four five and he would read the cues from the body language of his handler knowing that he got the right answer and he would stop so as I said he was a pretty clever horse the reason that I'm telling you
all this is because on from many respects machine learning and AI in general that we are having in the world today is kind of on the same place machine learning is very good at solving specific problems sorry
machine learning is very good at solving specific problems whenever you try to give them larger context problems it breaks and it breaks horribly and we will discover a couple of those paths today so what we need to know in order
to understand what we're talking on in this talk first of all some basic common
language when I'm saying machine learning I mean a system that on the one end we are inputting lots and lots of information but we're also including metadata on top of that information in that context it means if I have a system built to differentiate between cats and dogs then I will feed it photos of cats and I will also give it that these specific photos have labels of cats these dogs are labeled as dogs and then the system will be able to learn from those inputs and labels and in the future when we'll really see a new input you know this is very similar to what I've seen previously and therefore this is a dog or a cat or a banana or whatever deep learning which is another buzz watch which you might have heard it's kind of the same thing but now we have no labels so we're just force feeding the system orders of magnitude more inputs and let the system decide by itself how to classify them so we give it 100 thousand 1 million 10 million input images and it will decide by itself oh this is a flamingo this is the hedgehog this is a girl building those classification by itself and when we are saying artificial artificial intelligence what we really mean is Arnold Schwarzenegger and the Terminator a machine that can think that it can reason that has context we are light-years away from this right now in our current state of technology we don't have a system that can really look at this picture and say oh the same system that will say this is a picture of the girl she's a flamingo there's a hedgehog at the bottom it's from a book by Lewis Carroll she's about to play cricket and in general she's very confused we don't have anything similar to that and therefore we don't have anything close to artificial intelligence however
everybody mixes artificial intelligence and machine learning I do as well and you will hear me throughout to talk saying ml and AI repeatedly in the same context I always mean machine learning there is no AI if anybody is selling you AI you should have a very different conversation so mostly I system were
designed to solve very specific problems so they were very good at solving that problem they are not good at the complementary part of the problem for whatever else so this is something that
we took a look at and the reason that this comes to be is because the way the system is built so I want to give like a very high level understanding of how such an AI system looks from a kind of a mathematical perspective so this is not
going to be complex I'm not going to scare you too much but I do want you to
have some good understanding good information about what a machine learning model really is so generally speaking we have a network a graph builds of a couple of nodes those nodes may be interconnected maybe more maybe less the amount of connectivity between the nodes is determined by the weight assigned to each arc the node itself holds a certain value and usually when we are discussing machine learning models we are talking about an input layer and output layer and the hidden layers which are between them so this is a very simple approach and when we're discussing data layers we are actually meaning what the system encodes or what what kind of information it has but yeah it understood about the world presented through its inputs when it was trained
what really determines the way that this system behaves is not just the values that the matrix that assists the specific representation of the model host holds at this specific time but also the weights or the amount that each node is contributing to the outputs so those weights are actually determining how much importance each node is carrying when computation is being run through all the way to the output in the end the output aggregates all of that information and we get a representation from what a machine learning system understood for example I will introduce an input image of a cat and it would tell me this is a cat with a 76 present confidence in that prediction the reason it only has 76 percent because there are other pathways to the network that led to other kinds of results maybe he thought it was a cat with average 76 maybe it was a dog with 15 percent so there are very different ways to traverse that graph and each way in the end computes into a specific value and we can look at those values and they give us more information about what the system does in reality the systems are
very complex so I showed a very simplified model here the reality is
much more complex than that when we are talking about modern machine learning systems we are talking about neural networks that are constructed about hundreds of different layers millions of parameters sometimes it's when we are talking about deep learning we are talking of networks of networks lots of different complications out of different ideas of how to make this more robust or more interesting which I will not go into today you can catch me later and I can fill your heads with lots of nonsense so what do we need to know
first of all when you're saying model and we mean a couple of different things one is what is the topology how does the graph look like what is connected to what how many layers are there the second thing is the weights what is the relative importance of each node inside that graph and for light lastly is the function the transfer function when we are transitioning from a scene from one layer to another layer inside the graph we have a transfer factor and usually a nonlinear transfer function which also introduces non-linearity to the computation and that adds more complexity to the system and enables the system to learn or encode more information into that matrix that I described earlier the bottom line is is that the mixed matrix that the representation of the model holds the intellectual property so if I'm designing the system that is trying to detect tumors in x-rays the IP the important stuff is holding that matrix in that model everything else is a framework to help to make that computation the real data the real important data is in that model and we will go later about that part so
how do we actually manipulate that data how do you actually get to their data a bit of background about linear algebra everybody remembers linear algebra show of hands well great because I had no intention to go into it okay so what what do you need
to remember when multiplying two matrices you just get a matrix matrix the values are the product of the rows and columns and that means that our various ways to accelerate that in the end from my perspective effect the vector is a single dimension where matrix an array is a two-dimensional vector a matrix or in the way that I look at it this is a vector and this is an array bottom line both are memory buffers of a certain size with a certain encoding and representation I don't care about the underlying math when I'm trying to go after that model I just want to know how its encoded in memory so so far we've always looked at a machine learning and AI is kind of like a big voodoo machine so we are introducing various kinds of inputs might be images audios binaries text whatever and you will get some sort of predictions for example I will say Alexa add something to my shopping cart and that audio sample would be uploaded to the cloud Amazon will do their own thing with it and in the end I'll have a specification meaning this is the most probable sentence the most probable utterance that matches that voice input okay but I want us to go a bit further
here and to understand how that mechanism really works so we have our inputs and we really can take a large image file a bitmap or a voice sample an mp3 or whatever it is and input it and multiply it by a matrix it doesn't make sense so what we need to do is to encode it in a with an intermediate representation so the way that we do it is that each kind of input and data scientists have different flavors of how to do that are encoding information with an IR and then taking that IR into the matrix multiplication part after we have the matrix multiplication all of the different algorithm functions weights etc the computation is over we are looking at the output and then we are matching that output into something that is human readable so the calculation from the matrix might have been 17 with a confidence of 6 or 97 or whatever and now we'll take that mapping 6 it maps to the label of a dog or a cat or whatever so the matrix doesn't know what the cat or dog is but it knows that 6 is one sort of representation and 7 is a different sort of representation and then we can output from the system the prediction meaning as specific classification usually attached to a binary string description but also the confidence or the amount of confidence that the system has in that specific prediction sometimes you'll get 10 of those maybe one of those maybe five of those depending on how the system is configured when we are training the
system what we are really doing is inputting more and more samples into the training data set and we are recomputing again and again and again and again the values for that matrix and that means that the matrix values start with a specific might be all zeros might be a random which is more probable and then we will modify those inputs again and again in order for them to better match the outputs that we are predicting for the system so when I'm training that system I do this I don't know maybe 30 million times until I'm able to get a good classification that matches what I know about those samples so my training data set is very important but also the way that I train the system is very important it's are those are both intertwined capabilities but when we go
out into the real world we don't have all of that we just have a deployment and that deployments usually a framework that just takes inputs encodes them into an IR goes through the algorithm mapping and then outputs the whatever information it was on the other end kind of like a very deterministic system and there's a good reason for that because we are not training the system anymore and you want to use it at scale the
point is that when we are looking at more at the models and when we are looking at code they're not the same thing and there isn't and are not the same thing is that when we're looking at the code or the binary execution code flow we know that okay we'll do this this then there will be a jump there me and compare maybe a jump nonzero or whatever we can read code we can understand what it means we can discern the logic from the machine representation but when we are looking at the model which is a matrix with set of values I can't discern anything about the system it's a very complex problem given a matrix can you go the other way and discern what kind of inputs build that matrix it's it's a much harder problem than breaking a sha-1 collisions the other thing is that when we're doing code we are usually looking at data structures so we are very familiar with data structure in how they are represented with machine language the matrix that we are working with the monotony in the model holds the same kind of data structure but not in a way that we are able to identify that so whenever it encodes more information into that matrix it loses the representation of the original data structures and we only get the latest current version of that matrix and the way it was updated without values it's very hard to look at the model and understand what build it what kind of information caused that specific matrix to come to be and the reason for that is that it encodes so much information and it forgets all the rest so it's a very nice representation of a one-way function sort of a one-way function just
to give you a notion of what it looks like in real life this is a model a very famous model called ResNet 50 so you can see like the label at the top but the rest is just binary information and when we're looking at it this is the only thing we care about this is binary information we can hold it we can manipulate it we can access it but what
can you do about the code because in the end the matrix doesn't live by itself there's a framework that needs to do the O's manipulations need to run to the hierarchy to do the mapping to the outputs etc so the model is not living in an isolation it has a lot of interdependencies and those the dependencies are code judges regular software and we are kind of good at doing code reviews maybe not so much we are very we are very bad at doing that for models and when we are looking at those models and codes and interaction between the framework and the model it's very difficult to understand where your code starts where the framework starts where your model begins it's a lot of mishmash between different interdependencies and different factors weighing into the final decision or prediction of the network and we really can't understand from the matrix what this means so I want to give an example of how how this comes to be and why this is so important so a small baxter background story a small example this is from their mythology study they did a study about skin tumors and they invested a lot of money yeah they invested a lot of money into trying to find those images and train those images to help us machine learning system that be able to to be able to
train on these and to identify those tumors and in the end they built a system that was very good at detecting rulers so this is like the real-life use case of what happens with data scientists you have a lot of data but the having data doesn't really mean that you have good information so there's a very strong distinction we have big data and big information nobody has lots of good information and
the reason that I want to mention that is because a lot of attacks are based on the same same kind of in principles we
are going to discuss five specific attacks today there are many many more and we have other presentations doing other stuff but the reason we wanted the
focus on these kinds of attacks is because we believe that those are the most important attacks we use CBF FS 3.0 3.0 to do these kourin's and when we talk to customers and partners and clients etc we discover that nobody cares about this prioritization this is actually what they care about so what people care about out there in the real world is not about somebody ddossing their machine linear system they care about somebody stealing their IP they care about somebody modifying their IP and they have no way to know that it happened so what we're going to discuss next is house ethics are built and then and here I will turn over to Elsa thank you guys so first of all I'm sorry of my boys Vegas is taking a toll on me cool so how do we build on that that we first need to know some stuff we are going to see it in a few slides what are the areas that we should target whenever you are building an attack and what are the areas that we have access to so let's first start to understand what is our tech surface and what's our attack objectives and we could go go either against the system infrastructure that it's running the models or going against the math and the algorithms
so just to recap a little bit we have in the infrastructure they the input is parsed and became kind of a mentor and I are so over here it's fully infrastructure and afterwards when it's the mapping between the output and the label it's also in the system level whenever we're talking about the matrix duplication and the output itself we are talking about the algorithms and we are going to talk a little bit more about it so let me start by the first and most
important part as we all remember we get an input that could be a picture or some words or some video or whatever but it needs to be converted into a matrix into an AR so it's the first unnoticed important part because if we cannot do that you cannot continue the second thing parsing is hard I mean if you had ever do vulnerability research and take a look at parsers you will find something most of the bugs in parsing exist because it's it's not simple you are not the one that developed the the representation so you need to understand what was behind and apply to your system Oh most important if you are an AI developer you are not the file format developer you do not develop parsers you it's not your field of expertise if you were is if it was your expertise is with you what you would be doing for a life so the most common thing that happens is your brain and dependence into your project it's very traditional just to say cool now I need to do parsing of bitmaps let's bring leap P and P I now need to do parsing good whatever I just going to bring this library and it happens and I respect that I mean if I were asked to person image I wouldn't know work nowadays I kind of know how to start but probably I would have bring an external dependency so again I'm bringing outside libraries into the machine learning server stack and this is very important to understand because whenever we're bringing this library we have written a very common problem in the industry which is the supply chain management the patch management how are we going to keep track of all the patches to all the file formats that we need to support or how are we going to verify that patching a certain layer that it's doing certain file format parsing doesn't break my representation afterwards and it's a very hard problem and not only that a common framework must have support for multiple file formats I mean if I were to develop a framework for machine learning that only supports a very specific file format it wouldn't be compatible with anything else so we need to have support to a lot of things so if we now know that parsers that parsing is hard and that file formats should be accepted by different machine learning systems we can do something very classical and very traditional from the exploitation world which is fuzzing those libraries so when we started with this idea of that fuzzing these libraries against distinct file formats the first things that that we identified was that the framework that we were fastened which is called buffer which had full coverage of all the functions that we were interesting in taking a look at was extremely slow why it was slow because every time we were trying to run one of those arbitrary malicious file formats representations to be able to trigger a crash it would run the entire end-to-end process so then we said who is actually doing the image parsing for cafe and then we say open computer vision so let's take a look at open computer vision project which it's a more limited coverage because now we don't go through the now we're going through specific this project and there might be cold pads that AFET doesn't use but the speed was good enough and the wood could have interval directly against the library that was being used for the specific file format extremely fast but we don't know where are the code paths and it was rural problematic so at the end we stayed with open CV oh and of course one of my favorite ones Disko upstream I mean many of those libraries that learning projects are using are not even maintained anymore so if you go upstream and just take a look at the heat reports or the proper parts you are going to find some very juicy stuff the issue is that you don't know if it's actually patch or somebody find it already or the code but is relevant to what we are taking a look at so at this moment we
found certain crushes and when we had certain crashes we went into the explain development phase and when we were indexed for a development phase we go to remote code execution and the question was could we use this remote code execution that we found before to be able to approach one of those vulnerabilities that we understood that are the risks in machine learning world
so let's right we'll take a look at let's try to do some demos we're going to start with our denial of service in this scenario we are going to abuse a memory leak where they input it's a couple of case and you are going to see now what is out just to clarify the system that we are using here is using an input image to the API then a machine learning model is doing the computation on that and it hands out a prediction so we are using that API to upload our own malicious image into the system so to be able to demonstrate this we had to build this API and we are going to see it two
screens and I'm going to explain a little bit what is screen so over here you are opening top to see what is the performance of the system and it starts
like this everything is normal afterwards you just go to the library where everything is running and run a bitmap that is going to do something something and now this is part is the video as you can see CPU goes to full the memory starts filling out and it's going to continue filling out for a very very long time so what I'm going to do is just make a first forward in the video and remember we're still talking about ten case bitmap file that we are taking a look at so it's still running if you see now we are using something around six gigabytes of memory for a 10k image and this is bad I mean imagine this kind of memory link and in a few seconds we are going to see the crash now you see 688 GG bytes we were using this is fat and you know the
business impact is following services downtime cause you are not you don't really know what it's happening in the system you just everything is working as usual but it because a lot more because maybe you are running it in the cloud or whatever but let's be honest we all came here to see the remote code execution so similar scenario in self immemorially guarda are going to exploit a memory corruption but in the hip and let's talk a little bit about it again left side WL
right we go to the library where everything starts we run the classification against malicious input and something is happening now so at this moment who are now going again back to the video we connect to the machine where it's running it because we've been to the shell import 1 1 1 at this moment I made a mistake and instead of writing house they might wrote host but we have the same host we do an LS and this is very important they can you because what they were seeing all the files in the system and we're going to return a little bit about it so at this moment with a malicious input we have full remote code execution [Music]
yeah but it's it's still not really relevant because we were telling that what it's what does NRC helps me in a machine learning world so in this scenario we are going to do something similar to what we did before
again we connect we get into the system we go to the library work we have the files we wait a little bit because it's the video it's a little bit slow and now when we run the system we are going to take a look at the files that are here so please take a look in a few seconds at the classification file and as you see here have all the labels and these labels are related to the output that the system gave me before now when we run the exploit in a few seconds we have a segmentation fault and now we are going to open exactly the same file and the mobile will always return the world hacked I mean it cannot do classification anymore and this is bad because I could modify any label that I want and the model at this moment we do whatever I want and the last one that I
want to talk about is the IP theft I'm not going to show a demo I don't need to do it but if you remember when I did alas I was able to see all the files that exists in this file system so the same way I could see them I could just copy them back to me and I had the model and I have the old a IP of the system which is is bad so yeah maybe there are
things really the king however we don't
always have an RC and when we don't have an RC we do something very smart we go to guy okay so assuming that we have an RC in the system which we've seen what we can do but let's talk a bit about what we can do when we don't have an RC on the
system so the first kind of attack that I want to share with you is something called a cloning attack a cloning attack means that I am using a service let's call it a machine learning service in the cloud as an Oracle I can ask it question to the API I can get responses and results back and I can use those inputs and outputs to train my own system that which you can see here in the bottom and that means that I can accelerate my own development of my model using someone else's IP sitting somewhere in the cloud effectively cloning it creating a functional clone of that system I'm not doing anything illegal I'm not hacking anybody system I'm using the api's as they were intended to be used but I'm stealing the IP away from the system by asking questions what would you do if I gave this input what would you do if I came with that input and I'm learning from those results and I'm taking away whatever they've spent so much time to study there are three
different approaches to how you go about cloning the first one is very easy you have full access to everything you know the data set you know the model you know the topology you know that transfer functions it might seem unreasonable but more often than not AI companies release papers about what they're doing they are built on top of open source projects they reference their dependencies they did not start from scratch you can get a pretty good understanding of exactly what they've done without knowing the actual weights but you know exactly what they've built and you can just query the the system and build your own weighted system with those results a gray box attack is a bit harder and that means we don't have actual access to their specific training data set and we may not even know exactly what kind of topology they've used however we do have domain knowledge of the system for example if somebody is designing a traffic recognition machine learning system and is using the American traffic sign database the government governmental national traffic sign database well okay I'll go and download Brazil's traffic science database they're pretty similar and I can still do the same attack even if I don't have the kind of data so just having similar data is enough in order to clone this attack and we have done this very successful what we are working on right now back home is a black box attack meaning we have no idea what's going on we have an API it's doing something it might tell us what it does it might not we don't know what kind of training it's used we don't know the data set we don't know what kind of label the system using we might be privy to a label one and able to but there might be 15 others that we are not aware of and what we're doing right now is how do you attack such a system you don't have full knowledge and so far we've been seeing pretty good results maybe next year but what if the attacker
has access to the data set itself that's a pretty interesting attack so if you
have access to the data set itself you can introduce back doors and backdoors in machine learning a very very interesting what do I mean by a back door remember the example I told you about the dogs and cats classification system do the same thing but now introduce bananas with a label of a cat the machine learning sustain on it and when I validate the system I will give it input in input images of cats it works dogs it works I have no way to know that an image of a banana will be classified as a dog to put it in a different way let's assume that I have a network analysis try a machine learning program and it's looking at all the network traffic of your enterprise and now I've trained it with a secret backdoor that whenever it sees a special packet header a special magic number in the packet header it will assume that anything that follows that magic packet header is completely benign non-malicious don't look at this I'm not really here I can do that if I have access to the training data set and when I as a user somebody who's buying a piece of an appliance or a piece of software from a vendor I look at that machine learning model at that matrix I have no way to know that there's a backdoor in the system and that is very different from the current software product because if I have a software product I can invest in reverse engineering I can do code reviews I can do certifications whatever I can get some level of assurance that there is no back doors there in machine learning system it's just a matrix I don't we have a way to go back and check it there are also an entire class of attacks called adversarial examples which some of you might have heard especially in the in the realm of vision systems this is like 95 percent of all research into adversarial system is around machine learning in vision systems but I want to share some information about other kinds of systems that you might not be aware of the basics is the basic problem of adversarial examples is that the problem space is orders of magnitude larger than the solution space in other words every input maps into some output so a dog maps into a label of a dog for the system but there are many many other inputs that also map into a label of a dog this is a simple collision attack with a very large problem space and the reason that I'm keep saying it's a very large problem space because finding collision is super easy okay it's not like breaking Juwan it's actually very easy to find other inputs that will give you the same kind of results and another
thing that I should note is machine learning systems are optimized and trained to find the local minima in math speak anonymous tricky same - finding the strongest signal in in the input so if the strong it's signal in the input is what it understands that characterize the dog it will say a dog but if I can influence the signal in the system to actually encode something else it will focus on that even though there is an image of the dog in the same picture I just need to find what's the strongest
signal so in vision system there was a debate for a very long time does it to even apply in a real world context and what do I mean by that there's been a lot of studies of take a stick here put it on something now it Singh thinks it's something else but you know in the real world objects are three-dimensional and there's lighting problems and zoom and very other difficulties and people say well you can't really do it on 3d objects it can be it cannot be done well surprise surprise this is not an actual turtle this is a 3d printed turtle but if you look closely at the top shell you see like red spots these red spots are the strongest signal in that object and they cause the system which is a Google's inception v3 here to classify it as a rifle as you can see at the leftmost bar so whenever it's turned up to the camera the camera the machinery system classifies it as a rifle because the strong signal is actually of it being a rifle even though we you see that this is a turtle and I not
want to show you another example and this is from an audio perspective so I want you to listen closely once I find my mouse without a data set the article is useless let me put pump up the volume without the data said the article is useless okay you could all hear him saying without the data said the article is useless now I'm playing the same thing with the attack listen closely with that ad that said the articles useless show of hands who heard the attack Wow two people with super hearing like that that said here are the course useless so the thing that most of you could not hear here is that there is a high frequency band here that encodes a different kind of data and that data is in the original a audio sample he said without the data set the article is useless but what a machine learning system is will here is ok Google browse through evil calm ok so when you're thinking about your echoes or Amazon alexis or series or whatever this is a completely viable attack so if anybody remembers the famous South Park episode making a really laugh out of the Lex and all of those different systems Cortana serial cetera this is because the machine learning system has no way to differentiate between one human speaker to a different human speaker to a TV speaking to somebody superimposing malicious audio on top of really audio machining system are not built to do it not designed to differentiate another
very interesting example is a dude who gave a presentation last year here called the Hiram Anderson and he built a virus compiling system he wrote some code and he compiled that code and he uploaded it into a service called virus total what virus total does is takes that the binary sample runs it against a large number of anti viruses he waits the responses and gives the score so if it's a virus and most of the anti viruses thought that it was a virus he would get the high score like 0.75 and everybody would know that this binary sample is malicious however he built a machine learning system that kept changing the code and measuring the output the same kind of Oracle attack that I mentioned earlier we found a different set of inputs that even though the code is still malicious is the same functional code now the anti viruses are classifying it as being benign so it's not very difficult to circumvent AI based systems and I want
to talk for two minutes about privacy
and this is a very simple example just to get the notion of what privacy means in the AI world so in our example we have a company that built a diabetes differentiating system so they took a lot of samples lots of people and they studied them and now they have a system that when Joe here comes to the system and it's classified either has a secure M diabetes risk of 7.4 or maybe has a diabetes risk of 35.3 and they're selling that system to various insurance companies and now insurance companies can check their clients and decided to provide them a policy or not depending on their risk score however now Fred comes along but Fred was part of the original data set but because it was part of the original data set the machine learning system knows him it already trained him so the score is going to be significantly higher and whenever I can see this kind of significantly higher scores I know that that person was part of the original database what reason is information was so privacy leakage is very real in these kinds of scenario so you might think that this is a very contrite scenario it's not it's like very real world but I want to give you another example and
this is a study called model retrieval or model inversion where what they've done is the same kind of Oracle attack I described earlier where they try to understand what kind of training data was used to train the day.the the machine learning model itself so you can see here at the bottom right that they got like a super composition of the different face images that were used to train the system but that super composition is not very far from the specific images in that training dataset so if I wanted to leak information out of that dataset or from other kinds of machine learning system I can really learn quite a lot but doing these kinds of attacks so what's our main point -
what's the key takeaways I want you to take on this presentation the first of all is that we really don't have a trust model for machine and we need a trust model here because even though we went forward and we designed lots of very different use cases for AI we need to bring security and privacy into it the second thing is that the way that the frameworks the design of the end-to-end system is today nobody cares of our security nobody cares about privacy everybody is running forward with developing not thinking what it means and people are using a lot of untrusted sources to fit into their machine learning systems the transportation of the of that inputs might be super secure they might be collecting data from sensors over TLS but if the data source is compromised what do I care about the TLS connection so nobody's really doing authentication all the way to the data source in the end you need to validate your data if you're not validating your data one of the attackers might come in and give you malicious data and you have no way to know that malicious data has been introduced to the system and we can do lots of very cool stuff if you have no controls in place and last but not least you need to understand the dependency tree of your machine learning system because if you are taking it for granted that your dependencies are secure you're going to have a very bad day a very bad day because as I mentioned a lot of the libraries baked into the frameworks today are no longer maintained I've unpatched nobody wants to touch their pet site data science project in order to update it it might break it might need to be retrained that's a high cost and if that's the case it makes my job much easier what you need to
remember in the end AI is just a buzzword okay it's just someone else's code and code is code and we can hack it we can break it and we can activate exfiltrated I want to
acknowledge some of the members on our on our team who contributed to this work on morality then it's raised the adults appeal and Oleg who all had significant contributions
I will not expect you to actually read this slide but when we release the entire deck you will have it come talk
to us we'll be waiting outside thank you