Machine Duping: Pwning Deep Learning Systems
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 93 | |
Author | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/36228 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
DEF CON 2477 / 93
4
6
7
11
15
20
26
33
34
35
36
39
40
46
49
53
58
62
63
66
68
72
79
90
92
93
00:00
System programmingVirtual machineInformation securityVirtual machineSystem programmingQuicksortMachine learningProxy serverHacker (term)Information securityCategory of beingMathematicsData conversionVenn diagramImplementationPresentation of a groupStatisticsSlide ruleCryptographyTheorySpacetime
01:48
Software testingBasis <Mathematik>Object (grammar)Machine learningSystem programmingAxiom of choiceObjekterkennungRight angleSound effectExtension (kinesiology)CASE <Informatik>SpacetimeVirtual machineInformation security
03:04
Task (computing)In-System-ProgrammierungMachine codeDifferent (Kate Ryan album)Functional (mathematics)Fitness functionExecution unitComputational intelligenceLinearizationLinear regressionPredictabilityExtension (kinesiology)QuicksortMultiplication signNatural numberTask (computing)AlgorithmLogistic distributionVirtual machineMultiplicationMedical imagingMachine learningSystem programmingVideoconferencingPoint (geometry)Dimensional analysisNetwork topologyFunction (mathematics)Complex (psychology)CASE <Informatik>Decision theorySpacetimeBitTrailResultantExpert systemMilitary baseNumberProduct (business)FreewareMessage passing
07:03
outputFunction (mathematics)Enterprise architectureLoginInternetworkingFunction (mathematics)Revision controlExecution unitCartesian coordinate systemCircleTheoryFunctional (mathematics)Connected spaceWave packetoutputWeightResultantEnterprise architectureLine (geometry)DiagramLinearization
08:20
outputFunction (mathematics)ComputerLinear mapAverageArtificial neural networkPredictionSocial classNumberoutputFunctional (mathematics)CASE <Informatik>Matrix (mathematics)Function (mathematics)Execution unitProbability distributionScaling (geometry)Vector spaceMultiplication signProxy server
09:19
Model theoryMaß <Mathematik>Gradient descentGradientPredictionArtificial neural networkoutputComputerFunction (mathematics)AverageLinear mapTotal S.A.Error messageWeightMathematical optimizationWave packetError messageSocial classCASE <Informatik>AlgorithmWeightGradient descentoutputPropagatorFunction (mathematics)Artificial neural networkStochasticPredictabilityMachine learningWhiteboardExecution unitMultiplication signDisk read-and-write headMaxima and minimaSpacetime
11:14
Demo (music)DatabaseDemo (music)DatabaseSoftware frameworkVisualization (computer graphics)System programmingOpen sourceVirtual machineFacebookTask (computing)Machine codeComputer animation
12:25
Router (computing)Mach's principleAsynchronous Transfer ModeValidity (statistics)Software testingMachine codeBitWave packetLine (geometry)Demo (music)ConvolutionFunction (mathematics)Model theoryWeightFunctional (mathematics)Source code
13:07
Numerical digitÄquivalenzprinzip <Physik>AreaProgrammable logic arrayArc (geometry)Uniform resource locatorElectronic mailing listDigitizingModel theoryDifferent (Kate Ryan album)Pairwise comparisonSet (mathematics)Standard deviationBitReal-time operating systemObject-oriented programmingConfidence intervalPoint (geometry)Computer animation
14:27
Pattern recognitionDigitizingImplementationSocial classClosed setAnalytic continuationConvolution
15:01
MultiplicationArtificial neural networkConvolutionMatrix (mathematics)Different (Kate Ryan album)Transformation (genetics)CASE <Informatik>AlgorithmLevel (video gaming)Artificial neural networkMultiplicationMedical imaging
16:02
Medical imagingGroup actionPattern recognitionFacebookCharacteristic polynomialArtificial neural networkShape (magazine)Pairwise comparisonFuzzy logicGraph coloringConvolutionPixelFunction (mathematics)Greatest elementInformation securityResultantFormal languageOrder (biology)Matching (graph theory)Computer animation
17:31
MultiplicationPerceptronArtificial neural networkMedical imagingGroup actionKey (cryptography)DiagramFerry CorstenMereologySquare numberDigitizingArtificial neural networkMappingPredictabilityConvolution
18:08
Function (mathematics)outputMultiplicationPerceptronRekursives neuronales NetzLoop (music)outputWordFunction (mathematics)Frame problemMultiplication signWeightRecursionMusical ensembleCASE <Informatik>Artificial neural networkSemiconductor memoryElectric generatorRecurrence relationAlpha (investment)Buffer solution
19:16
MultiplicationPerceptronTerm (mathematics)Read-only memoryMultiplication signSingle-precision floating-point formatSemiconductor memoryFrequencyArtificial neural networkQueue (abstract data type)Context awarenessSystem programmingData conversionFunctional (mathematics)Data bufferRecursionRecurrence relationPredictabilityData storage deviceExtension (kinesiology)Term (mathematics)DiagramCausality
20:27
Error messageWordDiagramInteractive kioskSet (mathematics)Cartesian coordinate systemSpeech synthesisBit rateBroadcasting (networking)Line (geometry)Multiplication signPattern recognitionData conversion
21:54
VideoconferencingCoefficient of determinationCubeAnalogyPhase transitionWave packet
22:46
Phase transitionAsynchronous Transfer ModeModel theoryCubeCausalityModel theory3 (number)AnalogyForceProxy serverMultiplication signResultantLine (geometry)Position operatorVirtual machineBlack boxVector spaceEmailPixelAlphabet (computer science)Reinforcement learningFinite-state machineMaschinelle ÜbersetzungCoefficient of determinationRight anglePhase transitionTranslation (relic)Machine learningNumerical taxonomyAreaWave packetAngleMedical imagingSampling (statistics)Information securityExploratory data analysisSystem programmingExistential quantificationLecture/ConferenceMeeting/InterviewComputer animation
26:03
INTEGRALAreaDecision theoryBoundary value problemSocial classDigitizingPattern recognitionMeeting/Interview
26:36
EmailDesign of experimentsSinePattern recognitionDigitizingMedical imaging
27:14
Wave packetMachine codePoint (geometry)DigitizingGoodness of fitStandard deviationPersonal identification numberModel theoryMultiplication signPixelTrailMatrix (mathematics)Medical imagingBefehlsprozessor
27:57
Design of experimentsFunction (mathematics)Medical imagingPoint (geometry)
28:31
Design of experimentsFunction (mathematics)Model theoryMachine codeVirtual machineDifferent (Kate Ryan album)Finite-state machineSocial classMedical imagingPixelImage resolutionBitComputer-assisted translationSet (mathematics)
29:19
Set (mathematics)Coefficient of determinationMedical imagingRight anglePerformance appraisalWindowSocial class
30:04
Complete metric spaceNoise (electronics)Different (Kate Ryan album)Normal (geometry)Medical imagingPixelWritingBuffer overflowDegree (graph theory)Vector spaceLibrary (computing)Moment (mathematics)Letterpress printingLengthPerturbation theoryRepresentation (politics)NumberComputer-assisted translationBitType theoryReading (process)Shape (magazine)Gastropod shellDemo (music)CalculationOpen setArithmetic meanSimilarity (geometry)
34:11
TensorPerturbation theoryPredictionModel theoryScale (map)Order of magnitudeoutputTraverse (surveying)Linear mapoutputModel theoryManifoldOpen setVirtual machineMedical imagingMappingBlind spot (vehicle)Mathematical optimizationTensorPerturbation theory2 (number)Order of magnitudeTouch typingOnline helpPredictabilitySpacetimeMatrix (mathematics)MultilaterationFrequencyVector space
35:51
Computer-generated imageryModel theoryArtificial neural networkTask (computing)Speech synthesisPixelFunction (mathematics)Model theoryMathematicsLevel (video gaming)Enterprise architectureWeightTask (computing)Service (economics)Library (computing)Software frameworkDigitizingSampling (statistics)Pattern recognitionMachine learningLimit (category theory)Artificial neural networkConvolutionSpeech synthesisRecurrence relationBitOnline service providerWave packetMedical imagingVirtual machineSoftware testingUltraviolet photoelectron spectroscopyMeeting/Interview
37:17
CAPTCHAMotion captureProjective planeCAPTCHAMultiplication signPerturbation theoryWeb 2.0Different (Kate Ryan album)Sampling (statistics)Model theoryPredictabilityPerformance appraisalWordRankingConfidence intervalElectric generator1 (number)Spherical capComputer animation
38:45
Motion capture1 (number)Software testingService (economics)Model theorySubstitute goodGame theoryServer (computing)WebsiteElectric generatorComputer-assisted translationSampling (statistics)Machine codeMedical imagingArtificial neural networkPlastikkartePredictabilityDifferent (Kate Ryan album)Software frameworkMachine learningMoving averageHeat transferWeb pageWave packetEnterprise architecturePoint (geometry)Network topologyDecision theoryCASE <Informatik>BitOpen setDreizehnInformation securityMultiplication signSystem programmingLetterpress printingWordFitness functionFunction (mathematics)Meta element
44:09
Computer animation
Transcript: English(auto-generated)
00:00
So today we're going to be talking about machine learning. No one has not heard of machine learning right? So in specific what we're going to be talking about is how to bypass deep learning systems. Machine duping is what I'm choosing to call it. It's a really corny name so if anyone has better ideas come to me after the talk and we can talk about it.
00:22
So everyone has heard of machine learning. My grandma knows what machine learning is. At least if you've been reading the news you know that the best Go player in the world is a deep learning system. So that's kind of weird. No one saw that coming in the last five years. But let's not have a slide on what machine learning is and what machine
00:42
learning can do for us. That's something that's for a presentation two years ago. Let's instead look at what we're trying to achieve in this talk. So everyone wants to be the flying carpet beagle in the middle of the Venn diagram there. Most of us in this room have some sort of hacking skills. People hacking or computer hacking. Uh we do stuff.
01:02
We're we're implementers so uh I'd say we're in that category over there. Hackers. Um security researchers have some kind of theoretical background. If they're working on crypto and stuff then they have some kind of math and stat skills. Um I don't necessarily consider myself that. Um data scientists don't have any hacking skills but
01:22
they they do a lot of stuff in in companies trying to maximize uh conversion. They have math and stat skills. But what we're trying to do today is to help all of you guys and convince all of you guys that you really want to be in the center of the Venn diagram and it's gonna be important and increasingly important in the next few years to
01:41
brush up on math and stat skills and to know about what's going on in the machine learning space. Especially for security folks. So whether we know it or not we are interacting with machine learning and deep learning systems on a day to day basis. Um if you don't like it you have no choice. Um Google Now, Apple Siri, Amazon Alexa they're all
02:03
things that have uh been uh covered by the press. Very high profile things. Some common some more common use cases of deep learning are in object recognition in self driving cars if you if you know Joe Haught and his cool comma dot AI startup. Uh they're using deep learning to recognize objects on the road and build a really cheap uh self driving car
02:25
system. Uh obviously on the top right hand corner you see AlphaGo beating the world champion at Go. And you have also other pretty interesting use cases um like in medical research where they're using deep learning to predict the effects of newly
02:41
developed drugs in patients. Um and I just have an example from Nvidia there. Uh also in the security space there's also lots of stuff if you've ever set foot at RSA or black hat expo you know what I'm talking about. Everyone is saying they're using machine learning, deep learning to do stuff. Uh the extent to which that is true I cannot vouch
03:02
for. So why would someone choose to use deep learning? Again forgive me if you're an expert at deep learning this is a DC 101 track so I'm gonna be spending a little bit of time going through some basics of deep learning and machine learning and then I'll go into the interesting stuff uh which is my research. Why would someone choose to use
03:22
deep learning over more traditional machine learning methods like SVMs or linear classifiers, clustering algorithms? The first thing is that when you use a deep learning algorithm you get some things for free that you otherwise would have to spend a lot of time doing. Um the most important thing that everyone would uh point
03:40
out to you is that you get feature engineering for free. Um again the extent to which that is true you have to try it out and implement it and it depends on the use cases that you're you're you're using it for. Um deep learning helps to select the best features and you don't necessarily have to spend a lot of time doing feature engineering. If you talk to any data scientist or machine learning engineer uh working
04:02
in a large company uh then they'll tell you that most of their time is not actually spent on algorithms. They're not trying to increase the efficacy of this so and so cutting edge algorithm that uh they're using to in in their company's product recommender systems. What they're actually doing is feature engineering and data cleaning. So they're like janitors you know. Like I'm a janitor too. I I spend most of
04:26
my time cleaning data. I spend most of my time doing data feature engineering. So deep learning gives you that for free and that's why it's so appealing I think. The other thing which I think is the main difference between deep learning and other kinds of machine learning algorithms is that uh it touts the promise of one infrastructure for
04:43
multiple problems. So if you think about it deep learning really is just one just one infrastructure. There's multiple layers of linear units and uh each one of these linear units interact in a different way with different linear functions to give you the result that you want to learn different things that you wanted to learn. Compared with
05:01
other kinds of machine learning algorithms like clustering you and uh SVMs and and logistic regression decision trees all of these require vastly different code bases. Whereas for deep learning the the differences in infrastructure are parameterized into the number of different layers the number of units in each layer the functions between each
05:21
layer and other things like that. So it's sort of one infrastructure for multiple problems and I think that's what gives it it's flexibility. The last two points are perhaps more relevant in today where there's so much data to deal with. Um deep learning allows you to do hierarchical hierarchical learning. So you can split up the
05:41
task of learning across different layers and we'll see an example of that later. So for example you have more shallow layers learning vastly different things from the deeper layers and you can extract out the outputs of each intermediate layer to uh exactly find the thing that you are looking for. The last thing is it's efficient and it's
06:01
easily distributed and parallelized. If you're looking at algorithms like clustering there's no straightforward way to really distribute it across systems and when you're dealing with terabytes petabytes of data this is a problem. Of course there's message passing algorithms that that uh help clustering out with that but they're a lot more complex. So deep learning allows you to distribute these uh problems up distribute
06:23
your infrastructure up and distribute the computation. Of course it's definitely not one size fit all uh deep learning is not something that you want to use for any random problem if you're trying to predict let's say the prices of oranges uh against time you wouldn't be using a deep learning infrastructure that's like you know using deep
06:42
learning to um you know predict a problem space of two dimensions you wouldn't do that. Uh what you would do deep what you would use deep learning for is in uh problem spaces that have multiple dimensions. So like hundreds or thousands of dimensions usually these things come from nature. So you can think of images, audio, video and prediction problems that are in a very complex problem space. So let's just spend a
07:06
couple of minutes to go through uh the two steps involved in training a deep learning architecture a deep learning infrastructure. So this is uh a diagram that a lot of you may have seen before it's it's it's a four five three neural net architecture it's a
07:20
very simplified version of it um so ignore all the all the nasty white lines between those those units. Each circle represents uh a linear unit and there's uh activation function in it. Activation function just means that if you input a certain value into this circle then uh it either outputs uh it outputs a certain real numbered value um
07:40
according to uh the the function. And each connection between two units between layers is uh is is uh weighted by the weight W and also there's a bias unit. Uh what the purpose of the bias unit is simply uh to help to skew the results of the output by a certain amount. If you can think of uh just linear algebra you have Y equals to AX plus B. This
08:04
bias unit is equivalent to B which controls the intersection between the X axis and the Y axis. So all this is just theory don't don't don't worry about it if you just like take any one of the ten thousand MOOCs out there to learn about deep learning. But let's just look at uh a simple example of how training works so we can uh go into how to
08:23
bypass them. So the first step is the feed forward step. Um each unit receives the output of neurons in the previous layer. Uh in the case of the first layer it just receives the input. Um and then uh the bias added to it the way it the output of the first the output of
08:41
the first unit is weighted by W1 and then it goes on and on. And eventually when it comes to the output layer uh it outputs logits which are just numbers which is an array of numbers. It's fed through most of the time a softmax function for classifiers which just scales it into a probability distribution function. And so in this particular case you
09:01
see that um this dummy uh matrix this dummy vector that's output is 0.34, 0.57, 0.09 and this just means that uh if you have three classes 0, 1 and 2 uh this classifier predicted that class 1 is the predicted class. Uh however that is uh wrong in our case. So
09:21
according to the labels cause this is training so you know the labels. So uh the error is 0.57 uh because the ideal case would be that you predict with probability 1 that uh the output is actually 0. So you feed it backwards and back propagation is uh really the crux of deep learning. So what back propagation is it's a pretty well it's not the
09:45
most straightforward algorithm that you'll buy you'll come across in machine learning and when you're taking some MOOCs like the Stanford one on machine learning by Andrew Ng he'll also say that back propagation is a pretty hard thing to grasp. I think the easiest way to explain it the easiest way to think about it is that back propagation is just assigning
10:02
blame. So let's say you uh you were the head of the board of directors and you had a bunch of people on the board that were giving you uh suggestions that were advising you and stuff. Um and some of them just talk bullshit all the time. Uh you wanna listen to them less. So that's what this is doing. Uh so basically you're feeding input data through uh a
10:23
deep learning network and you're trying to figure out what gives you wrong answers. And you do that by sending input data and then seeing when the answer is wrong you know the answer you you know the right answer. And you're seeing what answer is wrong and then you trace you trace the path that this answer took through the network and you find
10:43
out exactly which units are responsible and how much they're responsible because of their weights. So let's say you find that a particular path of units is particularly bad at giving you recommendations or predictions for something that you're trying to learn then you just decrease their weights and you block them out. So all of this uh can be
11:03
optimized with certain algorithms like stochastic gradient descent which is just basically uh trying to find the local minima uh in in in a particular uh problem space. So there's gonna be lots of demos in this talk so if you have been ignoring me for the last 10 minutes uh you can uh look up now. Okay is this big enough? Let's make it a little
11:31
bigger. Okay so what we're gonna look at here is an example of uh a deep learning system that is really accessible to everyone. This is TensorFlow um it's uh Google's uh open
11:43
source uh deep learning machine intelligence framework. I think it's uh probably the easiest deep learning framework to use and what we're gonna be walking through is a small example of how to do uh deep how to use deep learning to solve the easiest task uh the task that's used most commonly in most tutorials and examples and then we'll look at how to
12:04
pass that. So TensorFlow okay. So what we're trying to do is use the MNIST database which is um which is created you know like 20 years ago I think by uh Yann LeCun who's now the Facebook AI director and uh we'll be having some visualizations done with uh this
12:22
pretty cool uh visualization tool. So let's look at the code a little bit. What this is doing is basically taking in the training data and the testing data and labels and then you're creating a validation set all this is is is uh just uh sugar and then here is the
12:41
actual definition of the model. This is the model and the layers are defined uh line by line. You have the first convolution layer then you have a pooling layer, convolution 2, pool 2 and so it just goes down from top to bottom and you get from shallow layers to deeper layers as you go down and as you see the logits and uh the softmax
13:01
function actually creates the output of this uh neural net. So this is a demo of MNIST classification. What MNIST is is just digits, handwritten digits. Uh it looks something like this. So these may not look like digits that you or I would write like I would never
13:20
write a 7 with uh with a weird thing at the end uh but you know what can you do? You can always train a a model on on different handwritten data sets but uh this is a standard and it's used for comparison between between uh researchers and in in academia. So let's see here. This is uh a real time um a real time classifier predictor for for digits. Uh
13:45
so if I were to write like a 2 here let's see if it predicts it correctly. Oops. Ok 2 even though there's that weird thing. So it's pretty good. Let's do like 7. Oh 7 that's great. Only .7 accuracy. I mean confidence. Let's try something that's a little bit
14:06
more challenging for the model. Like a 6. Ok so this this is a wrong this is a wrong classification because it thinks it's a 5 with .886 uh confidence. Let's see if we can
14:20
make it better. Nope. Never mind. So you can see this is this is not the cutting edge in uh handwritten digit uh recognition. Like if if uh I think uh this this particular this
14:41
particular implementation represent correctly classifies 90% of digits. Um so that's nowhere clear clear close to what a human can do right? I mean if you or I recognize 9 out of 10 digits then you should see an eye doctor. But anyway that's that and uh let's
15:01
continue. So what that was doing um it was using convolutions and this is what you call a convolutional neural network. So these are just different flavors of neural networks. Just different algorithms that researchers publish papers on um to get tenure and stuff. But this is a really cool algorithm um that was developed 20 years ago I think. Um and uh what
15:24
this does is that it uses convolutions to uh gather insights on different details uh different uh levels of details in an image or in stuff that have adjacency relationships. So what are convolutions? If you remember your Fourier transform days from school or
15:42
not um convolutions are just filters. So uh if you apply a filter a convolution on a matrix let's say in this case it's a 2D matrix then what you have is you do a matrix multiplication and then you end up with a single value for the convolution applied on the particular space that you're applying convolution on. And so what this allows you to do is
16:03
layered learning. This is an example for facial recognition where the layer the shallow layer is actually at the bottom here and then you're going out as you go deeper into the network. Uh in layer 1 you're learning very very uh very very fine features like the shape of your uh eyebrows no the the shape of your eyelashes and the shape of
16:22
and maybe the wrinkles in your face as you go further up as you go further down the network then uh because of the convolutions and pooling then you actually get uh more uh zoomed out features like the shape of your eyes, the color of your eyes, whether you have uh uh like a mole somewhere or something. Uh at higher layers then you get the
16:42
shape of your face uh more general uh characteristics of you like if you have a mustache or not. So um this is interesting because uh you can extract the results out of intermediate layers to do certain things like uh I was at a talk by Facebook security team once and they say that in order to find uh images that are spammy so when when
17:03
spammers try to spam their network with images they often tweak the images a little bit or change the language or text in in certain ways that you can't just do a pixel by pixel comparison. So there are certain ways that that you can solve this by shingling or by doing some kind of fuzzy matching uh but the most efficient way they found and the most effective way was actually to pass these images through a neural
17:24
network and then get the second layer out and compare the outputs of the second layer. Then they were able to reliably find images that were spammy and then group these images together and then have a human actor come in and see whether this is actually a spammy image or not. So this is an ex- this is just a diagram of the
17:42
convolutional neural network that uh was used to to uh to classify the the digits um basically if you look at the small squares that are zoomed in uh one part of the digit is uh one part of the digit is uh fed into feature maps and then you do
18:01
subsampling on that and then you perform convolutions, you perform more subsampling and then you do uh a prediction on that. So besides convolutional neural networks there are also things that are called recurrent neural networks. These are slightly more complicated but not actually that complicated. What you have is just recursion in the neural network. So instead of feeding in through uh instead of feeding
18:24
in input one way through a neural net you have recursion in it so um the output of each time step and each intermediate output actually gets fed into the next time step. So this introduces the concept of memory. And memory is important because when you learn things you don't necessarily learn things one frame at a time. That'll be
18:40
really weird. Um and uh this allows you to learn things like uh you know the music or or things that like audio or video which have some kind of uh relationship between frames and relationship between inputs. And so this is an example of a generative uh recurrent neural network. Uh you can teach a network to spell words. And
19:02
so in this case if you see um that you already have the letters Y O L then O is likely to be the next word because of the memory of the network having Y O L in its in its uh buffer. So there's also stuff that's more on the cutting edge. This was actually
19:22
used uh to a certain extent in AlphaGo long short term memory. So when you're looking at things with slightly more context but you don't want to scale the depth of the network or the depth of recursion indefinitely then you want to use things like LSTMs long short term memory networks. Um because this allows you the concept to arbitrate um longer term concepts. Arbitrate longer term data to store for a longer time and
19:46
not just have uh have a single uh FIFO queue that you that that you store uh your your memory in cause that's not how we learn. So to make good predictions we need more context and you can think of uh things like uh a system that converses with you. When
20:01
you're talking with somebody and someone mentions that he's from let's say France uh 5 minutes ago in a conversation. You don't just forget that after 5 minutes cause your memory buffer is full. You have to remember things like that and so the beauty of LSTM networks is that they have certain gated functions in the recur in in the recurrence of the network that allows you that allows the network to learn what's important to remember for a
20:24
longer period of time and uh what's not. So this is uh just a simple diagram of how deep learning has helped uh with speech recognition over time. The different lines in this diagram uh this is uh the y axis is logarithmic by the way. So the different colored lines
20:43
represent different data sets. Uh the holy grail is definitely the red line. It's conversational speech. So a very very uh a very very sterile kind of speech is uh red speech or broadcast speech where uh every word is enunciated to to you know to to to to
21:02
the nail. And um all of these over the years have seen pretty good performance. You see like up to 2% word error rate which is WER. Word error rate uh performance in air travel planning kiosk speech was a pretty weird data set. Um and then for conversational
21:20
speech um you see in 2011 it got it went down all the way to about 20%. So that's that's great. By the way the conversational speech data set if you ever um if you ever have a chance of of finding it or looking at it I it's not out in the public but it's actually pretty weird. Um it's actually from blind dates. So so uh when when I was when I
21:44
was listening to it it was it was very interesting how uh some of these uh dates turn out. But yeah there's some disturbing stuff in there. So okay now for the now for the fun stuff. How to pwn. So there's a short video. This is Gunter. He's uh a dog.
22:11
He's a he's a mini schnauzer and uh he's he's my best friends dog. So let's just uh see how this analogy pans out. Let's see this video please. Okay. Cool. So this is Gunter. He
22:35
loves ice cubes. I'm not sure if all dogs loves ice cubes. They think like he thinks it's alive. So this is the training phase where I'm basically teaching him that the
22:43
clink clink sound in the bowl represents that there's an ice cube in there. Sorry. The clink clink sound in the bowl represents that there's an ice cube in there and he knows he's gonna get an ice cube because he had the taste of it. Then I mislead the model. First I show him the ice cube. I put it back in the bowl. And I don't actually throw
23:01
it. I I throw something else that's not an ice cube. And he's confused. The model's bypassed. So Gunter is the model now. And this is kind of a like a lame analogy. I draw some stuff I'm doing. Forgive me. Uh but what we see here is that uh we the dog doesn't know that I'm throwing the ice cube. He just thinks that I'm throwing the
23:24
ice cube because he sees like the big hand motions. Um ice ice doesn't have much smell and he he doesn't have great eyesight. So he sees me throwing something and I've done it a few times before. So he thinks that an ice cube is there at the other end of the of the yard waiting for him. So that's very similar to what we're gonna be
23:41
doing. Um we're gonna be feeding in images to classifiers um that will mislead these classifiers. And these images are crafted in a certain way that will uh facilitate certain classifiers from being misled more than others. But first let's look at the attack taxonomy so we can look at the attack the attack vectors and stuff like that in
24:04
security speak. Um there are two general kinds of attacks that you can do on these machine learning or deep learning systems. You can have causative attacks or exploratory attacks. Causative attacks are relevant when you have access to the training step. Um so for example when you're looking at like an online translation engine like Google
24:23
translate uh they actually do rely on some kind of online reinforcement learning when you see that something is a really really bad translation which happens pretty often I'm sorry. But um when you see something's really bad you can report it as really bad and you can maybe give the right answer or something that's more relevant. And so this
24:41
is how it how it does reinforcement learning. Um in Gmail when you have when you receive email that's not marked as spam but actually really is spam you can mark it as spam and I urge all of you guys to do that because that actually helps a lot with the with the reinforcement learning model. Um and uh this will help to train the model to recognize such examples as as spam in the future. So you can see how you can
25:05
influence a model in such a way. That's a little less interesting that's what I that's what I that's what I did last year in in talks. Uh but what's more interesting I think are exploratory attacks. So in this attack model you have no access to the model you have no access to training phase and what you're doing is just a black box attack in in in
25:23
some scenarios where you're feeding it just weirdly crafted samples that look correct to a human but uh the machine just gives you a wrong result. And so that's really that that really throws some people off even from machine learning researchers because um they think that the deep learning model or the machine learning model is learning
25:41
exactly in the way that the human learns. Like we don't learn uh to recognize alphabets or letters by uh looking at the pixel or the angle between uh the horizontal line and the slanted line in an in an A. Um we learn it in a more general way and I think it's still active research area into how to represent these things in
26:02
machines better. Uh so this is still the dog keeps coming up. Uh this this is still active research area uh and uh that throws people off. It's also targeted and indiscriminate attacks when you try to uh move the decision boundary in a certain way to cause an intentional misclassification or indiscriminate attack when you uh just try
26:25
to decrease the integrity of of uh these classifiers. So this is a simple example of um misclassification. So in the early days MNIST and a digit class digit recognition used to
26:41
be used in recognizing digits uh in in checks. And uh I know these these digits don't look very realistic but I'm using stuff from the MNIST data set. And what I did beforehand was to was to generate some adversarial images and then fill in two copies of the checks. One with normal images and one with adversarial images. So if we just go
27:04
back a bit and look at this. This is the adversarial one and this is the normal one. They look pretty identical. Just look at the digits portion. So it's 9378 and this is some simple code to use for pre-trained MNIST model with the standard TensorFlow uh MNIST
27:22
example trained with that. It takes about 4 hours to train this model which is trust me really good speed using a CPU in the deep learning world. You see something that takes longer time to train later. And so what we're gonna do is to just basically read the check. Um this just divides the image up into a pick in into a pixel matrices and then
27:46
uh it reads digits. So it's loading the model. It's predicting uh the digits 937800, 9378.00. So that's that's correct. And so if you look at that, that's what it is. Okay.
28:04
And now we're gonna do the same for the adversarial image. Okay so you expect this to be
28:21
the same but no you actually expect this to be different because it's called adversarial. Um so the output of this is something totally different. 0524.84. So it looks the same but it gives you a different output using the same model, using the same code. And uh
28:41
that's adversarial machine learning. This is something a little bit different. It's the CIFAR 10 dataset. So CIFAR is um is a dataset of images. So what you're trying to train the model to do is to recognize images uh in 10 classes. There's CIFAR 10 which is for 10 classes of images. There's CIFAR 100 which is for 100 classes of images. The
29:04
classes are are here. They are uh like you have things like airplanes, automobiles, birds, cats, deers. The interesting thing is that these images are not high resolution images. You don't take them with your phone. They're 32 by 32 pixels large. So this is the actual size of it. Actually maybe bigger. And so we're looking at two sets of images
29:23
here. Um dog and automobile. Um just chose them cause dog is cute. So uh what we see here is that well if you see the preview window on the right here uh that's actually
29:40
preview on on Mac uh it's not 32 by 32. I think Mac OS does some aliasing so your your pictures don't look like shit. And uh what we're doing here is to eval this and what this evaluator does very similar to the MNIST classifier is that it classifies this this image into a class. So it tells you whether this image is of a dog or of a or of
30:04
something else. So you see in the at 10 uh image it classifies it as a ship ship. In at one it classifies it as a ship as well. So you can see that the images are pretty similar um but there's actually differences in them. Let's classify the automobile one just
30:27
for completion. Automobile. Okay this is what it looks like. It's a pretty shitty image. Automobile okay that's correct. And what the different numbers after add means is just to to what degree they are perturbed. So to what degree we're injecting stuff in
30:45
the image to uh make the classifier think that it's not what it is. So you see it becomes a little bit more grainy but to a human it still looks like a car. I mean you wouldn't say that that's a that's a cat. Or you're weird. Yeah? So let's look at a
31:02
exactly what differences uh exist between these images. Let's open the python shell. Uh scipy. Let's read these images in and then uh look at what the difference is between them. By the way a lot of python libraries actually don't read PNGs or don't write
31:25
PNGs exactly pixel for pixel. So if you're trying this at home use these libraries. So this is the this is the standard representation of of the PNG. It's just um a 32 by 32
31:40
pixel uh representation. There's 3 channels R G and B and um the the the value of each pixel can be between 0 and 255 which is pretty standard. So we're reading the adversarial image now. And and we can see that let's print it out. We can see that
32:04
okay the numbers are slightly different from before. Let's print out exactly what's different. Now let's look at the size first. You can see that the the shape and the the the shape of these 2 images are the same. 32 32 3. And let's let's calculate the
32:20
differences between the adversarial image and the normal image converted to uh N64 to prevent any overflows. It just is boring you out just like zona for a moment and I'll wake you up. So doctif 1 let's print this out. You can see okay the differences between these images are of between 1 and 2 pixels or 1 and 2 pixels and URI
32:44
wouldn't be able to detect that but the classifier learns things differently so it can tell when there's a when there's a pretty significant difference or when there's a calculated difference between these these 2 images. Let's save this. Okay typo. I was lazy to redo the demos uh typos there. Im save okay save the image and let's
33:05
actually look at it. This is the difference between the 2 images. So if you add the normal image and this noise vector and this noise image you actually get this. Yeah. And
33:20
let's look at uh the the differences between adversarial 1 and adversarial 10. Um you would expect that uh F10 has a larger has larger perturbations. Okay more typos. I have to calculate doc F10 first. Okay so instead you see now that instead of um having
33:59
perturbations of of of length 1 or 2 then now you have larger perturbations. Okay and
34:10
same for the automobile. So why can we do this? Basically it's an open research problem and not everyone in the research community agrees on why you can do this uh but mainly
34:23
it's the concept of blind spots. Uh machine learning models learn in ways that are vastly different from how humans learn and uh there's this concept of the data manifold which is the mapping between the input and output. Um and there are gaps there are pockets in the in the data manifold that allow you to do such things. So how do you generate
34:44
images like that? How did I generate images like that? Um the intuitions are just 3 steps. The first thing is you have to run the input through a classifier model then based on model prediction derive a tensor, a vector, a matrix that maximizes the chances of misclassification. You can do this in 3 methods that we'll touch on a bit later. And
35:04
then you scale the perturbation tensor by some magnitude resulting in a perturbation which is the noisy image that you add you add to your original image and then use it as an adversarial image. So that will result in an image that tricks that tricks classifiers but not humans. And obviously if you scale the perturbation tensor by a larger
35:22
magnitude you have a higher chance of tricking classifiers but then you also have a higher chance of the human detecting that this is this looks weird. So a couple methods you basically traverse the manifold to find blind spots um in in input space. Uh there's there's also optimizations for that that help you to uh do this more
35:45
efficiently and do this in a period of seconds instead of hours. And then there's also like better better optimizations that allow you to look at how much each particular pixel actually affects the output. This is called the saliency map and I think it's really cool. Um and you only change you change those um uh pixel values by a smaller
36:05
amount than you than the other pixels that don't affect the output as much. So it you can affect the output more without changing pixels as much. So we'll look at the threat model. Uh the more you know about the the model of course the better you can do against something like this. Um if you know a lot about the architecture uh the if if
36:26
you know the training tools used even the framework or library used then you can simply use that and train the model and and generate some adversarial images and you'll be good. So that's easy. The hard thing is when you have only labeled test samples. Let's say you are doing you're dealing with an online service like Amazon
36:44
machine learning as a service or other machine learning as a service start ups and you want it to induce some kind of misclassification on them. Then that's a bit harder but you can still do pretty well. So you can do a lot with limited knowledge. You can make good guesses and infer the methodology from the task for example. If it's image
37:02
classification digit recognition you can use something like convolution neural net. Speech recognition then you use something like recurrent neural networks and if it's general services like machine learning as a service then you would use a shallow network because these networks have to be easily generalizable. So what if you can't guess? Can you still do anything? So this is a small example of a capture crusher. A
37:24
capture crusher is uh this really cool project and what I'm gonna be doing is reading captures with capture crusher and and uh testing them on captures that I'm generating with cool PHP capture. So let's just generate some captures here.
37:43
Okay this is the evaluation model um it just reads in the samples and then tests them one by one and gives a prediction um it prints out the actual label and the predicted label so you can compare them on the command line and then precision at one
38:00
just means the top prediction so the top confidence for the for the prediction. So usually when you run such classifiers they'll give you like a ranking of um we think this is the most likely and this is the second most likely so precision at one just means just compare the precisions for the top most likely. Generate some captures
38:22
with cool PHP capture uh generate cap generates captures that are pretty similar to what you see out there um you can train the model to to work better for different kinds of captures, different kinds of perturbations but uh I have problems reading some of these so I think that that qualifies as a good capture according to what I'm
38:40
seeing on the web now. Okay so they're just random captures let's generate some new ones to use for our tests PHP capture again okay okay then let's run it. So this is the
39:03
training of the model um training deep learning models take a pretty long time uh this is an output I don't want to bore you for 30 hours uh but you can see that I started training this model at uh on July 12th 553 AM and it completed completed July 13th 9 AM so
39:27
that's about 30 hours okay so it does pretty well model accuracy 98.2% so this is like no humans involved it's just reading just reading captures reading the images and then
39:40
it's learning how to read them so you know like that death by capture these services use humans and interesting idea is to use this and make some money with a server farm so you can make money solving captures by uh using this tool okay so you see it makes correct predictions skip forward a bit okay so in this case you know there's only 10
40:07
samples so it predicts all of them correctly uh you can see that uh for the last for the first sample it actually read IACTGB and let's look at the actual image let's look at
40:24
the last image okay no okay so let's look at tricking this model the interesting thing is that um you can generate adversarial samples for these models with live model it's just
40:45
uh edit of the of the tool and it's just a walk through of the code um it's online and then let's test it out and you can see that now it's pretty good it's pretty good
41:14
something very different so if let's say you were facing some kind of uh capture solving tools problems on your website um then you maybe wanna use something like this uh if you
41:27
suspect that someone is using deep learning to bypass captures on your site then this would be an interesting thing to do where you would predictably break these tools and decrease their accuracy by a lot so this is an interesting cat and mouse game because the
41:41
deep learning models uh the deep learning models are used to bypass your capture can just take these new images and train them and this is how you would make your models more robust you would take um these adversarial images and train it on your model to to to make them perform better against adversarial images so there's no end to this so why can we do this two things transferability is the first one so the adversarial samples that
42:05
fool a particular model uh have a good chance of fooling some other model so even if it's a vastly different architecture uh let's say you're training your model using um a decision tree um and you see that you can bypass uh you can bypass this model using
42:25
adversarial samples generated with a deep learning network with a pretty good with a seventy nine point three one percent accuracy with seventy nine point three one percent uh chance so this is this is kinda weird why why can you do that like it's still an open research problem the second thing is that substitute models you can always
42:43
just use substitute models uh to train the the target model and generate adversarial samples with this target model a substitute model and then you can use them on these networks open research problem so what this means for us is that deep learning algorithms are susceptible to manipulative attacks and you shouldn't make false
43:01
assumptions about what the model learns uh you should always evaluate the model's resilience and not just its accuracy and these are some ways that you can use to make your models more robust and I'm introducing this framework today called deep honing which is a meta spot for machine learning and uh it's just this there's a
43:23
github page which you can find deep honing it allows you to generate adversarial samples for arbitrary models and use this to test your network so please play with it and contribute uh this is important because more critical systems rely on machine learning and
43:43
thus there's more importance on ensuring their robustness and we need people with both statistical and security skill sets to evaluate these systems and to actually know when someone is trying to bypass and know how to protect it so in other words learn it or become irrelevant that's it thank you
Recommendations
Series of 2 media