Okay, say something no the audio is definitely broken again Now say something time with the audio issues. There we go. The audio seems to be working now. All right, awesome

Yeah, I almost feel like we gotta have whoever Is the poor soul stuck with jockey and this in the future almost needs to have two laptops like one with like Twitch and everything going and one with just like OBS and zoom. Yeah

Yeah, that's nothing can confuse OBS's tiny little brain but then the problem is when you forget to mute that thing and then this thing and then there's one of your tabs is doing the wrong noise and you're like Oh, oh, this is wrong Yep, spin you're being heroic. It's just

Yep, we are live by the way, the internet is listening to us talk. So enjoying the banter. You're still being heroic Yes Yeah, so for those of you watching Sven has been the the main guy troubleshooting all of the audio issues and fighting with the stream for

basically all of Def Con and it's a Testament to his hard work that things have gone as smoothly as they have You've made it a lot easier that things were pre-recorded because can you imagine trying to hunt people down Hunt speakers down and then their sharing is not quite right and all that

Yeah, no, I mean this year at least they're not passed out in the bar How many people have been passed out in the bar None of ours are they know of but we've definitely hooked a few of them out from the nearest bar Yeah Well, you know

So Expecting anyone else or we want to get this show on the road. Yeah, let me well Let me pull up this thing for myself I'm gonna try a new layout

As soon as I can find the thing So while I'm looking for it Rich do you want to introduce the paper to everyone? Oh Sorry, so we're gonna do a couple things Hi Barton, so we're gonna start with just the four of us talking about it for a bit and then I'm gonna open up the

join the Voice chat in the Def Con discord. And so you guys will be able to talk on stream and talk with us and everything so For the first like 20 minutes is going to be just us and then we'll bring everyone we can in and have a bit

More of a Wild West thing. I'm gonna try out how that works and The other thing is I'm gonna play with the layouts a little bit. So That's the structure of it. We've also got a journal club coming up on this Wednesday We will put the link to that paper in the Def Con discord and then you if you want to participate you should join our

discord Link in the Twitter bio Yeah, it's in the Twitter bio And you can just DM one of us and I'll we'll give you an to send you an invite to the a village discord Please don't use the a village discord while Def Con is going on. Please be respectful of the hard work They have put in into building their discord and getting it running

And only use it like tomorrow with Tuesday So take it away rich Okay, cool. So this is actually a slightly older paper. It's from 2017 It is called summoning demons the pursuit of exploitable bugs and machine learning which is probably one of the coolest paper titles

I've come across in a while And it's by Rock Stevens and Tudor Dimitris's group. So the basic idea of the paper It's kind of an interesting spin on the adversarial ML papers That I think started coming out about the same time and in this rather than actually like using the ML

model itself to sort of guide inputs They're looking for more They're more looking for flaws in how the different components of the system hook up and to do that. They actually crack out an old classic American fuzzy well the fuzzer and they reinstrument a lot of it so that

weird actions from the ML components of the system register as crashes to AFL and so that lets them Essentially do sort of like a black box Adversarial attack if you want to think of it that way against the ML system and what they find is that in addition to sort

Of the usual stuff you can get where you have like inputs that look like faces that don't register as faces because they've had Some pixels perturbed. They also run into like image format parsing errors where a component of the ML system Won't parse for example a face correctly completely fails to find the face

But if you open up open it up in other programs, it actually looks good so I I think this is kind of In the vein of the same sort of things that Ariel Herbert boss talks about a lot where it's not just attacking the ML system itself. That's important You've got all of these other components in the pipeline from ingesting the actual image all the way through feature extraction

Pre-processing and so on and each of those is actually a point that you can attack and so in this paper They're they're basically using fuzzing as a way to Find some of those, you know broken assumptions or breakdowns in the link from from one stage of processing to the next

And I thought it was a super clever paper with a really good name and they dig a couple of CVS out of it too, so super cool application of sort of more classic security methodology to What from my perspective is kind of an adversarial and I'll problem so that's my

That's my I guess like two minutes summary of the paper. So I guess maybe to kick off conversation

Is so does this do you feel like does anyone have an opinion strong opinion one way or the other? if this falls into the same sort of general field of adversarial ML is sort of like Gradient based attacks, like I'm trying to think of the name escapes me off the top of my head like Fast gradient sine method or like these Jacobian based methods or saliency based methods

Or maybe even like you can almost argue. It's like a genetic algorithm based attack against a particular ML pipeline. Well to be honest, I thought that adversarial ML covered all this already. I was a bit surprised that it was new

Yeah, it's Genetic algorithm based because they're using fuzzy lock which is genetic algorithm based which is also interesting because they were network Extensions genetic algorithms might be more appropriate if you're searching spaces So there's some interesting stuff on the front that could be done. I kept wanting to say what happened next

Yeah Context of Healthcare, you know practical realities of securing systems like it seems like a lot more Emphasis right now

Is placed on more theoretical attacks where you know, it's more so once you have like Inhibitive access to being able to query this API. It's being explored. So it's nice to see a paper like this That's a bit more than an all systems set of things because at least from Like sort of healthcare. It seems like it's a more more likely encounter that you'll find

Yeah, I did one thing that I did really like was their focus on implementations rather than just theoretical properties, right because like in some sense a Fast like something like the fast gradient sine method would

Apply to almost any of these models, but here they're sort of like, okay This model was instantiated in this way and is doing these things and now how can we attack? Sort of that specific version of the model, which I thought was cool because it's Yeah, it's it's a lot less academic in some ways and a lot of the adversarial ML research seems to be very academic still

This was like down and dirty and how do we attack listing? It's kind of in my vein of stuff We just look at the system work out with how to break the system And take it to pieces. I We're going to take out the things that actually broke and then we're going to induce breaks for the things we care about

Yeah, yeah changing the target It was just beautifully done. I Thought it would be useful for defense as well because we're also looking now at things like No straits work

So Britney Prostokoff's work on attacking with robots What could we do to? Screw up their vision systems So how how could we we'd be mess that so there's some beautiful adversarial there, too Hmm, like they can't see they can't move. Yeah

Yeah, so actually I thought table one in this was I don't know. I'm surprised It's the table one It's I think it's on page four. Yeah. Yeah Yeah, they actually like I feel like again this is very much like Ariel's jam right they go over sort of the attack surface and the kinds of things you can do to it and it reminds

me a little bit of The contest there was so there I think it was late a late entry to the contest last year that was announced at the AI village where they were trying to do and elevations and one guy actually found Essentially a bunch of ways to just completely trash the future extraction so that the ML prediction act just wouldn't run on the

Samples he uploaded And so yeah, he was he was a little upset that that that didn't count as a win Which I think is justifiable right? Because if you can just like stop the analysis cold, right? That's almost as good as evading it. But yeah

It depends what you're trying to do if you kind of sneak into a system and it's a bunch of systems We just want to know the usual thing get in do you think get out without no one noticing And we we see enough of that in real life of the stuff. It's like oh crap. We didn't realize that was a problem before someone else adds them

There's also another thing for that like if you wanted to tack the ember The ember model with the ember data set it only looks at your headers and you can just change other parts like Don't even think about it in terms of adversarial ML just copy like a notepad dot exe's header and just put your own

Entry point function in and start launching weird stuff from there and that's a different way to bypass the thing That is also not academics also not aware of Yeah, so that's sort of like I guess that's

Maybe Like a you can classify that if you're gonna I'm gonna screw this up So if I'm getting the terminology wrong security people, please like yell at me, but it's almost sort of like a logic attack on The feature extraction rather than what they're doing here Which seems to be more going for four crashes, right? If you know how the feature extraction process works

Then you can sort of write your way around it deliberately rather than just throwing shit out and it's throwing shit at it until something blows up Some of it's blowing it up some of its Nikki in the back some of its like sneaking in the back conventionally in depending on your system

The other thing interests me is that only some of these became CVEs At the response to the people who are responsible for the algorithms that some of this was like won't fix and who cares And that's part of our Our journey, I guess is just like every other security thing is trying to get people to accept that yes, you may have to drop a little bit of

Responsiveness in your system, but you'll get it more secure. It matters Yeah I'm kind of reminded a little bit of some of the argument that went around for the the proof point

CVE where people were arguing whether or not that actually counted as like an information leak CV because so if if people aren't familiar the short version is The people who did it will pierce and someone whose name i'm forgetting. I'm, sorry uh, they found a way to essentially like Send an email through proof points

Spam detection service that would then bounce back to them and in the header of the email It had a score for for how spammy it looked And so that essentially was the information leak the fact that they could get that score back and that was enough for them to rebuild Essentially a proxy model for it um, and yeah over on

In in various like ai village discussion channels we had very strong debates over whether or not that actually should have counted as like an information leak cve because you know, it's In a lot of cases it's useful contextual information, but when you look at it in an ml context

You're actually enabling like a model stealing attack which again, it's one of these we're kind of in new territory as far as the security of these ml based systems goes and and A lot of people that are more familiar with conventional security. Maybe don't quite get what these various information leaks or um attack surfaces could do once you wrap it in

how ml space systems work And I think another thing we get out of this is by doing this sort of attack. We start showing where the things that break are And one of the things i'd love about the fact that ml sec exists is it makes machine learning better

Um just by the end of the spotlight we throw on it you chuck hackers at it They take it to pieces to show you where it's broken And hell we spend most of our life trying to get our damn systems to work But i'm going to give a little bit of history i'm old Yep jump in for just a second. Um, I just saw in the discord chat

So it was will pierce and nick landers who got the proof point cv. So i'm i'm very sorry that I forgot your name Nick, okay. Sorry, carry on I was I was just going to say about sort of um, some things they're finding that it's double float precision thing so Way back before we had I think we had python but we sure as hell didn't have psychic learner friends

We had matplotlib and not map not matmap We use matlab for a hell of a lot of machine learning stuff And there was a neural network kit in the matlab And that neural network kit suffered horrendously from underflow errors That you could break it really really really easily. So Aston university built its own

They built a separate kit because they couldn't get the first one fixed I mean this this sort of stuff like the float to double errors I mean let i'm crossing my fingers that actually all of those hidden faults That are probably still going on with your network stuff and deep learning

We're not even noticing because we just trust the systems Yeah, I mean even in even in python, right like Under the hood a ton of these neural network libraries and pi torch at least sklearn as well Um use pickle which is

Just broken kind of by default, right? It's a stack machine that you can write arbitrary code for and yeah People still share weight files, uh that are essentially pickles. And so, you know, you're downloading this off the internet and Running code and I keep threatening one of these days. I'm actually going to write a A pickle poisoner that is just like replaces calls to like the relu function with like some sort of stochastic

Function so that half the time it works and half the time it gives you garbage comment from A comment from discord. Yeah, numpy as well uses pickles. So it's it's everywhere. Keras. Yeah, is that what the

Save file is the numpy save file I thought that was secure I'm not sure. I know i'm pretty sure it has one One pickle thing in there one or two. It's got one or two formats that um, Yeah That are are not apparently there's a couple that the the psychic learn bomb that these guys found

Uh was numpy was part of that. I'm just flicking to the page page seven at the bottom if you're reading along with us One interesting thing is that they weren't able to induce like misclassification or uh from the Precision loss problem so they they've tried different values of epsilon and basically said that it's possible in theory, but

It's interesting to consider. How would you model? Um that epsilon to uh, based on this attack to Induce this classification. Well How much of this is induced by uh, basically all the machine learning code is written by academics and over-enthusiastic

like over-enthusiastic academics guilty Um who aren't experienced in Like it's incredibly hard to write numerically efficient code So you don't have time to like really learn how to like multiply matrices incredibly efficiently

And make sure you open files correctly so you've we've chosen a lot of lazy paths to formatting and opening things versus secure paths and And also you've got tiny numbers and people don't really you go waiting on in build your stuff and don't think about how those numbers Interact you just assume the system is going to magically deal with it

Yeah, and I mean there's also the fact that Writing secure code is very much its own skill set, right? You've got you know Some people are writing research code because they don't expect it to be used in anger They don't expect it to be used in a production context

Some people are writing numerically efficient code because that's their jam and god bless them um And then you've got sort of security People who are probably gazing at a lot of this with sort of a feeling of vague horror that they're actually going into production yeah, I know even um

Yeah in my in my day job, right a lot of what we've been doing recently is sort of going back and and Like rethinking how all of this stuff is put together so that we can like feel confident and actually like pushing out products containment There's a middle ground where you can enable basically more flexible code, but also execute it in a way that's um,

Perhaps minimizes the blast radius and I mean, I don't mean put it in a container obviously, but it's uh something more. Uh, To find of like running code in secure enclaves for instance the work with uh silo, uh, uh type of workflows or Do you see anything where um You try to mitigate the impacts of this code

And could we have code checkers? I mean literally just some way some sort of test data sets But we know this stuff is bad run it through through your system Yeah Yeah, I mean we I I had a great amount of fun. Um Again at my day job actually like taking our feature abstraction code and fuzzing it just to see what would happen with it

And you know, fortunately it turned out that that it you know, I spent more time than I care to think of beating on it and it it turned out fine, but Yeah, like if it hadn't what would we have done then right? It would have been And I think the idea of like building test vectors of just like, you know, really bizarre

But technically conformant files for this sort of thing is is an awesome idea But extending that to The uh actual data science part like how often have you Issued a pr about some data science thing and sent it to someone and they were gone. It looks good to me

But like a lot of the stuff like there's some stuff that you can check easily but like a lot of the uh, Like a burt like the code that goes into burt is incredibly complicated Very hard to check when you make a small change, you could have broken things fundamentally because it's uh, You know software

How do you like write a unit test for I broke burt when you input this one slight thing and now it uh, Causes this whole class of things to be uh, false positives false negatives Like how do you yeah, I mean there's a machine learning There's a I mean you can do

Um, I mean essentially you you do test vectors, right? You say, you know If I load because the weights you can consider the weights are obviously like part of the model You say i've got this set of weights if I load in You know this set of features I should get this exact set of outputs up to like numerical precision which again, it's it's numerical precision is is kind of its own nightmare, but

um yeah, but then if you change the weights, right and then then kind of all bets are off and that's That's like the one of the painful things about Debugging ml even when you're not under adversarial conditions is just I retrained the model

um How do I still trust it? I mean there's a difference between I messed up my models, which is like we do this all the time And somebody is actively trying to attack my models or may actively try to attack my models using known known vulnerabilities Because sure as hell half the system is out there haven't patched any of this But how many machine learning developers even know unit testing or even know good software practices

Like they can't a lot of us came from academia. I have a bht in math not software engineering So I have a uh a philosophical question. I want to pose to the panel from uh discord, um and yaga asks, what is an a terror and oh, sorry, what's a terror?

This is terror right here. Um, what is an error and what's an attack and Like is there Any meaningful distinction between the two? I guess I would follow up I I think if you're looking at adversarial images I mean the fact you have like the little image patches to screw up your system is an attack. It's not an error

You just don't randomly get sort of extra patches on a stop sign Yeah, no matter how much undergrowth you got on it I I think the difference is intention Mm-hmm. Yeah, I mean that's the same thing in security right bugs can be I mean exploits are essentially software bugs, right or I don't maybe that's being too general, but if you

You know a subset of software bugs or exploits and you use them to make the system Do things you don't want it. It shouldn't be doing under normal circumstances But wasn't intended to be doing yeah, right and sometimes they're useful like

Would you consider a well-informed person building an attack against a Machine learning system where they understand the machine learning system They understand the feature extract and all that stuff and they have they've built a you know, a false positive Um, is that an adversarial attack against the machine learning or is that like an exploit?

Uh, like where would you classify the mistake? Is it like a software mistake or the uh, like or a Exploit or what? I think that's the tricky bit about ml models, right? I mean they're inherently statistical so

It kind of gets back to like testing after you retrain it right so I might if If you have if you have a single example and it turns out to be a false positive And that's maybe that's one of the bridges that I feel like as a data scientist I have to to cross a lot when i'm talking to security specialists If I have a model and it fps on a single file

That doesn't call into question the efficacy of the model, right? You have to look at the statistics of what the model does right and if you say well yeah, it got this fp and that was a pretty boneheaded fp but For 99 detection you get one fp and in 10 000, you know negative samples

that's a pretty good model and so being able to sort of like Drive that switch from hey, here's the you know, here's the good bad um To hey, you know, you have to think of this sort of heuristically and statistically Uh is is a big part of just like communicating as a data scientist, I think

But but I mean, no, sorry Sorry, go ahead. Sarah. Yeah, if you've got somebody actively attacking then maybe some pre-processing is enough to To cut down the attacks. Yeah Yeah, for sure, right All of that goes out the window when you've got someone Maliciously trying to yeah Yeah, but also my question is like so if you have like a fancy ml

from an academic paper bypass of um your model where versus someone trying things What would be When is it like a machine learning mistake or versus a like a mistake in your featurizer and stuff?

If you look at the Decision boundary basically for something in the indexing how the perturbations, uh relate to that Yeah, I kind of feel like Maybe to to pull it back to this paper a little bit If it gets through your feature extraction process in one

shape There it like in one piece then maybe like It's a modeling issue and if it like crashes your feature extraction process or it gives you something that Like you wouldn't expect from your feature extraction process then then it's then it's a featurization bug, right?

Like if you if you feed garbage into a model, you can't blame the model for producing garbage outputs Which how would you basically what are some methods for quantifying like, uh, unexpectedness in this scenario? Is it more like error counts or like uh latencies in response? Like what would you pay attention to if you were trying to understand?

If something was misbehaving You know what when you see it, yeah, like if you take a look in the in the paper, right? They've got an example. I want to find it. Um Figure two, right? So they show an example of two different images one is an image of uh,

Sort of the very top of a person's head and then the rest of it is gray And one is a picture of a person's Person's head. Yes. There you go. Thank you um H6 if you're following along right so open cv has a bug that these guys found by their uh guided fuzzing technique

which Means that open cv can't properly load the file and then you run it obviously you run it through feature extraction and it doesn't get anywhere so It's like all these different individual components that you got to worry about, right? And I think on that topic there was a from hacker factor in discord

He's got a question. Are you making a distinction between the ml model versus software that drives the ml system? Um, he says fundamental error versus algorithmic and I I think here right? This is what this paper is going going after right? It's saying the software that drives the ml system the implementation of the system Also has bugs and they find them here and you know, if you go to table table two

Right, they've got in open cv. They've got like two heat corruptions. They've got a weird rendering bug that they found Page six. Thank you. Yeah Right, so all these different like they literally have like heat corruption bugs that they're discovering with these funky inputs. Um

And that has nothing to do with the model. That's your you're attacking the plumbing around the model, which is But you can use that plumbing to go and tweak the model Right, exactly. Yeah, so that's that's what the um, the third line down does that's the the figure figure one figure two I forgot figure two

Yeah, yeah So you break the rendering and then your the rendering goes to the feature extraction and feature extraction You can't get anything because it's just got a bunch of gray pixels to look at And then the model right garbage in garbage out the model can't do anything because it doesn't have good features So i'm going to try to open this up to discord

uh If you are in the different discord go to a village general voice We'll try Unmute you And get you so you can talk If you're human plus that you may that's the correct way to do it, but we have a workaround

So head over to ai village dash general dash voice to ask any questions If you can't we'll still be paying attention to the journal the text chat in ai village dash journal dash text the moment of truth

I'm watching journal club text. Yeah, me too Audio managing audio is i'm so glad I don't have to deal with it They Um I wish I had seen this paper Before I started like really trying to solve adversarial machine learning as a security problem

Because there's a lot This seems a lot easier like you to attack for a lot of attack people I think a lot of attackers have more knowledge in how to break a Parser than how to break a neural network

Mm-hmm So this to me feels like more of a realistic threat model of like your uh pipeline was They they broke your pipeline. You did something wrong in aws. You did something wrong with your uh Container setup and it's parsing your images incorrectly now

yeah I mean that's tons of the bugs that you find in sort of real systems right as parsing bugs and Yeah, a lot of a lot of the research right now focuses less on that and more on sort of these theoretical properties of you know machine learning models, which

I mean, I don't want to dig on those. They're fascinating, right? I love these papers, but they're also Again, keep coming back to this paper This paper makes the really good point that these sort of theoretical models that have these nice properties are embedded in real world systems

And we know that real world systems are always kind of full of bugs that can be attacked and exploited And so it gives you sort of multiple entries Into the problem, right on the one hand you can exploit the pre-processor And maybe that gives you some sort of heap corruption or some sort of like direct You know denial of service or or even code execution

And on the other hand, you can break the feature extraction enough so that everything downstream is getting garbage data Or mildly corrupted I mean I I say again with like the sneaking in you want to do low and slows on some of this Again I am really like the idea of using this as defense. I mean there's a whole bunch of

applications where the bad guys are using machine learning based systems I like the idea of breaking their systems Yeah as I would great hats on Yes Yeah, and yes, sorry go ahead yeah just like yesterday's paper with the forks masks

Yeah, so, um Will has a comment. He says i'm surprised a paper is the vehicle for this knowledge there's a whole security industry that has advice for this stuff and I think yeah, this is part of what

We're trying to do right with ai villages draw a link between security and machine learning, right? So We have a lot of like wicked smart security people Who know about all about these implementation bugs and then we've got a bunch of like really smart academics and machine learning people who know about like sort of these

You know feature-based attacks or weird properties of ml systems and getting sort of that knowledge transfer going on so that we can Find out where those intersect and how those two, you know, sort of sets of attacks inform each other I think that's like a really rich area for future development and it

Kind of makes me sad a little bit that this paper came out in 2017 And got comparatively little attention When it really is going right to the heart of a lot of really big questions in this, you know in ml slash security so In practice you see a lot of data science teams disconnected from both like infrastructure application network security

sort of teams and so like the whole uh mlops sort of formation of teams that can interface between uh data scientists and uh implementation has has been emerging but i'm curious what the panel thinks as far as uh, what's the appropriate like

Role a person on the cross-functional team. Do we want to have security people sitting with data scientists as these things are being developed? and uh Yeah, I mean weirdly enough I used to be One of the people going into large companies working out where to put the data scientists and There was always this argument between you want to embed the data scientists out into the rest of the team

And have them work with because you're informing the rest of the team everywhere um But then you've got data scientists out on their own and we're kind of you know You see in this paper that we get ignored quite a bit on when we're on our own Or having these unicorn pens full of data scientists who talk to each other but not really to anybody else

And there's a sense of you you put people out but you make sure they're still part of tripe so you literally build Tribe or whatever the the good word now is for that It used to be called tribes across the village cross. Yeah village. Yeah village So you build villages for the data scientists to keep connected to each other, but they need to be out in

the rest of the teams Because because otherwise, you know, we we our work is everywhere. We should be everywhere too That that's from a help belief though Yeah, I mean, I think it's but it's it's got to be sort of a two-way street right on the one hand Yeah, there is kind of for for data scientists some sort of ml

Practitioners, I think There yeah, we do sort of tend to stay in our own little bubble and don't think about these sort of Externalities, I don't know what the right word for it is but like these other ways, you know It's not just going to be like fast gradient sign based attacks. It's going to be like no someone broke the parsing but then at the same time

and this is kind of going back to maybe a little bit of the hype panel yesterday like having People that aren't data scientists Understand kind of like what the models can and can't do and what kind of behavior to expect from them so that they don't Freak out when they see something weird happen and be like, oh, you know

Whatever we we flagged this dll as being malware when it's not we're clearly under attack Right, and you're just like it's statistics. Sometimes it makes mistakes Well, just explaining. It's not magic It's hard, but it's not magic. Yeah, it's hard. It's useful. It does cool stuff, but it's a tool

Iyaga is talking about implementing the security development life cycle And this secure software development likes life cycle into model creation and upkeep. It'll make a data scientist cry Yeah this I had

Issues with just being like cool we need to like just write a few unit tests to make sure that the Functions will kind of do what they're supposed to do Um, there are there's a bit of pushback in our for some of our more academically minded Folk of like just a little bit of basics like that

um which may have solved some of these like open cv issues because I But interesting that um, if you basically treat The life cycle of like a data project as more internal products for like a company and treat it as a product Which has which serves a purpose provides value and then uh is is integrated with other ways of developing software products then

Data science and machine learning they just become tools and engineering toolbox But then everything else falls within like methods of like continuous integration and testing and delivery So then perhaps the risk rather than the unicorn depends Yeah I mean that's what we're doing. We're doing things like uh hypothesis based. Um

Development which is like the next thing up from behavior-based development Uh, which is next thing up for test-based development and test test-driven development So you very much like work out what the universe you'd like to see is go do the go to the maths on it Build the systems in it

So there there are ways of working Yeah Yeah, so maybe looping sorry, go ahead it's fine go ahead Yeah, so I was gonna loop back to the paper one more time And talk about the disclosure experience that they they mentioned towards the end of the article so basically they they got three of the vulnerabilities got new cves, um, because they

enabled arbitrary code execution or denial of service attacks Right. So those are both very firmly within the wheelhouse of what people Page seven page seven. Thank you Very much within the wheelhouse of what security people think of is. Oh, yeah, this is that's a bug. That's an issue That's a security gap that we need to fix. But um, there were a couple of other ones that they found

Um where they could impact or manipulate the prediction and then they they call out in particular the mouth here memory corruption They found where basically they can rewrite the feature vector, which um Allows you to basically like make the model give you any output you want essentially

Uh doesn't have to have anything to do with what the actual file that you were analyzing was and those specifically are the ones that didn't get the Didn't get a cve didn't get called out and mostly got labeled as won't fixed or as won't fixed So that's sort of like, you know, we were kind of bashing the data scientists earlier saying

Oh, you know, we got to like think of this as part of a security system, but then It goes back the other way, right? We need to be able to communicate to people that as data scientists as machine learning experts Look, you know, this is this is bad, right? And how do we convince people that? You know something that allows arbitrary misclassification arbitrary predictions to be to be produced is

as serious as or maybe not as serious but But has some degree of severity just like a remote code execution or a denial of service I like the example that eric brought up in his talk of a turtle rifle Uh, like that is a very effective way of communicating potential problems Whereas if for those who haven't seen it, uh, like a uh a turtle, uh,

A 3d printed object could have uh be applied to a turtle to make it misclassified, uh, being classified as a rifle you can imagine like systems that uh, security of uh and stuff to induce certain events I mean, it's like any other ethics discussion we have to talk about consequences

And then track it back and just keep pushing and pushing the examples real examples if we've got them It's it's hard Yeah, I mean I guess like that's You you have a joke But we're not really a joke that like airline regulations are written in blood

Right. And so when you've got all of these safety requirements It's because a lot of people got hurt before someone was like, oh we ought to do something about that and it would it would be nice to think that we could we could find a way to not have like Ml disclosure requirements written in blood right like maybe you know, let's let's

we're we're at a place now where we could maybe like jump ahead of the game a little bit and and Get an understanding of this out there so that we don't have to see people hurt before people are like Oh, yeah, we should maybe take that seriously. I actually have a historical example for that too for being old So I ran one of the unmanned air vehicle safety teams back in the day before uavs were sexy

and We didn't have the Privilege I guess that the s the rest of the airspace industry had So aerospace regulations are literally written in blood. So an aircraft would crash into something else or

Hit the ground and you would change the regulations and that's literally how the regulations were written same with fire safety fire happens change the regulations Um and We Were told explicitly and understood explicitly that

We couldn't just put uavs in demand airspace have them crash into stuff and then rewrite the world rules We literally had to think all the scenarios and Write the rules for how to do it safely And it's whole space safety not just individual aircraft safety before we're allowed to fly in the same spaces

So there's some precedent I mean there's maybe some looking in those spaces to see how it was done well As a good way to look But a lot of it's just getting the will Now nothing in focus is retention like a whole bunch of airliners about to crash but yeah

Yeah, you know we could maybe get there before that point that would be So, um That's where I wanted to go next. Um, so yaga has the question. How do we force open source software to fix itself? We don't fix it ourselves. That's why it's open source and the authors actually make a comment in the future work. Um,

Where were they? They say that it's unclear who should be responsible for fixing them as well. So Um when they found a bug in in the malheur feature processing that was because malheur relied on libarchive Uh, and the bug was actually in libarchive and again this gets back to this notion of like

You have all these different components that feed into the system um So even though the bug was in libarchive it affected malheur. So where do you Where do you put the responsibility for fixing it? And how do you convince? You know, like whoever's maintaining libarchive. Look, this is serious enough that you know, yes, I don't think it doesn't affect you directly but

It does affect this other system and it could be a critical impact like how do we How can we navigate that sort of um uncertainty? How does linux and uh, like a one two and others separate open source software ecosystems do it?

Do they have liability? I mean, I know for a lot of systems the end users especially if they're big end users just get involved with the open source communities and work in there, but Uh if you're using Even like python and your physical system crashes who's responsible

Yeah, I mean a lot of licensing argument a lot of licensing agreements basically say like yeah You agree that you're using this at your own risk, too. So You know, basically everyone's going around like not me

Which you know, it's fair right and people have like they have they can't assume liability for for everything that they do but um You know, they can't agree to affix affix every single thing But at some point right there's there's a balance right if something's widely adopted It's widely used and there's like a bug three steps up the chain. That's that's causing it to behave weirdly

It feels like somebody ought to There ought to be like some impetus to fix it somehow Low tail companies like fortune 500s and whatnot that use these open source libraries in theory, you know, this is ai so you have a lot of Boosting productivity like and there's a lot of value that's being derived and especially if they're starting to use it for critical applications Or decisions

so I think the onus responsibility also needs to fall in the end users of like widely adopted packages and then somehow make it making it very clear to decision makers in those companies that hey if you're using this to Predict insurance rates or something and uh, it fails. That's what you could potentially face and then somehow Channeling that into the projects themselves Repurposing that large companies that make a lot of money using open source tools should actually give back to the open source community

Smells like communism to me, it's kind of terrifying that we all just laughed when you said that Yep, sorry, I gotta get my gotta get my commentary in once in a while

Uh, these are complex systems and like just communicating The impact of some small like overflow bug to somebody who only cares about like the bottom line is is is a problem maybe there needs to be like a Awesome ml failures github repository or something like that where you can find it We haven't had that Sounds like actually a pretty good weekend project for someone. Yeah

I mean like, you know when we started up the ethics stuff years ago It just just takes some people determined to make this thing a thing a visible thing. Yeah. Hey community Anyone want to take this one on? But so how many like solid examples of ml failure do we have

If we want a lot of it is compile those. Yeah, how much do we actually know? Yeah that's um I want to say uh andrew davis, uh over in different discord had a uh, pretty good comment on it Which is that a lot of time when ml fails it kind of fails silently, right?

It just you you don't flag something and it comes back to A lot of the earlier discussion about is it is it a bug or is it like normal just sort of like statistical? failure right a one percent A one percent failure rate means that if it's not failing one percent of the time you should be vaguely surprised, right?

so yeah, like I think it's possible that there's a lot of ml failures that We just kind of assume are in the statistical noise a lot of them are probably kept quiet because you know There's potentially like pr risks or pr damage, right or you know, like like actual like gross You know liability attached to it

Yeah, I mean it's it's an open question how much is is actually sort of flying beneath the radar out there I mentioned a rental company say or a background check company who uses ml going uh, so we didn't give a lot of people their uh, Approve, you know their credit check approval

Because of a mistake and we want to publicly apologize for that When it's an ml mistake and they can just be like well the model just was yeah Sweep it under the rug fix it quietly never mention it I think that would be an ethical nightmare

Yeah, I mean there's lots of Startups, um that have proposed to do things. I was one I was about two three years ago, which was like, oh, we'll use ml to like pre-screen your babysitters by like digging through their social media profiles and stuff right, and I have my own opinions on how likely that was to succeed versus how long it how much how likely it was to simply just like

recommend white people but um You know, like how do you how do you even like measure or quantify failure Failures in that case, right because it's all going to be completely internal and all you're going to like all you can do is

you know Essentially like try and like get a proxy model to be like aha here are the features, right? I've submitted a thousand different profiles and here are the features that seem to be like triggering right and that's You have to like Affirmatively dig for it right those failures aren't going to be obvious unless you're actually doing something like that

A recent streak of sort of facial recognition, uh failures that lead to Uh in character s there are several other characteristics, but that elicits like visceral gut reactions like yes, this is wrong So I think it's in some cases just like seeing it like the outcomes like there's a helps Understand whether it's like a This gets yeah, I mean

So the bias question like how do you tell if something's biased or Just like for this one instance. You were wrong. It was wrong. Oh, man Yeah, we're gonna have to clean that one out for me yeah racial bias in images is oh god How much longer do we have for this conversation

Yeah, we can solve it in eight minutes we've got eight. Yeah. Yeah, no problem. Yeah. I mean the the problem The problem in a lot of the bias stuff right is like the terms aren't even defined sometimes so like The the the example I keep coming back to is like recidivism and like predicting rearrest rates

Right and someone might point out. Hey, look we we you know, this is biased, right? It's clearly predicting rearrest rates that are much higher for this group Right and one argument we would be well, that's clearly wrong, right? They're you know, it's biased because it's trained on Biased data and the counter-argument would be like look it's reproducing the data, right?

We don't live in a perfect society This the data is what it is and within that box you the ml has done as much as you can ask it to do Right, and so it always come it always boils down to these normative questions and in this is my own totally personal opinion that

You you can't like talking about unbiased systems unbiased ml is is almost inherently impossible right, it's it's a question of how Is it producing the kinds of outcomes that you want and how is how is the ml system moving power around?

Right, who is it empowering? Who is it disenfranchising? And you know, who is it helping and who is it? Who is it hurting and Chasing after some like vague notion of fairness, you know a fair system or an unbiased system is

Is kind of like a distraction from like more fundamental questions of like who's being helped who's being hurt? What? What's being reinforced or whose concerns are being downplayed? And who's accountable? I guess also because of like reference recent twitter's term between Everyone basically like is it a mal engineering or is it actually the data scientist or is it throughout the entire chain? Yep

But yeah, and I mean Sorry, god, i'm talking too much I'm, i'm seeing over in the chat that we already have awesome ml failures in the ai village as a repo And stellar athena seems to be um getting in there already

Suggestions for implementation failures ethical failures hci failures and security failures You got that stella Yeah Yeah, I think uh, do we do we have a category for because we were playing around with this cat does not exist Um a couple days ago. I think we maybe need a category this ml should not exist

Yeah nightmare fuel Yeah, well, I mean i'm thinking of things yeah, you're right there are how a bunch of machine learning systems So just should not be on this planet. Yeah, and anyone asked to contribute to it should have been like perhaps not um But yeah, I mean as as far as responsibility we would never have had facebook if anyone had had that call. Um, anyway

I mean hindsight, but you can look at like really uh short-term stuff like um Even just like genderify, right? It was should have been pretty obvious everyone involved that this was just like a losing proposition from the start Yeah, and the recidivism through face images stuff from china. Oh god. Yeah. Okay, so way to replicate somebody's bias

We do have we do have I I just so we do have a uh a workshop coming up in a couple minutes so I think I want to kind of like Put a pin in this. I think if everyone's enjoying this conversation stella's having an awesome ethics in ai panel Which is coming up in a couple of hours. I'm not sure someone someone fill me in

Two o'clock two o'clock. Thank you. So everyone should absolutely turn tune in for that if they're enjoying this discussion Um, but yeah, I guess I want to just like go around Just to wrap this up. Um and See if we have any like last minute like big idea takeaways from this paper that people thought were really really cool

Uh, I love the way they did it The idea of attacking the system and looking at taking it apart it's just kind of my thing and I like the idea of using to attack systems that I It shouldn't exist Yep Sven

uh I really like the idea of attacking the feature extractors and I think We should also add like bypassing the feature extractors by just doing something that you know, the ml system is going to miss uh because You understand it well to this whole thing

cool, barton Yeah that also really enjoyed the paper and discussion. Um, I I think um to me, uh, What's clear is that it can occur in many levels of the um, uh, basically stack so just Instrumentation and stuff become very important to monitoring what happens and then I would really like to see full on work that uh applies this to tensor flow by torch basically deep learning libraries and seeing how you can

play with Um intermediate representations and things like that to do certain things Yeah, for sure. Yeah the reminder that like these ml things ml Products don't exist in some abstract theoretical space and are implemented in real code in real systems that come with their own flaws was

Yeah, like, you know it but being reminded of it sort of viscerally by look we have cves is always is always a great thing so Cool. Um, so I think this is we probably want to leave a few minutes for people to fight with the audio As we transition over to the workshop. So, um, I think we want to call that a panel or a discussion. So

Awesome, thanks everyone. Thanks. One second. Oh before we jump off, uh, we have one on wednesday at 5 p.m. Pacific time uh, and we have the I'm guessing I think we should go with your paper rich the slide deck on Yeah, the anomaly detection

Outside the closed world on using machine learning for network intrusion detection by robin subbon and uh, veron paxson so we'll post that in the The defcon discord it will also be posted to our discord and twitter So if you want to participate you should find your way over to our discord. We can't post a link to that in The defcon discord, but the links are places and you can just dm us for an invite

Uh, and we'll be discussing that for probably an hour and a half on Wednesday and we might be able to get one of the authors in even though that this is a 10 year old paper How far we have come All right, thanks everyone catch you later