from the company. But yeah, let's talk about back dooring pickles. So 11 years ago, Mark did a talk at Black Hat called sour pickles and in it he talks about how to do deserialization attacks against pickles. Not a whole lot has changed since then.

Back then, 11 years ago, there was a warning message, it said pickles shouldn't be used from an untrusted source. Makes sense because it is code. Pickles are code at the end of the day, it's assembly language. And the attacks that he talked about were predominantly deserialization attacks. So this is something along the lines where a web server gives you a cookie, that cookie

is a pickle, you can replace it with a pickle you craft in order to get a shell on the web server when they load it. So yeah, not a whole lot changed, as I mentioned. Pickles still have this warning message, it's a little more verbose, gives you a few more ideas on what you can do, but at the end of the day, if it's coming from an untrusted

source, you shouldn't load it. Because they are still code, that part hasn't changed at all. But what has changed is in 2016, machine learning libraries started to be released that use pickles as the way that they save their models. And so this creates something along the lines of a macro in a Word document or in an Office document where you can send it to someone and if they load it, then you get a shell

on their system and do whatever you want with it. Yeah. So I've mentioned models. Just really quick, what is that? Models are used in artificial intelligence and machine learning. For what I'm talking about, they're a combination of layers and weights that probably has very

little meaning to most of you as well. So what is a layer? A layer is the mathematical equation that lets you give it one input, like an image, and it goes through this equation and gives you a label. Like cat, dog, chicken, ostrich, whatever it is. And these layers we represent as code in Python specifically for the Python frameworks.

The weights are the coefficients for that equation. It's really just the data that's trained. So when we're training models, this is what we're training. We're training the weights. And that can just be viewed as data and because it's a combination of a bit of code and a ton of data, it makes sense to store it in pickles as long as you ignore the fact that researchers are going to be distributing it and sharing it as they do research.

And this is exactly the opposite of what that warning message said we should do. Just another little note. A lot of the frameworks tend to save a model as multiple pickles in a single file, which if you do anything with it, can get confusing at first, but yeah.

So how do you make a malicious pickle? This is just randomly, like, just arbitrarily malicious. It doesn't do anything except for be malicious. This is an example from a blog from 2011. And what it says is when you pickle up exploit, this is how I want you to recreate exploit

when you de-pickle it later. You re-create it by calling subprocess.po with bin SH. Now obviously that's not going to create an exploit class. It will just be an exploit, in this case to give you a shell on a web server. So yeah. That's not what we want to do, though, because I wanted to find a way to place a backdoor into a model that still works exactly as it's supposed to and gives you a shell as

well. So this is the simplified version of the attack if all you want is a shell. You don't care about it working still. And it gives us a nice example to talk a little bit more about pickles. So this output on the screen is thanks to a library built into Python called pickle

tools. The module is pickle tools and it's built in and has a function called dis for disassemble and it produces this output. The first box in the output is getting the function pointer to subprocess.po. It writes subprocess and P open onto the stack and then it calls stack global which takes

those two pointers, collapses it all down into a single pointer that points to the function. After that in box two we mark the beginning of all of our parameters, write them all onto the stack as well, call tuple which again collapses all down into one pointer that is all of the parameters. And then the last instruction is really the most important.

It's reduce and that calls the function with the parameters and so it takes the two pointers on the stack, pops them off, throws them away and puts onto the stack just the one pointer to the return value. So now that, you know, that's understood a little bit more, how do we make a pickle more

malicious? There's a really cool tool called fickling. It was released last year in the AI village and it's made by trail of bits. It's really good. It allows you to inject code into the beginning of a pickle or the end of a pickle and it attempts to detect if there's malice within it. Unfortunately for me it's a little more complicated than required and so it ended up being easier

for me to write a solution that only injects in instead of trying to help patch some of the bugs with how it worked. And then the second issue which would have been easy to fix if that was it is that it can only inject into the beginning unless you want to replace the output in which case it can only inject into the end.

So why do I say it's a little more complicated? So as we've covered, pickles, you can't just load them because you'll get a shell on the system or you could have somebody doing that. So they use a symbolic interpreter so that it won't happen and that works really well but you have to reimplement everything about pickles in order for that to work and I kept

running into bugs and ultimately it just became easier to do something else. And when I was trying to patch it, like, the tool they created is what led me down all this research in the first place so I really can't give them enough credit. It's just I found it was easier to use something else, unfortunately.

Now why can it only inject into the beginning? As I mentioned before, when you reduce down, so on the top of the slide in the blue box is the code that fickling adds. In this case it's execing print high. Nothing fancy but it works for the example.

So when that all runs, it leaves one pointer on the stack and in the beginning of a pickle that doesn't matter because it all collapses down, creates an object and that object's returned and you have one value left on the stack that as the error message on the bottom says it's not empty so it's technically wrong but it doesn't matter because everything works.

But if you put that same code anywhere else, it's going to alter the stack in a way where the instructions that follow it don't work correctly because the stack is effectively corrupted for what they're trying to do. And I wanted to be able to just add code arbitrarily into a pickle.

So just a few more things about pickles and then we can get to the attack. Pickles are an instruction set, not a file type. So for detection this starts to matter because pickle doesn't have any magic bytes that you can detect to identify it as a pickle. There's no specific file extension because PyTorch is .PTH and not everything is a .PKL.

It can be anything. MPZ sometimes has pickles and sometimes it doesn't. So you don't have just one file you can look at. Also there's no forking or conditional logic which makes it so that we can inject in the middle because we don't have to worry about it jumping around, byte offsets don't matter.

You can just add code in and it should be fine. As we've shown you can import any Python callable. Subprocess isn't imported when we exploited it earlier. Pickle actually imports the subprocess module and gets the function for you. So if it's installed on the system you can use it. And there's an instruction called pop that takes a value and discards it.

So I keep saying that I had a simpler solution. On the right side of the screen are the 50 lines of code that I used to inject into any pickle. And 50 lines of code was easier than writing patches. And so when I wrote this I had a few requirements.

I wanted it to not be obvious to the user or intrusion detection system because I don't want to get caught. I wanted to parse the pickle without loading it but at the same time I don't want to use a symbolic interpreter because I saw how that can get complicated. And I want to inject into an arbitrary location. Mostly that's just to help support the first point.

But also because I can. It makes it harder to detect. So how do we avoid being obvious to a user and IDS? Well we spin off our own thread so it hides in the background. We don't want to stall the pickle. We want it to load and everything to work exactly as the user expects. If they're loading up a model and it's supposed to generate a picture of a cat, they need

to see a cat otherwise they're going to get suspicious. So we have our thread running in the background that is running our agent to call back to our C2. Also size isn't a concern at all. Most models are hundreds of megabytes or more like gigs, maybe more than that. So adding one meg or one kilobyte, nobody is going to notice most likely.

That's my assumption if I'm wrong, so be it. But I do use compression and that's not to actually make things smaller. It's because base 64 makes a really long string that you're adding into the file and if you run strings and you see a long string, you might get suspicious. It makes binary data. So it was just a simple solution.

And then finally, as long as we don't launch memey cats or anything that's super malicious and obvious and going to get detected, should be fine, shouldn't be detected. So how do we load the pickle without using a symbolic interpreter? So the output I showed before, there were some numbers on the left side of the disassembly.

Those are the byte offset into the pickle. And because every instruction, there's no jumping, there's nothing like that that we have to worry about, we can make this byte offset change, make this byte offset change and nothing is going to go wrong. It will still work the same, so long as the stack is exactly as it's supposed to be.

So what we do is we pick an arbitrary line from this disassembly, grab the number, write that many bytes into the output file, write our malicious instructions into the output file and write the remainder into the output file and you have a pickle that will run your instructions so long as you're not ‑‑ so long as the stack is exactly as it was when

your instructions started. So how do we avoid leaving a trace in the stack? Well, I only use pickle instructions that alter the stack. If you know anything about pickle instructions, you might be wondering why I don't mention anything else and that's because I just don't use it. It does exist if you want to know about it.

Mark's talk covers everything and it's really good. For our examples, we only use the stack and so that's all covered. And because we only use the stack, that's all we have to clean up. And so at the end, we add a pop instruction and it pops off whatever ‑‑ well, in this case, we're adding a single value to the stack, pops that one value off and everything

is exactly as it was initially, so everything works exactly as it's supposed to. So this is a bit of an example. Here I'm downloading a pickle that I've injected an agent into for the mythic C2 and I'm loading the pickle and it takes a second, but I'll get a call back. This call back is running in the background while the GAN is still loading.

When the GAN finishes loading, you see a picture of a cat. I can tell it to start running calc, which I did. And when the agent calls back, calc will start even though I'm able to look at a bunch of different pictures of cats while it's working. Calc pops and yeah. In a real attack, obviously you're not going to pop calc because that's obvious, but you

have an agent running on their system. You can do whatever you want with it. The hard part about the attack is figuring out how to inject the code into a malicious agent or inject the malicious code into an existing pickle. The way that you go about doing this is really up to you, I guess, but there's three ways

I see it working out. There's the supply chain attacks. If someone's distributing a bunch of pickles on a website or however and you can modify them, then you can get a shell on all of their customers, assuming that they don't detect it. There's also watering hole attacks. If a bunch of data scientists have a shared network share, they keep passing pickles or

pickled models back and forth between each other and you have access to that as well. You can modify it and the next one of them that loads the one that you modified will spawn a shell on their system and you have access. And then finally there's phishing. Instead of targeting a business person, you can send an awesome research paper with a real model that really works that's actually the research to a data scientist and when

they go and load it up, you can get a shell on their system as well. And since research is done by a bunch of different countries, not always who are friends but are often sharing data, that's a bit of a concern. And then after that you just wait for a call back and do what you want with it.

So, what can we ‑‑ what can we do about this? Well antivirus software doesn't work great for detecting this. Pickles aren't like a file type. A coworker of mine wrote a YAR rule but because it's just a full instruction set, you can do anything you want with it.

You can make the code do anything you want completely. You can load any library. So it's essentially detecting an arbitrary Python file except written in a different form that's binary and harder to read. So the YAR rules that we wrote are pretty easy to bypass if you know what you're doing.

Maybe there's a way to make them better. There's probably ways to make them better but I still don't think you're ever going to be able to completely detect every way you can make a pickle malicious. So all you can really do is verify that the integrity of the file is exactly as it was when you created it. So that doesn't help anyone who's getting a file from you unless you have a way to share a hash that's more secure than the way you show the pickle.

Which maybe you do. And so you can do that. But you sort of run into an issue of how does anyone know that it hasn't been modified? Fickling has a feature for check safety. But as I mentioned, there's a lot of different ways you can make something malicious.

And so the last line of their output is don't unpickle this file if it's from an untrusted source because it can still be malicious. The examples I make aren't things they thought of so it doesn't detect them. So what can you do if you have a pickle that comes from a source you don't 100% trust?

My recommendation is that you do not load them. You just shouldn't. If you really want to, there's people on the Internet that have examples of how you can create a list of approved functions that will only those functions can be loaded by the pickle. But at that point you're creating a Python jail and there's a lot of CTFs where we've

seen people breaking out of Python jails and a lot of issues that we've seen in Python jails over the years. So I wouldn't recommend that. But you'll find that option on the Internet if you really, really want to try it. Yeah. So what do I recommend? Well, if you're creating something new, then you can release your layers as code.

Pickles don't really hide what you created in any way, in any meaningful way. It can all be extracted by anyone who wants to. So you can release your layers, all the code that you write, as actual Python code, whoever is going to iterate off of it will have an easier time doing it and they won't be vulnerable

because they can look at the code and make sure it's not doing something sketchy. And then you release your weights as binary data. If you wanted to do this in PyTorch, on the right-hand side of the screen is some code which I'm also releasing at the end. And that code will, for PyTorch, save the state of the weights and write it out to a NumPy blob for you and load it.

And if you look closely, you see the NP lib, the super deep function inside the NP library that you're not really supposed to call. That's because the default one will fall back to pickles if you have something in it that makes it want to. So it's not really a good function to use.

But you can, it exists. And then if, you know, this has been around for six years. So there's a lot of pickles that already exist that are malicious, well, not malicious, that could be made to be malicious. And it's sort of unrealistic for me to expect them to just delete all of them and start over, as much as that would be the safest solution.

So if you have some of those, then you need to protect them like they're unassigned executables because effectively that's what they are. That means verifying that their integrity is exactly what you expect it to be. If you're offering people the ability to download it, it needs to be over an encrypted channel

because otherwise an attacker in the middle can make it whatever they want it and get a shell in the system. And if an adversary gets access and could have modified it and you have no way to verify that they haven't, you sort of just have to delete it all and recreate them, which sort of sucks. So don't let that happen.

And some of you that know about machine learning models might be thinking, okay, I don't use pickles, I use onyx or torch script or something else that still supports arbitrary layers. And arbitrary layers are going to be in some way still code. So most likely they're also still vulnerable, it's just going to take more work to do it

because there's not ten years of research showing you how to just add malicious code into it. For example, onyx already has a POC, thanks to Lovato, not sure if you can read the link at the bottom, but that's his GitHub where he has proof of concept on how to add exploitable code into an onyx file.

If you're someone that wants to do this kind of research, onyx and all the other ways to save models could be something interesting to look at to add malicious code into as well, because I'm pretty sure it's possible even though I haven't tried myself. So, yeah, if anyone wants to see the injection code, I have it on my GitHub.

If anyone uses mythic C2, I made a wrapper, so all you have to do is upload the pickle you want to inject into, it will inject a mythic Medusa agent into it and you can put that where you want and wait for a callback, and if you have any questions, you can ask,

but also hit me up on Twitter and I'll do my best to respond.