Rehabilitating Pickle
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44995 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 201810 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
Software developerComputer-generated imageryRouter (computing)System callDefault (computer science)Module (mathematics)Level (video gaming)Server (computing)Software testingModulare ProgrammierungRoundness (object)Integrated development environmentCuboidInformation technology consultingPoint cloudPhysical systemWhiteboardCodeWind tunnelResultantRandomizationMiniDiscState of matterScripting languageRemote procedure callProgrammschleifeLaptopFunctional (mathematics)Library (computing)Goodness of fitSystem callChainComputer-generated imageryConnected spaceMoment (mathematics)QuicksortWritingDot productDeterminismProjective planeInstance (computer science)DemosceneTask (computing)BitInternet service providerLatent heatLine (geometry)Computer animation
05:14
Communications protocolType theoryData structureCompact spaceSource codeStructural loadComputer virusRevision controlASCIISerial portType theoryCodeString (computer science)Data structureFunctional (mathematics)Task (computing)Well-formed formulaMultiplication signStructural loadSocial class2 (number)Cache (computing)Data dictionaryTupleCuboidQuicksortBlogDefault (computer science)Virtual machineHost Identity ProtocolSystem callPhysical systemLibrary (computing)Standard deviationRemote procedure callInternetworkingLine (geometry)Core dumpProcedural programmingQueue (abstract data type)RecursionKey (cryptography)RootComputer fileBenchmarkCASE <Informatik>Source codeGoodness of fitNetwork socketInstance (computer science)Computer animation
10:28
CAN busModule (mathematics)Web browserJava appletComputer animation
10:51
Structural loadComputer virusSource codeRevision controlModule (mathematics)OpcodeCommunications protocolStack (abstract data type)Object (grammar)ChainComputer fileOpen setCore dumpMultiplication signCommunications protocolCASE <Informatik>Module (mathematics)CodeBinary fileArithmetic mean1 (number)Extension (kinesiology)String (computer science)Virtual machineStandard deviationDefault (computer science)Line (geometry)Library (computing)IntegerMereology2 (number)Web browserDenial-of-service attackContent (media)Presentation of a groupForm (programming)Remote procedure callProcess (computing)Social classFunctional (mathematics)LengthComputer wormBitStructural loadInterior (topology)Computer animationDiagram
17:10
Revision controlTerm (mathematics)Data storage deviceBlogYouTubeInformationComputer programModule (mathematics)Web browserProcess (computing)Data storage deviceCASE <Informatik>AdditionInformation securityMiniDiscResultantComputer worm2 (number)DatabasePhysical systemNumberFunction (mathematics)Order (biology)Sound effectSoftware testingLimit (category theory)Object (grammar)Element (mathematics)Flow separation10 (number)Semiconductor memorySelf-referenceFilm editingExtension (kinesiology)Term (mathematics)Right angleComputer animation
23:29
Revision controlComputer fileSystem callLine (geometry)Core dumpCommunications protocolSource codeStructural loadSet (mathematics)BlogYouTubeMiniDiscInformationComputer programCommunications protocolMoving averageUniform resource locatorGodMoment (mathematics)Projective planeObject (grammar)Structural loadIntegerPresentation of a groupBitBlogArtificial lifeDenial-of-service attackCode1 (number)Exterior algebraSlide ruleVector spaceTerm (mathematics)Demo (music)Data storage deviceVideo gamePrototypeCore dumpWindowExtension (kinesiology)CASE <Informatik>Multiplication signRange (statistics)Instance (computer science)Sound effect
Transcript: English(auto-generated)
00:04
Good afternoon, everyone. My name's Alex Wilmer. I work for a company called CGI. Please come to us for all your consultation and outsourcing needs. Today I'm going to be talking about the Python pickle module, which is possibly a slightly controversial topic.
00:24
Has anybody ever used pickle? Has anybody ever been told, never use pickle? OK, that's pretty much 90% across the board for both. I want to start with a bit of backstory.
00:41
On our project at CGI, we make heavy use of an orchestration tool called Ansible. And Ansible is very nice if you're doing DevOps, because you can write roles and playbooks, and they will configure your servers into some deterministic, indepotent state,
01:01
and they will do it across multiple environments. So you might have a dev environment on laptops. You might have another test environment in the cloud on Amazon, for instance, and a third one that you're deploying to, and we can reuse the same playbooks. So it's very nice, and we use this heavily.
01:27
The problem is, because we use this heavily, our Ansible roles and playbooks got heavy. We have them running lots of tasks. We have them communicating with lots of servers, doing lots of steps, and a lot of stuff that's in loops.
01:41
And it got slow. Every round trip to a server was costing us milliseconds. And when you have hundreds or thousands of round trips, those milliseconds add up. And thankfully, just shy of a year ago,
02:01
a new project appeared on the scene called Mitogen, and it has within it a provider for Ansible that can speed up your Ansible connections greatly. And there are testimonials from people who have used it. My playbook's gone from one and a half hours
02:22
to three minutes. It's that dramatic. So great, Ansible is fast again. There are a couple of caveats that we're having to overcome. So if you use Mitogen just on its own, no Ansible, this is the sort of thing you'll see.
02:44
You'll, in Python, set up a chain of hosts that you're connecting through until eventually you hit the box that you want to run something on. And Mitogen will, behind the scenes, tunnel all your connections. Doesn't matter if you're using Docker or SSH
03:02
or some other odd thing that only you use. It will do the tunneling for you. And unlike Ansible, it will do all the tunneling without having to write things to disk at each stage.
03:21
In this example, we're ultimately calling os.system with a random Python script. Equally, that os.system call could have been any other Python function call. Could have been a standard lib module. It could have been a third-party module. Mitogen would have set things up
03:42
so that when Python, on the other end, tries to import that module, Mitogen will be there ready and have provided it. So you don't need to install Mitogen or your third-party modules on the boxes that you want to run this code on. Essentially, Mitogen is a zero install
04:00
remote procedure call system. This is Mitogen without Ansible. But when integrated with Ansible, you carry on using your Ansible inventories and playbooks without needing to write Mitogen-specific code.
04:22
The problem that I alluded to earlier is that at the moment, Mitogen uses the pickle module to return the results. So when master sends the request
04:40
to run os.system, the result of os.system is pickled on the remote system and then the pickle is sent back up the chain and the master system unpickles it. And as you may know, this can be problematic.
05:00
So if you want to use Mitogen with Ansible, it can be as simple as installing Mitogen and setting these two lines. But with the caveat that you need to be aware that when you are doing this, you are essentially trusting all the systems
05:20
that you are controlling with code execution on your bastion host, on your master. The benefits of running Ansible with Mitogen can be quite dramatic.
05:41
This is a fairly artificial, a slightly artificial benchmark, but we see vanilla Ansible in the red line taking a total of about a minute and about four and a half megs of traffic to run a task 100 times on a remote host.
06:01
The same things doing it with Mitogen, 90 kilobytes, eight seconds. This is our goal. This is why we want to use Mitogen for this. But we would ideally do it without having the trust issues
06:24
of potentially giving all our remote machines code execution on the box that we are running Ansible and Mitogen from. So for anybody not already familiar with pickle, it's a built-in standard,
06:42
it's a library built into the standard library, import pickle and then has a very simple pair of dump S and load S functions. You provide dump S with an almost arbitrary
07:03
Python data structure or type or user class if that user class has the support and you get out a byte string. You can send that byte string over a socket, you can write it to a file, you can write it to a queue or to a cache or to celery.
07:22
And when you load on the other, when you retrieve it and load it again, you get back almost exactly what you put in. Pickle has the advantages that it works everywhere Python does, it preserves types that the likes
07:41
of the JSON and YAML don't. For instance, tuples go in, tuples go out, date times go in, date times come out. So it makes it very convenient, very tempting to use pickle for remote procedure called systems such as this.
08:01
It also handles recursion as a bonus. We're not specifically using that but it's something nice to have. And the serializations are quite compact. Typically a pickle of some nested dictionary will be less than half or maybe a quarter of the size of the equivalent JSON.
08:24
However, I want to reiterate the standard advice. Don't use pickle. It is by default dangerous. If you unpickle data from an untrusted source,
08:42
you risk blowing your foot off at the hip. This is the standard advice. It's included in the Python documentation. It has been reiterated by various blog posts, various white papers that you will find
09:01
around the internet. An example of how easy, how trivial it is to get remote code execution through unpickling is in fact, I've lifted this straight from the standard library. What's happening here is somebody has sent us a pickle
09:22
which contains a call to os.system and in this case, it's echoing hello world. But it could equally have executed rm-rf. And if you're running as root, you've lost everything. Otherwise it could have posted your SSH keys
09:41
or your top secret formula for Coca-Cola or finger licking good chicken into your adversaries and goodbye your business. Just to give a quick example of this,
10:00
my colleague Rick has sent me a pickle to try. Let's load Rick's pickle.
10:20
I'm guessing it's some sort of joke, a dancing ASCII, something I don't know. No, I wasn't expecting that. You see, a pickle can do anything Python can.
10:43
If there is a module with a callable thing, then pickle can invoke it and that includes the web browser module. So if we have a look at the contents of Rick's pickle, then it's very similar in form
11:02
to the echo example we saw earlier, but in this case we're using the web browser module. So what can we do about this? The standard mitigation is to subclass the unpickler class.
11:25
We create a whitelist of Python built-ins, Python functions that we are willing to allow pickle to invoke and we only let things on those whitelists be invoked
11:42
when we unpickle. This works to an extent. A lot of the Python standard library has not been audited, has not been checked for ways
12:01
that if you put certain things on your whitelist, they won't be able to drill out through those, use dunder, get and possibly reach built-ins. Once they reach built-ins, they can do just about anything.
12:22
The other trouble with this technique is that it is opt-in. By default, if you have other people dealing with pickles in your code base who aren't aware of you must use the restricted unpickler subclass, never call pickle.loads directly,
12:41
you still open yourself to getting rick-rolled or worse. And assuming we fix this, there are other attacks that most articles, most presentations about pickle don't go into
13:02
because frankly getting remote code execution is job done as far as adversarial people are concerned. If they can get code execution, then nothing else matters. But if we can come up with something that doesn't allow code execution by default,
13:23
doesn't require extra means to whitelist things, then what other attacks remain? The ones that we've found so far fall under denial of service and the possibility of data exposure and weird machines.
13:46
So, first example, I mentioned earlier that, or I showed earlier that pickles can be pure ASCII.
14:09
That's because this pickle in question is what's called a, termed a protocol zero pickle. So we've got new line delimited strings
14:21
and no binary data. But equally we, in Python 3, default to higher protocols. So if I import pickle and dump a string,
14:48
it will default on Python 3 to protocol 3, which is a binary protocol. The string you see there is length prefixed. That makes it much faster to unpickle with this protocol.
15:04
And the protocol that you use is determined when you create the pickle, not when you unpickle. So if I want to be backward compatible with Python 2.2,
15:21
because who doesn't? I can create a very similar pickle, but this time it is using the new line delimited. And this protocol zero is much, much slower to unpickle.
15:44
And the attacker chooses the protocol. So benchmarking this, we see that the worst case is unpickling integers.
16:01
If your attacker wants to send you a one megabyte pickle that's just a very, very, very large int, it'll take you the better part of 10 seconds to unpickle that at protocol zero. At protocol one, it's quite a bit better.
16:21
It'll take you under a millisecond to unpickle this. So we would like to be able to just not allow protocol zero because protocol one or protocol two is what supports the binary encoding and that will get us all the way back
16:40
to certainly Python 2.7. Python 2.4 I believe also supports it. Once we've got protocol zero out the way,
17:00
we then have the problem that our attackers can send us self-referencing pickles. This is a very small payload, similar to a zip bomb, if anyone has come across those, that just repeats the same element again and again and again and again but by referencing it.
17:22
And by referencing references of references of references, we can make a very small payload that expands to an extremely large object in memory or on disk. Now pickle handles this surprisingly well.
17:49
So I have a billion last payload that I've already prepared. This is a pickle two payload. And it's a couple of hundred bytes long.
18:04
Let's unpickle it.
18:21
That was easy. The problem comes when you try and send this on. If you want to log this to disk or you've got another system that accepts JSON
18:40
and you try and encode this to JSON, the problem will be that as soon as you try and serialize the result in anything that doesn't support self-reference, then, well, I've just locked up B-Python.
19:05
The problem there is that B-Python is trying to syntax highlight the output and the output is, I can't remember if it's several tens of megabytes or hundreds of megabytes in size.
19:20
We can't just consider ourselves when we are using these. So a further mitigation you may want to do if you are having to deal with untrusted pickles is to find some way to remove self-reference,
19:42
although there are possibly use cases where you would want to keep it. Excuse me one second while I kill this. That's good.
20:05
So what else can we... The other problem of which we're aware
20:20
is that there are a much larger number of pickle payloads that the pickle module will accept unpickling than the pickle module will generate. So our attackers don't have the restriction
20:41
that they need to restrict themselves to what the pickle module will output. They can handcraft pickles and, in fact, that's how the os.system example and the web browser example were generated earlier.
21:04
We need to be able to test our thing, whatever it ends up being, against handcrafted and also randomly generated pickles in order to try and prevent effects
21:23
that were never thought of when the pickle module was coded. To that extent, we've been trying to fuzz test the pickle module with limited success so far, but we'll be re-looking at that in the future.
21:50
Final thing that I want to reiterate is that the use case we have here is a pickle for RPC results.
22:04
So the pickles are living very short lives on the wire. They are not being put into long-term storage and they are not living longer than the processes that created or will be reading them have lived.
22:23
In addition to any security considerations about pickle, there is a very important thing put well here that if you are using pickle and writing the results to disk or to a database or anything
22:40
where the pickle you created could be read days or years later, you will run into problems. You should not use pickle for long-term storage regardless of the security.
23:01
So the question becomes, should you use pickle? The answer is still no, but perhaps in the near future, you might like to use the module that I'm working on called pickle light.
23:24
Pickle light is essentially taking the pickle module and cutting bits out. We're giving it a lobotomy. So at the moment, this is still very much a prototype.
23:48
But for instance, if we try and generate
24:13
a protocol zero pickle, which might be a vector for denial of service attacks, denied.
24:24
Oh, wait, I did, thank you, live demo gods. So any attempt to create or to load
24:47
a protocol zero pickle, such as, wrong window,
25:03
thank you, any attempt to load a protocol zero pickle
25:25
fails. So in this case, we haven't been protected against the, we haven't been protected against Rick's joke here because we've whitelisted anything.
25:42
Instead, we have been protected because he happened to use protocol zero. But likewise, if we try and dump a set object,
26:08
which normally would require this load a global object and call it effect, then pickle three will fail
26:22
because we've excised the bit that the Rick roll relies on. Instead, we need at the moment to use a very little used feature of pickle light, sorry, of pickle where we register extensions.
26:55
In this case, we're going to register built-ins set
27:03
and we're going to pick an arbitrary integer. That one might be in a reserved range, I don't remember, but no one is using these at the moment and we can choose better ones later. What that has now done is allow us to dump and loads
27:36
objects that we have explicitly whitelisted
27:41
and we have had to explicitly whitelist them. This means that there's no opt out of arbitrary code execution, there is only opt in to selected objects.
28:02
Definitely too much live coding in this.
28:27
Anyone spot any typos? Thank you. So we have managed to round trip an object that is not natively supported by pickle
28:40
without opening the door to arbitrary code execution. Thank you very much for your time, everybody. Slides will be available at the URL you see there very shortly.
29:01
If you'd like to know more about Mitogen, please visit the URL there. There is an additional project that is working on sanitizing pickles called Picara and there will be a presentation about that at PiOhio this weekend, which I will be eagerly seeking out.
29:27
If you'd like to know more about pickle attacks and what you can do when objects are whitelisted, there was a very good presentation by the company Sensepost in 2011
29:41
and if you are looking for an alternative for pickle for long term storage, then there is an excellent blog post by a lady with the name of Evie and I recommend checking out that. Thank you very much.