Edge Side Include Injection: Abusing Caching Servers into SSRF and Transparent Session Hijacking
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 322 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39710 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Inclusion mapContext awarenessCache (computing)SpywareServer (computing)InjektivitätRevision controlFormal languageInclusion mapSpywareCache (computing)Graph (mathematics)Server (computing)Information securityWordSign (mathematics)Latent heatRoundness (object)Context awarenessWeb-DesignerHuman migrationSocial classConfiguration spaceClient (computing)In-System-ProgrammierungRight angleAddress space
02:29
Right angleCartesian coordinate systemCache (computing)Server (computing)Primitive (album)Web pageComputer fileSingle-precision floating-point formatElement (mathematics)Dependent and independent variablesFluid staticsWeb 2.0Computer animation
03:22
Cache (computing)Server (computing)Dependent and independent variablesProxy serverEmailLatent heatEmailGroup actionLatent heatInclusion mapServer (computing)Cartesian coordinate systemRight angleDependent and independent variables
03:52
Web pageInclusion mapContent (media)Cache (computing)Web pageServer (computing)Computer fileMultiplication signComputer animation
04:42
Web pageInclusion mapClient (computing)Server (computing)Computer networkCache (computing)DataflowLastteilungServer (computing)LastteilungCache (computing)Web pageCartesian coordinate systemClient (computing)Content (media)2 (number)Computer fileStructural loadChemical equationComputer animationProgram flowchart
05:26
Variable (mathematics)Server (computing)InjektivitätVariable (mathematics)Exploit (computer security)Server (computing)Cache (computing)HTTP cookieAttribute grammarDependent and independent variablesMetadataVarianceDatabase transactionPerspective (visual)Content (media)Cartesian coordinate systemOcean currentWeb browserString (computer science)Arrow of timeException handlingIntrusion detection system2 (number)Parameter (computer programming)Scripting languageExpected valueQuery languageArithmetic mean
06:56
Local ringInjektivitätServer (computing)Graph (mathematics)ImplementationComputer wormInformation securitySelf-organizationServer (computing)Computer animation
07:26
HTTP cookieServer (computing)EmailArithmetic meanHTTP cookieServer (computing)PlastikkarteComputer animation
08:03
Demo (music)Proof theoryLocal ringHTTP cookieServer (computing)Medical imagingDirected graphReal numberProof theoryWeb browserWeb 2.0EmailComputer fileUniform resource locatorLogin
08:41
WhiteboardMessage passingLocal ringServer (computing)HTTP cookieWeb pageCartesian coordinate systemEmailInternet forumMedical imagingComputer fileDatabaseWeb browserReal numberSource code
09:48
Message passingHTTP cookieSource codeProgram flowchart
10:03
WhiteboardMountain passDemo (music)HTTP cookieOracleCache (computing)MereologySuite (music)Server (computing)Proof theoryHTTP cookieComputer configurationComputer fileWeb browserServer (computing)Cache (computing)Cartesian coordinate systemWeb 2.0BlogArithmetic meanScaling (geometry)Web pageOracleContent (media)ImplementationLatent heatMountain passLogicFunction (mathematics)Source codeProgram flowchart
11:48
Inheritance (object-oriented programming)WhiteboardOracleGraph (mathematics)InjektivitätServer (computing)Web pagePhysical systemVector spaceHTTP cookieContent (media)Cartesian coordinate systemArithmetic meanObject (grammar)Perspective (visual)Function (mathematics)CuboidWebsiteElectronic mailing listDependent and independent variablesComputer wormSystem administratorSource code
13:13
Inheritance (object-oriented programming)WhiteboardLink (knot theory)OracleMessage passingLocal ringCache (computing)LeakHTTP cookieDemo (music)Arithmetic meanWebsiteContent (media)Computer fileLeakHTTP cookieProof theoryCache (computing)Network topologyFunction (mathematics)Source codeComputer animation
13:33
Core dumpRule of inferenceCache (computing)Server (computing)Web applicationDoubling the cubeWeb browserSoftware developerDependent and independent variablesArithmetic meanCore dumpProduct (business)Rule of inferenceType theoryInformation securityRight angleFirewall (computing)
14:55
Content (media)ExpressionLocal ringWebsiteData typeServer (computing)EmailCodierung <Programmierung>CloningHTTP cookieComputer wormRepresentational state transferSound effectDependent and independent variablesServer (computing)Cache (computing)Local area networkComputer fileWebsiteImplementationE (mathematical constant)AreaWordInclusion mapXML
15:47
Server (computing)Electric currentRevision controlMetra potential methodInheritance (object-oriented programming)DataflowClient (computing)Content (media)Dependent and independent variablesAreaType theoryServer (computing)WebsiteComputer fileXMLComputer animationProgram flowchart
16:21
TwitterDependent and independent variablesServer (computing)Computer animation
17:03
KinematicsComputer animation
17:22
Human migrationServer (computing)Graph (mathematics)Cache (computing)Web 2.0Heegaard splittingSoftware frameworkMechanism designDependent and independent variablesEmailSoftware testingComputer fileTwitterGoodness of fitPoint cloudUML
18:27
Software bugUniform resource locatorGraphical user interfaceMultiplication signSign (mathematics)PasswordDenial-of-service attackJSONUML
Transcript: English(auto-generated)
00:00
Uh, Luis here. This is his first talk at DEFCON, and uh, give him a round of applause! Cheers! And uh, he wants to talk to you about exploiting cache services, cache servers. Um, so uh, I'm gonna let him go, and uh, enjoy. Alright. So, thanks everyone for
00:29
coming. Last day of DEFCON. Let's do this. So, this is edge side include injection, abusing caching servers into server-side request forgery and transparent session hijacking.
00:40
So, I know that's a very long title, uh, meant to mention some of the many things that you can do with edge side includes, especially when you're injecting it. So, for the rest of this talk, we'll be referring to edge side includes as ESI, for convenience, because that's a mouthful. Um, so during this talk, we'll learn about edge side includes. Uh, we'll talk about the problem that it was created to solve. Uh, we'll then talk
01:03
about the problems that it created by introducing it in an unsafe manner, and then we'll talk about mitigation and migration. So, my name is Louis Diermarcille. I work at GoSecur in Montreal, and to give some context on what ESI injection is, basically, it's a new class of attacks. Of course, it targets ESI enabled caching servers, so
01:22
it's not a widespread attack, uh, unlike what James Kettle presented at Black Hat two days ago. So, this is really targeting ESI engines. So, this was discovered by mistake, basically. Uh, one of my colleague, uh, Laurent Desoni at GoSecur, was tasked with reviewing the caching configuration for one of our clients. So, our client, which
01:42
is a large ISP, basically wanted a cache overview of the security features, and we kept seeing references to ESI kept coming up, edge side includes. So, we're a bunch at GoSecur, and we never heard about that. None of us ever heard about that. So, we started looking into documentation, and we saw that the first and final specification
02:01
from ESI was in 2001, and I don't know if you were doing web development 17 years ago, but security was not invented yet. So, we started looking into, basically, vendor documentation, because the specification was so old, we thought, this can be right, and we kept seeing stuff like that. Word art, word art in documentation is always a
02:23
good sign from an attacker's perspective, because it tells you, well, I wasn't even in high school when this was done. So, okay. So, the documentation isn't going to help us, so I'm going to explain it to you. So, basically, let's look at this very primitive web page example. So, you have a weather web page, and to the end user, this is a single HTTP response, right? But to the end, to the ESI server, the ESI
02:45
caching server, this is multiple fragments, and these fragments were invented to do one thing, which is invalidate individual elements of a web page instead of invalidating the whole document. So, when you think of caching, usually you'll cache a static file, and ESI is invented to cache dynamic files. So, for the forecast, Monday, Tuesday labels,
03:06
you can keep that as a static fragment, but the forecast, for example, 27 degrees, you can invalidate that within the next hour. So, with this knowledge, we know that there has to be a way from the application server to tell the caching server where fragments stop, where they start and where they begin. So, this is done through fragment
03:23
markers. Those fragment markers are in the HTTP response, and they look like this. So, you have an ESI tag followed by an action. It's basically an XML tag, right? It's instead of having a secure layout, you just have tags inside of the HTTP response, which is going to get stripped once it's evaluated by the ESI engine.
03:43
So, these tags are parsed when a specific HTTP header is sent by the application server. So, you're not able to inject ESI everywhere, but usually when they use ESI, it's implemented everywhere. So, let's look at our first feature. We have ESI includes. ESI includes are, in my opinion, the most relevant ESI tag. So, you have two files
04:02
to show how this works. You have page 1.html and page 2. Page 1 is sitting on ESI server. The second one is on another server called API, and you can see in page 1, you have an ESI tag, which is an include tag, pointing to page 2. Now, two things can happen here. You can have a cache hit or a cache miss.
04:22
If you have a cache hit, it means that the ESI engine can just replace the tag with the content of page 2. Easy enough. If it's not there or if the cache entry is invalidated because it's been too long, well, you have to fetch this file. So, the ESI engine is going to do a side request for this file to fill in the blanks.
04:41
So, whatever happens, this is what your end user is going to have. You're going to have the content filled by the engine. So, to illustrate how this works, let's look at a very example of a cache miss. So, you have your clients, your load balancers, and your servers. So, your client is going to request 1.html, and the caching server has a cache miss, meaning the file is no longer valid. It has to go and get it.
05:01
So, this is done through the upstream server. So, the load balancer requests this file to the application server. This is sent back with an ESI tag saying, hey, please fill in the blanks with 2.html. So, this tag is parsed on the caching server, and the side request is sent to the API server.
05:20
Now, the API server responds with the contents of page 2, and the ESI engine is able to fill in the blanks. Our second feature before delving into exploitation is ESI variables. So, they're a very simple feature. It's a very simple feature. It has no attributes, so no XML attributes, and basically the content of the tags gets expanded to access metadata about the current
05:41
HTTP transaction. So, you're going to be able to access stuff like this, so the HTTP user agent, cookies, query strings, basically anything relating to the current HTTP transaction. So, now we know about ESI includes, we know about ESI variables. We also know that the tags are sent by the application server, and they go through the caching server,
06:01
and this is where they are parsed. But there's a very important question that we have to ask ourselves, which is how can the ESI engine know which tags are legitimate and which tags are injected by a malicious user? Think about cross-type scripting. It's basically the same thing, except we're not exploiting browsers. So, that's a very important question, and that's the problem. It's that it can't, and you're able
06:22
to inject ESI tags and do basically whatever you want with the cache server. So, to illustrate this, let's look at a very basic example of ESI injection. So, you have the content of the city get parameter that's echoed back in the HTTP response. Now, the caching server is going to parse anything that is sent there. So, you can put ESI bar in there pointing to the user's PHP session ID. If you do know PHP,
06:44
you know that this is HTTP-only cookie, meaning that JavaScript is not able to access this. So, if I'm able to access this from a caching server perspective, I can effectively leak a session cookie and effectively take over the account, and this works as expected. So, let's try to build a payload in order to do this.
07:01
So, we're going to look at two ESI engine implementation. First of all is we're going to look at Apache traffic server. So, I looked at this one for two reasons. First of all, because it's used by high-profile organizations. So, Showdown tells us that it's used by Apple, Yahoo, and Comcast. The second feature, the second reason is because they have
07:21
the initial ESI stack implemented, but they added bonus features, some of which are security features. So, our first security feature is cookie whitelisting. So, even if you're able to inject ESI tags, sometimes you're not going to be able to access the cookies because they're not whitelisted. If you want to access the cookies, you have to preemptively configure the traffic server to say you can access this cookie.
07:42
But by reading the documentation, you also find out about another ESI variable called HTTP header. It allows you to refer to any header. Smart. Meaning that you can access the cookies. So, the whitelisting doesn't work. It's so easy to bypass. So, that was fixed when I reported it two months ago.
08:01
Pretty quick fix, so good for them. So, let's build a proof of concept to do HTTP only session hijacking without JavaScript. So, I built an image source tag. You can do basically any tag that requests an HTTP header, but this is fine. So, it's pointing to evil.local, which is an attacker enabled server for which I have a web server pointing there.
08:21
So, the file name that is going to be requested by the victim's browser is an ESI tag pointing to their own header cookie. So, when this is going to go through the traffic server, it's going to expand the value with the session cookie. And then the browser is going to request this URL. I'm going to access this in my HTTP logs. So, now let's look at how this would look in a real world example.
08:43
Now it's working. All right. So, I built a very simple message board. So, you have your victim on the left. You have your attacker on the right. It's two different browsers, so there's no cookie contamination. On the middle, you have basically everything that is stored in the database. So, you have hi, hey. So, you can see what it looks like before being sent
09:01
to the application server. So, now our attacker is going to put the evil.local, well, basically the petal that we just looked at. So, the file name has the header for the cookie. And it's going to hit sent afterwards. So, this is going to pollute the database of the application server. Now, when they send it, you can see in the database
09:21
it's sent properly, it's stored, everything is working fine. Then the attacker, by refreshing the page after sending it, basically attack themselves. So, they're going to leave their own cookies because the browser is going to send a side request for that image, which is not really an image. Now, we're going to wait for the victim to refresh the page and we should effectively steal their session.
09:42
And as you can see, the session cookie appears. So, we just stole their HTTP only cookie. We're able to take over their session completely and become that user. So, we're going to look at the session cookie. We confirm that it's HTTP only. We're going to replace our own value with the one that we just stole through ESI injection.
10:00
We're going to save that cookie and save it. And then, once we refresh, we should become the victim. There you go. So, you have HTTP only. So, this is HTTP only. Cookie hijacking without JavaScript
10:20
through ESI injection. So, that's nice, but you need to inject a page for which a user is going to travel to. Sometimes, that's not always easy. Sometimes, think about self XSS. The impact could be great, but you're only attacking yourself. So, let's try to crack the impact up a notch. So, I looked at another ESI implementation which is Oracle Web Cache.
10:40
So, Oracle Web Cache is usually sitting on top of WebLogic application servers. It's not necessarily sitting on top of that, but we've seen it sitting on top of that. So, I looked at it because it's usually high scale application and also because the initial ESI specification is implemented and they also have bonus features. Unlike ATS, they went with the least secure option which is they added the ESI inline tag.
11:03
This tag is pretty easy to understand. It allows you to override the engine, the ESI, it allows the ESI engine to override any cache entry with arbitrary data. So, here, we're going to override jquery.js with arbitrary content. What a great idea. So, jquery.js is going to be filled with the content that you see here
11:21
which is basically an AJAX request. So, once the user is going to refresh, the file should trigger AJAX request to our evil.local server and this is going to get expanded meaning that I'm going to get their cookie again because there is no cookie whitelisting in Oracle Web Cache. So, now we know we can override file and we can make the browser do anything.
11:41
So, the browser is going to request this file because we took over jquery.js. So, let's look at a demonstration of this which is already running for some reason. So, we have the same application server but now it's super safe because the sysadmin is
12:01
they notice basically that HTML was not being stripped so ESI was also being injected. Well, there was a possibility of ESI injection. Now, we can see that the victim is confirming that jquery.js exists and the content is valid. So, we have jquery3.3.1. Now, our victim is going to refresh. The attacker is going to put, sorry, payload just to see if HTML encoding
12:21
or escaping is working and as you can see, the attacker is no longer able to put HTML charset meaning that ESI injection is effectively mitigated. So, that's a problem for an attacker's perspective. Now, the attacker sees a new feature which is a usualist. This usualist will reflect anything that is in the search box. So, that's a pretty good vector for either
12:41
self-xss or ESI injection and we can see here that HTML is not escaped. So, we're able to put our ESI payload in there which is going to override jquery.js. So, the attacker puts it there, summits it it's echoed back in the HTTP response and now if everything works properly jquery.js should be overwritten.
13:01
Now, the attacker has effectively polluted jquery.js. Nice. Now, the victim can just refresh the page and once anyone refreshes any page on this website, they're going to send us their cookie. So, there you go. We just overwrote an arbitrary file with arbitrary content meaning I can either deface the website
13:20
and steal anyone's session using ESI. So, that was basically a proof of concept to overwrite arbitrary cache entries and leak HTTP-only cookies. You can use our script, but as you can see it's not really necessary. So, now let's talk about mitigation. So, if you like web application firewalls, you have ModSecurity.
13:41
It's a pretty popular product. It's gotten way better in the last years and if you use the OWASP core rule set, you're good for ESI. We talked with one of the developers of the ModSecurity team and they basically said we already strip anything that is XML-like which includes the the charset for ESI, so you're good with that. If you don't want to use a WAF
14:01
or if you don't use Apache, you can use proactive escaping and what I mean by proactive escaping is you might think that since you're okay, like you're escaping HTML everywhere, so ESI is the same charset, I should be good. Well, not necessarily because when you think about it, contextualized effort of escaping will often ignore HTML
14:21
and JSON, meaning that HTML is never escaped in JSON because the content type is already telling the browser don't interpret this as HTML but we're not exploiting browsers. We're exploiting cache servers. So here, I can put an ESI include tag in a JSON response and it should work fine, right? There's just one small problem. We have an invalid
14:41
ESI tag because of those backflashes because of the double quotes but ESI engines have a very flexible syntax which is nice and I can just drop them. So this will allow me to do server-side request forgery and a JSON response. Let's look at a brief example of this. So you have effective REST API sitting on slash API slash me
15:01
which is going to respond to basically a small JSON payload. So you see that my full name here is Luizio Marcille and I can overwrite this with an ESI include tag. This ESI include tag is going to say please fetch REST server slash server status which is just some server sitting on the local area network of the cache server and you can see in the response in the reddish area
15:22
it went ahead and fetched that file for me meaning that I can do server-side request forgery with ESI includes. Now most ESI engines will not allow you to do server-side request forgery on arbitrary hosts so you have to whitelist them prior to doing ESI include of them but some implementations will just allow you to do server-side request forgery through anything
15:41
for example, squid cache just allows you to do ESI included whatever you want so that's pretty nice. To illustrate what this looks like I just changed the content type to text HTML so that you can see that the JSON response is the greenish area and then the reddish area is basically the content of the server-side request forgery through ESI injection. So to illustrate
16:00
how that worked I used the same imagery that I did before so you have your slash me which responds with an ESI tag saying please get that file for me the file is fetched and then the content of step five and three gets concatenated together and I get the content of the server of the effectively the server-side request forgery get it content by ESI includes
16:22
okay so if you want to do ESI injection you need to first identify if you're messing with a ESI enabled caching server someone on Twitter called Alex Berson came up with a pretty smart solution which is leveraging ESI comments ESI comments basically are tags that are going to get stripped by your ESI enabled server
16:40
so if you have this HTML comment looking tag which is ESI and it's removed from the HTTP response then your ESI engine has removed it if you do the same thing but instead of ESI in the comment you put something completely different for example foobar or just foo and this one does come back then you're probably messing with ESI engine if you
17:01
don't want to mess with manual detection because it's pretty painful you can just use automatic detection so you have burp active scan plus plus burp upload scanner and iQnetics which all can detect ESI injections I'm not sure how iQnetics does it but I know that burp active scan plus plus and burp upload scanner are using the aforementioned heuristic
17:20
so it's pretty reliable if you think that ESI injection well ESI as a feature is a good example of a robust caching mechanism and you want to implement that I personally wouldn't go with ESI because basically it's pretty broken but you can use Cloudflare workers which are basically JavaScript files
17:40
setting on edge servers someone on Twitter called Lucas Rider came up with modern ESI if you may which is basically a JavaScript file that's going to allow you to do basically fragmentation of your HTTP response so you can see on the bottom screenshot you have a fragment pointing to footer and that footer is specified in the HTTP header
18:01
which is so much more secure than just pointing to the resource and the HTTP response because if you've done any web testing in the past 10 years you'll know that HTTP response splitting or just injecting HTTP headers is so much harder than it was 10 years ago so most frameworks will just not allow you to do that so if you have to inject in two places instead of just
18:21
one it's of course much smarter and it's probably a lot faster too so this is basically ESI injection there's a lot of research to be done with this we documented this I think in April at this URL so basically you have our prior research we analyzed I think half a dozen ESI engines some of which
18:41
are pretty famous if you may so you have Akamai, WebSphere, Varnish, Fastly and Squid we found a whole bunch of bugs like denial of service with Squid, we found basically XSS, filter by password Chrome so a whole bunch of interesting stuff so if you want to go ahead and search for more ESI
19:01
bugs I think you'll find a lot of them I think this is my time and I might have one minute and a half for questions