Edge Side Include Injection: Abusing Caching Servers into SSRF and Transparent Session Hijacking

Video thumbnail (Frame 0) Video thumbnail (Frame 3729) Video thumbnail (Frame 5039) Video thumbnail (Frame 5806) Video thumbnail (Frame 7059) Video thumbnail (Frame 8145) Video thumbnail (Frame 10407) Video thumbnail (Frame 11150) Video thumbnail (Frame 12076) Video thumbnail (Frame 13033) Video thumbnail (Frame 14702) Video thumbnail (Frame 15072) Video thumbnail (Frame 17703) Video thumbnail (Frame 19827) Video thumbnail (Frame 20334) Video thumbnail (Frame 22364) Video thumbnail (Frame 23664) Video thumbnail (Frame 24516) Video thumbnail (Frame 25577) Video thumbnail (Frame 26061) Video thumbnail (Frame 27666)
Video in TIB AV-Portal: Edge Side Include Injection: Abusing Caching Servers into SSRF and Transparent Session Hijacking

Formal Metadata

Edge Side Include Injection: Abusing Caching Servers into SSRF and Transparent Session Hijacking
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
When caching servers and load balancers became an integral part of the Internet's infrastructure, vendors introduced "Edge Side Includes" (ESI), a technology allowing malleability in caching systems. This legacy technology, still implemented in nearly all popular HTTP surrogates (caching/load balancing services), is dangerous by design and brings a yet unexplored vector for web-based attacks. The ESI language consists of a small set of instructions represented by XML tags, served by the backend application server, which are processed on the Edge servers (load balancers, reverse proxies). Due to the upstream-trusting nature of Edge servers, ESI engines are not able to distinguish between ESI instructions legitimately provided by the application server and malicious instructions injected by a malicious party. We identified that ESI can be used to perform SSRF, bypass reflected XSS filters (Chrome), and perform Javascript-less cookie theft, including HTTPOnly cookies. Identified affected vendors include Akamai, Varnish, Squid, Fastly, WebSphere, WebLogic, F5, and countless language-specific solutions (NodeJS, Ruby, etc.). This presentation will start by introducing ESI and visiting typical infrastructures leveraging it. We will then delve into identification, exploitation of popular ESI engines, and mitigation.
Server (computing) Context awareness Injektivität Spyware Client (computing) Latent heat Sign (mathematics) Roundness (object) Cache (computing) Information security Address space Social class Context awareness Server (computing) Graph (mathematics) Formal language Human migration Inclusion map Cache (computing) Word In-System-Programmierung Web-Designer Revision control Configuration space Right angle
Web page Email Server (computing) Dependent and independent variables Email Group action Proxy server Computer file Dependent and independent variables Server (computing) Primitive (album) Cartesian coordinate system Element (mathematics) Fluid statics Cache (computing) Latent heat Latent heat Cache (computing) Single-precision floating-point format Right angle
Web page Inclusion map Cache (computing) Server (computing) Computer file Multiplication sign Web page Content (media)
Web page Ocean current Server (computing) Injektivität Computer file Client (computing) Web browser Parameter (computer programming) Perspective (visual) Metadata Variable (mathematics) Attribute grammar 2 (number) Cache (computing) Intrusion detection system String (computer science) Arrow of time Lastteilung Exception handling Dependent and independent variables Dataflow Server (computing) Chemical equation Content (media) Variance Client (computing) Computer network Database transaction Cartesian coordinate system Variable (mathematics) Exploit (computer security) Inclusion map Cache (computing) Lastteilung HTTP cookie
Implementation Server (computing) Injektivität Server (computing) Graph (mathematics) Self-organization Local ring Information security Computer worm
Email Server (computing) Computer file Server (computing) Demo (music) Plastikkarte Web browser Login Web 2.0 Proof theory Medical imaging Arithmetic mean Uniform resource locator HTTP cookie Local ring HTTP cookie Directed graph Proof theory
Web page Server (computing) Email Computer file Real number Database Web browser Cartesian coordinate system Medical imaging Internet forum Whiteboard HTTP cookie Message passing Local ring
Whiteboard Message passing
Web page Server (computing) Implementation Inheritance (object-oriented programming) Computer file Mountain pass Demo (music) Web browser Function (mathematics) Perspective (visual) Web 2.0 Latent heat Cache (computing) Whiteboard Computer configuration Cuboid HTTP cookie Oracle Physical system Proof theory Injektivität Dependent and independent variables Scaling (geometry) Suite (music) Server (computing) Graph (mathematics) Content (media) Electronic mailing list Mereology Mountain pass Cartesian coordinate system Cache (computing) Arithmetic mean Vector space Logic Website HTTP cookie Object (grammar) Oracle Computer worm
Inheritance (object-oriented programming) Link (knot theory) Demo (music) Content (media) Function (mathematics) Leak Cache (computing) Proof theory Arithmetic mean Cache (computing) Whiteboard Network topology Website HTTP cookie Message passing Local ring HTTP cookie Oracle
Email Expression Server (computing) Firewall (computing) Web browser Content (media) Rule of inference Product (business) Website Local ring Information security Data type Rule of inference Dependent and independent variables Server (computing) Software developer Sound effect Core dump Representational state transfer Web application Type theory Cache (computing) Arithmetic mean Doubling the cube Right angle Computer worm
Email Expression Server (computing) Implementation Inheritance (object-oriented programming) Computer file Local area network Codierung <Programmierung> Content (media) Local ring Metra potential method HTTP cookie Data type Area Dependent and independent variables E (mathematical constant) Server (computing) Content (media) Cache (computing) Type theory Word Revision control Website Electric current Cloning
Server (computing) Dependent and independent variables Dataflow Computer file Server (computing) Content (media) Website Client (computing) Twitter
Dependent and independent variables Email Server (computing) Computer file Graph (mathematics) Twitter Web 2.0 Cache (computing) Goodness of fit Mechanism design Human migration Kinematics Point cloud Software framework
Graphical user interface Sign (mathematics) Uniform resource locator Multiplication sign Password Denial-of-service attack Software bug
Louise here this is his first talk at Def Con and give him a round of applause [Applause] and he wants to talk to you about exploiting cash services cash servers so I'm gonna let him go and enjoy all right thanks everyone for coming last day of Def Con let's do this so this is edge side include injection abusing caching servers into server side request forgery and transparent session hijacking so I know that's a very long title meant to mention some of the many things that you can do with edge side includes especially when you're injecting it so for address of this talk we'll be referring to edge side includes as ESI for convenience because that's a mouthful so during this talk we learn about edge side includes we'll talk about the problem that it was created to solve will then talk about the problems that it created by introducing it in an unsafe manner and then we'll talk about mitigation and migration so my name is Louie domicile I work at goseck here in Montreal and to give some context on what ESI injection is basically it's a new class of attacks of course targets ESI enabled caching servers so it's not a widespread attacks unlike what James kettle presented at blackhat two days ago so this is really targeting es engines so this was discovered by a mistake basically one of my colleague Laura is only at go secure was tasked with reviewing the caching configuration for one of our clients so our client which is a large ISP basically wanted a cache overview of the security features and we kept seeing references to ESI kept coming up at shot includes so we're a bunch at go secure and we never heard about that none of us ever heard about that so we started looking into documentation and we saw that the first and final specification from ESI was in 2001 and I don't know if you were doing web development 17 years ago but security was not invented yet so we started looking into basically vendor documentation because the specification was so old we thought this can be right and we kept seeing stuff like that word art word art in documentation is always a good sign from an attackers prospective because it tells you well I wasn't even in high school when this was done so okay so the documentation isn't
gonna help us so I'm gonna explain it to you so basically let's look at this very primitive webpage examples so you have a weather webpage and to the end-user this is a single HTTP response right but to the end to the ESI server the ESI caching server this is multiple fragments and these fragments were invented to do one thing which is invalidate individual elements of a web page instead of invalidating the whole document so when you think of caching usually you'll cache a static file and ESI is invented to cache dynamic files so for the forecast Monday Tuesday labels you can keep that as a static fragment but the forecast for example 27? you can invalidate that within the next hour so with this knowledge we know that there has to be away from the application server to tell the caching server where fragments stopped where they start and where to begin so this Islam through fragment markers
those fragment markers are in the HTTP response and it look like this so yeah the ESI tag followed by an action it's basically XML tag right it's instead of about having secure layout you just have tags inside of the HTTP response which is going to get stripped once it's evaluated by the ESI engine so these tags are parsed when a specific HTTP header is sent by the application servers so you're not able to inject ESI everywhere but usually when they use ESI it's implemented everywhere so let's
look at our first feature we have ESI includes ESI includes our my opinion the most relevant ESI tags so you have two files to examine to show how this works you have page one that HTML and page to page one sitting on ESI server the second one is on another another server called API and you can see in page one you have an ESI tag which is an include time pointing to page 2 now pooting is going to happen here you can have a cache hit or a cache miss if you have a cache hit means that the ESI engine can just replace the tag with the content of P - easy enough if it's not there or if the cache entry is invalidated because it's been too long well you have to fetch this file so the ESM jain engine sorry it's gonna do a side request for this file to fill in the blanks so whatever happens this is what your end
user is gonna have you're gonna have the captain filled by the engine so to
illustrate how this works let's look at a very example of a cache miss so you have your clients load balancers and your servers so your client you're gonna request one that HTML and the caching server has a cache miss meaning the file is no longer valid it has to go and get it so this is done through the upstream server so the load balancer requests this file to the cache to the application server this is sent back with the ESI tag saying hey please fill in the blanks with to that HTML so this tag is parsed on the caching server and the side request is sent to the API server now the API server responds with the content of page two and the ESI engine is able to fill in the blanks our
second feature before delving into exploitation is ESI variables so they're very simple feature it's a very simple feature it has no attributes so no XML attributes and basically the content of the tags gets expanded to access metadata about the current HTTP transaction so you're gonna be able to access stuff like this so the HTTP user agent cookies query strings basically anything relating to the current HTTP transaction so now we know about ESI includes we know about ESI variables we also know that the tags are sent by the application server and they go through the caching server and this is where they are parsed but there's a very important question that we have to ask ourselves which is how can assign engine know which tags are legitimate and which tags are injected by a malicious user think about cross-site scripting it's basically the same thing except we're not exploiting browsers so that's a very important question and that's the problem it's that it can't and you're able to inject ESI tags and do basically whatever you want with the cache server so to illustrate this let's look a very basic example of ESI injection so you have the content of the CD get parameter that's echoed back in the HTTP response now the caching server is going to parse anything that is sent there so you can put ESI var and arrow pointing to the users PHP session IDs if you do know PHP know that this is HTTP only cookie meaning that javascript is not able to access this so if I'm able to access this from a caching server perspective I can effectively leak a session cookie and effectively take over the account and
this works as expected so let's try to build a payload in order to do this so we're gonna look at two ESI engine implementation first of all is we're
gonna look at apache traffic server so i looked at this one for two reason first of all because it's used by high-profile organizations so showtime tell us that it's used by Apple Yahoo and Comcast the second feature the second reason is because they have initial the initial ESI stack implemented but the added bonus features some of which are security features so our first security
feature is cookie whitelisting so even if you're able to inject ESI tags sometimes you're not gonna be able to access the cookies because they're not whitelisted if you want to access the cookies you have to preemptively configure the traffic server to say you can access this cookie but by reading the documentation you also find out about another ESI variable called HTTP header it allows you to refer to any Heather smart meaning that you can access the cookies so the whitelist thing doesn't work it's so easy to bypass so that was fixed when I reported two months ago pretty fixed quick quick fix so good for them so let's build a
proof of concept to do an HTTP only session hijacking without JavaScript so I built an image source tag you can do basically anything that requests the HTTP header but this is fine so it's pointing to able the local which is an attacker enable server for which I have a web server pointing there so the file name that is going to be requested by the victim's browser is a ESI tag pointing to their own Heather cookie so when this is going to go through the traffic server it's gonna expand the value with the session cookie and then the browser is gonna request this URL I'm gonna access this in my HTTP logs so now let's look at how this would look in
a real world example wow it's working alright so I built a very simple message board so you have your victim on the Left you have your attacker on the right it's two different browser so there's no cookie contamination on the middle you have basically everything that is stored in the database so you have hi hey so you can see what it looks like before being sent to the application server so now our attacker is gonna put the evil that local well basically the pedal that we just looked at so the file name has the header for the cookie and he's gonna hit Sante afterwards so this is gonna pollute the database of the application server now when they send it you can see in the database it's sent properly it's stored everything is working fine then the attacker by refreshing the page after sending it basically attacked themself so they're gonna leave in their own cookies because the browser is gonna send a side request for that image which is not really an image now we're gonna wait for the victim to refresh the page and we should effectively steal their session and as you can see the session cookie appears so we just told their HTTP only cookie we were able to take over their session completely and become that user so we're gonna look at the
session cookie we confirm that it's HTTP only we're gonna replace our own value with the one that we just stole through ESI injection we're gonna save that
cookie and save it and then once we
refresh we should become the victim
there you go so you have HTTP only so this is HTP only cookie hijacking
without javascript through ESI injection so that's nice but you need to inject a page for which a user is going to travel to sometimes that's not always easy sometimes think about self XSS the impact could be great but you're only attacking yourself so let's try to crack the impact up a notch so I looked another another ESI implementation which is our whole web cache so our web cache is usually sitting on top of web logic application servers it's not necessarily sitting on top of that but we've seen it sitting on top of that so I looked at it because it's usually high scale application and also because they'll initial ESI specification is implemented and they also have bonus features unlike 80s they went with the least secure option which is the added the ESI inline tag this tag is pretty easy to understand it allows you to override the engine the ESI and all of the is I'm trying to override any cache entry with arbitrary data so here we're gonna override jQuery j/s with arbitrary content what a great idea so jQuery s is gonna be filled with the content that you see here which is basically a ajax request so once the user is gonna refresh the file should trigger ajax request to are evil the local server and this is going to get expanded meaning that i'm going to get their cookie again because there is no cookie whitelisting in oracle web cache so now we know where it can overwrite file and we can make the browser do anything so the browser is gonna request this file because we took over jquery GS so let's look at a demonstration of this
which is already running for some reason all right so we have the same application server but now it's super safe because the system is they notice basically that HTML was not being stripped so ESI was also being injected well there was a possibility of ESI injection now we can see that our victim is confirming that jQuery ojs exists and the content is valid so you have to create retouch we don't want now our victim is gonna refresh the attacker is gonna put sorry pelo just to see if HTML encoding or escaping is working and as you can see the attacker is no longer able to put HTML char set meaning that Injection is effectively mitigated so that's a problem for an attackers perspective now the attacker sees a new feature which is a user list this yuta list will reflect anything that is in the search box so that's a pretty good vector for either self accessor or ESI injection and we can see here that hTML is not escaped so we're able to put our ESI payload in there which is gonna override jQuery ojs so the attacker puts it there some it's it it's echoed back in the HTTP response and now if everything works properly jQuery object should be overwritten now the attacker has effectively polluted jQuery edge is nice now the victim can just refresh the page and once anyone refreshes any page on this website they're gonna send us their cookie so there you go we just
over wrote an arbitrary file with
arbitrary content meaning I can either deface the website and steal anyone's session using ESI so that was basically
proof of concept to override arbitrary caching trees and leakage to be only cookies you can use our script but as you can see it's not really necessary so
now let's talk about mitigation so if you like web application firewalls you have mod security it's a pretty popular product it's gotten way better in the last years and if you use the Oh a spec or rule set you're good for ESI we talked with one of the developers of a security team and they basically said we already strip anything that is XML like which includes the the char set for ESI so you're good with that if you don't want to use a woth or if you don't use Apache you can use proactive escaping and what I mean by proactive escaping is you might think that's since you're okay like you're escaping HTML everywhere so ESI is the same char said I should be good well not necessarily because when you think about it contextualized effort of escaping will often ignore HTML and JSON meaning that hTML is never escaped in JSON because the content type is already telling the browser don't interpret this as HTML but we're not exploiting browsers we're exploiting cache servers so here I can put an es I include tag in a JSON response and this should work fine right there's this one small problem we have an invalid saitec because of those back flashes because of the double quotes but ESI engines had a very flexible syntax which is nice and I can just drop them so this will allow me to do server-side request forgery in a JSON response let's look at
a brief example of this so you have effective REST API sitting on slash API slash me which is gonna response basically a small JSON payload so you see that my full name here is Luigi or muscle and I can overwrite this with an
es I include tag this is I include tag is gonna say please fetch rest server slash server status which is just some server setting on the local area network of the cache server and you can see in the response in the reddish area it went ahead and fetch that file for me meaning that I can do server site request forgery with ESI includes now most ESI engines will not allow you to do server side request forgery on arbitrary host so you have to whitelist them prior to doing any assign include of them but some implementations will just allow you to do server side request word read through anything for example squid cache just allows you to do E's I included whatever you want so that's pretty nice to illustrate what this
looks like I just changed a content type to text HTML so that you can see that a JSON response is the great greenish area and then the reddish area is basically the content of the server start request forgery through ESI injection so to
illustrate how that work I used the same imagery that I did before so you have your slash me which responds with ESI tag saying please get that file for me the flowers fetched and then the content of step 5 and 3 gets concatenated together and I get the content of the server of the effectively the server site request forgery kept it content by ESI includes okay so if you want to do
ESI injection you need to first identify if you're messing with a ESI enabled caching server someone on Twitter called Alex via song came up with a pretty smart solution which is leveraging ESI comments UI comments basically are tags that are gonna get stripped by your ESI enabled server so if you have this HTML comment looking tag which is ESI and it's removed from the HTTP response then you're assigned gen has removed it if you do the same thing but instead of assigning the comment if it's something completely different for example foo bar or just and this one does come back then you're probably messing with ESI engine if you don't want to mess with manual detection because it's pretty painful you can just
use automatic detection so you have verb active scan + + burp upload scanner and a kinetics which all can detect ESI injections I'm not sure how I kinetics does it but I know that verb active scan + + + where apples can are using the aforementioned heuristic so it's really reliable if you think that ESI inject well ESI as
a feature is a good example of robust caching mechanism and you want to implement that I personally wouldn't go with ESI because basically it's pretty broken but you can use cloud fair workers which are basically JavaScript files sitting on edge servers someone on Twitter called Lucas Rider came up with modern ESI if you might have which is basically a JavaScript file that's going to allow you to do basically fragmentation of your HTTP response so you can see on the bottom screenshot you have a fragment pointing to footer and that footer is specified in an HTTP header which is so much more secure than just pointing to the resource and the HTTP response because if you've done any web pentesting in the past 10 years you'll know that HTTP response plating or just injecting HTTP headers is so much harder than it was 10 years ago so most frameworks will just not allow you to do that so if you have to inject in two places instead of just one it's of course much smarter and it's probably a lot faster - so this is basically a
sign Jackson there's a lot of research to be done with this if we documented this I think an April at this URL so basically you have our prior research we analyze I think half a dozen ESI engines some of which are pretty famous if you may so you have Akamai WebSphere varnish fastly and squid we found a whole bunch of bugs like denial of service with squid we found basically XSS filter by password chrome so a whole bunch of interesting stuff so if you want to go ahead and search for more ESI bugs I think you'll find a lot of them I think this is my time and I might have one and a half for questions [Applause]