Can everyone hear me up the back? I heard there's some sound issues but lots of thumbs in the air. Thanks very much. Cool. So a lot of, a lot of us have probably heard about domain fronting. It got a fair bit of attention recently when signal messenger made some noise when I think Google said, hey, we're going to put an end to this. So there was

lots of questions about what is it. There was an academic paper floating around which I don't really understand how to read papers like that. So I asked my friend to help me out and he explained it to me and I sort of figured it out. So I thought let's bring it here and explain it in simple terms to maybe people who don't even necessarily know what

TLS is. Um, or just have a rough understanding. So I will be maybe to some people going to a, a basic level. Um, that way more people can understand. So hopefully we get the

right balance. So um, what is it? I, I put on my dictionary writer hat and sort of dictionary definition and it's um, abusing an implementation detail of shared infrastructure to disguise the true destination of a HTTPS transaction. So um, there's a few, there's a few keywords in here. Uh, implementation detail is one big one and shared infrastructure is

another. And we'll, we'll go into why these, these parts are important later. And HTTPS, as far as I know there's probably not really any other protocols that share, uh,

different paths to different things within the same place the connections get terminated at. Um, domain fronting is not new. It's been around for quite a while. It didn't do an exhaustive search but uh, it has uh, this is the oldest thing I could find that uses it. It's uh, some kind of proxy that's uh, written in Python and um, it's

some, the code, the code of GitHub, there's the commit message says abstract host substitution. Um, maybe that's the old name for it. Um, cool, so let's look at some of the advantages of domain fronting. Um, what? Sorry, I messed that up. Sorry, uses domain fronting. I mentioned uh, signal messenger. They primarily seem to use

that for bypassing censorship. Um, lantern seems to do the same. I think that's some kind of a VPN. Um, and then there's uh, meet client which is an OBFS proxy for Tor. Um, I presume pretty much everybody here has heard of Tor, which is usually used for, you

guessed it, bypassing censorship. Um, but also um, for evading detection. So uh, the OBFS proxy, there's different plugins for Tor basically that let you pretend you're different types of traffic. Um, all kinds of interesting ones, I wish I could go into

them today, but that, that's a huge rabbit, rabbit warren. Um, and then malware, they'll be bypassing censorship. Obviously, censorship isn't just a country saying no, you can't go to this website, that's also corporations saying no, you're not going to Facebook at work. Um, and also evading detection, malware doesn't want to be going to

hey, you just got hacked dot com. And, so, advantages, why is it useful? Well, it's uh, it can't be detected without breaking TLS. Um, someone has to actually perform some kind of a man in the middle attack in order to be able to see that you're doing that. Um, so,

so, one potential case here would be, if I'm in, if I'm running in a corporate network, which doesn't want to invade the privacy of their staff by intercepting their TLS communications, um, I won't be able to tell that this is happening, so, you know, maybe a

piece of malware could use it to, to evade my firewall from detecting it. Um, it uses existing infrastructure on the server side, so when I've been playing around with this, it's, uh, mostly CDNs like, uh, CloudFront and, um, not CloudFlare anymore. Um,

not Google Cloud anymore either. More on that. And, uh, it's compatible with anything that can be tunneled over HTTPS. So, if you can think of a way of tunneling your traffic over HTTPS, you could use domain fronting to, uh, to send that traffic around. And, it's, it's very simple and easy. It sounds complicated, but once you, once you realize how simple it is, you, you'll just walk away and go, cool, I'm going to go do that

at home, it'll take you five minutes. Okay, was it not useful? Um, bypassing censorship where the TLS is man in the middle, so if I break your TLS session, I can see you doing this, I said that before. Um, and it's not useful for hiding web traffic in a

normal web browser. You can, you can do it in a normal web browser, but you, each webpage you go to is going to load things from all kinds of different places. You need to, uh, find suitable domains to hide behind and not every website would have domains that match. We'll, uh, so, yeah, it needs to be on a shared infrastructure and, uh,

the front domain for that, that destination needs to be known. So, uh, I'll, I'll explain what I mean by front domain, uh, a bit later when I go into the demo. Cool. Oh, and the security is void. Uh, the reason for that is you need to basically trust a

different certificate to where you're actually talking to. Um, again, we'll see more in the demo. So, how does it work? Um, well, let's, uh, go back to the days of HTTP 1.0 and, you know, obviously when your, your web browser was doing, you're trying to browse to

www.mywebsite.com and, uh, you do a DNS lookup, your DNS server returns a server address and you connect to that web server and say, hey, get me the home page and it returns it. The problem here, of course, is we needed one IP address per domain we were hosting. So, if you wanted to host both, you know, website A.com and website B.com, you

needed to have two different IP addresses and some of us couldn't do that, uh, or, you know, and then we obviously want many millions of websites. Maybe we would run out of, uh, IPV4 even faster than we currently are. Um, so, they introduced the host header. So,

basically, that just changed this, uh, sequence diagram slightly. Well, not at all, really. It just added this extra piece of information on the, uh, on the, the second green arrow here where we just specify, hey, I want to go to website A or website B. That way, that way we could share that, uh, IP address across both of them. But, it's

2018 and we encrypt web traffic where we're supposed to. So, uh, you know, we, this thing called TLS, I guess, was its original name. I'm sorry, SSL was its original name and, uh, now known as TLS. Um, basically, it just takes HTTP as it was, wraps it in a

secure layer that validates, you know, where, where you're talking to. In some cases, it validates you yourself if you decide to do bi-directional authentication. Um, you know, am I talking to my bank or am I talking to someone that claims to be my bank? You know, it takes care of that for you. And it encrypts the traffic between you and the server, as

we know. And without, without changing the protocol. So, as the application developer, it doesn't matter if I'm using HTTP or if I'm doing IMAP or POP or whatever, you know, TLS was designed to just go over the top. However, sorry, getting ahead

of myself here. So, TLS handshake is pretty complicated and I don't understand anything but the first two that's happening there. And, and, and it's not, not relevant for domain fronting. Basically, what happens here is I use TLS handshake, your client says, hey, I

wanna talk TLS. And then the server goes, sure, I am your bank. Definitely your bank, not someone else. And they do some kind of, uh, special packet dance and arrive at some, uh, encryption keys that they both agree on. And from, from there, everything gets

encrypted, right? So, it's a layer, it's hidden from me as the application developer. But, uh, everything from that point is encrypted and as the application developer, it just looks like the same protocol that I came up with, in this case, HTTP. So, forget about all that stuff that I don't know. And we pretend it's not there and

just focus on the first couple of messages. Now, if I'm, if I'm sitting on a network watching your traffic, ok, so maybe I'm at your Starbucks Wi-Fi, maybe I'm your ISP, maybe I'm a government, maybe I'm your, uh, your sysadmin at your company, I can't see what's

happening down the bottom here. Um, that part's encrypted, obviously. But, what I can see is, these first few messages are unencrypted. The client hello actually has a part at the beginning that says, hey, I wanna go to google.com. And the reason for that is

because at the beginning, when we, when we look back at our HTTP 1.0 versus, uh, HTTP 1.1, is, I've totally messed this up, I'm sorry. So, the, the reason for this is, when I say,

hey, I wanna talk to TLS, the server immediately needs to know who it is representing. So, if I have website A and website B both on the same server and my client says, hey, I wanna talk to TLS, I need to send it back, back a certificate to say, hey, I'm website A or website B. So, that meant TLS had to be extended slightly to

identify which server we're talking about. Now, am I talking to google.com, am I talking to yahoo.com? So, that got, it's an SNI extension, so I can, I think, when do

the xbox 360 cannot, uh, that didn't support it, I just know that if you, if you wanna support xbox 360, it doesn't support this, you need to have one IP per, um, so that's roughly how long ago it got introduced, but it wasn't a standard thing. So, say I'm, I'm

on your network, I'm sniffing your traffic and I'm watching you use HTTPS, I can see that you're going to google.com because it's sitting there in that very, uh, first message that you send in your TCP session, um, to google. So, that's, that's the just to, just to recap that, the entire contents of the HTTP session are encrypted, um,

which is all that stuff, um, for web developers, maybe early on, certainly early on in my web development career, none of this was clear to me, like, where all this stuff went, it just somehow magically arrived and then you got it from dollar post in PHP

somewhere, um, and then what's not encrypted is the domain name of the server we're talking to, that's exposed by SNI. So, that's how you're, even though you're using HTTPS, your ISP can see what you're, what you're looking at and the server certificate also isn't encrypted and I can't remember, but I think TLS 1.3 is supposed to encrypt

the server certificate. Anyone here? No? No? It does, thanks. Um, cool, so let's take a look at, let's take a look at what makes it work is the implementation of HTTPS stacks and

how they usually get shared across, uh, customers, say, if you're using Amazon Cloud front. So, very simple single server stack, like, let's say I'm, uh, when I was 14, I used to run a web server under my, under my desk, which was just Apache, um, and it basically, it's this, this first thing on the left, so, sorry, first thing on the

left is a client and then we got the web server, which is, like, Apache and whatever, and that pulls stuff from an origin and that's generic because the origin can be a file system, it could be a Rails application, it could be a Django application, doesn't matter, um, it's, this is just to show the separation of concerns. So, you see

this web server is responsible for HTTP, HTTP TLS and, and caching, um, and when you're running one website, uh, it's, it's all, it's all pretty simple and yet, what we like to do, usually for performance reasons, though, is delegate the TLS. So, TLS is, you know,

reasonably heavy on the CPU and perhaps what you're doing in your application might be very heavy on the memory. So, we, often, often we split this out and this is often what's happening, uh, at somewhere like CloudFront or Google Cloud where, um, they have

lots of customers and they want to have a very fast end point with an IP that lots of many, lots of customers use that just does the job of terminating those TLS connections and getting them to the next link in the chain, which is, um, the next web, web server that goes and then fetches the content. Um, and so, when we delegated it,

this is where I can steal that little NSA graphic I, I love. Um, so, this is how, um, how I think Heroku kind of works, is that that's a shared infrastructure provider. I haven't tried if domain fronting works for Heroku yet, um, that's probably something I'll do

right after this. Um, but, uh, you basically, you talk to that reverse proxy whose job is, hey, let's terminate TLS here and then it goes to something else whose job is, uh, look at the host header in, in the request and route it to whichever one the host header matches. So, what, what happens in the CDN networks, uh, often is the

reverse proxy decides what certificate to send back to the client, um, based on the SNI header, that thing you saw in my YShark screenshot and then the next link of the chain decides where to send it based on the host header, um, which they don't always

have to match and in domain fronting, they basically, we deliberately make sure they don't match. Um, so, if we were to make, uh, I'm sorry, this is way too small to see on the screen, but if we were to make a request to www.good.com and say in SNI header, hey, I want the certificate for good.com, then I get my TLS session established and then

I say, okay, get me the homepage and the host is www.evil.com and what will end up happening is if that reverse proxy, that should satisfy the rig, the TLS session, then the router would look at the host header and go, ah, I want to go to evil.com and, uh, this

seems to be roughly how it works behind the scenes at Amazon. Unfortunately, I don't work there, I don't know anyone that does. I wish, I would love to know why it works there. Um, so, I've already, I've already ruined that one. Um, and again, the

remembering what we saw in the, in the Wireshark, I would only see, if I'm sniffing the network, I would only see the packets going to good.com. Why does it work? Um, an anonymous Google software engineer said it worked because of a quirk of our software stack. Um, and Google has since done what they can to get rid of domain

fronting. So, it highly depends on implementation. Um, in order for it to work, the shared infrastructure must not check for a mismatch between SNI header and host header. Apparently, that's what CloudFlare does to stop it and it kind of makes sense to do that. Um, and also, HTTP requests must be routed separately to how TLS route, the TLS.

So, both those layers need to be dealt with separately. So, to put it together, what do we need to actually do? All we need to do is connect like normal to one host and set the

host header to another and they've got to be on the same infrastructure that can route between them. So, we find evil.com needs to be accessible via the same infrastructure as something innocent looking. Um, and the infrastructure needs to have the right implementation quirks. So, let's say there's plenty of websites on a, on a

popular CDN like, uh, Jqueries on a CDN. Um, and that same CDN I could go and sign up for myself and put my malware command and control server behind that, that CDN, which would allow me to use somebody else's domain who uses that same CDN to go there. And I

keep giving away the, um, I think that's better. So, if I need to go to the finding them, there's, there's loads of websites there. If we go, if we go to say Alexa top, top 500, we should be able to just do reverse DNS lookups to, you know, uh,

google.com points to this whatever. Uh, this customer is using Akamai. I think Facebook uses Akamai. Um, also would be a good, another good one to try. And they're, they're easy to find and sign up for. So, what makes some domains better than others? Um, it

depends really what we're trying to do. If we're evading detection like malware, we want to have something that looks pretty business as usual, you know. If I, if I'm in a company that sells, uh, apples, it probably doesn't look very suspicious if I'm going to fruit.com. Um, or maybe something innocuous, you know. If I'm, if I've infiltrated, uh, a

company and I'm trying to exfiltrate some data and that company is also a marketing company that uploads a lot to YouTube, maybe I could hide my stuff through youtube.com. If I'm in a, in maybe a country that blocks, blocks a lot of access to sites, um, which is, which is Signal's problem, is, uh, they were, they were in a

country which was blocking, blocking their messenger and they also apparently block websites. They chose a web, an e-commerce website which, um, they thought would be, would, would have collateral damage. Um, so if that country were to block that site, um, that would, that would negatively impact, um, that, that country as, as a whole.

So perhaps not be, perhaps be a reason why that wouldn't get blocked. And then maybe you could do a combination of all. You find something that's, you know, it looks like business as usual, it's innocuous and it's got collateral damage. Probably a really good one to go with. So, um, I've, I've talked a lot and I'm going to keep talking.

But instead of, uh, boring slides, let's, uh, fire up this SSH session and I'm going to try and talk and hold the mic at the same time. Alright, one second. Alright, so what

we're going to start with is, uh, I, I have a root shell here and I'm going to start off a, uh, a TCP dump so you can see the traffic that's actually leaving this

machine. So it's, this is just an empty EC2 instance in, in Amazon. So there's, there should be hopefully no other traffic on 443. Um, so this command here, for those not familiar with TCP dump, TCP dump, uh, takes network traffic that's going through your

network hard and logs it somewhere. Um, and I'll just briefly go over the arguments here. Minus C4 means stop after I've seen four packets and so there's three for, three for the TCP hands, three for the TCP handshake and then the fourth packet is our first message from client to server which is our client hello which has the SNI header in it. Uh,

minus A means give me ASCII output, show it straight to the terminal and is like a performance optimization like don't do a reverse look up to, uh, the IP address. Um, and I, our interface F0 and then we're looking for TCP port 443. So let's run that and then

switch to another screen and then we're just going to run a curl to pixum dot photos. It's like lower Mipsum but for pictures. Um, so curl minus S just means, uh, shut up, don't do anything except get that and give me the output to standard out and then we're going to pipe that to, uh, MD5 sun so we can look at, okay, what was the content I

actually saw? So the content I actually saw, in this case, or the hash of it is, uh, you know, C8E, uh, EAD. So the purpose of this is just basically go get me this webpage, make a hash of it so I've got an idea of, uh, what I saw and then let's get

another webpage. So in this case, um, protect your privacy now dot com and obviously different hash. So these are the two hashes and then what we're going to do is, uh, take a look at when I, when I do a domain fronting which, which one I get. So, hopefully this makes sense. Now just realize I've messed up my TCP dump. Okay, so only

the first, only the first curl request got shown up, uh, only the first curl request got dumped here because, uh, it stopped after four packets. So you can see right here, this is,

um, this is my SNI header that says, hey, I'm going to pick some dot photos and then let's run the TCP dump again and switch to the other screen and run, I'll just run this curl command here. So when I'm going to protect your privacy now dot com, you can see on the other side, you've got in the same, roughly the same place, I'm going to

protect your privacy now dot com. Alright, and it's in my history because I was testing it out. Cool, so basically, let's do some domain fronting and turn over to, let's get TCP dump running again. Okay, so, uh, all we've done here is we're still

establishing our TLS connection with protect your privacy now dot com. Um, so you can see that most of this curl command is just the same as the one above it. The different, the

only difference here is we're telling curl with the minus H, uh, flag, change the host header to pick some dot photos. And so when I run this, the, the MD5 sum of what came back is actually pick some dot photos, but if I go and look at the other side where I've, uh, dumped the packets, it actually just shows protect your privacy now dot com. Um, so when I

said it was really easy and you can go and do it in 5 minutes at home, there it is. Um,

so, uh, there's, there's a few risks here. Um, obviously one of them is the reliability. When I try going to protect your privacy now dot com, I don't know that they're going to keep it on that same CDN as pick some dot, pick some dot com, sorry, pick some dot photos. Um, they could change that at any time and they could point

it somewhere else. Um, so a potential solution to that is you could have a list of backup domains. If you were shipping an app that, uh, relied on domain fronting, maybe you, you're going to have a list of other ones that still work. Um, but a bigger problem is you can't validate that the server is authentic because if you're connecting to, uh, protect your privacy dot com but then really intending to talk to

pick some dot photos, the server you validated who you're talking to is protect your privacy dot com, not the photo site that you're trying to go to. And, and also, since

you're connecting to that place that you don't, you didn't set up yourself, they could change that certificate or CA at any time, uh, so you can't pin it either, which you, you'd normally be able to do if it was, you know, a self-signed cert or whatever. So, also, the traffic's visible to the infrastructure provider. So, say, if you had traffic

that you needed to hide from, uh, Amazon cloud front, for example, um, or, or it was sensitive, because you established your connection with site A, that infrastructure provider just by default, uh, is able to see what, what you did. It's, uh, it's kind of

obvious if you're used to using a CDN, but it's not that obvious sometimes when you, when you do these tricks, I don't know, I guess I forget about all these sort of things. Sensitive data could be stolen and malicious payloads could be injected. So, basically, just treat it as an unencrypted connection. Encrypt and sign all your

messages, um, maybe come up with your own way of, uh, validating that you're really talking to who you think you're talking to. Cool. So, uh, I actually saw a DOS attack happen via domain fronting. It was a combination of, uh, some not quite configured correct, uh, not quite correctly configured infrastructure and, uh, domain fronting. So,

what happened was, uh, we had a very, there was a very normal web server set up, you know, you start with one web server, you refactor your application and you make it two web servers and then you, you know, you eventually end up with a, a cluster of them and then you end up going, okay, we need a CDN and so it was put behind a CDN. Um, and

so they marked the CDN IPs as trusted proxies to the web server, um, as, as you do because obviously TCP connection hits the CDN, uh, endpoint and then the CDN makes another connection. Um, and then attackers found a nice slow web page and they

decided to make thousands of requests to it. But they used domain fronting which in the case of the CDN that was being used and, uh, for some reason was going between two different IPs. So, it was going from one CDN endpoint to another and then to the

web server which it didn't expect. Um, so, basically the connection diagram would look like this. You get a client way over to the left, connects to a reverse proxy, that's one TCP connection and the reverse proxy then connects to the web server. So, normally, um, because the web server can only see the connection from 2222, which, so,

for those that can't see the, uh, the IP addresses, um, they're a bit small. So, the web server sees the reverse proxy's IP address from that actual TCP. So, the reverse proxy usually adds, uh, into the X forwarded for header, hey I'm forwarding it for 1.1.1.1. And,

and the web server knows that because the actual TCP connection comes from 2222 that it can trust that 1.1.1.1 is really who it's talking to. Um, and that's what I just, so, in this case, what, what was happening was the user was hitting, uh, the, the front

domain, the, the domain fronting domain that was being hidden behind, that added X forwarded for 1.1.1. It, uh, got forwarded to the other one which added another X forwarded for header and then passed it up to the web server who didn't really

understand that there could be more than one. Um, so, uh, that allowed, that basically opened up an IP spoofing vulnerability, um, because the web server was thinking that the user's true IP was in fact the IP of the CDN endpoint. Um, so, what we expect to

happen usually is that header has one address which is a true IP of the user or at least that's what the web server was expecting and that the web server reads that header and it knows who it's talking to. In actual fact, there was two, it didn't understand it and it assumed, it assumed that the CDN was the real user. It got worse because we had

failed to band configure and it ended up banding the whole CDN and taking the site out because it thought the CDN was tossing it. And, uh, yeah, the root cause, misconfiguration, we'd forgotten, look at the, look at the full list and, uh, because it went through

domain fronting which is a, uh, a way that it wasn't expected to work, it invalidated that, that assumption. So, in order to plug that, know your infrastructure, um, X forwarded for is actually a de facto standard. Your infrastructure provider might do it differently to how you expect, um, I think CloudFlare I

think has a different XCF connecting IP or something, it's completely different. And wherever there's a possibility to get proxied multiple times, make sure that, uh, the chain of your, the chain is trusted. Um, so, you know, if you've got a CDN endpoint that

maybe hits another one, that hits, uh, a load balancer, that hits another load balancer, it needs to be able to draw a line between all those endpoints and go, uh, okay, I trust this one which trusts that one and, and find its way to the, the true IP. Um, so in that case, NGINX is, I, I don't know the other ones unfortunately, but it's real IP

recursive if you want to Google that. Um, and CloudFlare's header connecting IP. So, the future of domain fronting, I don't think it'll last much longer. Um, Cloud, CloudFlare has already said, um, they're getting rid of it, it can't be relied upon. Um, it's

implementation details which could change. Uh, like I mentioned before, Amazon might just suddenly start doing something differently one day. Um, Netlify, Beluga, all those CDN providers, whatever you use, um, they might find a more efficient way of doing it or decide to get rid of it. Different regions of the world might use different

infrastructure. Uh, I think I noticed this when, uh, I was in China, I went to Apple dot com and I was like, oh, I wonder where that's going and I had looked, it was completely different to the result I was getting from Perth. It was back in my own place in some other CDN elsewhere. So, if you were writing something that needs to be used all over the world, uh, they'll be, they might be pointing to different

infrastructure. Um, and it's also actively possible, sorry, it's possible to actively prevent it from working. Um, which is, Cloudflare is deliberately doing this by checking for a mismatch between the SNI and host header. Or, by using SNI for r-routing the

request, if you remember the several layers of, uh, where it decides on its final destination. Um, it could just simply use the SNI header, which I believe was the percent behind, uh, HTTP 2. Um, it seems unwanted, Amazon and Google, uh, have, uh,

r- appear to have responded to pressure against it. Uh, Google's already broken it deliberately. Amazon said they will. And Cloudflare, for some reason, says it's a risk to their customers. It would put our traditional customers at risk. Um, I presume that means people behind, uh, corporate firewalls, they, cause those corporate

firewalls can't necessarily ban malware traffic if it's using domain fronting. Um, so, domain fronting, maybe not that useful for making your own personal HTTPS more private. So, a couple more suggestions here. Uh, TLS 1.3 has explored the possibility of, uh,

encrypting the SNI component. It didn't make it into the spec because it required an extra, uh, round trip in order to, there, there was, there was actually two proposals. One was an extra round trip to negotiate a key first. Uh, the other one, I think, was based on a static key. I didn't really pay much attention to it. Um, um,

however, I did read recently that there was, uh, there was work done on this, uh, by Cloudflare, I think. And, this was only in the last couple of days, I think, that came out. Um, use a conventional tunnel like a VPN if you set the tunnel up yourself. Uh, tunnel traffic other ways. There's, there's other, there's other protocols

out there for, for achieving this. Um, I've seen some pretty, obviously, old school malware, uh, command and control, IRC. Um, a friend of mine tried to do IP of a Facebook messenger. That was pretty horrible. Um, HTTP over XMPP. And, uh, just coming

up with, uh, crazy silly ideas can be, can be pretty fun. Um, but I, I hope you all walk away from here understanding, you know, how SNI facilitates sharing, uh, infrastructure, um, that uses TLS. You know, working around that, uh, problem of needing one IP per, per host. Um, how third parties can see, uh, where your HTTPS

traffic is, is going without doing a man in the middle attack. Um, how to find, uh, domains and, and actually do domain fronting yourself. And, and also how to, how to protect yourself from, uh, misconfigurations like, uh, like what I saw before. Um, where

basically the infrastructure just didn't do what, what was expected. Um, thanks, thanks for having me. Uh, any, any questions?