Are we on? Oh, we are, too. Morning, everyone. Thanks for coming along. The context of this talk, there's a few different parts in here that sort of trigger it. One was, as we see more and more web developers going and starting to directly touch infrastructure,

I think there's a lot of things that we've been able to sort of ignore for quite a while. We've typically, you know, go back not too many years ago, you build a web app, you hand it over, there's some ops team, they do that server-y thing and they get it running on the internet. Now we walk up, we get Azure websites, so we get AWS or something like that. And we're bolting domain names onto websites.

And I still see a lot of web developers out there who hit some problem and they kind of look at it and they're like, is it my Wi-Fi down? Is the server wrong? Does that damn DNS thing not work still? And I really wanted to start to just open up more of this. The other thing that I find really interesting about this is a theme. And I don't want this, this is not all just a networking talk. But as a theme, there's an amazing amount of magic

that happens in that first 50 milliseconds. You type in google.com, you press enter, like done, and all of this magic has happened. And the amount of stuff that happens in there is just amazing. The problem with that though is that we spend a lot of time just turning it off and back on again.

There's a great essay, it's up at stilldrinking.org slash programming sucks. And it's a very, very good read. I'll put the link up at the end if you haven't seen it before. And it talks about that you get even a one year old PC here or older, nobody in the world actually knows every single thing

that's happening on this machine. That's why we tell people to turn it off and back on again. What I want to show through this is how easy it is that we can actually keep pulling back the layers and none of it really is magic. A lot of this web technology is just ASCII moving around in really simple protocol that was written 30 years ago. So it's not that complex.

So to kick off with that, there's a lot of stuff that happens when a request starts. First one we've got is DNS. So we're gonna throw out some other protocols that might happen delivering google.com. HTTP, HTTPS, any others?

ARP, nope. Sorry, PKCS. Okay, we've got the technical person here. We can play this game, but we're gonna end up with a list like this.

And this is just the start of it. And alongside this, we have a bunch of other protocols and technologies that all need to make it work. Because we're not just going and plugging in a network cable anymore, we're doing it over Wi-Fi or we're doing it over Wi-Fi that goes to a hotspot that's going out over 4G or LTE. And we've got IPv6 getting in the mix. We've got all this underlying plumbing

that generally we get to skip over. I really like this quote as well. Any sufficiently advanced technology is indistinguishable from magic. That's why it just kind of feels magic. All right, so why don't we start off with jumping into DNS as the first one.

We'll assume for a moment that we've got a fully initialized kind of bootstrap network connection. We're all connected, good to go, and we need to look up something. DNS is typically described as the phone book of the internet. For people of my generation, that's those things that get delivered outside your apartment block that you use to either prop up modems or you just prop up your monitors or you just leave them straight in the recycling.

But I think that's really kind of a boring description of it because it implies just mapping names to IPs. And DNS is actually used for a lot more stuff. What it really is is a distributed key value store before key value stores were cool. And it's no SQL because you can't query it with SQL, but it does have its own query language.

And it's incredibly powerful and resilient because of that. Now, I assume everybody knows the whole two hard problems in computer science joke. I've seen it around here a few times this week, location validation, naming things, and off by one errors. Well, I don't understand why people find DNS so hard.

It's exactly what it is. Right, so let's jump in and have a look at some of this. I'm gonna jump into a PowerShell window, clone my display, and start resolving some things. Now, one of the other things that I've been trying to get myself into a habit of

is getting out of the old commands like ping and nslookup and all these things built into cmd.exe. So I'm gonna show you a number of PowerShell commands as we go. First thing I'm gonna run is just resolve dns name microsoft.com.au, because I'm from Australia, so everything ends in a .au. I'm gonna ask for the name server records

at the end there. So Microsoft, it gets delegated out. It's got some name servers. I'm assuming most people have a basic understanding of DNS, that there's name servers, they return records, and off we go. As we walk up the stack, somehow .com.au has to work. So we can go and look at that. I can actually just ask for com.au, and that works.

And then ultimately, au has to work. So we can continue to walk up the tree. Whoops, there we go. For this one, because I'm getting down to a single name,

the command line starts to get confused about whether I wanna go and do some form of local resolution or what. So I need to explicitly tell it to go out and do DNS only. How do we even find au, though? So we've got our root servers. I'm gonna go all the way up to resolving,

just name blank, but it says that doesn't work. Now I'm getting timeout periods. This will be fun doing a networking talk with no networking.

Cool. So conceptually here, though, there's a bunch of root servers, which are quite useful. And those root servers get published at the top of the internet, and there's 12 of them that go and delegate off, and we get this tree that runs all the way down. What's interesting about the dot, though,

is actually how that works as a termination character. And a dot in DNS indicates just the start of the next block. And where this is actually useful, is I'm gonna just jump in and show you what some of the underlying protocol looks like. So quintessential protocol analyzer is Wireshark. How many people have actually used Wireshark before?

How many people have actually got a successful debugging result versus just seeing lots of stuff on screen? A lot of nice hands, guys. Because I designed for failure, I'm going to open a recent capture, which I've done. This is Wireshark. And I'm gonna filter at the top here

by just saying I want dns.query.name. I'm gonna go look for outlook.office365 when I've resolved that before. I hit enter and it goes and finds me a request. In here, down the bottom, I've got the actual, so I've got some packets at the top there. Down the bottom, I've got the actual query.

And when I break this apart,

in the actual query here, the encoding that it uses is where it's doing outlook.office365.com. It sends the number seven, the word outlook, the number nine, the word office365, the number three, com, then it sends zero to indicate the last component,

which is that final dot. So what this actually means is that, we'll see if our network works now, I can resolve microsoft.com.au, but I can also resolve microsoft.com.au, dot. And this has always been a fun little trick with web servers like IIS Express that only bind to local host,

is you can actually resolve them also with local host dot. So you can do http colon slash slash local host dot, and it will still resolve to the same place. And a lot of the detection logic in different applications won't detect that as local host. So that can be a fun little thing.

As I come out and resolve some of these, I'm just gonna, we'll just do google.com.au for now, and I'm not gonna explicitly ask for the name servers. I'm getting two different records that come back here at the top. So in this table, I've got google.com.au, now I've got my A record returning ipv4. We're also seeing more and more of these

quadruple A records coming out above that, which is ipv6. One of the common ones that we come across as we try to use any of these cloud services is everybody asks us to use a CNAME. I see a lot of people that assume that kind of a CNAME is another type of these, it's a bit of a redirection. And we run into problems about not being able to use that

on what's called the apex or the root domain. What a CNAME says when we return that at one of these is it says everything about this entire node in my distributed key value store, you need to go and look over here instead. And what that actually then means is if I was to resolve something else off here,

let's do type MX, so I go and get out, say, the mail server records that map to it. Because we redirect the entire node, those all get pointed somewhere else as well. So what does this mean? Let's say we've got mysite.com and we wanna put in mysite.com as a CNAME

to mysite.azurewebsites.net. What that's gonna say is when somebody wants to look up where do they send mail for mysite.com, they're gonna go and query mysite.azurewebsites.net and it's not gonna return them. So this is why, well, for a lot of these applications, we sometimes struggle to put them on the root or the apex domain.

There's a number of DNS providers out there now which are starting to provide a lot of other infrastructure on top of just kind of basic DNS. So DN's simple is quite a common one. And they're starting to introduce records like alias and pool. And these are getting snuck in as just kind of the, as if they were part of DNS.

But these are actually a server-side concept where when we use an alias record, DN simple will go and resolve the other record for us and then basically cache that and then return that to other people as if it was an A record. So they start to become quite useful as a new concept, especially where we're working with cloud services.

Right, spinning through, I wanna get through DNS reasonably quickly. So I'm just gonna power up through this a bit. One of the common scenarios that we see is also around load balancing. So I'm gonna go and say resolve DNS name and I'm gonna ask for outlook.office365.com.

And I'm gonna ask for an A record off the back of it. And what I get that comes out in that table at the top there is actually a collection of records. So the first one is saying that outlook is a CNAME for lb.geo, so load balancer geo. And in that there is returning me a CNAME for something else which is then ultimately getting me an answer

which says you need to go and look at outlook EMEA West 2. So they're actually doing DNS-based load balancing here to go and direct people off to the right traffic. Now that could potentially get cached. So we need to make sure that as I move from place to place, my laptop could be holding a cache. So we need to make sure the service can respond from everywhere.

But DNS actually becomes a really powerful thing for going and distributing traffic in a very efficient way. There's another service, how many people use Visual Studio team services, VSTS? Not many, okay, a few, cool. So if I go and say resolve DNS name mashit.visualstudio.com,

I get an answer here where they give me an A record. So that works all right. But if they sent all the traffic for all their accounts to a single address, then they've got this kind of complex load balancing problem whereas they have really nice segregated accounts. If I go and ask for the address for something like retify.visualstudio.com,

I get out pointing me to a specific cluster. So that's saying TFS production scale unit US scale unit three. So they're pointing me direct to a specific scale unit where my account is hosted. And they're doing that via a wildcard, sorry. What they're doing here is they're priming the DNS and saying this account is on this scale unit.

But then when people create a new account and perhaps that hasn't propagated yet, they're falling back to a wildcard entry that says, if they come in and we don't know which scale unit they're going to, send them to that other IP address and then we'll go and redirect them back on the server side. So they'll go and move that traffic around. So it becomes quite a nice way to actually go and segregate accounts and move traffic.

Now as we start to do dev where we have multiple domain names in play, we often end up in that ugly hosts file. I'm seeing a couple of smiles up there of oh, that old hosts file. Now I don't really like the hosts file

because first of all you have to go and run as admin to launch Notepad to then go and dig around in this ugly part of your file system to get to the thing and to go and update it. And it's not very friendly in a development environment where you want to check out F5 run off you go. So one of the things that a group of us set up a few years ago, if you go to readme.localtest.me

is we thought why don't we just run a DNS server on the internet that returns 127.001. So what we've done is we've used a DNS wildcard record. So everything except readme.localtest.me goes and responds with 127.001. So I can do ndc-oslo.localtest.me

and I get 127.001. So we can use as many different DNS names as we want in our local development process. Yes it's hitting a live DNS server but it all resolves to localhost. One of the problems as I go into different environments as a consultant is that I often can't trust

what the setup is that teams have gone and run. So I might go and resolve something like evil.localtest.me and it doesn't return 127.001. It's returning something else. And then often you'd go off and dig in the hosts file and go and check there if you think to check that.

One of the useful things about a lot of these PowerShell cmdlets, these newer ones over the CMD versions is that they have various switches for things like no hosts file. So I can go and see the different results there. Very useful little switch. I don't trust what I'm getting back off the local machine. I had one client once that was taking us ages to work out a problem. And we had the front end web servers

that on the web server they also had a bunch of service layer and they were hosting it side by side. And the service address that the web app was connecting to was HTTP local host blah, blah, blah. It took us a while until we were looking at a command line and went, why does local host resolve to 127.002?

Because they thought it was a great idea to add that as an additional IP address to every box and to remap it to .2. So you can't even trust local host. Interesting little tidbit. The host file, one of the reasons it lives in such an awkward place in Windows is it's not actually designed as an override. The concept of a hosts.txt is what we actually

had before DNS, like going back 30 years ago. People would share around their hosts.txt and update them. So that's just the underlying infrastructure there. Going back to that geoload balancer concept, there's an interesting concept around the idea of sort of glue records. So in a lot of HTTP 1.1 scenarios,

we're going and using things like static domains, resource domains, CDNs, all these other things. And often people will talk about DNS being a little bit of an expensive hop at the start, because somebody hits your website and then they have to resolve all these other domains. One of the, if we just go back and run that outlook

scenario, when I go and run one of these queries, you'll notice that every time we return a C name, it's returning back kind of the next part. So I only asked for one thing, and the DNS server came back and gave me a bunch of responses in one hit. So an interesting trick we can do here, if you really want to play with micro performance,

is actually being able to go and take the idea of when somebody asks for www dot, I'll just zoom that up. No, I won't zoom that up. Windows 10's awesome, yeah? Here we go. www.mysite.com.

Instead of going and returning that as an A record of some IP, one of the things you can do is say this is a C name to CDN dot my site, oh sorry, not CDN, more so like static dot my site dot com, and then static dot my site dot com can then go and actually return the underlying A record, because often we see environments where people have

multiple domain names in order to get lots of local connections pointing into the same load balance, which then goes and divvies up the traffic across a farm or something. Obviously that doesn't work if they return to multiple places, but it's kind of a cool trick of how you can force extra DNS records down into somebody's cache locally when they query something. Talking about caching, PowerShell we can run

get DNS cache, clear DNS cache. There's one which I really like though, which is developers dot google dot com slash speed slash public DNS slash cache, and you can wander up here, you can whack in a domain name and you can hit clear and you dump Google's DNS cache for it,

which gives you a pretty wide swath of the internet when you're trying to update various records as well. You have to tick the little I'm not a robot button though, otherwise you'd walk up with whatever domain name you want. All right, so getting out of DNS land, one of the first things we needed for this to work

was that we needed to know some DNS servers that we could go and query. So I've been querying up to the name servers on the local network here and they go out. The underlying protocol that does that is DHCP. A lot of us usually see that as just enable automatic configuration on my network. In Wireshark that's known as bootp

because it was originally the bootstrap protocol. There's also some really interesting information that can come down in DHCP. Typically we see just things like here's your IP address, here's your name server and so on. There's some really interesting extensions to it around things like VoIP where DHCP servers are starting to return location information.

As in civic location, what's the address that you're currently at so that the phone, when you pick it up and dial 911, knows where to route people to to come and help you. So interesting little use of that protocol. Let's get into IPv4 and ARP quickly. So I'm gonna run get net IP configuration

and this is gonna go and tell me how my adapters are configured. So this is the equivalent of ipconfig or if I want the equivalent of ipconfig slash all, I can go and say detailed and I get bucket loads of information. Now there's lots and lots of net commandlets

here in PowerShell. Just as a quick one, if I go and run get command noun, say give me everything that ends in net, there's 332 of these commandlets. So you can find pretty much anything you wanna get through PowerShell. So going back and looking at this though, in my config here, I have my IP address

and I have my default gateway. Now, how many people are familiar with gateways and subnet masks? Sweet, we'll skip subnet masks then. What's really cool though, is how they actually get calculated at high speed on the internet.

If I extend my display, whoops. BGP is the protocol we use to go and publish out

how do we find different places, we need to evaluate subnet masks, that's basically a really big problem. There's 600,000 of these things active as of the last time I took a snapshot back in August. So how do we query them quickly? Typically, we think we wanna query something, we'll put it in RAM and it's gonna be really quick, but that's not gonna work at the core routers of the internet. There's too many of them, too much to go and query

and we have to loop through each one or something. And the random part there is the problem. So there's actually a really interesting type of memory chip that gets used in routers, which is known as content addressable memory. So the way this memory actually works is you go across and you say, here's the piece of content I want, tell me what address that it lives at.

And then they have this concept of TCAM, ternary content addressable memory. So it works in three states. So the way these bits of memory actually work is if we assume subnet mask, subnet masking for a second and how that works with IP address, subnet mask and we work out the subnet, we store a whole bunch of subnet information,

but they have three states in this memory. They have one, zero and question mark, represented as an X. So what they're able to do is they go up to this piece of memory chip and you go in and say, I'm looking for how do I get to this IP address? Now I'm obviously shortening half little bytes here, but imagine you're walking with a whole IP address.

And what the TCAM chip actually returns is one and three as the location that it exists in of where it matches. So it's saying that if I look for 1001, that's available at slots one and three. So I can go and then look at those routes and evaluate it. It's very specialist type of memory chip that's only used in these routers. I thought it was pretty cool.

This is what one of them looks like. It's a pretty amazing chip. They do 360 million searches per second per table hosted in the thing. This one has 40 gigabits per second of transfer rate. So that's just evaluating routes at 40 gigabits per second. They have fairly long keys. That thing there for a sense of size is 27 millimeters square.

And it's basically only used in this type of routing. So I think it should probably come with a Genesis logo, because it's kind of like how that Renaissance thing looks. One of the interesting problems this causes, well, one of the problems caused by these chips is they're incredibly fast, but that thing only stores 250 megabits of information,

not even 25 megabytes. In this day and age, 25 megabits still. It's about 350 US dollars a chip when I looked them up. So we can't put that in too many different network devices, otherwise they're gonna get expensive. And we have a growing set of BGP routes around the internet. And there's been a couple of magic transition points that we've hit along the way.

There was one that we hit, so that's 1994 to now. And it was a point, I can't remember the exact date we hit, but it was back in 2014 where we crossed a threshold. And they basically, there's a whole bunch of people in the background that we don't think about running around upgrading all these core routers of the internet, because they were running out of memory storage space

in these TCAM modules, all that underlying infrastructure. So in terms of how do we get an IP address onto our machine and how do we find out, so we've got an IP address through DHCP, how do we find other machines? That protocol is address resolution protocol.

So if I just run a Wireshark filter for ARP, it's amazing how simple this stuff is and how few bytes they are. There's basically a series of messages that run out and I've got my machine here doing a broadcast and it's just going, hey, who's got 192.168.0.1? Tell me, tell me, tell me. And these just go out as multicast packets.

And then once I've got an IP address, my machine also goes out and yells, hey, has anybody got this IP? And then when it gets a response to that, that's when the world falls apart and you get a little pop-up going, IP address conflict, something went wrong. And then at the end of that, people come back and respond with what have I got?

And all of this table is available to us if I go and run get net adapter. I can find out where my adapter is, there's my Wi-Fi, interface index three. And then I can go and ask for get net neighbor of the other machines on the network around me. Ah, sorry.

That's the fun of swapping between slides and code. Right, so I can run get net adapter and I can see my Wi-Fi adapter there, interface index three, and then I can go and run get net neighbor

and I can always go and filter that. It's also a really interesting way to see everything that's on the network around you. So mostly we don't need to think about ARP but because you have all this broadcast stuff happening around the network, you're able to see what other devices are there. Now where I've found this useful recently is actually doing a bit of IoT work. I had to take a Raspberry Pi, I connect it to my Wi-Fi at home

or plug it into the network, where is the damn thing? All right, you gotta go and plug a screen in, find the IP address or you gotta poke around in your router for that attached devices panel or something like that. I don't really like giving devices on my network a static IP. So what you can actually do here, you know what the MAC address is, it's basically printed on the bottom of everything, there's a little sticker on the chip usually

or it's not hard to work out and then you can go and look it up. Now even if I don't know the MAC address, that's the physical address of this block of stuff here, they all start with a prefix that indicates the vendor. So you can actually go and just look up by vendor type, show me all the Broadcom chips floating around in my network. Oh, that's probably my Raspberry Pi over there. This is exactly what tools

like the Windows IoT dashboard do. I don't have it installed at the moment, I've reset this machine. So it's a little tool that just says here's all the Win 10 IoT devices on your network. And that's all it's doing, it's just looking at your ARP table. Doesn't even go and actively query the network at all.

A little bit more time in routing. So as we go and work out how to get to all these devices we have these different routes that get it set up in our route table on our machine. This is my current route table. So this is going and saying, what I'll do is I'll actually filter this to

interface index three, which is my Wi-Fi adapter. So these are all the different ways that I can get out of my Wi-Fi. Now, a common scenario, development, we use lots of VMs, right? We use lots of VMs, we use often a number of VPNs, we often have people who are running on MacBook Pros

with Windows on top, all of that. And every time we connect to the VPN, now traffic gets really slow, or the VM gets confused, or all these sorts of things. These route tables are really, really easy to buck with. And I actually find that quite useful. So, unfortunately though, we need to go and elevate to do it.

So the first thing I'm gonna look for is say, find net route, how do I get to something like 66242415, oops. I can go and ask it, say, for this IP address, how do you plan on getting there? And it's gonna come back and tell me

the way I'm gonna get there, I've matched it to this route, I'm gonna go out over Wi-Fi, out over this interface, and off we go. So if I'm getting confused between, no, no, I don't want that to go over the VPN, or I don't want that to go over my Wi-Fi, I want it to go over this network here, we can move these things around. So, what I'm gonna do is just run an elevated PowerShell prompt,

random little side bit, the way I got that up so quickly under WinX, I have PowerShell here. If you're on Windows 10, right-click Settings, there's a little toggle here which says, replace command prompt with something useful. You just toggle it on, it's off by default. But that's really nice, because I just go WinX I and I get PowerShell, WinX A and I get PowerShell as admin.

So what I can do in here is I can say, I want to create a new net route, and when I go to somewhere like 66, 240, 241, imagine that's kind of a core part of my corporate network or something, and I've provided a range, I want you to go out over the Wi-Fi interface index three,

the next hop that I want you to go out is say 192.168.0.1, so what's my gateway address, and then I set the route metric for that, which is the priority, lower number wins, something like five. I go okay, it's gone and put that in,

and then if I go and say get net route, interface index three, that's now in my routing table and it's been ranked by priority there. Really easy little one line command where I'm actually able to go and say, I want that to go to the VM and I want that to go to the VPN. One of the projects I was on, the VPN we had to use

had a whole bunch of horrible config in it, they didn't expose, they had the firewall super locked down, so they didn't expose their internal DNS, they expected us to put stuff in our host file, the routing said send everything via default gateway, it was a nightmare. So what we actually did was we just had a script like this in task scheduler in Windows, you can configure a script to run based on an event

in the Windows event log. Every time you connect to a VPN, the RAS client, the remote access services client, writes an event. So what you do is look for that event number. We had it that every time we just went connect to VPN, it just cleaned up all of our config by running a script straight away afterwards. It's quite handy. Before I forget about this though,

I'm gonna go in and say remove net route. And yes, I do want to remove that.

Talking about VMs, one of the other useful things I find is sometimes we've got multiple VMs, we want to move things between them. Often the shared networking concept in VMs is actually really, really useless or quite painful to work with. It's a lot easier to take a VM and just bind it straight to a real network adapter, you get a better result.

One of the ways that I approach this, if I go right click start button or Windows X device manager, you can come in here and go add legacy hardware, you start to drift back into Windows seven days. And in here, if you look for network adapters, you actually get an option. There's a Microsoft one, which is the Microsoft KM test loopback adapter.

You can add as many of these as you want. And it just gives you more adapters on your machine. And then you can just set them with whatever IP address you want. You can bind VMs to it, you can drive all your routing between them. Becomes quite useful.

All right, getting on to, back down to sort of the physical layer and getting on to Wi-Fi for a second. We in our offices for the company where I work, we don't really have wired network connections at all in our offices.

A lot of people don't come to our offices, this is the first thing. We just do everything via Wi-Fi. Generally works pretty well. It's been a bit of a nightmare to actually operate that though. This is some interesting location analytics out of our Cisco kit that we use. And this is for our Melbourne office and it's in a nice kind of high-rise tower. We're up on level 17.

And this is showing the number of devices and probes going past our access points over a week. So on average, it's saying we see 756 devices pass by every day and we only have 54 that connect. Reasonable amount of load on the router, right? Or in the airspace. This is our Brisbane office, which is on level three of an intersection in the CBD.

On average, we only have 16 people in that office, but we have 4,500 devices that walk past filling up all the airwaves, going in and sending little probes. It's also a great way to actually, this goes and tracks people as they walk past, how long they dwell in our office, all sorts of freaky things that happen in every retail scenario you walk past.

Now, Wi-Fi is not a particularly intelligent protocol. The way it actually goes and schedules packets or tries to get them onto the wire is that I try to talk and if somebody else talks at the same time and we shout over the top of each other, I just wait for a few moments for a random period of time and then I just talk again. We talked over the top of each other.

So I wait a little random bit of time and then we talked over the top of each other. And that's all it does. It's just this dumb retry process, which is why it goes and kind of fails catastrophically when it does start to fail. What it also means from a scheduling perspective being very inefficient, if when we compare it to protocols like 4G or LTE,

the way they work in much, much higher density is they actually have a whole underlying, another channel where they just do packet management that says, your turn to talk, your turn to talk and so forth. Now, the interesting consideration in that from a mobile perspective, if we're sending assets down to mobile devices is that we're much, much better,

specifically in a 4G environment, but in most places, to send a single resource that's nice and chunky because it can just schedule all those packets in one hit and send them down. As soon as we get chatty on a mobile network, we take on a lot of overhead about having to, can I please have some air time? Wait, yes, you can. Now send your data. Okay, I'm done.

Request some more air time, off we go. There's a lot of work happens on the mobile device there. So batching requests becoming particularly important. What I lead into getting up to sort of TCP now is the wire protocol moving stuff backwards and forwards. And more talk about this from an experience

around some performance we had. I'm just gonna jump into, yes, we're on the screen. One of the things I'm gonna do first of all is just say get net TCP connection.

And this will show me everything on my machine that's currently connected. Where this is useful often I find is wondering about, we had a debugging case the other week where somebody was saying, I'm not getting information going out to serilog. How do I know if it's being sent? A lot of times things will dwell in here for a little while or they're in a thing called a close wait state. So this is a really easy way

before you start digging in logs just to see is my process talking out to something else at all. It's also an easy way to see what's listening on the port. I can go and say, show me everything that's in a state of listen. And I'll go and filter that down. Typically the display that comes out of this command I don't find terribly useful

in terms of the columns it chooses. So I say format table, give me all of the local properties, all of the remote properties and the owning process. And that gives me a much better view there. So that ft is just format table in PowerShell. The other one when it sometimes gets quite long,

say I'm not going and running state listen is if I just run get net TCP connection on its own is if I pop it out to ogv, open grid view, sorry, out grid view, you actually get a nice little pop up that comes up and you can go and do filtering and criteria and stuff. A lot of people don't think about firing up a UI out of PowerShell.

Now, when we have problems connecting to things, the typical tool that people fall back on is they go ping, right? I can jump in and I can say ping google.com. Yay, it pinged. Now that's going and using ICMP, which is its own protocol. It's not actually proving if I have a problem connecting to the web interface, it's not really telling me anything.

It's telling me I can get traffic to the box, but it's not telling me that that port's open at all or anything like that. First of all, getting up into PowerShell land, getting away from ping.exe, we've got test connection, which is the PowerShell equivalent. It gives us that back as nice objects. But the next thing that we can do is actually use test net connection.

Now this goes out, tells me a little bit more information. It tells me this is the IP I went out on. I went out over wifi. This is the IP I went to and I got it back. What we can also do though, is we can say something like common TCP port, HTTP, and now it's changed. Instead of just ping succeeded, they haven't even bothered running a ping. They've just gone and done an actual TCP connection and said, yep, I can successfully connect to that port.

So this is particularly useful. You don't even have to remember port numbers for things like RDP. Now that's gonna go on fail. Funnily enough, Google.com does not expose RDP to the internet. They've seen Troy Hunt's talk. Now we had a bunch of assumptions here. You'll see how long this is taking to respond. It doesn't fail instantly.

It's just waiting, waiting. I'm going back a number of years here, but there was a particular kind of experience which has stuck with me for quite a while. So I want to tell the story of that. I'll swap the slides back correctly this time.

As Windows freaks out over display adapters. So the problem we had was we were delivering a major website out to a broad consumer base. And we had what we were referring to as the intermittent page delivery problem. You'd be browsing, browsing, browsing, click, all of a sudden nothing responds. And this was the distribution of our request times in milliseconds. Now the numbers on the side there are a bit small,

but this line here, kind of two thirds of the way up, that's 100,000 milliseconds. Probably a bit long for an average web request on an e-commerce site. And we just had a whole bunch of noise in the traffic here. Now the first thing that actually really started to get some clarity for us was when we went and moved that and just collapsed the time scale,

took time off, and just plotted all of those points on top of each other. We start to see some interesting clusters here. And this started to lead us down a path of understanding what was happening. I'll explain in a little bit why logging didn't work for us. The architecture that we had was we had a group of front end web servers that would talk to a load balancer,

which would then go across a group of search servers. The problem ended up being in the search servers. On each of those machines, on each of the search boxes, we ran two indexes. One that was active at any one time and the other one was getting updated. And then we would swap the indexes in and out as part of the kind of almost real-time updates. It was a constraint of the search platform we used.

And we ran each of those on separate ports. So you'd hit the load balancer, you'd ask for port 16100, you'd get one index. If that failed, you'd fall to 16101. And our code was written as, try connect. If it fails, fail over to the other one. And it should just fail over nice and quick and instantly. And it worked this way in development. It was great.

The problem that ended up happening, sorry, what was happening in development was we would go and send a SYN. Hi, I'd like to open a connection. And the one that was down would come straight back and go, nope, sorry, I'm not here. Actively cancel the connection. We'd fail over, we'd connect to the other one, and off we go and run.

The problem that we were having in production is somebody in their firewall configuration in the data center, which, of course, the data center was different. And this was going back a few years before cloud was a thing, had decided to go and block reset packets. So what this meant was that we would try to connect to the port, we'd wait. And TCP is designed for a very, very slow period.

It's designed from pre-56K modem days, go back further than that. So it waits three seconds. And then it goes, you know what? I'm not going to fail. I'm going to send another one. And then every time, it doubles the retrainable. And what was happening in that period for us was that we had our second index, or the index we were trying to connect to was down. But by the time we got to the end of this,

the index had come back up. But it'd only take about 30 seconds for it to toggle over. So the index had come up. And eventually, it'd respond and go, oh, I'm here now. So we never performed failover in production at all. Because we had an assumption that we would get an active reset back from it. And we deferred that assumption just down to the network. We weren't actively telling the system

to switch between nodes in the cluster. And we'd never tested in production that we were getting that failover capability. Now, the path that we got led down in this initially was we went and we eliminated the concept of search being slow from our problem. Because we pulled the logs on the search server, and it said that everything was insanely fast and there was no problems. We didn't go and test that from the side of the web

server as effectively. Or when we did try to connect to it, we had a perfectly fine experience. The other problem we had was when we were looking at this chart here, we were modeling out all of the request times in IIS. And they include the time to last byte. So it includes the download time to the destination device. And that went and muddied up a lot of our information.

So we got a lot better debugging data when we started to measure time to first byte of anywhere we were doing web performance. Just trying to measure how quick are we at delivering the first byte. And then everything after that we can't control. We also had a problem once where we had a,

if you left the web page open, it automatically logged you out. So you didn't leave all this commerce information up on screen. And we deployed this and it was fine for two or three weeks and nobody saw our problem. And then we got this really angry phone call in the call center. Something in our deployment hadn't dealt with some cookie state properly. And for most people, they just reloaded the page and off they went. Unfortunately, somebody had gone on holidays

and they'd left the web page open and they had cable internet. And internet in Australia is very expensive. And they had decent internet. And we pushed something like 150 gigabytes of infinite redirect loops down to them as they continued to reload the login page. And it cost them something like four or $500

when they got back from holidays. And they weren't very happy. So we had to give them a very nice gift voucher and pay for their internet bill for them. So that was a, what I now see every time I look at this chart is that distribution of three, nine, 21 stepping up

as you increment those retry times. So it starts to be a common distribution anytime we have networking problems. All right.

So TLS and SSL are obviously all the rage at the moment. How many people are familiar with Let's Encrypt? Few hands, cool, fair number. So trying to get SSL rolling out in more places. There is a group called Let's Encrypt where they're going and providing basically free,

or they are free SSL certificates. And it's all designed around heavy automation. They rotate quickly. So we're starting to see more and more services turn them on. One of the interesting things I saw the other day is my blog is hosted over on WordPress. I just have a C name from blog.tatham.com.au

to tatham.wordpress.com. And I have never set up SSL for that. Bad me. But also that would have previously involved me going and getting an SSL certificate, handing it over to WordPress, them holding the private key. One of the interesting things I noticed the other day is all of a sudden there is actually a lock icon

on my blog, and I did not go and do that. What they're doing is they're actually using Let's Encrypt under the covers. And what I found really interesting about this, it's a fully valid certificate because they've been able to prove domain control because I've delegated my C name to them. You'll see here it's issued to TLS.automatic. Because they don't want to have lots and lots

of different certificates, they do fairly high density hosting. The interesting problem, this is actually currently, well maybe it's not a problem, it's just interesting when I first reacted to it. Current leakage is when you look in the subject alternate names, I share a certificate with all of these other people that are actually running on that same endpoint as me. And it goes and tells me all of their host names.

So SSL is becoming an interesting place on the internet. But it is certainly getting a lot, lot easier to do as people can go and dynamically actually provision certificates based on just proving ownership of the domain name. And that comes back to DNS being such a critical part of actually proving ownership of something at the same time

because if you own DNS, you can own verification for it and prove to somebody that you're in control. And every time you put a C name record in, you're delegating responsibility to somebody else. So WordPress was actually able to go and issue an SSL certificate on my behalf without me even realizing because I delegated part of my domain name to them via a C name. Kind of cool, also kind of scary.

What it does mean is also that, or what it does highlight is just how easy it is to actually go and swap out or to issue various SSL certificates along the way. Now when we're using things like Fiddler, Fiddler can inject its own certificates and you can say that those are trusted on the device

and it will meet all of the trust requirements in the chain. And then I can convince tools like Outlook to keep running, but I can go and look at the traffic sitting in the middle of them. One of the common things in mobile scenarios that a lot of people do is actually start to pin the certificate. So they take a known thumbprint of what the certificate should be and embed that in the mobile app.

And that's designed to make it harder for people to go and inspect the traffic of that mobile app talking back to your services because if somebody goes and swaps out the SSL in the middle and then says that their device trusts it, you can still have the app go, no, no, that's not my legitimate one. So certificate pinning becomes quite interesting. Also a core concept of HSTS,

but that's not gonna fit today. HTTP, no explanation required, but I do wanna show a cool tool. So another little tool that some of us built was this website, httpstat.us, or just HTTP status. And this is designed as a debugging tool

to make it easier to go and respond with different codes. So all we actually do is, if you go and request httpstat.us slash 401, you get a 401 comes back and you get an authentication challenge. Or you go 403, you get a 403 forbidden and so on.

So useful little debugging tool. One of the things that we, there's my 401 challenge. One of the ones that we do support of course is 418. I'm a teapot. How many people are familiar with I'm a teapot? Cool. So the RFC for I'm a teapot is actually very useful from a testing perspective of proving,

so this was the first of April one year, April Fools. There's an RFC which is the hypertext coffee pot control protocol. And it has an error code that says, I'm a teapot, sorry, not gonna help you. But it is actually an interesting one in that it pops up in a lot of frameworks as a test scenario for testing just how generically that framework can handle every different type of response code.

Or is it hard coded to a particular set? So I kind of like it from that perspective there. Of course, HTTP is starting to go away now. We're seeing HTTP two pop up in a lot of places that fundamentally changes a lot of the semantics to how it moves over the wire becomes a lot more efficient. There was a talk yesterday on HTTP two

from developers perspective, which if you didn't see it, I encourage you to go and grab the video and check that out. Cause it starts to affect how we treat some of the web performance decisions that we've made over the years. So I guess the summary I wanna put around all this

is hopefully you've got a number of useful little techniques and small tidbits along the way there. It wasn't gonna be a networking deep dive at all. But if you haven't really played with some of these different parts of some of these protocols before, none of it's particularly complex. The end of the day, it's all small bits of ASCII, tiny little bits of binary math

that somebody came up with 30 years ago. It's all available and documented via RFCs. And it's just peeling back those layers and debugging them. Wireshark doesn't need to be a scary tool at all. All the RFCs are available. I've always taken an interest in going and peeling back those layers and playing with them. I do read a surprising number of RFCs.

But I wanna get away from that mindset of one of the kind of quotes I showed right at the start of the whole nobody understands it so we just turn it off and on again. And where I see people going and just resetting their network adapter or doing IP config renew or saying all the servers down cause I can't ping it. I want people to start to really peel that apart

a little bit further and go through that kind of check process. So when something's not working, it's can I connect to the port is the first question. Do I have a route to it and so forth. So thank you for coming along. Two interesting kind of references. One there that GitHub link, github.com slash Alex slash what happens when.

It was kind of the inspiration for this, but it is insane and worth a read. So he put up a markdown file, just one markdown file in a git repo. And it was the basic idea of we do DNS and then this happens, then this happens. And people have gone through and sent more and more and more pull requests.

So that repo now gets to the point of when you depress the G key and interrupt is sent via the USB protocol, and then it hits this buffer in Windows. But if you're on OS X, then this happens and so forth. And it's a pretty cool read. The other one there that is a still drinking.org slash programming sucks is just a fun little summary of wow, isn't it so complex.

And when people go and say I do a tough job and you're a developer, it's easy. You sit at a desk every day. It's a really good summary of I guess all of the different things that we go through and just how challenging programming is. So thank you for coming along.