All right, I guess we'll get started.

Can you guys hear me okay? And everyone can hear me fine. Okay. All right, cool. Oh, by the way, before I get started, this list is hard to see. This is a zone that I created manually because that predates authors.bind.

But those are the first guys that started bind as student projects. Let's see, three or four of them for student projects, than a few other developers that came right after them. But that's a different story.

All right, cool. So today I'm going to be talking about DNSSEC, DNS security protocols. I am not going to be talking about TSIG or SIG0, but I'm going to be talking

about assigned zones and related to that. I have a few questions before I get started. I'm curious, how many of you have deployed DNSSEC? Come on, any hands? All right, because my other questions are like,

have you done an authoritative server? Have you, oh, I guess another question. Has anybody enabled their validator to start verification? Okay, all right. A couple of people. All right, that's good, that's good. And we'll see later on that it's pretty easy to get started with. And also it's very difficult to go with too.

So just for fun, this is DNSSEC RFCs only. So this is not the DNS RFCs, but just DNSSEC. That's pretty thick, over 300 pages. And so it's, and there's maybe another 100 pages to add to this because this is a couple of years old.

Well, I work for ISC. I work on the bind 10 project. I'm a release engineer, technical writer. I maintain our build and test farm. Before that, I was a support engineer for ISC for one year. This predated most of DNSSEC use, but we did have some customers

that were testing DNSSEC in their labs. And so I did a lot of troubleshooting work. Earlier today, I went to a presentation about logging, about using the same log messages in different places in the code. And, you know, it's a nightmare to track down. Well, we have this exact same problem in bind nine. You know, you have a form error, which might get turned

into a serve fail, or you have a serve fail. We had like over 50 places with the exact same log message. And from a user point of view, there's nothing to do about it. And so the administrator, then again, there was nothing they could do about it either. So that was a little bit of work I did for bind nine is just adding unique logging. And we'll get into that in a little bit.

And also, I'm a NetBSD developer, and I'm on the board. So a few things that we won't cover also. I'm not going to cover dynamic updates with signing. And hopefully, you guys have some basic DNS experience.

So how many of you have adminned a bind server? Okay. What about other DNS servers? Okay. So we basically got almost everybody hit here. So that's good. In my examples, I'm going to be using bind nine. But there are other DNS-capable resolvers

and also authoritative servers. All right, first, we'll talk about some of the threats to DNS, and we'll talk about later on how these apply to signing, whether they help or not. So we have packet interception, man-in-the-middle attacks, people on your wire

who might be able to modify questions or responses, eavesdropping to spoof responses, and we'll talk about that in a moment, ID guessing and query prediction, name-chaining, where they return part of an answer,

like they give you part of an answer, that will point you to their own authoritative name server. And then once you start querying their authoritative name server, they've sort of got you caught for other things. ISPs might accidentally or purposely provide different answers. And we know this has been an ongoing debate where ISPs purposely provide their own answers.

And yeah, even the ISP in my area of Verizon, they do that. So okay, I don't use their name servers, but that's their choice. Denial of service attacks, signing doesn't help that, and signing actually can add to the problem.

And we'll talk about that briefly later. Another type of attack is not modifying data, but it's actually removing data. And an easy example here is if you remove the MX records, most mail servers will drop back to send just to the regular address record. And so what if that system is already controlled by the attacker?

And so that's an example. A few things about DNS components. The query information has a source address in the port of the query, and that's from the recursive server. Or it might not even be a recursive server.

It might be a stub type client. It has a destination address, you know, port 53 that's also asking. Then the identifier to keep everything unique is a query ID, but it's only 16 bits. Then there's a query name, the class. In our case, we just use internet class and the type, whether it's an address record,

if you're asking for an MX record, SSH, FP, things like that. And the dig command, it's pretty handy. It shows you all these things except the source. What's interesting is the response that comes back, it needs to provide the same original information, and that's part of the pattern or the way

to synchronize all the responses it receives. So it has original query ID, class name, and the type. And they also should come back from the same address and ports that were originally used. So now I'm going to give you a few examples how you can break it yourself.

And I have used some tools and some scripts to poison caches, and this gives you some ideas. So you need to know the query ID and the source port. And the other information is generally easy to control or know, and you can control it by flooding your own series of queries to it.

So now you know which questions it's going to ask, so you can control a lot of it. It is a race, but again, in many ways, it is a technique. And of course, it's obvious. If you have a packet sniffer somewhere along there, it's really easy for you. Brute force or random chance. So when I say brute force, if you flood responses, you know, tens of thousands,

hundreds of thousands of responses, the odds are you're going to have a match. And your whole goal to poison it is to beat the real response back to the resolver. And then once it's in the resolver, let's say you're a big ISP

and you have a million customers, and then from then on until the time to live expires, they're feeding your answers to all their customers. A few years ago, maybe some of you remember, there was a lot of buying news, but it was more than just buying. It was DNS news, Kaminsky techniques for predicting and forging a DNS response.

And he showed ways where he could spoof a cache in like 30 seconds. And so just the very basics of it, you get the cache to ask a question. You get the resolver to ask a question. And it might be as simple as doing SMTP connection over and over and over again

because most SMTP servers might respond, and you can give it, you know, some information, SMTP, and then you've started the DNS queries. Based on the birthday paradox, you realized it didn't matter if you went through a whole series of query IDs. It didn't matter if you went through a whole series of source ports.

Just keep on flooding the same ones over and over again, a small pool. And he was able to do it in 30 seconds. And instead of providing just an answer, he gave him a question that would ask for further information. So he'd provide something that delegated to his own authoritative server.

And then he would just give it a huge time to live. And yeah, a lot of name servers have a cap of the maximum time to live. Well, let's say the cap is a week, and then they've got you cached for a week. I guess I don't cover it in other slides, but what are some goals with this? Well, they could point you to a spoof of your bank or a spoof of your iTunes.

Anything that you log in, and yeah, it might look the same. It might redirect you to the real one, but they might be able to capture some information and potentially use that to steal from you or all types of things.

The DNS changer worm. This was in the news starting, I think, in November. The FBI, working with some other governments, they caught, I think, six out of eight people that were involved with this. They had over 100 name servers that they had traffic pointed to.

In this case, over four million systems were pointed to just their name servers. It was via some worms on Windows boxes. And also, they targeted cheap wireless routers that actually millions of people have in the United States and redirected them to rogue websites. And they know that they earned at least $14 million just

by pointing them to sites with advertising. It's $14 million. But in some cases, they will also be able to steal money other ways. That's also called Operation Ghost Click. What's interesting about it now? Well, ISC runs the name servers now. And slowly, other groups are trying to convince all of the ISPs to clean up, you know.

And it's just, it's all their customers. And so, I think at least 500,000 compromised systems are still using it. Our turn off was in May. But now, we got a court order to keep it running until, I think, July. It's sort of interesting.

Maybe some of you might know somebody that has a Windows box that's pointing to the wrong name servers. I'm sorry. Where did it come from?

Oh, I don't know. Not us. Yeah. The alternative is, is we don't provide any name service. And four million computers would actually think that the internet is down. Yeah. It's probably true because that's four million bad Windows systems.

Yeah. We don't know if DNS security would help in many cases. But some browsers and clients have DNSSEC validation built into them. And so, if they chose to use a DNS capable client, then they could see if they're going to their own website and if it's been modified or not.

The very quick short description, DNSSEC offers a standard DNS and it proves that the data is not modified and came from the official source. And we'll get into detail. Yeah. Just before we get into setting up DNSSEC, we'll talk about some other use cases.

And some of these are pretty interesting. DANE, or DNS-based authentication of named entities. Some clients, and there has been patches for Chromium already, they can use DNS to fetch initial keying information. And so, this is something to replace certificate authorities.

It's more lightweight than the regular HTTPS key exchange. And so, it's an alternative way using DNS and it's pretty interesting. Also, another thing is using it for SMTPs, or start TLS, and it also stops it from being compromised and there is known attacks.

Another thing is for BGP route origin verification. It announces which routes that it prefers and announces these versus via DNS and that's been signed with DNSSEC. And so, that's called Rover. Do any of you use a secure shell fingerprints in a DNS record?

OK. And so, when you log in via SSH to a new box, it doesn't know who you are. It'll show you your fingerprint. This is a way to automate or verify that you're hitting the box that you really wanted to log into. And it's not just logging in. You might be, you know, remotely running commands or all types of things on that system.

And so, you could do it via that. Also, the GNU PG, and maybe PGP does also, but it can retrieve certificates and public key information via DNS. And so, you could have that via DNSSEC signed.

And then, some people use it to make sure that they're hitting their correct packages repos. And so, I've heard of one distribution of Linux. I think it's called CERO, C-E-R-O, that uses that. And so, those are some interesting cases.

I have not heard of progress on that. But yeah, I've heard of that though, yes. And so, hopefully, you guys can invent some more ideas and more uses.

All right. So, we'll get into this a little bit more. So, a DNSSEC actually adds new resource record types. And it also adds some message header bits. And these are used to verify that the data matches what the zone administrator actually put into the zone in the first place, and to make sure it hasn't been altered in transit.

It does not provide a secure tunnel. In the DNS world, you could use TSIG. There's also a DNS crypt, a DNS curve. There's some other ideas for tunneling DNS traffic. And it also does not hide the data. So, it's not encrypting the data whatsoever.

It's designed with backwards compatibility in mind, and I'd say not 100%, but very close. And so, if you do not want to be a validator, then you could just continue to use DNS the same, even if your upstream provides DNSSEC records.

Basically, when you ask them, they won't send them to you. And it'll be fine. And also, if the destination content, I had mentioned about pointing you to a spoofed website.

A lot of people may not even check if the SSL certificate is in place. They may not care if it's HTTPS or HTTP general users. But if you wanted to make sure that type of content, there's other technologies. The first thing about DNSSEC is it requires eDNS0.

Now, these are extensions to DNS. And this allows you to have new flags, new return codes, new label types. And it also provides ways to announce how many bytes you could receive and send. And this is important because historically, DNS is usually small packets,

usually less than 512, and many firewalls or router operators kept that in mind. Luckily, eDNS has been around for over 10 years. Most servers support it, and all the root servers support it,

and most of the infrastructure supports it. But I have seen over the last year some big customers roll out DNSSEC live, and then a day later, they have to turn it off because something between their production environment and someplace else on the internet, some routers or something are squashing their DNSSEC traffic. And because they had validators enabled, it was incomplete.

And so it was actually breaking their DNS. We'll talk about examples of that later. So in your case, make sure your firewalls don't block these packets, larger than 512 bytes. And yes, there are some broken servers or network devices that have problems.

I haven't noticed anything here on campus, but many times I go to different hotels and motels, and I see it often. But it's not just DNSSEC, it's DNS in general. They do all types of weird things, and a lot of DNS might not work. We'll see this a little bit later, but one of the new header bits is DNSSEC OK.

Basically, this means that you're a validator, and yes, I'll understand DNSSEC-related traffic coming back to me. I'll understand those. It doesn't mean I'm going to do anything with it. It's just saying I'm OK to receive it. Just to back up a little bit, just in case some of you have seen documentation for

or have encountered the SIG record, key record, and next record, those are old and outdated, they're obsolete. Those are not used. And so the new standards are 2005 and on, and those are the ones I'll show you in a few minutes.

Interesting to know also on here is some of the threats were documented starting maybe 1989 and then in the early 90s. So these threats have been documented and announced in different places like this over the years, and it wasn't until Kaminsky's showed some different ways

to do it that it became really big news. Oh, yeah, so one of his ways to limit it is to make sure that the source port range was large, and so that's one of the ways to alleviate the problem. The problem still exists. So if you are in a network with very high-speed bandwidth,

you could still do this same attacks to spoof. The spoofing is not fixed, and so you need a solution to work around it such as DNSSEC. And yeah, it's because everyone's using DNS, and it's hard to change a protocol that the entire world uses, so we're just extending it.

Very briefly, it uses public key crypto. There's a private part, which you keep on your own systems, and you might keep it in HSM. You use that to sign, create your signatures. Then there's a public key that you share to other people, and they're using that just for the verification parts, and we'll see that upcoming.

The new resource record types are RSig, DNskey, DS, Nsec, and Nsec3. This lecture, I will not cover Nsec3 at all. Just mention it briefly, and Nsec, just in a little bit. And the rest of those, we'll get into some detail.

All right, so the first part is a validator. When it sends its query, it's going to use the DO bit, and it's going to say, okay, I understand DNSSEC, go ahead and send me what you have, if they have something. On the server side, they're enabled with DNSSEC. Their zones already have this information. So it'll return a signature that's already been set up ahead of time.

So generally, I've never seen an implementation that does it on the fly, so it's generally done ahead of time, but it'll provide a signature that goes along with the answer. Now, if some of you know the CNAME, you're not supposed to have other information

at the same point as it. That's one thing that's been modified by the specification. You can have a signature with a CNAME, for example. The signature is a hash of the entire record set, so just not an individual record, but entire set, and it's encrypted. The hash is encrypted using a zone key, and so that's your signature, a digital signature.

Oh, I already mentioned it, but if you have four NS records at the same label, same class and type, you'll just have one RRSig for that. And again, it's automatically sent back an answer.

And so we'll look at some examples here. You can see the first thing here is, well, we'll talk about it later, about the authenticated data a bit, but the first thing you should notice here is answers too, and we're using EDNS0. You can see this in this dig output.

You can see the flags was passed back, DO, you can see the question. Now we're getting to the part where there's two answers. There is an address record and a corresponding signature. And also, your authority section, you might have other sections of information that are given back to you.

They might have also corresponding signatures to that records being provided. Notice in this case, it was signed information. It's still less than 512 bytes. Yes? So in this scenario, they have to give you back all the data that went into that signature other than having to be able to verify it, right?

Yes. We'll see that in detail. And so very quickly, we'll go over the different parts of just the RRSig. So the RRSig identifies the record type, the algorithm used, the number of labels,

whether it's www.1.2.3, you know, it has a number of labels. The original time to live, because these are regular DNS records, they have their own time to lives. And so it's nice to see what the initial time to live was, just in case that was modified also.

You have an expiration time of the signatures, and you have an inception time when it should begin. You have a key tag, also called a key ID. This is used for administrator or for maintenance purposes, just so you could look to compare things visually without having a tool to validate or verify for you.

You should not rely on it. It's just for your own sake, just to match things up. And we'll see examples. The signer's name, the name of the zone that signed this record, and then the digital signature. And so what I just explained, but you can see that information there, you can see the inception time, expiration time.

If it's a shorter value, then it would be an epoch time. One thing to note is the expiration is before the inception time when you read it. And that's hit a lot of people. They look at it and go, oh, something's wrong here.

It's because they're reading it out of order. And this example here, this says labels three. It's because it's three labels, www.isc.org. One thing to note, this lecture won't cover wildcards, but wildcards is also covered with this.

You could sign wildcards. All right, so a DNS key, and this is generated with the key gen tool. It is for the zone and not for the individual labels. Like you wouldn't create a different key for every single record or record set.

So again, the signature is generated with the secret key, the private key, and then the DNS key is the public key, and that's used to validate that signature. There's also a hash of the DNS key, and that's called the DS key.

And we'll look at that in a little bit. There's some flags, 257 and 256, the key signing key and the zone signing key. From the validator's point of view, this does not matter. This is only for the convenience of the tools maintaining your zones.

It's for convenience for the administrators to set it up. And we'll see that in a little bit. So we'll look at some examples here. So we want to look at the key tag here, 44016, and it's right here in the signature.

And we'll go a little bit farther. I'm doing another dig, and this time I'm asking for the DNS key record. And it came back with a couple DNS keys, and you can see this one matches up.

And so in this case, it matches up. That would be the public key used to validate the previous signature. We'll talk about the other key ID in a little bit. What's important to know is everything's signed.

It's just not your original questions, but there's other information, too. The keys themselves are signed. You know, we need every single part of this signed. And so the public keys are signed, too, and you can see that there also. And just to talk about this example here is. Now, the signature, DNS key, the signature identifies which one was signed,

which is the DNS key above. Oh, the original time to live, yeah, 3600. You can see it's been in the cache for a certain amount of time. It's already counting down.

All right, so the DS record is used to start building a chain of trust between a parent zone and your own zone. And the parent might be, you know, some third party, and usually, it is. It contains a hash of what's called a key signing key. It's one of these DNS set keys.

Now, the DS record is created by the sign zone tool, and we'll see that. The DS itself is also signed. That means it has its own signature hosted in the parent zone. Then you provide it to the parent zone. And this is an important point. It's returned with delegation records.

So when you ask a parent for information, it says, well, okay, I don't have this zone. Go ask someplace else, you know, by giving it NS records. At the same time, it'll also provide you the DS records and all the signatures for this information. Now, the NS records themselves are not signed.

The delegation is not signed, but the DS records are. And we'll look at TCP dump, and hopefully, this is fine for you guys. And I had some other examples a lot with TCP dump also. So we see a address request to a parent.

And so it's a normal request you would see, and this has happened millions of times every second. But in this case, the DO bit was set, and so in return, the answer, well, here's the delegations, go ask them instead, NS, go ask them instead. And by the way, there's a DS record also associated with it.

Here's another example. This is just a dump with a cache on bind nine. And this is from a simple query. It had the NS record, and then it showed the additional, the DS. And so those are sent automatically. You could also query for them and ask for them, and a validator would do that.

Yeah, I'll skip through that. The insect record. This is to indicate that a resource record does not exist, or to indicate if a certain individual record type does not exist. The first thing that's important with a DNS signed zone is the entire zone is sorted

into canonical order, so it's all sorted in order. So then it can create a, these new records, insect records, and create an entire chain representing every single record in your zone. And how it works is one label will refer to the next label all in order.

And so by asking for something, it can simply see, well, is it between this label, or I'm sorry, in between this insect and that insect, it will know that there's a gap or something missing in the zone. Also, the insect record identifies the types, and so then you can see, oh, no, the MX record is gone, someone has done something, for example.

So since there is a chain, some people are concerned that, oh, okay, well, I don't want anybody to know what's in my zone. Well, it's so easy. Now there's a chain of insect records, they can crawl it, and so they can know everything in your zone.

And it's very simple. And so the alternative technology for that is insect3, which creates a hash instead of using a regular label name, but hashes for each one. And so, yeah, you can crawl it also, but you don't know the original names anymore. Okay. Oh, well, by default, it's used generally by default.

So I think everybody would just use insect.

Most people don't care if your zone information is crawlable. Yeah, it's already in the public, but some people do care. The people that do care, like they turn off their zone transfers, for example, and only allow it from certain systems. And so, whatever your own desire is.

All right. So the authoritative, I'm sorry, some new additional message header bits are AD for authenticated data, the validating resolver, it'll check to make sure a chain exists.

We'll see detailed examples of this shortly. And if it could verify all this, it could set an AD bit and return it back to you. You know the common bit, like if you're asking an authoritative server, it'll send you an AA bit, so now it'll, if you're asking a recursive server, you get an RD bit. Well, here's another one, the AD bit.

Now, let's say you want to do troubleshooting or testing, or simply it doesn't work because there's a problem in it. You could turn off the checking. So in your original query, you could do DIG plus CD, and basically it means turn off that checking, just send me the information.

Now, it would be good advice if you ignored the AD bit. That's because you're trusting whoever you asked, and what's in between you and them. There's no, at this time, no protocol or any way to secure the AD bit itself, unless you use a different tunneling or other mechanism.

So you can do the validation on your own. Or you could use a TSIG between your stub and your resolver, for example. Or if it's on your local system, between your stub and your resolver there, that would be fine. A very quick summary, again, of the keys.

The keys are used to prove the integrity of the DNS data. Again, the private key is used to encrypt. The private key is not shared via DNS. But the signature created with it is shared via DNS with our SIG.

The public key used to verify that is the DNS key. It's used to decrypt it. There's a hash of that key, which is stored in the DS record, which is stored in your parent. These two are associated, so the DNS key and the DS are associated. They can verify that they're, they can hash the DNS key and verify it's the same one.

And they have signatures also. So this is a chain which begins the chain of trust, or the authentication chain. Now your validating resolver needs a starting point. Because depending on how far you want this chain, the alternative is what's called islands of trust,

which I'm not covering, but you need a starting point. And we'll show an example of that a little bit later. Well, there's the starting point or the trust anchor is a DNS key or a DS record. You retrieve it securely from a trusted source. And let's say you use Unbound.

You trust their source code. So you will trust them to provide you a starting point or this trust anchor with their code. I mean, if you don't trust them enough to use the key that they provide, then why use their code in the first place? And then we'll say the same thing about bind. If there's a broken chain, and in some cases, if there's a broken chain

where there is a DS record in your parent, but the child does not have DNSSEC or it's wrong, then you've just made everything underneath it invisible, serve fails.

And so if the chain's broken high, let's say it's broken at a level of like .com, or .hb.com, you know, all the labs under hb.com could disappear. The chain of trust must be from data, oh, oh, remember the chain of trust may be

from data on a caching name server. All of these are resource records. They have time to live. They're going to live in your caches for a while. And so it doesn't need to recreate this chain of trust each time. It has this information. And so they may not be getting it from authoritative server every single time. It's already cached. And again, the weakest link defines a strength.

All right, so our trust anchor is the DNS route itself. It's been signed since July 15th, 2010. There's more than 80 secure delegations provisioned in the route, for example, .org. But there's many, Brazil, and on and on and on.

Lots of ARPA is signed. For the registry and registrar, how you share the information to provision them in the route is by giving them a DS record. And so a lot of that is still in flux. There's not a consistent way to share this information.

If you need it, hurry it up by encouraging your parent to provide a mechanism for you to get this information to them. Because of it is manual every single time, let's say six hours or something like that. If you're in an emergency situation, you know,

manual way to recover is not fast enough. So we'll look at this trust chain example here. Oh, yeah, I want to talk about signed parent a little bit more. So I use joker.com, and I contact them a few times.

And yes, they were DNSSEC enabled, but they did not have any mechanism for me to share it with .org and things like that. The way I got into .org was via emails to .org's administrators. And so now I feel like I'm really out of luck because there's no way I can maintain it that way. Luckily, Joker recently enabled some new mechanisms, and so I'm going to figure

out Joker's APIs and see if I can work with those. Some of my coworkers use GoDaddy, and they're able to get their DS records into their parents via GoDaddy. And so, and there's others too. And if any of you know of any, go ahead and share them.

So we'll look at this entire chain starting from root. And so you can see it's root with the period. It has the DNS keys. The DNS keys are signed. This is the beginning. So n by nine or another validator, you have this trust anchor stored on your system.

But then it also does a real query. To make sure it gets it from the real source. Sorry. Just making sure I didn't miss something there.

Then the next part of the chain is .org. And it has a DS record. And notice the 21, or 21,366. So remember that key ID. And then if you ask them for their DNS key, well, we found they have a lot of keys.

And one of these keys matches it. And the reason there's a lot of keys is some of them might be old. They're not using them anymore. Or some of them might be planning for the future, which is good advice. If one of your keys gets attacked and it is the one that you use,

it would be nice to already have a DNS key already available so you could begin signing with it. This is important, again, is because, well, okay, 900 seconds is not very long. But with your time to live, if these things are cached, you need to know ahead of time, or you need to plan ahead of time.

Let's see where I'm at here. And so going down, further down the chain, it goes on and on. And so in this case, we have a DS with 12892.

And then we have DNS keys with also the same key ID, 12892. And so you're going to the child. How's that DNS key for that? Now this next slide, this is a bad example because the parent and the child are served from the exact same server. So when you ask it, it's not giving you the in-between hop.

It's not giving the in-between delegation. It's just going to give you the answer. And so my point is we have this signature. And its key that was used to sign it was 3947. But it is not associated with the parent that I asked of previously.

So in this case, you would figure this out by, we see this, the full name, it's four labels. This one is three labels. It says bind10.isc.org. That was the signer of it. So you can ask it for its DNS key.

And so you can get that intermediate hop. And the validator does all these steps for you. 20 minutes left, okay. All right. So the type of results you get from the validation. If it has this trust anchor, if it has a chain of trust, and everything's checked

with the signatures, and the DNS keys match to the DS and the parents, all the signatures are verified, then that's called verifiably secure. And it could be marked with authenticated data. It might be considered verifiably insecure. You have a reason.

You verify you have a reason for it. You have a trust anchor. You have a chain of trust. And then you have proof that you're not providing the DS. And I'm sorry I don't talk about that here. But if you have some way to prove it, one way is insect3. You could opt out things out of DNSSEC signing. We have some proof that proves that it doesn't exist using insect, for example.

Or if you simply don't have a chain of trust. Now in the case of the chain of trust is broken, and it's supposed to be there. And then you get what's called bogus. But bogus has so many different reasons. It could be expired signature, unsupported algorithms, some missing data.

Something's missing that insect says should have been there. It may be a mistake, corruption. It might be somebody spoofing it. It might be an attack. There's so many different reasons. Sadly, the only response we get in DNS is called a serve fail. So from an administrator's point of view, hard to troubleshoot.

From an end-user's point of view, let's say it's impossible. I mean, unless they have some tools that do it for them. And I already mentioned invisible subdomains, for example. At this time, there's no easy way for the resolver to report specific failure reason.

I believe there's some drafts to extend this. And so we'll be able to have a way to report the specific reasons. All right, so you want to set up DNSSEC. Luckily, Unbound are buying nine. They already are DNSSEC capable. And make sure your network and even upstreams or whatever,

they don't lose or squash the eDNS0 traffic. And remember, I mentioned it's 10 years, and most devices handle it. Also, you don't want them to squash other related traffic. Like I've seen many devices where I do a query for DNS key, and the device will lock up.

It just won't even respond. It doesn't know what to do. And so this is common. Also, there is a increase in traffic size. There's going to be a lot more requests and a lot larger responses. And why is there a lot more requests? If it's not cached already, it needs to build that trust chain.

For high volume DNS traffic, prepare for increased bandwidth needs. At this time, most people are not hitting any type of problems with the resolvers where it's using up their disk space or using up their traffic or memory. But if you're a large validator, let's say Comcast or something like that, yes,

they're hitting their limits, and they have to roll out new validators all the time. And so they should be prepared. I think a good rule of thumb would plan for 10 times more traffic and 10 times larger zone files. But when I say traffic, depending on how you set up TTLs and things like that,

that's a one-time hit, and it's not going to be continuous. Another thing is DNSSEC preparation. It's sensitive to time issues, being really off. So if it's within an hour, it should be fine. And you set up your tools to manage re-signing the signatures.

And if you sign them 10 days in advance, then you have lots of leeway. But if your system clocks are reasonably accurate, it should be fine. That is a bootstrapping problem, though. If you want to use NTP when you first start up your system, well, how are you going to get that information if you don't have DNSSEC?

And so that's a bootstrapping issue. All right, so using BIND9 as examples. Well, DNSSEC enable is already on by default. Validation is also already on default in all of the current stable versions. There's also a way to trigger on and off using RNDC tool,

validation on, validation off. And so these are enabled by default. But just because they're enabled by default doesn't mean any of you are doing validation yet. You might be doing the very first step is you need to have a trust anchor. And so if you have not configured a trust anchor manually, then it'll stop there.

So BIND9 does include a trust anchor. And if you wanted, you could copy and paste it into your name.conf file. Or you could just use the syntax here, DNSSEC validation auto.

And it will use what's called RFC511 updates. And it'll manage this. And how it manages it, it'll try to keep it up to date. Yeah, and I don't know anything about the rollovers with the root or if it's even happened yet. But we'll look at that later.

Also, you could use alternative called DLV, DNSSEC look aside. This is where you could use it for testing, but you might use it in real production. Probably not on the internet, but real production in your offices or with your customers. What it does is it provides a way to look someplace else.

Well, you've looked at your parents. They don't have the information. Well, let's look someplace else. And they'll serve a record called a DLV record, which is just like the DS record. So a parent provides a hash. And there's different services that provide DLV, DNS hosting. And ISC does provide one.

And I don't know how many people we have in there, but it might be over a thousand or a couple thousand. And so it's a good way for you guys to start testing if you want to just use a DLV before you can convince your parents to get your DS records in place. You can just use DLV.

Testing validation. Well, the first thing to note is once your trust anchor is in place and you do a query regardless of using DNSSEC switch, your validator might already have it enabled and start doing validation. But if you want to do it from the command line, you can use DIG plus DNSSEC.

And then this answer came back. It has the extra flag, in this case, the AD flag, authenticated data. And I mentioned it also earlier. You can disable the checking by using the CD flag. And you get a serve failback, try to use a CD flag and see if you get the information.

Cruising along with a lot of information to cover here. Enabling it on the server side. And it's already enabled by default. It's ready to go. If someone sends a query with the DO bit set saying that they'll accept DNSSEC traffic,

your bind server is already ready to go. If your bind server has signatures, it's going to send them on. This is an important note. Make sure your secondaries also have it enabled and supported. And I have seen this.

This is a real problem. If you have a DS parent saying yes, it's signed, but then all of your secondaries won't provide the information, then it's possible you're going to return serve fails and make it invisible. So make sure your secondary support is to test all of them.

All right. So here's a fun part, real problems in the world. And so I've been following it and troubleshooting it for around four years. And I see different situations and troubleshooting as much as I can. But then there's also mailing lists where people point out whenever they see problems. It's interesting.

And there are a variety of different reasons. But so I'll go through these very quickly. E164-ARPA, a signing tool had a bug, so the new signatures were not published. Well, remember they have expiration dates. All right. So they lost their DNS set, or they lost their DNS for whoever was validating.

ARPA expired. They had a known failure, so they turned off their monitoring. And then the real thing came along, and they didn't notice it. IP6.ARPA, they had a mismatch between their DS and their DNS set key. .GI, .BE, signatures expiring.

.KG and .TH had start times in the future. And it's hard to do, but their time zone adjustments or they didn't use the tools correctly. They should have used UTC time, and something was off. And so they started serving. So any validators that were set up, then they would lose everything.

The good thing is now is there's not a lot of zones using DNSSEC, so it doesn't hurt a lot of people. But potentially, it could be really bad. .US expires signatures. Yeah, they had a key management system that was automated. They removed an old key from use,

but they never re-signed the records previously using it. And so what happened is that caches had a mismatch of signatures and keys. Now UK had a similar thing. They had a failure in their hardware security module. And so it felled over to an alternative way to do signing.

They had a different signing key, so they had a mismatch. Now UK, another mismatch, and they fell over. .ORG announced mozilla.org was signed. But mozilla.org was not signed yet, so they published it in the wrong order.

And so for DNSSEC implementers, they can't even get to mozilla.org as an example. And in these examples, I didn't put the time to lives on these things. Let's say you're using somebody else's validator, and this time to live was two days. You're out of luck unless you know what to do to switch over.

For a regular user, they would have no idea what to even begin with. They can call their ISP. They wouldn't know either. It's pretty tough. Some more examples, .gov. This isn't real important, but .gov, when they first started, they shared their keys

to many people who manually configured them in place. They had some government rules where everybody had to deploy .gov, but it wasn't in the root, and so everybody in .gov had to configure it in their validators manually. So then start rolling keys and, yeah, start breaking things because it was all manual versus automated.

Some crypto errors generating signatures. Providing a DS based on a key that wasn't even available via DNS. NOAA, weather.gov, a whole onslaught of .gov sites have had problems over the past couple years. NASA signed with an old key.

Yeah, ISC had some problems too. Overlooked some reported errors, and so a mailing list, a server expired. And yeah, that's a list used by people playing with DNSSEC, and so they can notice it.

Also, some implementations of validators, checking all the DNS keys, checking the DS records. It was so exhaustive and aggressive, in certain scenarios, and we called it rollover and die. It was an amplification attack, over 40,000 requests per a single query, out of control.

Also, in some cases, like when .net introduced a new DNS, sorry, DS records, it caused validation problems until that cache data was expired. And so some of these were code related.

A learning experience, but we're not the only ones. And so just to summarize these common problems, automated maintenance, program failures not detected, DS doesn't match the DNS key, signatures expired, signatures haven't even been start time yet, slave servers not knowing DNSSEC.

The failover systems, using different keys, or keys that weren't published yet. It's okay to use a different key, but if the key is not available, that's what I mean, to the public.

The last point here is important, is it time to lives versus expirations. And so you might have signatures expire, but what if it's time to live is a lot longer and it's cached for a long time? Hopefully, your validator would time out if the expirations timed out also, but.

All right, so very quickly, some of the command line tools, we have key gen, sign zone, set time, revoke, DS from key, key from label, and insect three hash. I'm not going to be covering all of these. And set time itself is something used by administrator, it's not part of the protocol,

but it's to help manage your DNSSEC keys. The two tools you'd use for playing around are key gen and sign zone. From the server side, without the tools, some of the features, you could add keys via dynamic DNS.

Also, adding records via dynamic DNS can sign those on the fly. Also, depending on the keys you provide, it can sign the entire zone via dynamic DNS. We provide DLV for the alternative trust anchor I mentioned.

RFC 5011 support. Also, we have support for different crypto hardware or hardware modules, storage modules. We have automatic resigning and expiration of dynamic signatures. So, it doesn't wait until it expires, but it can do it ahead of time.

Polishing and removal of new or old keys. It could randomize or scale signature regeneration over time. This is important. Let's say you have 100,000 signatures in your zone, and they all expire, you know, at the exact same time. And so, by now, we'll look at it and start resigning those scaled or randomized over time.

And yeah, all of it done before it's needed, but you can't let it try to resign everything at the same moment. A lot of work. We have a bump in the wire inline signing support. It could automatically sign or reassign zones that you maintain with other tools. Let's say you have tools already used that don't know anything about DNSSEC.

So, you could use the inline signing to do this for you. And also, if let's say you bring in a zone as a secondary, you can use inline signing to sign it for you then. So, it's a nice way to deploy DNSSEC without learning it first.

So, I mentioned KeyGen generates the keys, the private and the public. It generates multiple files. So, you should get into the policy of maybe having different directories for every single zone. A lot of people already do that, but it's a way to cleanly manage it.

And just very quickly, you run the KeyGen tool. You can forget about the options here, but you tell it to zone, the name of the zone, and it creates the keys. And so, we have the original zone file, very small. We had a couple of keys here. And you can see I'm looking at the public key. This is the key that you would put in the file to start publishing it.

How you could put it in the file? Oh, sorry. That's one slide ahead. So, then you also will create a key that will be used for that DS record. And this is not used for all of your individual signatures,

but it's only for what's called the apex. The apex of your zone. And this is important, again. This is only for management. The validators don't care about this. But let's say you want to change your key for all of your own records. Well, you wouldn't want that to be the same key that was used in your parents' published DS.

Because then you would have to work with them and change them at the same time, if you didn't plan it correctly. So, I'll go for one more minute. So, your random devices are adequate. And some devices, it takes so long, I just use you random.

But that's something to consider for your randomness devices. The algorithm choices, these are just from the surveys. So, you can see the algorithm choices or the bit links used by the 89 TLDs that exist right now that are assigned.

So, an easy way to do it, you can use the include macro to load it in, or you can just manually put that DNS key into your zone. So that's the first step. The second step is run the signed zone tool. The increment, that's your SOA record, it increases it for you.

The name of the zone, it creates some new files, a key set, DS set, the DS set is what you give to your parent. And you have your new zone file with the extension .signed. The original file was 2,000 bytes, this new file, 24,000.

And your own configuration, well, if you just have the file pointed to your previous one, in my case, descending.com, it wouldn't be the signed version. So I give it that new file name. And so, you can see it's just a few steps, very easy to do. At this time, you could do an RNDC reload, your zone name,

or restart your name server if you wanted. And it will just start working. You can start troubleshooting it. I already mentioned the DS records. This is what it looks like. Again, you'd share this with your parent. There's no standard mechanism yet for sharing them with your parents.

Well, I'm out of time, so let me just quickly look. Mm. Yeah, just so very quickly here. So I have a serve fail. I didn't know what to do.

And so I could query it again using the CD flag. I could get the information. So even though the validator failed, CD allowed me to still get the information. I could see the signature. In this case, I could see the date. It would already have expired. It's just an example. And I did this just for fun for testing.

Another thing in from the administrator point of view, there is very, very verbose logging. After a flush, one look up, 104 lines of logging. But the thing is, that doesn't help anybody except the administrator. And so Comcast, for example, has reports every single week

now of validation problems. So it's interesting to point out. It's complex to implement. Yes, but you could implement something basic in 5 to 20 minutes. Has a very steep learning curve. But if you use the defaults, again,

it should be easy to do initial deployment. The resource requirements are not significant unless you're extremely hitting your peak right now. There are several documents about it. You might want to look at RFC 4641

or the draft that's replacing it, operational practices. And I'm out of time. I had a bunch of TCP dump. I could show one more example of that if anybody wanted to look at that. Or does anybody have any questions at this time?

OK. Yes, I can put them online. Any other questions? OK. So I have all my DNS set up such that all the time they're using CPS, the SRI records and all that sort of stuff, auto-generated

to make sure it's all correct and stuff. I meant to redesign all that out there. In the long run, you might want to. But in the short term, you might be able to get around it by just creating the keys only and then using the bump in the wire service.

And yeah, let's. Oh yeah, we have a bind version. But yeah, Freebird is an alternative. The example, very short and quick. Run the key gen twice, one for your own keys

in your own zone, then one for the key for the chain of trust for the DS. And then the name.conf, you can see two more entries here. In-line signing, yes. Auto DNSSEC maintain. Once that's enabled, boom, it'll sign it. And you might have journaling files before doing dynamic updates.

It's the same concept. It has a journaling file, a separate zone file that it keeps in sync with your original. So if that answers your question. Any others? Yeah, so that's the DS record.

Well, there's no chain created yet until the parent has the DS record. So you can set it up inside all you want. The chain only exists when the DS record is there. And so it doesn't break it. If the DS record got removed, that would break it.

And so you could test it without it, yes. And one way to test it again is using DLV. Or I didn't show example here, but my manually pasting in your DNS key into your name.conf file on your validator.

And so you could have a beginning trust chain. Any other questions? Who here will try to deploy DNSSEC in the next year, within the next year? All right, cool, cool. And feel free to shoot me any questions you want. Or you could use bind lists for doing that.

But there's a lot of helpful people that could help you out troubleshooting it. Thank you much for listening to my presentation.