Thank you So hello everyone Just before we start. I'm really interested who of you came only because of the name of this talk Because I'm doing research. I'm interested whether the name sells or not But yeah, I'm young

I'm a software engineer at NS1 and I would like to talk about a tool that we open source recently It's called flamethrower and it's DNS performance testing tool This whole started about two years ago We started working on a new like custom DNS server for a company and we needed to test particular

features and the existing existing Tools like DNS perf like flamethrowers inspired by DNS perf Was didn't didn't get every or didn't contain everything we needed Particular thing that we are interested in was like source for distribution DNS perf uses one single port so

When you actually want to test this in some real production setup, you realize that the traffic patterns are completely different than from real clients we also wanted something that has much better TCP support and TCP in general and the DNS port used to be a second-class citizen

Like the support is not always perfect in DNS servers. So we needed a tool first To test it to make sure that our server was worse like well or as we expect we also wanted something that can generate realistic traffic patterns and

Since the beginning we wanted to integrate this tool into our CI CD pipeline Because as you are working on a server adding feature, we are interested how the performance changes The flamethrower lives at github and our DNS work like organization

We Open-source is like early this year at the beginning of January DNS work for you. Don't for you who don't know the organization is or the ORAC stands for operations analytic research center is a community of of people From like DNS work so operators software vendors

Top level domain Like registrars and so on so we are happy that we can host this this project there and shortly after we open-sourced flamethrower The s perf moved to a DNS ORAC as well and yesterday also DNS was moved there

So I think flamethrowers and pretty good company So I would like to start with just simple quick start just to give you some idea what this tool does because later I will talk about some internal architecture and I don't I want you have to some idea. How it is like works from the user perspective. So

On the left side there is some sample invocation of the of the utility You're on the binary is called flame and you give it some parameters. So in this case We are sending some queries to server a NS example to test and we are saying sending one same query over and over

The record name will be foo foo example to test and we query for the SOA type the dash queue configures the query rate, so this means 2,000 queries per second and Dash L like limits the execution time. So this will run for five seconds and So the flame will like first output some some like basic information about what's going on and then it will

Generate like short stats every second and at the end it will like print out Like the result in like human readable form I run it At the airport before I before I left to this conference so you can see for this query a lot of surface

It was probably some DNS firewall in the way Like internally the tools for them in C++ You need relatively new tooling you need C++ 14 or it should work with C++ 13 But it's likely that we will soon use something that requires C++ 17

we use LDNS as a DNS library to construct the packets and We also use UVW which is really nice wrapper for lip UV The UV is like a sync IO framework It's used for instance in node resolver. It's used in node.js

It's pretty nice library and UVW provides Like pretty nice interface for C++. So it's much easier it uses Lambdas it uses It can handle the memory allocation for you. So so it's pretty nice And if you have some time at the end time, I might get get into some details about UVW

Like internally front over it's it's not a rocket science. There's only only a couple of components There's some kind of payload generator that prepares the DNS queries that will be sent Then there is a traffic generator that handles the networking so either creates

Like UDP sockets TCP sessions sends the queries this closely interacts with with the rate control because the payload generator can or the traffic generator can send send the queries in some batches and

Sometimes you won't just like put limit on on top of that. So that's that's what the rate control is there for and There's Yet another component to collect the metrics in format that is useful for instance for like machine machine processing So I'll talk about about the individual components the payload generators

These these are the ones that we we implement it The static one is the simplest one that I showed at the beginning the quick start You just sense the query for the single single name and type On like over

File is another packet generator is actually compatible at DNS perf so for DNS perf you could like write the text file that has Like record name and record type on each line so we can process the same same format as well There is a generator that Generates just random garbage non-valid DNS packets. So this is this is really really useful

It probably like won't crash your DNS server, but it's useful for instance if you want to test your your monitoring Because what we realized when we are testing our service that for instance the tooling that we use to collect metrics Doesn't see this kind of queries at all because it's not a query

It's it's malformed and the parsers in the metrics collectors are not expected. Just just random junk Then there are like three very similar similar generators that generate valid DNS packets, but with random domain names, so the number queue name just

Uses like prep ends a random numeric label as a first like label for the name Random queue name Does something very similar? but It's like generates again random possibly binary

Query name like DNS specification says that in the domain name you can have only like letters numbers underscore Whatever, but in theory you can put even binary binary in the in the query names like binary values Like for instance binary zero like most of the DNS software can handle that

It's not a little bit specification, but it's it's interesting To play with this as well And then random label which Most of the people in DNS know like two years ago the Mira attacks It was essential it is like random random prefix attacks for some domains

So we can simulate this with with this payload generator as well So you generate your traffic and this come goes into like the traffic generator which actually handles the the networking At a moment the flamethrower can run only in a single thread so

So all the like operation on several sockets are like Driven by the by the IO. It's is the lip UV style like a synchronous spot in a single thread you are switching the context the max query rate we You should expect on a single core is like 100k

Thousand queries per second Of course, it depends whether it's on your laptop or if it's some high-end server, but it should be roughly 1000 so it's not much but it's enough for the best thing and The main configuration option for the generator are like these these three You can set up the number of concurrent

traffic generators this like depends whether you like later pick UDP or TCP, but essentially if you Create 10 for instance or I don't know hundred concurrent traffic generators, it means that for UDP flamethrower will open 100 UDP sockets

So you will see the traffic coming from 100 random ports If you do this for TCP, it will go open 100 connections and it will try to keep these connections alive so whenever the server closes the connection it will like reopen and a new one but there should be like 100 connections all the time open and Then you are sending like batches of queries on these either like UDP sockets or TCP sections or TCP connections

So this is configured by the next two arguments the dash queue Which is number of queries to send in a batch and then there is a delay to wait for the next batch So for instance if you want to send 1000 queries per second You can have I don't know 10 concurrent generators, which gives 100 QPS per

per Per the generator and now you can set up the generator to send 10 queries every 100 milliseconds and you get as a result 1000 QPS There's also the the rate control that I mentioned so this is how you can like

Cut Cut the the rate of the queries in case the generator is too fast You Can either set it constantly for a constant rate or you can write like a QP define like QPS flow so you can always set like

the rate you you want to see on the output and the duration of milliseconds, so This is actually what you see some you see on the graph below you can for instance Credis Credi speak so there is Like 2,000 queries per second for six for 60 seconds then 100 queries per second for 60 seconds

repeating so This is from one of our testing servers. There was some nominal traffic for of 600 QPS like this is the first First first minute and the second minute is it's like back to the 100 and so on so on Yeah, we don't sample we sample the the right at 30 seconds, so it looks

Like this, but yeah Then the metrics so I've already shown you at the beginning how the the output for for humans look like But if you run flame with dish Oh

It will print out a JSON like the results in JSON into the file So you will have for each second you will have one line of JSON that essentially contains Everything you see in the like human for at the end

But you will get get the same information for for a second so you can see like the development during during the run of the flame and We for instance feed this into the elastic search and you can later do some visualizations based on based on that I Didn't include it in the slides because I didn't find something really interesting

But yeah, so just so we have some idea. It's it supports, Jason Here are some examples A simple one that uses the the DNS perfect input format I included some some description like how it will What it will end up doing so in the first case you're opening like it's on UDP. You're opening

100 source ports or source source sockets, and you will send 10 queries each 500 milliseconds Which will end up with 2000 QPS Here the second one is slightly more complicated it uses

The QPS flow and it's is to simulate some some kind of peak so you start with 10,000 QPS for five seconds then decaying to 5,000 QPS 250 and so on and then there is some like 1,000 QPS At event for 40 for 40 seconds, so it ends will run one minute

But you will have some some peak of traffic you can use it to simulate. I don't know code cache or some failover and Yeah, the last one just shows Sending like random garbage on on TCP

With some right like 1,000 QPS Yeah, we have some ideas to improve this this to further We definitely want to improve the improve the use of the resources on on the the machine so we want to support multiple CPUs We want to be able to target multiple servers or IP range within a single run

We wanted to spoof the source address because this is useful for testing as well We'll probably add DNS over TLS DNS over HTTP If someone has a need for that

And Yeah, query rate is something that we control only with this QPS flow or like static configure statically at the beginning so we were looking into like extracting for instance the rate from from pcap and then just pretending Pretending that the rate is the same and of course like this is this this used to be

Or we started this To test particle features. We didn't focus on performance. So maybe maybe that's maybe that's something something we might focus on and Overall overall it like the tool needs some some cleanup it's not really a state-of-the-art C++ because we were all learning C++ at a time and

People from multiple teams contributed contributed to code So, yeah, that's it Thank you, and I'll be happy to answer questions and I will be also happy to Talk to you on a github if you send out send us some PR

Any questions for young? Yes Mathias both

Yeah, sorry, sorry, I should repeat a question I was talking about QPS and well you are interested whether it's UDP or TCP Like it can it can really generate The same QPS on TCP as well It can I didn't mention the explicitly but it can it effectively

Does TCP pipelining it opens the connection and tries to keep the connection open? For the whole run. So so you're just then sending the burst of pockets So that's essentially the pipelining and if the server closes the connection flame will reopen it a new one But the overhead of opening the connection is not that high so you can relate generally like

Thousands of QPS on TCP if the server is able to respond

Yeah Can I run flamethrower to find this peak where? Like the server stops being able to respond on on TCP or UDP No, you can't you have to do it manually, but it's it's actually nice nice feature request. Yeah, yeah small

feature request I would say do not focus so much on the high-performance stuff because everyone is always focusing on the high-performance stuff No one here ever runs a name server that does two million QPS But we do want to run DNS over T and TLS really well and we want to run DNS over HTTPS Really well, and it turns out that those two protocols have a lot of modes

So we can have many connections. We can have connections. We keep alive for a long time We have can have connections that come with tickets and that resume we have all kinds of TLS modes with zero RTT and stuff What the world super duper needs right now is a testing engine that can exercise all these modes Because we don't yet know what the browsers are going to do

And because we don't know we would love to have a tool that allows us to explore that landscape So here my feature request and if you do it, we would love to help. Yeah, that's cool Thank you one comment in one question are really waiting for the to import pickup real traffic for the testing and

Second the question is like in in DNS perf there was like a maximum number of outstanding queries So where it's rattled until the names are responded again, do you have like some such similar? No No, there's no way No, it will just Like it tracks the the number like internally we know the number but there's no function of that

It's like stop sending queries if there's some amount of outstanding what's in flight Any other questions Martin? Hi. Thanks for the call So I'm not so much of an DNS expert, but no know a bit about load testing

You may want to consider to print out the percentile instead of the average For your response times because the average usually doesn't tell you anything at all. Yeah. Yeah just as an idea Yeah, like in this in this JSON format, you will see the actual values So if you want to do some post aggregation on top of that you can

Not for each request but you will see a current starts for each second and there is a There is there is always a current like query rate and current response rate and the number of for instance time Queries that I'm out during that second

So I think you should be able to get a number you want That's a good point. Thank you I've lost question over here Well, maybe it's more or less a feature request. I'm sorry for that But did you ever consider not only checking the result code?

Also some other expected patterns in the answers, yeah, of course some bugs only pop up under high pressure Yeah, you're right No, we are checking only for the response codes. We are not checking any other content in a packet We could probably do that. I don't know if

Thank you