I think we're on your group. Thanks for sitting around. Like Simon said, this is the last talk. It's very different. Hopefully we can hear a little bit about how we do testing and how we do it.

The goal of this presentation is to go through all the different things that we do to test our SSL, TLS, and crypto library. What we do and maybe it will be useful in your project when you have questions. We're always looking to improve our testing as well. My name is Chris Conlon.

I'm a software engineer with Wolf SSL. I started with Wolf SSL when we were younger three people. I live in Bozeman, Montana. It's over in the U.S. I wear a lot of different hats. My main role is software engineering, but I also fit in for a support team. I help a little bit with our testing team. I do some pre-sales and marketing.

We have a stand out here. We've been visiting with open source users from the last day. I really have a fun time. If you need to reach me, you can reach out over e-mail at Chris at WolfSSL.com. I'm also on Twitter. I can't guarantee how responsible I'll be, but e-mail is definitely the best way.

How many of you have heard of WolfSSL or know what we do? About a third of you probably. We make an SSL, TLS, and crypto implementation. We wrote ours from scratch, from the ground up. It's written in C. It's roughly 10 years old now.

It supports up to TLS 1.2, DTLS 1.2. It's very optimized, so low resource use. The full footprint size is about 60 kilobytes. It works very well both embedded all the way up to a large scale server environment. We're currently still a fairly small company. We have 19 people who we've been growing every year.

We're based in four countries. Our headquarters is in Seattle, Washington. I've been both in Montana. We've got a guy in Portland, Oregon. Some guys in California, one guy in Tokyo, and one guy in Brazil. And a contractor that kind of works for us part-time in France.

So today's outline. I'll just go through a quick introduction and go into the types of testing that we do. Hopefully we'll have some time at the end for you guys to ask questions. Why is testing important?

Especially in security software. Security software is really one of the core pieces of your application. Especially if you're dealing with user data. You're trying to secure data that's in transit. Secure data that's at rest. You really need to make sure you want your users to trust your application. Indirectly, they're trusting security that you're using in your application.

So it helps to know that it's well tested, well vetted. It's widely used. And there's a lot of eyes looking at the code base. And one thing that we've learned is being open source security is very helpful. That means any of you guys can pull up our GitHub account and look at our code base.

We can have universities look at us and different security teams. So with SSL it's providing data integrity, confidentiality, and authentication. So we try to test all these different pieces throughout the next slides. Types of testing.

These are the types that we're going to cover today. We have unit testing, cyber speed testing, algorithm testing, and static analysis. Detecting memory errors, doing interop testing with other implementations. Peer review, third party testing, some fuzz testing, CI, and then just going through our nightly test cycle.

So first up is unit testing. So the goal of unit testing for us is to provide a quality tested API to our end dealers. So unit testing aims to test the almost granular function.

And for us, in a library, that granularity is an API function. And so we have unit tests for our SSL layer. We wrote our own test framework that doesn't have any external dependencies on the third party package.

So it makes it more portable across platforms. And it's ran automatically, well somewhat automatically, every time the user runs a make test or make check. We run, among other things, our unit tests. And so if you download our code, run make check, you'll see that we get API tests and end API tests.

Also run as part of that same executable as our cyber suite tests. So the goal of this is to test all the cyber suites that we support in our library. And a cyber suite, if you're not familiar with that, it's a representation of algorithms and protocols that are used in making an SSL or TLS connection.

And so it verifies the suite's going to work as expected. Here's some examples of cyber suite strings. They're kind of small, but I'll put the slides up afterwards so you guys can take a look at this. And in our normal build, we have 196 cyber suite tests that we do.

If you enable DTLS, which is Datagram TLS, that bumps up to 300. And so that's 49 and 75 suites, respectively, being tested. And this is testing against ourselves. So we spin up our example client and test that against our example server,

which is going to look something like this. You'll notice that first we test the regular cyber suite. Then we test that in non-blocking mode, so using non-blocking IO. We test it with client authentication turned off. And lastly, we test it with both those, client authentication turned off and in non-blocking mode. And so each four of these are run for all the cyber suites that we test.

So similar related to cyber suite testing is algorithm testing. So you want to make sure the algorithms are implemented correctly and that they're working properly. And so we have a couple things that we do for this. The first thing we have is a WolfCrypt test suite.

WolfCrypt is our crypto library. And what this does is run through all of our algorithms that we have implemented. It has a set of input vectors and matching output vectors. So these are NIST vectors. And the other thing that we do is we recently went through FIPS certification,

which is like a stamp of approval certification for a crypto library. And with that, there's a whole bunch of test vectors that come with that. So on a regular basis, we run all those vectors through the library as well. So again, this is verifying the correctness of the implementation of an algorithm.

And it's also helpful when porting because we can port it to a new platform, run these tests, and make sure the crypto is working on a new device or a new platform. Just to give you an idea of what those look like. So on the left, we have our WolfCrypt test. On the right is a little snippet of our FIPS test harness, where you're reading in test vectors that come from files,

and outputting them to a response file. The next thing we do is static analysis. So the goal of this is to find bugs somewhat automatically by using static analysis tools. And this is going to follow different code paths. So code paths that our developers may not have tested when they were working on a feature.

They'll test less used or less frequently used code paths. And to do this, we use the Kubernetes scan tool, the Clang scan build. And we recently started using the Facebook invert tool, which is a new static analysis tool being put out by Facebook.

So memory errors is another big thing, especially in an SSL or crypto library. You don't want a segfault to happen or your program to crash when it's doing some security operation, like making a handshake or sending data across a TLS connection.

So it's important that we vet these out. And we use an algorithm to do that, the memcheck tool. This can detect accessing memory that you shouldn't be accessing, incorrectly freeing memory, finding memory leaks, etc.

The next category we do is interop testing. We tested the algorithms. It doesn't necessarily test the correctness of the protocol implementation. This is all the protocols, the TLS protocols. And some of those RFCs are somewhat vague or hard to read,

and they can be interpreted differently by different implementers. So doing interop testing is important, because applications in the wild are most likely not talking to the same implementation on each side. OpenSSL might be talking to a wolf SSL, or a libRSSL might be talking to a boring SSL.

So we do regular interop testing with the limitations I've listed over here, which include OpenSSL, BoringSSL, SSL GnuTLS, and embedTLS, which used to be folderSSL. So if we're not testing against you, we'd like to be interop testing

if you have a crypto implementation. Or an SSL implementation. We'd like to throw you into our test suite. So if you fall into that category, please come up and talk to us sometime today or tomorrow. So verifying the correctness of algorithms in sector suites is important.

Another way to stop bugs getting into the homemade code branch is through a peer review. So the more eyes that you have looking at code, the better the code is in the end. And what we use is a fork and a pull request system. So all of our code is on GitHub. All of our developers in the development site

left a fork art code into their own fork, did their development on a feature or bug fix, and then submitted a pull request back to our master. And then that'll be assigned to someone independent of them who hasn't even worked on that feature or fix yet. And that person's responsible for testing it or viewing it, and then they merge it back into the master branch after that.

So having two sets of eyes really reduces the amount of bugs that end up in the master branch that then we don't have to find the rituals later on. And it's done on every commit that comes into our master branch.

Next up is third-party testing. This is different than interop testing where this is external parties like security researchers or customer user labs that are analyzing and looking at our code base. So we've had teams from Google and Intel look at us. I think Intel's gone through it almost line by line.

We have other customers and users who analyze it. We've done our fix testing, which is an external review. And we pretty regularly have university researchers and people working on theses or doctorate work to look at us and submit comments and feedback back. And being open-sourced really makes this easy

because anybody can look at it and analyze it and give their suggestions or feedback. The next category... So we added this, I believe, roughly two years ago. We started fuzz testing, or fuzz to sell. And so fuzz testing just guided the random inputs to the library

with a hope to make the library crash and find a bug or a point that could potentially be a vulnerability if released into the wild. And to do this, we use a couple different tools that we've developed in conjunction with one that Google has contributed back to us partly.

And the first one is WolfFuzz. This is an in-memory fuzzer. So it spins up a client and a server, and it communicates over memory buffers. So it doesn't actually go over a TCP connection at all. And it fuzzes all the public-facing buffers that you would have at a TLS level. So hash buffers, IO buffers, et cetera.

The advantage of the in-memory tester is that it's very fast. So last numbers I heard it was testing... It was fuzzing four trillion combinations every couple months, I believe. The second one we have is a network fuzzer.

So this introduces the TCP into the mix. So instead of using memory buffers, it actually goes over TCP IP link. The advantage of this is that you can test it with other implementations. So you can fuzz us on one side having another implementation on the other. The disadvantage is that it's a little bit slower since holding a socket is slower than just running over a memory buffer.

So we do continuous integration testing. We use Jenkins for this, which is a pretty cool package, and we like the work those guys have been doing. So on each pull request, we do continuous integration.

We run our FIPS build. We do a known configurations test, which includes common configurations, user configurations, and more frequently used ones. We run our Belgrin BEM check tests, and we do our static analysis of scan build. So all of these things are run on the pull request before it gets assigned to another developer to review.

And then lastly, our nightly test cycle is again done by Jenkins. So this runs tests that are more extended that we don't want to take up time during the day or that might be going too long to do on each pull request.

And so with these, we again do our known configurations test. We have an extended build options for tests. We use AutoComp to build and configure our library. And we have, our list of options is ever-growing. So we have a set of those that we test with multiple compilers and multiple platforms.

Compilers we test with GCC, Clang, and Intel's ICC. And I believe our platforms, we test the desktop platforms at the moment. And then we also do extended fuzz testing to go over, kind of a recap,

the testing that we do over three unit testings, Cypher Suite testing, algorithm testing for implementation correctness, static analysis, detecting memory errors,

interrupt testing with those implementations that we listed, peer review, third-party testing, fuzz testing, CI, and then test cycle. So we just wanted to give you guys a glimpse into what we do. We have a goal of being the most well-tested SSL and TLS library available to you.

And we think that makes it better for everyone, makes it better for open source users as well as people who are using it in a commercial project. We're a company, and we do provide commercial support to paid users. But we offer free support to open source projects. So if you're looking to add SSL or TLS to your project,

you can get in touch with us, and we'll support you through that process of offering. And you just have to send us an email, support at wolfssl.com, and that reaches the whole team, and we assign it to one of our engineers, and work you through any issues or problems that might come up. And again, if we're not interrupt testing with you,

please contact us. We'd like to throw your tool into our testing cycle and testing framework. And with that, I guess I'll open it up. Does anyone have questions about wolf SSL or about how we do testing? Thank you, Chris.

Questions? Hey, we talked a little bit earlier, but our conversation was cut off. So I'm curious. Most of the fuzzier testing seems to be at the input level. It's at the almost integration, testy level.

Have you considered doing property-based testing on units of your code instead of one single fuzz, for example, using a conjecture or quick check libraries against individual units of your code? Yeah, so doing it more at the function level versus like the external buffer that they're probably facing. Yeah, I think that is one thing that is on our list.

We'd like to do more testing. Usually it comes down to a matter of time, allocating resources, knocking everything out at a time. But thanks for bringing it up. If it's not there, I'll make sure we do have the chart list.

Do you have some more field testing, loading testing?

We do stress testing in-house. But we have customers who use single-fuzzy selling in very intense, high-production environments that have multiple hundreds of thousands of connections coming in. We have users that are processing hundreds of thousands of hours of like, quick traffic per day, per cent, using multiple-fuzzy selling.

So at the moment, we've relied on those types of customers to give us feedback on how we perform our hidden bugs. Since you've been applying such a big range of tests,

which strategy has been joined to be most successful to find bugs? Yeah, that's a good question. So which strategy has been most successful to find bugs? The fuzz testing has passed out quite a few, especially when we first downloaded it. There were a lot of, say you have a buffer

and you randomly mutate one byte in it, so that it becomes like a corrupt 2-less breaker, or some kind of handshake breaker. And that found several areas, I would say like one of the four to eight really important things, where it would cause the library to crash. But on a daily basis,

the continuous integration testing and those tests catch quite a few things, especially given that we support so many different platforms. Out of the box, we support probably upwards of 20 to 30 different operating systems. And the developer might not have all of those in mind when he's adding a feature for one specific platform.

Like say somebody's working in a Mac OS X environment, they're adding a new feature, they may not think to test it in Linux 64 and 32 and Windows, oh, and this freescale kinetics device over here. So it helps to have that in place, what I've seen.

So you said you're doing these interop tests with other TLS implementations, so if the other implementation behaves correctly,

these are more or less good cases. So maybe it's a case test of TLS, for example, if the other side doesn't behave like the protocol it specifies. So that's a good question. Right now, we're only trying to make a valid connection.

We haven't purposefully injected errors into that. With the exception of the network buzzer, we put another implementation on the other side. We don't currently have the network buzzer set up to automatically test against other implementations. So for that, we've had to manually fire up, say, a new TLS on one side and list them on the other.

But that's something that would be helpful to add for a test cycle. You said that you are more than happy to get other implementations and submit them to your tests. Do you do the reverse thing, submitting your code to other implementations and tests?

Yeah, we would be very open to anyone accepting our code and putting it in their own test framework. Is it part of your process for bug engineering? For us to do it. Or do you actually do it?

We haven't traditionally done it ourselves. So we haven't gone and put us in someone else's test framework. We're also doing some speed comparison tests. So if your church implementation is faster after some function fixed

or whatever, your hash functions, if they're recording, will fail. So a performance test. So we do test before and after something that we think is going to affect the performance.

So say we add a new hardware crypto support, and we're definitely going to test it before and after we get the delta performance there. Or if we add a new assembly optimization, we'll test it on several different platforms and support that and measure the performance gain or hopefully not the performance loss in that case.

We don't automatically test performance against other implementations. But I can tell you, our user base continuously asks us, how do you perform an implementation test? And we do have some things that we can send them on these cases. But it's more like a benchmark that we run at one time. This is a somewhat recent but not instantaneous result.

So that's what we're going to do before automated performance testing. Okay, one more event, please.