work for a company called GreyNoise, all the slides are grey, now they're purple, so I work for a company called PurpleNoise. Um, again, my name is Andrew, I work at GreyNoise, I also am GreyNoise, there's no one else in the company, it's literally just me, um, but it's kind of an open secret, and I like referring to the company as we, cause it makes

us sound more legit and professional, but make no mistake, it's just, it's a guy in an apartment doing all of this. Um, before I started GreyNoise, I was on the R&D team, uh, at Endgame, and prior to that I was always kind of private sector red team doing various different things. I've been staring at internet scan traffic for so, so, so very long, um,

and I'm not a data scientist, I'm not good at math, I'm not even good at stats, I'm not good at machine learning, I don't know anything about any of these things at all, but I had to learn a little bit about some very basic statistics to kind of do the thing that I'm trying to do and write this big ol', big ol' bastard of a SQL query that I'm gonna show you

guys. Um, so today what I want to do is go from, I want to talk about the process of going from, uh, a bunch of firewall logs in disparate systems to actually, you know, some kind of anomaly, some kind of indicator of an anomaly that is able to be correlated with an

actual thing that makes sense. So, just a bunch of Apache and firewall logs from that in a lot of systems to, hey, at this time and this place there was a giant uptick in people scanning and probing for those things and it probably had something to do with this vulnerability that is associated with that traffic or this event, this thing that

happened, right? Um, and so that's it and the way that I did it is with this, that's it, this giant ass SQL query. Um, will you guys do me a favor, if anyone in the room, who has already heard of Gray Noise before? Will you raise your hand if you've already heard of Gray Noise? Alright, so we got probably, I don't know, maybe 3% of the people in the room have already heard of Gray Noise. So, um, we do this thing where, and

I'm always gonna say we, it's just hardwired in my brain. We do this thing where we write these tweets anytime we see these upticks in scan traffic that are explainable that have value and this is how we do that. We just run this query. I mean, it takes forever but this is it. And so I'm gonna break this whole thing down and I'm gonna

kind of get, get there, right? So, um, what's internet background noise? Uh, it's basically, it's the baseline omnidirectional scan traffic that's generated by all these people that are scanning the internet. Like the Shodans and the, and the Merais and like the Censuses and like the WannaCries and like everybody scanning the internet creates this thing called internet background noise, right? So what does scanning the

internet actually mean? Really it just means querying all 4 billion, um, routable IPs, 4 billion odd routable IP addresses, sending them like a sin packet or a UDP program or something like that to try to figure out like a certain open port or protocol. Way back in the day it used to be that you, if you had like one box and you wanted to figure out what was running on it, you'd port scan it and that would basically be one IP,

many ports and you'd get information. Now mass scanning is the flip, is the inverse of that. It's one port every IP to figure out what's open on the internet, right? So why would people scan the internet? There's many reasons to do it. Find exposed devices, measure risk exposure like try to figure out how many Apache servers are there or how many people have this port open or how many versions of this, how many

places are this version of software running? Or I just want to hack a bunch of devices, I don't know, whatever. So who actually does it? Lots of good guys do it. Um, Shodan, Project Sonar, uh, binary edge census, I have like a, actually many giant lists of all the labels that I've provided in here and I'm gonna go through that. Lot of bad guys too, Mirai, WannaCry, Satori, Muskic, um, what's an anomaly? It just means like

anything more than we're used to seeing, like an actual, like an uptick, some, some out more, right? And what is a newsworthy event? That's just like a thing that happens that we're trying to talk about that falls into something of interest, right? This is like one list of the labels that we have already made in GrayNoise right now. This is like, in

GrayNoise, like what we do is like we're, we're looking at all of this omnidirectional internet wide scan traffic and we're trying to label all of this stuff. So these are some of the actors that do it. Here are even more of the actors that do it. I took these screenshots right before I came up here on stage. So I guess before I go any further, um, I'm gonna go and explain like what GrayNoise is and how it works. So it is a system that

collects and analyzes all internet wide opportunistic omnidirectional scan and attack traffic, right? Why? Well, the reason why is because every single one of those packets that's flying around the internet has its own unique and special story and that's actually true. There's, there's no such thing as random noise that happens for no reason.

Every single thing is explainable. Um, it's just that everyone thinks that it's just bullshit, but it's not. Um, you can look at these, the trends of this internet wide backscatter and scan traffic to figure out like what's going on to try to peel back like what, why are people looking for these things, you know. And, um, from a security operations center's perspective, if you're like a network defender or something

like that, then if you're actually looking at the firewall of your network or something like that, you need to be able to differentiate between the actual things that matter that are hitting you specifically versus like the things that are just hitting everybody on the entire internet. How many times have people been responding to an incident or something like that that ended up just being like a Chinese SSH bot or something like

that? And it's like, this is not a big deal. This thing is hitting everybody. This isn't some sexy APT shit. Um, so really what we're trying to do is we're trying to provide rock solid negative ground truth of what everyone should be expected to see. And this is actually something, um, that, um, Alex here over at Verizon, uh, said, uh,

yesterday, two days ago. And I love that negative ground truth. That's exactly what, that's exactly what we're trying to do. So the technical mission statement of Gray Noise is label every single internet wide scanner as either good or bad and put it, first of all, take the category of everyone and then try to label as many of them as humanly possible. Like what are you, what are you doing? Why are you doing it? Um,

this is kind of a state of the union of that. We're right now, uh, good. The markup of good is about fraction of a percent. It's about one tenth of a percent of internet wide scanners. Uh, bad is about 10 to 20 percent. It's labelable where we know they've done something that's violating some, like the computer fraud and abuse act. They're logging into a system that they don't belong, that doesn't belong to them. They're slinging an exploit. They're compromised and they're slinging an exploit on

behalf of somebody else. Blah, blah, blah. And then unknown. That's everybody else. That's the gray noise or I guess the purple noise. Um, so how do we do it? We have a big network of nodes in a gajillion different networks all around the internet. They're constantly shifting around in AWS and Google Cloud and Digital Ocean and all these different providers. They're always shifting around. They have no business

value whatsoever and they just hang back and they wait for people to talk to them. They're completely passive. Um, it's just like a ton of people with their ears to the ground listening for like these little teeny tiny minute signals and then aggregating all of those together in one place and doing all this labeling and analytics to try to find value in all of that. So again, we want to go from all of the traffic that's hitting

everyone to actionable, hey, this thing happened and it probably had to do with this, right? This is probably why. So this is what the raw data looks like a lot of the time. These are just IP tables logs and I ripped these straight out from gray noise. So at some point, a long time ago, those were gray noise nodes. Um, it's bad

opsec. And so now I need to figure out, okay, like, well, why, why is what we're doing challenging? What, what are some of the things we have to overcome? First of all, um, you have to get, when you're doing this kind of thing, you need to have a very diverse optic and you have very diverse set of data. Um, you need to have data in a lot of different places and you need to make sure that the anomalies that

you're measuring are, uh, being that they're justifiable in equal amounts of places that you are observing, which is to say that you have to avoid collection biases. If you have one little optic in one place and that starts seeing all kinds of crazy stuff, that is not an Internet wide anomaly. That's just probably somebody scanning your machine or something like that, right? But you need to really be able to correlate it across many

different places that have a different, that are different kinds of networks and things like that. So it's insufficient to just have cloud providers to do this. Really, you really need to have residential IP space as well, business IP space as well, because there is like an idea of macro targeting that mass scanners do. I'm not even going to get

into that right now. Um, one of the other reasons is you have to make sure that you're getting like an unbiased opinion with your data. You can't just install a honeypot on a network that you own because that device has business value. And if something has business value, then bad guys are going to pay close attention to that. They're not going to just like backs. They're not just going to like accidentally see that like you need to have things that have no business value, right? Um, you need a

lot of data and so you need to have a lot of volume and managing that amount of volume can be difficult, especially if you, um, you know, if, if you don't, if you're not used to dealing with relatively large amounts of data and putting them into databases and querying them and all things like that. And then money, like who cares

about this kind of stuff, right? So like for me, um, everything that I need to do, this is all I do. Gray noise is all I do. So like I need to be working, it's for, this is the kind of thing that I'm talking about right now that is like, it's more of like an R and D thing. It's less of like a thing that is easy to package and product and make money with, which is mostly why I'm like just like lobbing it out for free

for everybody. Um, so the solution, my proposed solution to this is collect all this stuff, put it into a database, average it over time, and then when we see any more unique IPs, um, any more than two or three times the normal amount of that, of what's expected over some period of time, then, uh, have it tell you and then do a little bit of research and try to tie it to some kind of event. So you're going to parse out the

data and this is really what you need. You need the time, the source IP, the destination IP, the protocol, the port, that's it. And if you really want to do this on a budget, you can do like, you can have like a unique constraint with this. Like, you don't need to record any of these data points more than once. You really just need them kind of once. Once you've seen somebody scan a box that you own on a certain port,

um, at a certain time, and then they do it again, you don't need to record that again. You just need to, alright, this is fine. This is good, okay? Um, and so then, what we're going to do is given a 30 day rolling average of the unique IP addresses scanning the internet for a given protocol pair, show me if there is an increase in unique IP address

count that is higher than three or four or whatever times the rest of the month's day to day increase. Which is to say, and this is where things are going to get a little tough for me because I am bad at math and I am still grasping this stuff myself. But what is the average number of unique IP addresses on the internet that are scanning

for a given port protocol over the past 30 days? What's the average daily? And also, what is the difference from yesterday to today? How many, how many times is it like a 0.9 X multiplier? Is it a 1.1 X multiplier of that rolling 30 day average? Um, and let's,

and then show me everything that is above four times the regular 30 day. And like, I can see some people right now that are already like, this is so simple. And, and it, and it is, I mean, I know, I know this is easy stuff, but it's really, really effective when you have really good clean data. Um, and so then there's, I mean, there's all kinds of like statistical trade offs, which in this case, like if you decrease the

window from 30 days to like a week, then you're going to be able to get, you know, a measure of the anomaly faster. Um, but it's going to be, it's, it's going to, um, it's going, it's going to, you're going to get it faster, but it's going to be less accurate, right? Um, it's going to be more chaotic and sporadic and volatile. Um,

if you increase the, if you increase the number of IPs that you need to see or above or below a certain number of IPs that you need to see, then you're going to miss some of the smaller anomalies. Um, so how did I actually did do it? I did it in SQL, which is the literal worst possible way that you can do this because I hate

myself and it looks like this. So I'm going to just basically break down this whole thing. Um, so the first thing that we do, I'm not a D a DBA and I'm like, not a lot of things today. Um, but I'm really not a DBA. So from, first of all, we're going to take a window of the last 30 days with the date or protocol

and the day and the number of times we've seen a distinct IP hit, um, hit us, hit one of a node that belongs to us in the last 30 days.

So where we have where date greater than current date minus interval 30 days, that's easy to understand. And then we're just going to mash it all together. But we have a having clause, which basically says it's, it's like just to avoid another gross sub-query. We basically are sort, we're, you know, grouping out in the, in the, um, in the group, in the groups after the group by. And so then we're going to say,

but I only want to see things where that I only want that count to be affected when we've seen some one unique IP address that has touched one over, over one distinct node. So don't show me anything that's only seen, that's only touched one of our devices. It has to be two or more, right?

And that's going, we're going to call that like, so wait, that's having stinks. Yeah. Okay. And then, so now that we've put that kind of into its own little list now, show me the average. I don't understand windows very well.

This is very like, this is very cobbled together from like stack overflow. Um, but basically I want to see everything where yeah, at the end I want to see all of the different, the days, the port protocols, the unique amount of IPs and what the both the month, yeah,

what the month, the average, the 30 day rolling average was for that given thing on that day. And then out of all of those things, show me everyone there at the very end, select date protocol port IPs round times mean show me everything that is

above a four X multiplier on the 30 day rolling average in the last five days. I never want to do that again. How can we make that better by doing literally anything else? Um, probably, I mean, this takes forever to run. It's inefficient. It's in SQL, it's gross.

It's slow. Um, it's not real time. It's, it's limited to dates, not times. We don't have any timestamps. Um, factors, very little information. Like an anomaly here is just like dictated by how many people are looking at something, but there's way more ways to calculate out an anomaly,

like maybe an ASN that's scanning the internet for a certain given thing or maybe a, you know, maybe a, I don't know, like a given organization or, or boxes that look like a certain thing. Um, et cetera. Um, and I mean, I'm sure any actual decent library you can just like cram into this and it would just do everything that I just spent the last six months working on just

immediately. Um, so then the correlation piece of it is like, okay, why does that, like, why did you see that? What, what is the, the, um, the cause most likely to, for that to explain that increase? Most often it's like a big botnet has just operationalized a new

vulnerability that they want to, they want to capture as many devices as humanly possible on the internet that, um, that are exposed to that vulnerability or something like that. Um, sometimes it's because a new CV comes out like heart bleed or, you know, like shell shock and everyone in their grandma is like, Oh,

let me scan the internet to figure out how many devices are vulnerable to this thing. Which is funny because when, whenever this happens, it's always security researchers are that big first wave. You can see them all like using this standard security researcher tool and then like the bad guys are way slower. They come in like months later because it's all these like bottom of the barrel

bad guys that are like, you know, kind of, it's a numbers game for them. They're just trying to compromise as many devices as possible. But it's always like, you know, it's like six months later. They kind of like, you know, gotten around to like figuring out how many boxes are exposed. But like, there's not, there's not a big jump. It's always the, it's always the people in this room that are doing it basically. Um,

or like a worm breaks out, right? It's like, and which we've seen with like Regman. Um, I mean, way back in the day, obviously, um, the, the great example would be, uh, Conficker and so it was six, seven back in the day. I mean, that was, if you were, if we had the same information, if we had the same optic then at the time, then we would have seen on one day,

many people are scanning for port four, four, five. And on this next day, that times a quintillion people are scanning for port four, four, five. Right. So, um, and we see other like worm stuff now, which I'm not really going to get too terribly into right now, but so like some easy hacks, if you want to do this, um, uh, to do some of the correlation piece is just Googling the port number to try

to figure out search, search GitHub to find the port number to try to associate it with something that you're seeing. Um, look on exploit DB for new exploits for a given vulnerability. Um, use Metasploit to figure out the default port number for different things. Um, that's an effective tactic. Look at CVE. Sometimes CVE,

the CVE actual page will contain the port that the vulnerability is on. Most of the time it doesn't actually almost none of the time it does, but one of the resources in those things will, will include that information. Check out router exploit, which is like Metasploit for only for routers. Um, so here's some of the things that we've found. Um,

some of the observations actually like over the last seven months, um, these are just like screenshots of tweets, but I'm going to go through kind of each one of these, a couple of the success cases. So like top left was, um, yeah, 52, eight, 69. We weren't the first one to catch this. 360 net lab was, and um, I have a shout out to them. They do, they do great work. Um, uh,

it was like a universal plug and play service that Satori had weaponized. Um, so we tagged everything that every, all of the new people that were doing that all of a sudden and like figured out a signature to be able to have that tagged as bad. Uh, top right, gray noise observed site 55 55. This one's whack. Um, it is the ADB Android, uh, debug bridge vulnerability. Um,

which is like ADB like in your Android device when you're, you know, doing something, there's a vulnerable, it's not technically a vulnerable, it's like a design thing where you can do arbitrary code execution if you have access to that demon or whatever. Back in February, we detected a big uptick in that of people like looking for it and,

but there was no real like smoking gun at the time. And then probably like a month ago we saw people start to actually exploit it on the internet. Bottom left. Um, uh, Oh yeah. Belkin and seven 50. That was just like a, some crappy router that had some vulnerability that we started seeing just like

this giant thing, giant uptick and we dug into it and we found that. Um, and then yeah, this D link 27 50 B, um, on port 8,000 we saw, we saw 1000% increase from one day to the next day. It was like, all right, something's going on here. Um, this was the ADB thing. I mean,

this is what it like, this is what the actual numbers kind of looked like of how many people on average were scanning for some of these. Um, so this, I mean, the reason that I'm putting a news article in here is not because I'm trying to show like how cool we are. I'm trying to show you that this works. It's effective. I mean, we wrote this and it was like, we like saw this and it matched perfectly with what the rest of the world was

seeing with, um, what some of like other people were reporting on. Uh, the same thing happened. This one was gnarly. Um, this was, um, red schmay, this microchip vulnerability that I think might've been like a vault seven thing. And so this one was, it was a nasty one. And so then, um,

the uptick in this was gigantic. Um, no one was really looking for it. The other fun thing with gray noise is like once we have these upticks, we can look back in time and figure out like who's scanned for this thing like ever, like who scanned for it like last year. Um, so yeah, and then this is just kind of a summary of some of the ones that we've been able to find. Uh,

these are the successful predictions that we've been able to find using this methodology. Um, I think I just went over all these Drupal GED and that was gnarly. We did the same thing with URIs that people request. So like, um, we were able to do the same exact thing to see when Drupal, Drupal GED was beginning to be opportunistically exploited. Um,

we see oral Oracle web logic is like the gift that keeps on giving. It's just, it's like permanently screwed up. Um, and it's just always being exploited. Um, and I think I talked about all the rest of them failed predictions. Uh, there was one maybe like four months ago where we saw this uptick in port

5647 TCP and the people that were scanning for that port also had the port open on their device, which is like a golden ass indicator. And so we were like, yeah, we got, we are clearly the smartest people that ever exist. Well, I was like, I'm clearly the smartest person ever exists. I was so wrong. And so I worked with the red hat people that it affected like red hat

satellite server. That was what it was the default port for. I was like, yeah man, like this is so cool. I'm saving the world. And then I ended up working with their team and they're like, yeah, that's not us at all. Like our devices are never exposed to the internet. They're deep inside of networks. And, um, the IPs that you published out are not running red hat. A lot of them are running windows. I was like, ah shit. Okay.

So then I had to kind of eat crow on that one. Um, blah, blah, blah, blah, blah. But they were, they were awesome to work with. They were really, really great. They were super cool about it. Um, they knew that we were like trying to do the right thing, but yeah, they were like, yeah, this does not affect us. And I was like, shit. Okay. Um, so I just kinda want to get a couple of tips for people. If you're going to be doing the same thing or want to do the same thing,

filter out the known good actors when you're doing stuff like this. So Bob Brutus or Harbor master on Twitter, um, over at rapid seven, like he, he was the first person that I know of that when he was when he to bring in known good labels when he's calculating upticks. So what that is to say is like, he doesn't, if there's a, if there's a good, like if show Dan's out there scanning the internet for a given thing,

and then they add a different thing from a thousand different places, he doesn't want that.