welcome to the blue team village fresh air got a treat for you Chris and plug gonna talk about how to not suck at vulnerability management if you do vulnerability management you know that this is going to be a really good talk and you're gonna be talking about how to know how to not suck at scale so how about starting off with some applause for these folks yeah testing testing hey thank you for coming by Def Con it's awesome and thank you for stuff with my other blue team village it's the first time so it's nice to have a lot of people here I go by plug and this is Chris so as you see is hard not to suck at will not ever learning management so really quick we work for oath which is a new company that is owned by Verizon we officially come from a CDN a portion of that Verizon digital media so we do a lot of stuff at scale and I think this is the reason why we decided to do that so let's just go for it

so to kind of set the stage we need to talk about the Korean landscape and the reason why we decided to go about this talk we're not gonna go a lot of we're going to talk a lot of the technical portions of the of the day-to-day operations but we're gonna probably ask you why you should consider the way you're running your vulnerability management program learning on the spot

so in this case what you see in the

screen is a lot of different companies and what they all have in common is they have been recently been breached well you can see in there is that these are the reasons why they got broken into or that or the reasons why the data was excavated or anything that happened to them that got our data exposed so you have a patch of availability you have backends that are exposure to the internet on secure servers software bugs and all of these things are fairly easy and they could have been or actively monitored by having a really good vulnerability management program in

addition to that do labs release these report basically talk about the cloud and all the finest that they have so if you have the cloud well not a million management applies to you and you should be aware of that we set the stage and the reason why we're doing this because we do security and is very important that will net a million management gets

the right resources and the attention that are required so we don't suck at it

now the one thing I really want to show you a scare is that management is not a check mark it's not a compliance check mark it does what you think it is you're doing it wrong it's not about that it's a lot more is not easy it's really not

easy it takes time it takes a lot of iterations and it's important you understand that because you're gonna go for a journey we need to set some goals

before you begin opponent up well let's actually asked some questions who's doing vulnerability management here all right that's fair those in response or who's in a blue team here and they kind of blend together management program one of those set some goals and you should be very simple with these goals and we're gonna define them this is what we planted so

let's go boys we have royal time identification that was the goal how soon can we identified a vulnerability and how fast can we triage that vulnerability the sooner that you know that there's a vulnerability the faster you're gonna react to it and the bitterness issues you're gonna make so now let's coil number one is faster edge what is good you have an incident you have to make decisions you have to make decisions real quick so management teams should do it too and this is the portion where you're gonna invest most of your time from early mitigation and

remediation what's the point on when all of this work if we cannot mitigate or ladies these are the tribbles and these are the tools you should consider you should work on your program or out recommendations all right so now a bony

management comes with some challenges and we want to address those and the first one is there are a lot of resources from which you get moment of all the intelligence what is vulnerability intelligence well that's a thing that we use internally but basically is any source where you can get information about a vulnerability blog post builder mailing lists and anything else that will give you that the information about a vulnerability so there's a lot of sources lots of information and that information is difficult so that's the one challenge you have to address the second one is building up actions that might not be available when I won't let alone it comes out or you might not be able to patch so you have to have any strategy to how to deal with that and you have to compensate for that so it's an important challenge and we're gonna try to give you tips on how to do that any further

layer there is one thing I'd really wanna address in here which is something called communal and LOD it's called system also known as cbss now this is what you see in the news all the time just come out then the thing is you should not trust that number and you should because it's just that it's a number in this context context is gonna come from a lot of the things you have to understand in your organization so you know wisely but don't go by it okay some context while rushing to over this course is to get a 5.0 good most of us traded like a 10 for more depending if you were to go - core system you would have done wrong I love that hi abilities that are on five that are very important that you might not be paying attention second ability that's the one that basically knows SMB right that's the one wanna cry use to score systems because they do is color systems you can use version two or three Masha is nine point three on this core system Russian 38.1 I'm gonna do something that affects usually the

system Linux very nice course this information about the vulnerability what do you do are you waiting for this course to come in and then do something about it so these are examples and you can find a lot more so please if you're doing management wanna take in consideration discord system use it as analytical value to aid you and provide context in here just you need an order to drive your program you are going to do a lot of things wrong now you have some prerequisites right things that you need to gonna have or you should have to do your program better this is king and you should use it in maximize it and take advantage of that if you can there's a lot of open source tools and you can use to your advantage and you should look for them at scale is to keep track of your IP addresses you know you own or a block surprise for you addition to your records and as you

actually what you're gonna try to do you

can kind of break down the things that you need to know into two categories external intelligence and internal intelligence hello sorry yeah you can break down says kind of two categories of things like internal intelligence and external intelligence so external intelligence is generally gonna be like public stuff so like us ends or red head security advisories and security bulletins for Microsoft things of that nature your need to parse them to make sense in your environment and your having internal intelligence which is essentially going to be looking at your assets and making profiles upon them in some manner to make sense to you that way you can relate the two together to figure out what which ones are your things are vulnerable and which ones aren't and then based on that data you or your tool or or whatever you gonna do you're gonna you're gonna try to go out and drive some remediations the important thing though is that I always remembered that you shouldn't you shouldn't get bogged down and trying to relate things together instead you should try to like automate as much as that as possible to make it make it easier so internal intelligence it's

kind of a buzzword out there you'll hear at other places it's it's it's nice be

be picky about the external intelligences that you're gonna look at like use the ones that are between parsable ideally and that are most most useful your environment we're in a Bluetooth shop so we used a bunch of security notices and we parsed them out and we were able to like pull some really good data out of it and then like compare against their environment they're almost all going to require parsing of some sort at least you know relate it to how makes sense to your environment so like if you're looking at you know like I moved to security notices you might want to you know also parse various bits of pieces of it yeah now there's a bunch of vulnerability intelligence feeds that you can use bunch of tools you can go direct to the source so like Red Hat security API or you can you can use these tools and then you can also there's there's a bunch of places you can get them so you know take a look at them peruse them find the ones that make the most sense to what you're trying to do and and you know pick one and start with it and then add to it as you go on now internal intelligence

isn't a buzzword it's something we just made up essentially the the most important part is that you're being accurate with the data that you're collecting from your own assets and that you know that you're you're collecting as much as you can about it you don't want to have bad data or you don't want to be incomplete you're always gonna be a little incomplete because that's just the way life is but you want to be as complete as possible and this is really we're gonna build a lot of your integrations with other tools like if you're if you got if you went on you bought a tool for this or whether you're like you're trying to integrate with another team or something this is this is where like all the pieces are probably gonna end up coming together

now there's there's a bunch of different ways to get internal intelligence - it's not just like I'm looking at all my servers you can grab like stuff from windows and like your network devices your domain records things that the things of that nature flow data is a big one that you can look at one of our one of the guys on our team who's not in this talk he does a bunch of great work with flow data and you can you can really learn a lot about what's going on here network that way so you know just there's things to keep in mind once you once you have these metrics you're gonna want to create metrics and data based on it it's gonna help you figure out what you're doing what kind of visual creatures so nice graphs are great and then when you when you have that data and you're trying to make an argument to someone to do X or Y or Z having the graphs and having the metrics they're gonna be nice and you know graphs are nice everybody likes graphs you try to keep in mind who you're looking at like this graph here is an example of one that doesn't help you make your point because just because there's too much there so try to try to make it look nice because it makes it easier to sell your stuff upwards something something about like this actually one of our co-workers who's in the former co-workers in the back actually made this set of graphs why should really helpful in and getting some things action by the businessman they could look at you know breakdown of what these vulnerabilities were or whether it came or their categories and things of that nature so you know simple is good yeah

[Music]

[Music] information that information I don't think so this is a cologne mistake that I see in different places there's no reason to have that information I'm gonna ask you for detailed information break it down but it will be much more innocent to summarize is this you know I was and that's it many like the significant version numbers now take a look at these

on the screen it's super easy to read a lot of important things it's just plain a sample whatever it is and everything like the more significance better right there how many of them do you have and also what is what is what is the risk on them that we pre score and it's super simple for someone to see oh my god I need to do something or I'm in good shape so please do yourself a favor and they're really good graphs or pay attention to your audience who are gonna read them these are the key for you to have people remediated things if you don't know them right you're not gonna get to remediate things properly

[Music] trusty tool which she is the spreadsheet you can actually use a spreadsheet to run a lot of this stuff on your program or your program so when the spreadsheet because a lot of stuff learning the Dillion comes in spreadsheets so it's a very useful tool Maxim I said use it and abuse it now interesting thing which is

scanning including to talk but after having people have a lot of questions and it's very important to press them so what are some things about papers about scanning you should install the discovery scans you should be very small take a small subnet do as moles can use very simple ports there's no reason for you to scan 65,000 for TCP UDP it's gonna take a long time it's gonna fail you know you're not doing any favor so it's most cans smoke ports if you if you pass that then you have the common core is that come on any map and use them right to figure out what is your what you have to your inventory so you know that you're kind of dipping the data to make sure that you're just calling the things that you should have or not the first time that exists in there so do yourself a favor it's Korres cans we want to blend ability scans you have [Applause]

it's a poor thing you have to ask yourself security it's gonna infrastructure make sure you secure that because that can be used for bad so you're gonna have any infrastructure secure it control it six it comes with a little challenges but there's a link in there than you can use to understand heaven you can scan ipv6 and then I can talk about it after the fact boys can type in a six and it's a lot of them so I can use some strategies simulations so before before you start your program or as you said your programming it's it's a chance yourself my name is told I am so why more thing you ask yourself is just use by something taking about the case when I explain that is not true you have a combination of toys the president approach it is use one toy I said today and hopefully you can use that hey hey so for when we were

doing our Volm Anna German process one of our former co-workers he he built tool called jellyfish internally hello it is Man O'War because there's another Python tool called jellyfish so we didn't want to encourage another name you could use to kind of manage like your internal intelligence side of the house where you're grabbing like your profiles of assets and things of that nature and storing them somewhere to analyze against the version that we made is missing some of the helper tools because it has some business logic in there I'm trying to get those open source maybe later this year you know when one time allows or essentially when reaching out to all of our servers and we're rounding a bunch of data back about them and we're in a database to be analyzed so the top of there is an example of sort of the data that we're collecting from a particular host the bottom line is an example of what we call in on it that's based on aus n there and then there's some graph said that at the end that doesn't actually make any sense but we have we went out to the USN site and we pulled down the data and we parsed it and then we compared it against our environment of profiles that we've grabbed under the hood I use I say vector so that when I talk about it you can just treat it as a crazy alien guy but I want to shout out Barron sorts of vivid cortex back in 2015 he gave a talk on his toe with the cortex and it gave me the the inspiration to write this part so and he used vectors in his thing so factors the the beginning data part

of it I called profiling we store a bunch of data about servers in like a 3-tuple so we keep collection type subtype and value then we record when we first saw it and we've also recently seen it so we'll have like essentially a first date seen in the last date seen for this tool it's important not to record like performance metrics that change all the time because otherwise you'll just be storing a bunch of different vectors and you won't get any of the benefit of storing the dates we also store IP data our network uses a lot of VIPs so there's very nice to be able to store just a bunch of VIPs that are associated with it and then we have the ability to like customize the stuff that we're pulling back so we have a bunch of essentially behind the scenes is just a bunch of commands that are bash commands that push up pull out a piece of data that makes up our collections you can see kind of an example of some of the stuff like that CPU - info is a CPU info command and like boot time is just it's a parsed date command it has like the time that that when it was booted and then like we pull it back into our interface and you can see the bottom one on the right there and we format in a nice table so if you want to go look at the stuff and then we we build a schedule to go out and ssh into a bunch of hosts we also have an agent still very experimental but it's in the it's edition in the code so you can take a look at it and improve upon it I'd pick poll requests so that's nice on

the audit side of the house this is where we're grabbing data from our external sources and then this audit part is what's kind of merging the two and doing the the comparisons essentially I'm breaking on my host set into a group or a bucket so I have a series of comparisons I decide which hosts and then I do a comparison against them to actually see like if it passed or failed the audit itself that's sort of the idea ideology I've followed when I built daily Fisher man o'war and it works pretty well I haven't opened sourced the part that automatically pulls stuff down so this is actually a live audit and I had to move some of this stuff because it really feels stuff but it should be alright you can see that it's essentially the top one there is pulling out 1404 host and the bottom one there is doing a comparison against a recent vulnerability and you know making sure that the version is higher

so you can also do other things in the audits like Intel to view reg X's or white lists or black lists and things of that nature so we have a few of those in there and then we display this stuff and like I said the USN scraper is still still coming also we built some api's

there's a lot of ApS in there so you can like integrate other tools which we've been working on we have one that's trying to you like statistically model like the differences in our servers trying to get that open source to but that's that's still coming and then like there's web pages allow you to do simple searches so you can go like Huntford like servers that have get installed on them or hunt for servers I have a package or I have a particular CPU type or things of that nature and then oh yeah so then along that

bundu specific logic and our environment specific logic that needs to be generalized before I can really some of the helper tools but effectively it should it should work if you write your own audits but yeah talk to me afterwards if you want to see more of that type of stuff and then if I were a smarter man I wouldn't actually be presenting that award I'd be presenting one of these tools specifically Hubble step would be it would have been a great fix for what I was trying to do if it had existed when I started working on this tool but there's a lot of great tools that will help give you data about your environment could tell oh if you're if you're a Red Hat person like that's built in like Red Hat satellite or the open-source version spacewalk OS queries a great one face back makes it and uses it and it goes out and grabs a lot of things - yeah set Zeus and if you're a Windows person wsus should give you quite a bit of data you'll probably have to pull things out if you want to do more of the threat hunting type stuff but you know it's there you're always going to need to customize whether you buy or build or whatever the case is and there's there's no one size solution to fit all so just you know make sure that you you're willing to do a little coding on the side or scripting at least to make it all integrate nicely because it really allows to do vulnerability management at scale if you do try it it will really make sense a lot of the stuff that we're telling you those goals the tool allows to do that so one of the things that is not very clean here is that with that tool one of the things that we can do is an inspector bull and I will just come out we can actually check our CPU and decide is that impacted and if so how many of the centers that we have are impacted and we can do it almost like this so that is really good because we can actually decide how we're gonna mitigate that and a patchable interval it comes out no problem we can actually do that so find a tool you're welcome to use this tool hopefully will be as useful to us to you that is to us and you have any you know if you have any concerns questions to solve what is known in the back but I want to also ask you something else early on we mentioned that there are all these companies that had compromises and all of them had also something in common they use the cloud and some other services so one of the things you should consider is management 2.0 which is go beyond the scanning we all have elastic that is one of these [Music]

should ask questions applications by our organization they know more than you and your service maybe even ask them for questions you can ask things my friend of them something asked nation shapes your negation will be better much more effective especially - at scale this is a difficult subject

interact with your organization to get things fixed and you pass like you have the make you a ticket I throw it over the wall and sometimes you want you want to you want to like shame people who know your tickets try to avoid that because then you would sent device them to avoid the shame and not necessarily to fix the issue sometimes you do have to be the bad guy and call people up for not doing your stuff if you're if you're using the the ticketing method but that you know try to avoid it if you can and then the other path is sort of self-service oftentimes if you give your org the ability to look at some of your vulnerability data and look at some of the findings and going back to the metrics and stuff we were talking about if your graphs and and and things are understandable you know they'll go out and they'll actually go fix them for you and like they they will preemptively like before you get to the point where you have to like go after them they will be able to go and they will attempt to to fix it keeping that in mind though accuracy is important in these things if you have a bunch of false positives and they go out and they put a lot of effort in to take a look at four or five issues and try to go fix them and then they find out oh they're all false positives they're probably gonna stop looking at your stuff so you know if you want to go this self-service right you have to make sure that you know your accuracy is is right and the bones you call out are actually real and actionable things and then so some of the questions are like should you remediate it like that going fixing the root cause or show you do some mitigation sometimes it comes down to like what can you do like oftentimes and you're doing foam management it's getting the answer is gonna be patch you know nine times out of ten if you come across a phoner ability the answer is gonna be patched but sometimes you can't patch for various reasons in that case you're gonna want to look in the remediations and then document your decisions not just because you can like hand that to an auditor and all the auditors do like it but also because when you add new teammates you can you can give them your your list of your your repository of documentation and then when they're wondering about how you did X in the past they can go just search and look at it and it makes it easier to kind of record what you did in the past and how well you did and then sort of kind of iterate upon it and get better each time you go through it it happens to all of

us in security and the reality is that one day one day things go horribly you know things go bad really really bad and the advice that we have for you is very simple if things go bad first don't panic you know the only one it's bad to burn shame don't blame anyone don't play that game it doesn't work the most important part is that find out what was the cause and do lessons learned that is critical for you when we're doing good lessons learn is very important that again you don't panic and you don't shame if you don't know that you're gonna learn from the mistakes and you're gonna be able to get better at what you do or what we do which is securing things again that applies to everything but one day one thing is gonna go horribly wrong and just keep these advice in mind now

there are some next-level ideas let's say that you you many of you maybe are doing already all of this you feel like well this is nothing you know I already I'm doing this well we're gonna give you some next ideas you know what can you go to increment and make your vulnerability management program better well why don't you gonna fly the program

I mean we have CTFs right people are capturing the fly well why don't you capture the vulnerability if you'd remediate those things that you can actually do you can create his chords for teams the ones that are doing better everyone loves swag right I mean someone's you got swag in here well you think that the people that are in a DAT bubble net ability is one so act to what they do in if you do it they're gonna love you and they're gonna want you to batch faster so if you can I'd recommend it doing that as well you really want to get into the animation business it was really gonna make your life super super easy not all the time but he's gonna make it fairly easy so looking at orchestration tools that can actually help you and do that if you have more cost-efficient tools and if you work at it scale is very likely that you already have so you have to secure

[Music] [Music]

sorry so yeah so the these are the final

tables if you just kind of spaced out for the rest of the talk of that these are the things we think you should shoot yes so like sharing work with your owners when you're in doubt go ask them blindly trust upstream scoring if you are going to lay on cbss scores try to please make sure to use the the the tempo oil and the environmental scores and those scores will change over time as you know time changes and as your environment changes validate your data make sure it's good and improve incrementally it's okay did not be great the first time around just you know get better every time you do stuff and the most important part is that if you're doing a building management like you are one of the most important assets so don't get bogged down and doing in like looking at like one mole that you can't quite figure out you ain't got a stick and move some time and get a little Mohammad Ali going so I think that's that's it we're going to be in the back taking questions I believe we'll probably be out by the bar so you know um meet us there Thanks [Applause] [Music]