Is it 0-day or 0-care?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 122 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40615 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Open setDatabaseVulnerability (computing)VolumeType theorySpacetimeInformation securityMetric systemSign (mathematics)Computer programTrailFreewareWeb applicationSlide rule1 (number)Vulnerability (computing)Web browserSemiconductor memoryDifferent (Kate Ryan album)Information securityPoint cloudNumberLevel (video gaming)GenderBlind spot (vehicle)Cartesian coordinate systemDependent and independent variablesStatisticsElectric generatorExploit (computer security)Software bugSource codeProduct (business)Profil (magazine)Type theorySweep line algorithmDirection (geometry)Endliche ModelltheorieWordComputing platformArithmetic meanDrop (liquid)Online helpBuffer overflowMobile WebQuicksortExtension (kinesiology)Right angleComputer programmingChannel capacityProcess (computing)Multiplication signScripting languageMathematical analysisVirtualizationWhiteboardConnectivity (graph theory)Context awarenessInternet service providerOrientation (vector space)Core dumpInstallation artSoftwareWindowBridging (networking)Physical systemData storage deviceDatabaseTable (information)Figurate numberInformationEnterprise architectureSoftware developerTraffic reportingRow (database)Goodness of fitPoint (geometry)Moment (mathematics)Software testingBitHacker (term)Intrusion detection systemRule of inferenceLine (geometry)RootCausalityCASE <Informatik>Instance (computer science)System callMilitary baseTwitterReflection (mathematics)Service (economics)View (database)Port scannerAreaPatch (Unix)Perturbation theoryCodeEntire functionTrailVector spaceHecke operatorElectronic data interchangeSocial classCloud computingDecimalMereologyCodecInternet forumInjektivitätConsistencyVirtual machineMobile appPay televisionRevision controlBlogLaptopFlow separationSurfaceLogical constantRaw image formatBasis <Mathematik>Term (mathematics)Pattern recognitionPeer-to-peerMessage passingDenial-of-service attackComplex (psychology)InfinityGroup actionElectronic mailing listTotal S.A.Limit (category theory)Real numberCivil engineeringTransport Layer SecurityExistential quantificationWebsiteWeightMathematicsPlastikkarteGraph coloringGame theoryEmailMetric systemCountingCalculusOverhead (computing)Position operatorLogic gateError messageCondition numberLocal ringGreatest elementCalculationAdditionProjective planeVarianceParameter (computer programming)ACIDPresentation of a groupData conversionLevel of measurementInformation technology consultingWritingPlanningCrash (computing)Expert systemDivisorPerspective (visual)Online service providerMotion captureBuffer solutionGrand Unified TheoryComputer-assisted translationPrice indexVector potentialFreezingPersonal identification numberConfidence intervalCross-site scriptingMedical imagingBit rateRemote procedure callCombinational logicEnumerated typeDataflowException handlingMultiplicationFreewareDisk read-and-write headValidity (statistics)Modulare ProgrammierungSystem administratorSet (mathematics)Reverse engineeringHypermediaCategory of beingDiscrepancy theoryUsabilityFront and back endsDescriptive statisticsSearch engine (computing)Barrelled spacePoisson-KlammerMeasurementProof theoryMultilaterationSubsetInterpreter (computing)Formal grammarCoordinate systemUniverse (mathematics)Inheritance (object-oriented programming)Volume (thermodynamics)Key (cryptography)Hydraulic jumpFlock (web browser)Phase transitionResultantUtility softwareIntegerArmAxiomControl flowWeb pageHidden Markov modelBuildingPower (physics)2 (number)Differential (mechanical device)LoginFinitismusBackupStandard deviationOpen setBoundary value problemMeeting/Interview
Transcript: English(auto-generated)
00:00
Thank you everyone for attending our panel today Usually I find that I get these like 8 a.m. Slots and it's really awful So I'm very very happy that it's a late slot. So thanks for everyone who has hung in there with us So over the last couple years several of us have been talking about Vulnerabilities in depth and we started down this path of saying does anyone really care about vulnerabilities anymore?
00:25
And it turned into this conversation. Well, it was zero day Well, then it's really important but otherwise maybe no one would care So we want to get some people here to get today. I got a pretty large panel of folks So I'm gonna try to keep them reined in as best as possible
00:41
And I want to get some people that had some, you know, good thoughts But we're really easy to manage that weren't that opinionated that wouldn't go off on rants and if you know anyone up here We'll see what happens, right? So Yeah fail already. Thanks for that So anyways, hopefully this would be a good panel for you I know a lot of times panels can be a real pain, but we're gonna try to make this interactive
01:03
Get some good topics that maybe aren't discussed as they should be So the first thing with a good panel is getting the right people up here And I think that we have that hopefully, you know all these people if you don't I'm not gonna waste any time in the session Introducing their bios. We got Brian from OSV DB Steve from CVE
01:21
Carsten from Sakuna art from cert Wait, man, Dan from HP tipping point Katie from Who's whistling for Katie or Dan? Katie for a Microsoft and Alex. I don't know where you are anymore. So I'll just call you Alex
01:41
All right, so we're gonna kick it off We're gonna try to do four questions about ten minutes each and then at the end There's gonna be some free-for-all if you really feel like you need to yell at one of the panelists There's a microphone come on up. Otherwise, you can hold it to the end your call So the first one we're gonna start off with is Does anyone that doesn't work on one of these things really care about vulnerability databases tracking or trending anymore?
02:03
And I'd like to start off with maybe Steve or Brian you guys want to pop your People care, but they don't know they care Near as I can tell At least on the CVE side. We rarely get many many complaints about About what's going on with us
02:21
Sometimes we have like blatant errors in CVE descriptions that we never hear of from from anybody So in a sense people don't necessarily care it seems like a really easy job to just like comb through thousands and thousands of vulnerability reports every day that Always most of which have one of the following four properties, which I call the four eyes
02:45
A Vulnerability reports are either incomplete They're inaccurate they're Inconsistent especially say between a vendor report and a researcher report not like we ever get those kind of inconsistencies
03:01
Incomprehensible some some of the stuff that we're dealing with comes in in a Broken English from you know people who live in the Midwest Poorly formatted advice Advisories where the more the most serious vulnerability They don't even quite realize what it is and it's buried in like a single sentence in a you know
03:22
Three-page screed about you know what they had for breakfast or something That's the kind of raw information that we have to deal with on every on a daily basis and Unless you really deal with that kind of stuff on a daily basis. It seems like really easy Oh, just scrape all these websites or taking all these emails and then just do a little bit of analysis in
03:42
You know in two minutes and then push this thing out Unfortunately, there's a tendency that all of us in the vulnerability information industry have Which is we kind of care about correctness and quality and in some ways This is why I think we're having this panel now
04:03
because Quality comes at a steep price so you want to talk about CVE and Are we still working on CVE these days? just Get right out and ask me is CVE dead no
04:21
Prove it. However, we are going through we are going through a Change so we're kind of in a cocoon and we'll kind of come out like beautiful butterflies or whatever But there's there are a couple realities. Okay. First of all, they're just the raw number of vulnerability reports coming out is Increasing significantly the complexity of the vulnerabilities that come out are are really difficult to sort of capture
04:46
Yeah, we could go along the lines of other people and just call things memory corruption Which is really code for some kind of buffer overflow that we don't really know how to describe exactly But at least on the CVE side what we've been doing is caring a lot about that kind of the
05:05
Academic strength aesthetics of what are the real root causes lying underneath these vulnerabilities? So so terms such as memory corruption We actually try and dig a little bit deeper what that means though is that there's a lot bigger analytical overhead in our pursuit of
05:21
Correctness and those of you who follow CVE on a regular basis. You may see Increasing levels of precision increasing levels of correctness, but it's come at a pretty high price One of which is the the entry to have people on our team Gets a little bit a little bit high in terms of the technical skills that are required And then and then the other price as well though is that you know
05:44
We're at a constant level of funding and the numbers of vulnerabilities are getting Reported or you know more vulnerabilities and more complex vulnerabilities. Something's got to give and in this case recently What's been giving is the actual number of CVEs that we've been that we've been publishing but this year especially
06:02
We're working a lot on modifying our processes To change that ultimately Brian. Yeah real quick Oh SVDB has two different ways to handle this the first one we tried was crowd sourcing So we would put memory corruption in the title and we figure hey, there's a lot of smart people out there There's a researcher who whoever else could clear it up. And of course, no one did so after a while
06:21
We went with our backup plan, which is just let second you do it all Any comment I just wanted to comment on whether does anybody actually care about bones It seems like everybody wants to talk about infrastructure, you know virtualization and cloud and mobile and puke so
06:41
At the end of the day all these various infrastructures are all just delivering applications Applications have vulnerabilities. It's it's the same thing getting delivered. It's still a web application It's still a web browser and they're full of bones and just because the infrastructure changes doesn't mean the vulnerabilities really change that much There's actually a blind spot that exists right now with respect to cloud-based services and
07:04
Services in general which is that us as vulnerability information sources don't cover those You know if there's a if there's a vulnerability and you know, the Google, you know, Google search engine or something like that You know, maybe it can be used to hack millions of people or whatever, but that's a that's an online service
07:21
That's not deployable software that goes into the enterprise So while I think we've been doing a good job in general tracking trends that that is one area That's a really big blind spot and it's gonna get worse and worse as the adoption of services increases I completely agree because if you look at the trends that have really occurred over the last Five years or so so many of the things that have actually moved our web base whether it's browser app
07:46
tracking a trend like SQL injection actually becomes quite fascinating because it actually moves as opposed to you know Some other stuff from Microsoft for instance When we got to the care like at least we hope that people care and that all the efforts that we make and in vain
08:04
At least there are some people attending today, so it's not completely co-care But sometimes we see ourselves as making an allergy like we're providing electricity. None of us are sitting in here now Excited about their slide, but we will be complaining if there wasn't
08:20
And the same when we do the vulnerability databases When we correlate all the information when our advisories get out when they're correct When everything is this is as it should be it seems like no one's really paying attention It's not the type of thing where people say oh, that's excellent. That's good But if we send out an advisory with just like a small typo
08:41
We within five minutes actually have a haha mail sent to us So at least with us we spend a lot of time making sure we get it correct Everything from the analysis of the core problem in the vulnerability down to yeah spelling So I think we kind of live in this I mean security people kind of live in this world where
09:01
Vulnerabilities each vulnerability, you know, it's beautiful snowflake right each vulnerability Yeah, they are each one unique But you know, it matters a lot to us as security people and security people working inside of giant mega corporations It matters to us each phone matters But to the broader world and you know, maybe to some people who call themselves security people, but don't actually care
09:23
You know, maybe they're talking APT in the cloud, whatever, you know Puke as you said right, you know, maybe those people they're thinking of themselves Well bones don't hone people exploits pwn people, you know, so they really only care about exploitable bones They only care about exploits. So, you know with all of these vulnerability tracking databases
09:43
I think for a lot of the population what they really call out of those databases is What's exploitable or what has an exploit out there as in what do I need to get off my lazy butt and deal with? Right now I love that because I think I saw two tweets on the zero day in WordPress And all of you people keep retweeting that metasploit for is out. So thanks. I know
10:03
Yeah So it's cert we we had the same same feelings as Steven and Carsten We wanted to be correct and count every vulnerability in the world Within maybe the last year or two I've come to the realization that that people do care they do because they want to count the vulnerability call it something
10:21
Scan for it. They want to be in compliance See if they can patch for it. They need to name it something sadly They don't really care how accurate the advisory is To a large extent if you're really gonna you're doing something hot and it's zero day and there's special mitigation advice And you got to do something, right? Maybe it matters But I think the big need is just having a label on the thing and being able to talk about that
10:44
And it's the same label so that we're all talking. You don't have to have eight different IDs You got one ID that you know rules them all which is probably should be see the places where accuracy has come into play At least what I've seen at CVE are you know, all this root cause analysis I think is kind of cool and and and I've had kind of a mindset of well
11:03
This may help influence how people think about vulnerabilities as these darling precious snowflakes that each and every one of them is But but when we do when we do get complaints on the CVE side of things in generally, it's it's two things Either the affected versions of the software. We might be a little imprecise about
11:22
and then characterizations of the severity of the issue ultimately the CVSS score and That kind of make that kind of makes sense, right because that's ultimately what people seem to care about It's how well what is gonna be the impact to my enterprise? I don't care if this is some you know brand new really really cool attack that deserves a pony
11:41
Is it gonna hurt me or not? This thing is is a nine point eight. I have to do something Yeah, it was a lot of that mindset in the enterprise user sort of community. So can anyone up here? Does anyone know how many public vulnerabilities were disclosed in any given year? I think OS VDB might be the closest these days
12:01
well if you're talking about just the overall numbers one of the reasons that We say that we have an accurate numbers because we're the only ones that abstract to the level we do where every single specific vulnerability in every script gets its own ID and everyone else across the board CBE, Secunia, everyone says no, we're gonna lump them together and
12:20
Yeah, that kind of moves into some of the stems fighting words Yeah There are lots of different ways of there are lots of different ways of counting vulnerabilities I'm not gonna dispute that you guys have a lot better coverage Than and any of the other sources up here because you guys really try and track everything and you know You guys put in really long hours into it Also, your analytical overhead in general is very minimal your work goes into let's compose a title
12:45
You know have fairly simple ways of like breaking things down That get a little bit more complex at least for at least for CVE when we're dealing with for example shared code bases And we're looking at two different bugs we may intentionally combine them in other cases we kind of have to look a little bit deeper to figure out if we need to
13:03
split them or not because In some cases, you know if we have two CVEs that are out there that are duplicates That's kind of okay. But if we have like one I hate duplicates, by the way, don't get me wrong But if we have one CVE out there that kind of combines multiple issues
13:21
Inadvertently then the utility of the CVE Goes down if people vendors are only fixing like one part of it and not the other part. So That said the way that you guys have structured things I think is really good because well You can be the closest to counting the total number of vulnerabilities that are disclosed using the way that you count things
13:44
Let's let's move to the next one. Hey, which talks about the trends and the tricks This is just a natural flow and plus, you know, so what are the vulnerability trends? It was average 8,000 a year My point being who cares if you can't even put a measurement on the number or how bad our vulnerabilities
14:01
I mean, we're just making stuff up which I'm gonna cover that. Okay. All right. Is that the next question? Yeah. Yeah So what are some trends and Vulnerabilities disclosures types volume that we're seeing and then are these security metrics even worth a damn So so what if you count this many, what does that really mean?
14:20
so so the first question is who out there thinks that Vulnerability statistics are helpful and useful and you actually do currently use them in any capacity anyone I'm sorry. I mean you are absolutely does this slide help you?
14:42
Yeah, my panel doesn't know the slide it's the The vulnerability counts from OSB DB. Okay, so one of the things I wanted to bring up about vulnerability stats Just as a quick idea There were eight thousand three hundred and thirty seven vulnerabilities in 2010. Does that sound like a useful statistic?
15:03
Only compared year-over-year. It's it's over time that I think it does become useful Okay. Well, I mean like the media will call up and ask us, you know, how many bones were there last year? So they knew what was the previous year, right? Well, then they will get to that and I will too So, you know, I have to footnote. Well, that's according to OSB DB. What about other VDBs? Well, there were three thousand six hundred and forty eight according to Sakuna. Well, wait, why the discrepancy?
15:25
Well now you have to get into different kinds of databases Sakuna's database is geared for a very specific use they have an entire customer base that Actually uses their database for day-to-day patching Notification, you know, it's an entirely different system than OSB DB where we're looking for long-term stats history
15:43
And we abstract the way we do. Yeah, like our customers They would like flock me publicly if we started doing what OSB DB do and send out one advisory for every single issue But they care about and how people like the idea of how to use our database is how many like
16:00
Actions do I have to take so if there are ten vulnerabilities being fixed by one patch? Then they just want one advisory listing the ten vulnerabilities and the patch they want to know which product is affected How many vulnerabilities how critical are they how do I fix it and Then there's that subset of people that actually care about the call details of each vulnerability
16:23
But they just want to know how do we fix it so we they don't want to have ten Advisories that tell them to install the same patch Right, and that's why it goes back to that is that OSB DB would be horrible It would be useless in that situation because it would saturate them So after that, the next question is that figure more or less than 2009?
16:43
And so I say there's seven thousand six hundred and seventy eight vulnerabilities in 2009 less than 2010s total according to OSB DB Next question we obviously get is is 2011 on par with the last well There's three thousand four hundred twenty seven as of July 25th. This does not appear to be on par so now all of a sudden we have one number that we start out with that as soon as you start putting
17:04
Context around it and you start even looking one year either direction the the stat starts to lose some of its meaning There's also an assumption in those stats that your analytical capabilities are keeping on par with the publication sources that you're monitoring So for example a couple years ago, I think you guys started scraping almost the bottom of the barrel looking on
17:24
Various various sites that no one else was looking at right that affected your numbers For that year and then the year future for us on exactly people do a lot of CVE based analysis counting the number of vulnerabilities without recognizing that You know We don't have the complete coverage that we used to have due to some of our actors that I talked about earlier
17:45
Not not to be a pedantic ass but the stats don't lose the meaning stats are just numbers What loses meaning what your problem with meaning is actual the model you're inferring meaning? In what you're doing there? Okay, so it's not it's not the numbers
18:01
It's your quest for knowledge out of the number right? Well, it's interpretation of them. That's why I say it's all about context All right, right Exactly, and I think that's what Dan I was just gonna say Dan Guido has been pointing out lately Of course, which vulnerabilities do you actually need to worry about? I think where the context and the numbers actually mean something is when you then pair them with attack data And then you can see are the trends similar completely different, you know
18:22
Yeah, active X disclosures going nuts and everybody's getting owned with active X same thing with SQL injection, but not the same thing with Pick whatever vulnerability type, you know So that's that's where I think it actually you get more of a whole picture because people actually want to know how am I getting? Owned I don't care if there's a vulnerability out there that attackers don't use
18:41
Yeah in a sense in a sense it's it's it's too bad that the media can't ask you smart questions Well, that doesn't make the numbers any less useless Well, the funny part is is that they ask these questions are like, you know Is there anything else you want to contribute to this and you know next morning? They wake up and they have a 17 page mail from me explaining all this and they're like, okay
19:00
Thanks for your time, you know the hell if they're gonna write about it This is the case It's the case of looking for the keys under the streetlight instead of where you think you kind of dropped them, right? This is the only data that's out there. So people people are looking for it So real quick to jump back to my example, you know, if 2011 isn't on par with 2010
19:21
The question is why you know, so all of us we know some of the reasons there's trends Look at phases like cross-site scripting SQL I there's certain years where people jumped on the bandwagon There was the DLL injection on Windows platform where everyone was finding software that did that You know low couple years ago with all the image ones image ones So that's low-hanging fruit that'll swing the totals. There's change in desires to disclose, you know a while back
19:45
Everyone's like well shit if I release an advisory that becomes free advertising for my company Eventually these companies realize wait, we're not getting business and then researchers are like Oh ZDI They'll give me all kinds of cash and hookers and blow for this, you know So all of a sudden they have a very different desire to disclose in the way they do it
20:04
There's bones we know about the time We have to dig into it Like Steve said there was a few years where I was a consultant and you know I didn't work a whole lot enough to live and I spent all my other time on OS VDB and those years our Numbers jumped dramatically because I was scraping change logs I would go through like the Apache bug tracker and if you've ever been in that thing with all of their projects
20:24
It's crazy. And yeah, I was the dumbass that actually went and said, okay I'm gonna search for the word security and start reading every goddamn ticket Apache's ever written with the word security in it pull out every denial of service every stupid little vuln Every race condition local permission area you name and we put in our database You still have to make a guess right because half the time it's like six words, right? Some of the big permission problem
20:46
What does that mean? Does that imply security issues or usability issues? Yeah, so not only do we have to write the entry then it was a techno disclaimer Says due to the vague wording of this. We're not sure if it's a security issue You know you add this up and yeah all sudden the numbers jump So then we get to what I call the security metrics factor who in here reads the security metrics mail list
21:07
anyone Yeah, okay stay the hell off of it that is the biggest waste of time of academic masturbation you will ever see As soon as you get close to a real statistic these asshats like Fred Cohen jump in what's a vulnerability
21:28
Wait, what do you mean we have to define vulnerability now, so he does he'll go down this path it Well, you don't know what a vulnerability is and I say well I've got an idea what one is and then he says well, there's an infinite amount of vulnerabilities These stats don't mean anything. I said well, it's not infinite and I gave example
21:44
I was like I have a 10-line program. There is a finite amount of vulnerabilities in this he says no There's an infinite amount. I was like 10 lines. It's not infinite, dude Trust me on that you know and so he will sit there and argue, and this is just one example You know in one way or another they will figure out a way to make all the stats useless
22:02
And you're wondering kind of what's the purpose of this list again? You know is it to get metrics, or is it just kind of like you know have a civilized flame war So long story short you know we take all these factors in and we come to the conclusion of what I think Jack Daniel said That that original stat. I gave you is about as meaningful as my cat weighs 134 miles per hour
22:26
You know without context these stats mean nothing metrics aren't very helpful I mean how many of you like you said how many of you really care that there was? 8,000 some vulnerabilities last year you don't Come on all that software no one there's also
22:43
I don't understand Brian Why are you getting spun up because the numbers are just the numbers which you're really bitching about is the fact that you don't have a model right so propose a model and Then I'll show you five ass ass it'll tear it down for stupid reasons and a bunch of panelists It'll tear it down for good reasons
23:01
And that's that's what we call scientific method I Call it academic masturbation So what one of the problems is also that people need to be aware of what they can interpret out of a given number Like we have a lot of those cases like oh There's ten vulnerabilities in product a twenty vulnerabilities in product B. Which one is the safest product?
23:26
and Then they even take the stats perhaps even from from our site And that's one of the problems we have in VDB so also people take those metrics And then they just start into a building shit out of it like oh They might even add because it's on the second side so kunya says this is more vulnerable than this night No, we don't we just tell you there's ten vulnerabilities in that product
23:42
There's twenty in that product if you want to start evaluating more What if I add for instance that in the product with ten vulnerabilities? They're all unpatched The product with twenty vulnerabilities they've all passed within a week So if we factor in time to patch which one is then the most safe the safest product
24:00
Some of you may have changed your mind now about which one it is If I then go and add the one that has ten vulnerabilities They were all classic stack based buffer overflows the one that had twenty issues There were more complex use after freeze which one is now the safest product which one would you prefer to use? Yeah, I think that's a that's a really good point especially from you know a big vendor perspective in terms of
24:22
If we've actually put in the due diligence to you know when we get a vulnerability report We've actually put in the due diligence to look for variants of that and we fix all of those two You know for it to be bucketed as something like oh well You know they just they just fixed they had more you know they had more bones But that that's not differentiating between vendors who actually do due diligence and find you know additional variants or additional vectors
24:46
You know whatever With those who just kind of do the lazy thing and patch one vector Incompletely you know whatever and then it it shows up in these in these counts as you know Lazy vendor only had this one. You know diligent vendor had a bunch more, so it's there's no real way to
25:04
Differentiate you know the lazy from the diligent in this model real quick in case anyone's curious. We're talking about Adobe I Was gonna say we we see it all the time through ZDI the researchers are actually quite good about testing the vulnerabilities And I can't tell you how many times they come back and oh yeah, it still works
25:21
They didn't they didn't patch it You know and or there's another vector whatever the case might be happens all the time Oh, I want to I do want to hear Katie about silent patching Hold on hold on. I have one more good mouthful of beer I Want to claim that a metrics are totally fine if you understand it, and it's your context and you wrote the metric
25:44
Start has this awesome metric that if anyone knows we still publish vulnerabilities once in a while it goes from 0 to 180 To decimal points of precision so you can tell which ball is more important because there's a number and you can sort them Which is totally worthless to everybody in the world except the people at cert and actually it only was worthwhile to us years ago
26:05
When we used it to decide whether or not to publish document a or document B It was very worthwhile for that purpose at that time, and that's it so No, it is context, but right my point is is that you have that one line vulnerability And then you have an 87 line disclaimer. You know Ryder saying. This is what it really means
26:26
It's subjective I mean all this a lot of this stuff is so that's kind of the trick if if I might care about some vol You know you don't care about at all. I might care about security as vols or Microsoft I'm gonna care about all the PHP includes you guys have maybe I do maybe I'm a PHP web app developer
26:40
And I'm you're sick That's my point though, that's how people are getting owned PHP yeah Then their metrics are bad because they're not telling them that PHP matters at the moment Yeah, that's that's kind of everybody always asked about Oracle vulnerabilities unless you're Litchfield nobody ever cared All right, unless you're a pen tester pointing out how broken every deployment of Oracle ever is nobody cares
27:03
But people get owned with cross-site and as cute You know everybody's PHP blog was getting owned left and right four or five years ago, and so Still are right exactly so that's my point I like the way OS VDB does it because I actually know what is the attack surface available to attackers? I think that's important
27:20
That's entirely dependent on who the researchers are who are concentrating on things and what they're concentrating on We if those of us who've been in this industry for since about 2005 or something like that remember a Latvian teenager age 14 or 15 who basically decided to spend 10 minutes testing all the software that he could download and came up with
27:44
800 vulnerabilities within the course of like six weeks or something like that and Just a couple years ago some guy for Debian basically used a super powerful vulnerability detection tool called
28:01
GREP I'm not sure what the acronym stands for and he found like 500 vulnerabilities or something like that right, and so we're still very much subject to Pretty much the whims and the fads that researchers happen to go through and even one individual Researcher can have a big impact on what these numbers are what were your four eyes again?
28:22
hmm the four eyes incomplete inaccurate inconsistent and Incomprehensible and you need a fifth ignorant All right, we're gonna move on What are your thoughts of the value of vulnerabilities bug bounty programs? vulnerability buying and selling impact on disclosure and we're gonna
28:42
Give it over to Katie and you can give you a little spiel and see what Dan has to say then I have a question for Katie So So the question about bug bounties and and that type of thing, so I don't know how many of you guys
29:05
Saw or heard about you know the the talk that I gave a couple days ago at black hat Okay All right. Well, I'll just I'll fill you in as I go. So so I Think that a lot of security researchers, you know have varying motivations for what they do, you know, it's not all money
29:25
How many of you out there, you know who do this for a living? I mean professionally Have Figured out ways to mint money on the back end of some financial system raise your hand Liars, come on. There's more of you. Anyway, if you want if you wanted it, you know if money was it, right?
29:45
There's a lot of unsavory ways that people with the dark arts know how to know how to get money Now what you know what folks like, you know tipping point do is is what we would consider the white market, you know of Vulnerability buying and it doesn't really the numbers don't really
30:01
end up Equaling anything close to what the gray and the black market will pay for right? so There's a lot of researchers out there who you know think that it's you know, it's important to get Recognition for you know, either publicly or among their peers so when we looked at the Possibility of doing some sort of a bounty program a nominal fee for you know for vulnerabilities
30:25
We looked at the motivations that were out there and we looked at the motivations for the researchers who are actually finding bones in our products Because not every vendor has the same, you know kind of profile of the researcher that looks at their code We're we're we're a pretty popular
30:41
popular Target for research right partner, right? We're yeah know what but we're we're a pretty popular target for research Other vendors might have different, you know different behaviors and different main motivators for you know Researchers who look at their products. So we looked at what what researchers do with our you know Why they do what they do with ours and what we found was this past year
31:04
We've had about 80% of our vulnerabilities, you know that were disclosed at all Were actually privately reported to us. So 80% were privately reported, you know, let gave us time to fix the issues And the other 20% were dropped a zero-day now in that 80%
31:22
Considering there are programs like, you know, Dan's over at CDI that would offer You know a comparable price to a bug bounty like should should we had decided to do so in that 80% 90% of those reports actually came directly to us. So even though they could have made a small amount of money
31:43
You know They actually the majority of the researchers who find molds in our products and want to give them to us to get fixed Actually prefer to come directly to us. So that's what we found when we took a look at that data Now we absolutely, you know are fine with the the researchers abilities to make money doing their vulnerability research
32:03
And I think there's some great programs like ZDI that are out there that you know We love we actually talk about quality of reports Actually the quality of reports that come from these guys is that is is really really good. So yeah No, no problem, but you know, thank you But actually but hold on
32:22
Agree with me later So but So that's what that's what we found when we looked at our data when we looked at our researchers, right? So instead of doing a bug bounty because it seemed like, you know, there's lots of ways for researchers to make that money We decided to do something different and that's what I talked about a couple days ago
32:43
So if you go to www.bluehatprize.com and take a look we decided to offer over $250,000 in cash and prizes for Mitigation research so we're looking for the next generation platform mitigations top prize gets
33:00
$200,000 so we're going to announce that the winners next year at black hat and the contest is already kicked off We've actually already gotten some entries to the contest and there are you know So top prize gets $200,000 second prize $50,000 third prize gets, you know, MSDN subscription worth $10,000
33:20
You know and money fame, I guess women I suppose if you would, you know, if money and fame bring women, you know But that's what we decided was, you know Sort of the best way for us to encourage the research community to do what it does but figure out you know ways to mitigate Exploitation because like I said, you know bones don't pwn people exploits do
33:42
And we wanted we wanted to encourage the research community to work with us like that Actually, I want to disagree because I think Microsoft is also setting a great precedent that they are rewarding not only badass exploits But the ones that are completely weaponized so your bug bounty does exist and it exists in the sense that I write an exploit
34:02
It becomes really good It owns 200,000 machines and comes part of a botnet now you guys offer a reward for information on the botnet So my motivation is now not just to write a Microsoft exploit, but to write a badass one That's actually a different thing. Yeah, you're thinking about the other reward that we have it has nothing to do with it
34:22
Yeah, but yeah, you're in essence. You were still offering money on what is fundamentally a very good working exploit against Windows systems Nope, that's not it. So I think you're thinking about the the rustock but botnet bounty That's a completely different thing. So that's a quarter million dollar bounty for info that leads to the incarceration of the people who wrote
34:46
wrote rustock Totally different. So what I'm talking about is this is a I just announced this like two days ago I'm sorry, you weren't looking but anyway listen So this is completely different. So We're you know, we're taking this approach where look there are open problems
35:04
You know in modern exploitation that breaks our platform mitigations things that break ASLR in depth right so return-oriented programming jet-spray that kind of thing There are open problems there that we're working on mitigating So what we're actually rewarding are, you know, take one of those open problems, right and these are for memory corruption vulnerabilities
35:24
Yes, I know I said it and you don't like it. But anyway take one of those open problems in the exploitation of memory corruption vulnerabilities and And come up with a with a novel mitigation, so basically next generation ASLR next generation depth that kind of thing, you know
35:42
SEHOP that type of research is what we're looking at and just just well, so He's asking the question The question is will the research be made public so they can be used in other platforms The answer is it is up to the inventor the inventor retains IP ownership of that research
36:06
We just get a license to use it. So the inventor gets to choose what the heck they want to do with their research They want to port it to Linux go for it my friend. Enjoy. You know what I mean? So yes, if the researcher who wins chooses to make it public they can do so they own the IP
36:23
100% from a vulnerability days point of view We can see though that to an extent bog boundaries Do matter and they do motivate people I've made a nice slide and and he killed it. So now I'm just gonna like describe it like this. I
36:47
Made a case for instance with the CA bright stall Had a fantastic track record 2004 5 6 onwards There were like 80 vulnerabilities being reported in one of their bright store solutions
37:03
Lapsops and desktops, I think it's called in in 2007 and it actually triggered us And there was a time where city I was was paying for CA bright store issues and a lot of them actually came via CDI And in the beginning of 2008 as part of our yearly report
37:21
We actually went went out and said CA bright store is a solution. We consider to be inherently insecure Not only because of the vulnerabilities because we already talked about we can't look at that alone but we also found a lot of those vulnerabilities my research team and We could just see the code was terrible. So we went out and said we consider this part inherently insecure
37:45
a while later CDI came up backed it up and also stated that they would no longer pay for vulnerabilities in in bright store After that, how many vulnerabilities have been reported? so either they magically suddenly just up the quality of their product or people just stop giving at them and found other places and
38:05
I think that Adobe shockwave is an interesting one us because that is certainly received a lot of attention lately also And if I understand correctly, you don't pay for shockwave anymore either Well, we had a presentation at Kansack West where we showed everybody how broken it was Yeah, so and I was also finding out those shockwave issues
38:23
And they have some problems in some of their components So and it's it's quite realistic to also expect that since CDI Won't pay for shockwave vulnerabilities anymore that that we will like to see a drop in it Because then people will find another target where they can get money So to a certain extent it definitely does motivate people to in choosing which target they want to go for and this is one of the
38:46
Kind of metrics that's much more informative about the relative security of a software package Then counting the raw number of vulnerabilities that have been disclosed. I can make this quite short I even questioned CDI when it came out if you go back to 2005
39:00
This this room probably a little would have looked a lot different The whole industry was different the number of reverse engineers and researchers on the planet was far fewer But it was a very naive position to think that that number was not going to grow that a black market was not going to spring up And if any of you have ever read for economics that it pretty much proves that people
39:20
There's very good positive response when then there is monetary reward And I think now CDI is quite proven year over year. It's more and more popular It's you know, we I think we do a good job You know Being responsible and and being you know popular both vendors and researchers But it's you know, if you look at everybody that's got their own bone programs now
39:42
I think it's been I think it's been proven that it's a model that works Well and for us, you know the the model that we chose for the blue hat prize Was something where we were looking at as a platform provider. We were looking at ways to To scale such that we were essentially blocking entire classes of vulnerabilities with some of the research that we hope to get out of this and
40:05
You know certainly, you know What simple was was hinting at is you know, were we going to share it with the community and quite frankly? We got a SLR in depth from the community. Why why shouldn't we give back, you know, so Absolutely. I think the model that you know that we've chosen and I think there's room for lots of models here
40:22
You know every vendor is not the same. Not every vendor is a platform provider You know what? I mean? So for other vendors other models might make sense But for us, you know It makes sense to try and make these changes that won't that not only will impact our platform and our applications that run on it But these are platform level mitigations that will also help third-party applications on our platforms and mitigate some of those issues
40:45
So for us, we're looking at this, you know in terms of sweeping, you know, we're Making much more difficult to exploit entire classes of vulnerabilities All right, this is a reflection of a growing trend in the area to move a bit more towards
41:00
Not only defense like you're talking about with the Blue Hat Prize, but also prevention in the first place, right? There are entire classes of vulnerabilities. We know about these in the common weakness enumeration We we document them, but we still have like 800 different CWE IDs Maybe 20 different ones for stuff that are related to buffer and memory corruption errors
41:20
All right myself, sorry, I want to get on the next question come on this is a good one I want you guys to talk about being the people that track and deal with researchers as well as vendors name names Tell us who they are How do you really feel about working with certain researchers and vendors and I know you guys are gonna be you're not gonna be shy
41:41
about this so Who wants to talk about the research quality and gender response? Okay You go first or last Yeah so I've had a few problems with Researchers and I think I'm the only one out of any of us up here that will actually reply to bug track and full disclosure and call them out on it and Part of that is you know
42:02
Yeah, quit being a dick and sending this really worthless information and also just kind of teach a lesson that if anyone's reading these lists Strive for a little better accuracy in your reports because it's not just reflecting on you But it's causing a whole lot of headache on the part of everyone else involved you know if Microsoft gets a report and I know that they've gotten probably hundreds if not thousands of these where
42:22
There's enough information and they're like wow, this sounds like it may actually be a an issue But the technical information isn't there and then all of a sudden they're in this like email back and forth And they spend two weeks all to figure out that well Oh, wait, you have to have local admin privileges to do this. You know, so, you know one of my
42:43
To name names, you know One of the the most recent ones for me was HT bridge and I'm sure that one or two of you were in the audience Hi, I'll respond to your mail from three weeks ago when I get home You know, they started releasing advisories and It's obvious They're using them as a way to promote their company and there's all kinds of really crappy stuff that they're releasing
43:05
Because they're going after beta products. They're going after real low-hanging fruit, you know, they'll find Yeah, well no not only that but they'll find like oh here's two cross-site scripting and two different advisories Oh, and we forgot you're not forgot. We just kind of missed the remote code execution You know in the serious bugs in it
43:22
And I don't know how many cross-site scripting issues I've seen reported that our error messages that clearly indicate RFI or SQL injection Yeah, and they're missing these left and right and you're looking at it Like, you know, if you guys would actually spend some time on this you would find some really neat stuff And you're not and then they also have this habit of you know As an example, it's like oh we're gonna contact the vendor and we're gonna give them two weeks
43:43
And the fact that we typoed on the email and the vendor never got it doesn't much matter You know, we're gonna go and release in two weeks. Anyway, bottom line is if you're discovering cross-site volumes, nobody thinks you're cool Yeah, yeah cross-site scripting is really old. It's really kind of lame and it's one of those that ask John Oberheide
44:01
Yeah, well if you're gonna do cross-site scripting just wait every 30 days and do one post with like all 750 of them. Okay, so if you can if you can own a mobile phone at Pona own then your cross-site is worth a crap otherwise Disclose it to the vendor or the website or wherever the hell ask for some swag and be done with it, right? And I'm fine with posting it to the list
44:21
It's just don't think that it's anything other than you know a novelty for most of these And the other big pet peeve is like SQL injection. It's like well, here's cross-site scripting I think will actually include the script code to exploit it and you're like, okay Well, this is valid and then when it comes to SQL injection They're like and the proof of concept is bracket SQL. I bracket
44:41
Wait a minute. That's not proof of concept. That's saying here's the script and here's the variable and wait a minute Why couldn't they actually put SQL? I exploit code in there Is it because they're morons or do they actually think oh well if we do that bad things will happen to the 87 Installs of this software that you've never heard of, you know, either way
45:01
It's a cop-out and yeah It gets really tiresome and I want to be clear that HT bridge has kind of been my whipping boy for the past year But that's just the tip of the iceberg, you know If I actually spent time to respond to all of these lame advisories, it would be more than a full-time job I gave up responding years ago just because the amount of time it took to do that, right? So we spend time responding but it's to our researchers. We don't do it publicly
45:23
We do that You know we accept about 30% of what is what is submitted to ZDI A lot of that is vulnerabilities that we're not necessarily interested in a lot of that is crappy submissions And we want to work with the community and we've you know, seen researchers come up through the years to make those submissions better That's obviously in our best interest but to call someone out
45:40
I will I will call someone out and then I will also give them kudos if any of you Were aware of the policy change. The only policy change we've ever had was ZDI. We now enforce a six-month deadline Because there were some vendors that were kind of sitting on their hands and HP And that's absolutely right. Yeah, and and so You know, that's that's it's actually been phenomenal for HP because
46:04
Everyone decided you know what we're one of the culprits and we want to do this better When the other culprits was real networks if you go back to last year and you see how many real network Vulnerability advisories we disclosed there were a lot and they took that policy change very seriously and look at how much better their software is
46:22
So yes, they were bad, but now they're good So that's that's positive we generally experience that like in the past 10 years I've been involved with VDB I Actually all think that researchers are getting better. They are getting better at Providing the details we need don't get me wrong. We're still killing about 25% of what is posted on the lists
46:43
But the level of quality seems to be be improving Now Katie has been baiting Steve and I for a while. So let's go back to the memory corruption issue That is one trend that so is going the wrong way More and more people using the term memory corruption Seriously, if you're a researcher then it's because you're damn lazy or you just don't really know what it is
47:04
There are a couple of valid cases where it's perfectly fine to case call it memory corruption, but it's been like a Thing covering in it everything from a stack based buffer flow to a use after free and how we've even sometimes seen it it's actually just the missing exception handing that just
47:20
Results in an application terminating so seems like being the standard thing. Oh, I ran a fossa Something crashed. I don't really know what it is memory corruption John sent The same from from vendors also And it's like come on. I mean the vendor should hopefully know what the core problem is
47:43
Please tell us is it a stack based buffer flow. Is it an integer flow is the use after free? What is it like don't tell us it's a memory corruption So I'll also chime in because you know, I mean obviously I'm here representing Microsoft a vendor but Microsoft also You know, we actually do vulnerability research on third-party products
48:02
I I founded Microsoft vulnerability research in 2008 to do this So we and we started releasing advisories on third-party products for vulnerabilities We found and work with the vendors to get fixed. So we see it from both sides, too You know We are both the researcher and the vendor and sometimes the coordinator will also MSBR will step in and coordinate
48:20
multi-vendor Super nasty apocalypse kind of issues, right and we'll try it. We'll try and do our best to coordinate there so we feel the pain from all three roles and disclosure a lot of the time and yes, some of you know, Some of the the researchers that we deal with are much, you know much more Able to articulate their issue than others, you know, but actually we have seen that
48:44
You know same trend where they do actually get better over time And are you see that on the search side? our We we stopped paying careful attention. We stopped counting vulnerabilities. We get maybe 30 direct reports
49:02
About 30 a month. So maybe one a day and we don't we don't run with all of them But probably half or more of those we go with it the only thing that really up, you know really bugs us is we get the Researcher who looking for some extra fame and their company's not famous enough yet
49:22
But maybe if certain has an advisory that'll help so there they'll be honest to make sure we publish something that has their name in there Hasn't happened a lot in the past couple years, but that used to really annoy me But do you think that the quality of the incoming reports to you has improved? No, it's it's all over the place. There are great ones and there are horrible ones and I
49:43
Know I can't measure enough to really say there's a trend either direction But my you know gut feeling is it's about the same We actually see we actually see something really interesting too in that a lot of researchers are They only come to us with one vulnerability ever and they don't that they got lucky maybe or they didn't like
50:04
You know doing vulnerability research anymore. I mean you don't actually We don't really know what it is that you know made them come to us just one time and then disappear Or accidental discovery, I mean something crash you bother checking it. Yeah, I think a lot of researchers don't look for variance either
50:22
I mean that that was a major pain when PHP application vulnerabilities first started happening You'd have one research one researcher go Oh, I you know looked at this PHP golf application with 10 downloads in its entire history And I found this cross-site scripting in these 10 different vectors and then 10 different parameters or something like that And then like, you know two days later some other person completely different
50:43
Reports 22 different vectors for the same vulnerability type and there's a little bit overlap But not all that overlap and it makes it very clear that you know The depth of the research is not necessarily there Yeah, and one of one of the lessons I want to say about you know Microsoft and the fact that we are in all three roles, you know of disclosure or vulnerability research
51:02
You know both the finding coordinating and the fixing side But as finders when we go to different vendors We've had to we've had to actually prove it just like any other researcher We've had to prove it to them sometimes by popping calc, you know this is this is definitely happened in the course of my you know, Microsoft vulnerability research where
51:24
Vendor just didn't believe us So we had to you know We had to show them so but part of that part of that mission for us is actually education for them, right? It's just like any other researcher. It's education. Like no really this is exploitable. I promise here you go
51:40
And they're like what why why is this calculator showing up on my desktop? I don't understand And then we use that as a way to start a conversation with them about secure development because we're saying we're saying to them Look, we've you know, we've taken our lumps over the years We've learned our lessons in the following areas and we'd like to help you because you run on our platform We'd like to help you get better because that makes our platform more secure
52:03
So we start talking to them about ways that they can catch these vulnerabilities earlier in the code But it's an educational process just like any other, you know researcher who comes to a vendor, you know and says hey Your flies down you might want to pull that up, you know We not only say, you know say that but we also, you know
52:21
We also definitely try to to make it so that they don't keep making the same mistakes over and over again Alright, so we're starting to get the hand signals, but I want to ask you guys thing real quick Real quick. Yeah, just as a heads up There are multiple vulnerability databases that do this the data is not public when OS VDB has a data set
52:41
We will make it public but one of the things that it's been fun tracking is what we call researcher confidence and OS VDB is actually gonna Eventually track vendor confidence as well. So researcher finds 50 vulnerabilities over the year and let's say 45 are accurate well that starts to give us a percentage, you know of success rate in finding a vulnerability and
53:01
At least one of the the VDBs represented here and it's not OS VDB tracks it even beyond that and when you start to look at These statistics, you know Steve Christian I were looking at the data and we're like, oh, yep. We know this guy Yep, that's accurate. That's accurate And you know some of these it's like it's amazing that some of these researchers that are well known and liked all of a sudden
53:22
Have a 60 or 70 percent success rate, you know How many of you know that it's someone has a 30 or 40 percent failure rate on reporting a vulnerability It's not accurate can't be reproduced or something else about it is wrong So down the road look look forward to that because I think it'll be very telling Not only what we deal with but a lot of the big names that you guys recognize, you know, it becomes neat
53:45
All right, so we're gonna be going to the next room here He's telling me no, but I want one comment from Alex and maybe art on what do you think about CVSS? that leads us into our room Two thing two things with that are wonderful about CVSS. All right, so I'll back up
54:04
My problem is CVSS is this it's a it's a attempt at formalization of something that doesn't exist I like the ratings There's nothing wrong with waiting and scoring and trying to figure out how smart something is But when you start multiplying ordinal values together, you break the fundamental light that the universe works
54:23
You just can't do that and you end up with you know jet engine times peanut butter equals shiny And you're telling me that the result of shiny the second problem with it is decimals aren't magic They're not unicorn poop. You can't just add them willy-nilly and suddenly it's a ratio scale. It doesn't work that way
54:41
the so the problem is that it may be right where you have a 15.4 is actually more severe than a 13.2 but when it is wrong because you're doing the wrong things with math, it will be Really wrong potentially and that that's dangerous. I like it. I wish they just wouldn't multiply things
55:03
Just give me a freaking baseball card scorecard like thing and let me look at it because I can look at that and digest it myself So there are two answers. I have two answers to that What one of them is that there's time in you? No, that was that was last night I
55:22
That's why I'm horse CVSS version 3 there are some rumblings within the special interest group without thinking about that for So for those of you who are stuck with CVSS version 2 with its you know warts and all if you have any concerns you can bring it up to bring it up to me or I'll name Katie as well or art because we're all one way or another kind of least indirectly involved on the SIG
55:45
The other thing is to address at least some of the limitations some of which you've alluded to Alex There's this thing called the common weakness scoring system, which isn't at the vulnerabilities It's at the when you find a weakness indication of the potential for a vulnerability
56:00
It still has multiplying ordinal values by ordinal values, but it has built into it continuous Values as well for those people who are sort of the expert users I think we need to recognize that most people are using CVSS, right? They need a score one way or another they all they care about is the score They don't necessarily care about a lot of the fancy math behind it
56:21
So my hope is that for CWSS some of our lessons learned can feed into the future of CVSS With that thanks for your time. Appreciate it. We'll be around find us for beverages and thanks again