Is it 0-day or 0-care?

Video thumbnail (Frame 0) Video thumbnail (Frame 1684) Video thumbnail (Frame 2855) Video thumbnail (Frame 11393) Video thumbnail (Frame 19931) Video thumbnail (Frame 21912) Video thumbnail (Frame 23592) Video thumbnail (Frame 33075) Video thumbnail (Frame 34905) Video thumbnail (Frame 42815) Video thumbnail (Frame 55543) Video thumbnail (Frame 68271) Video thumbnail (Frame 80999)
Video in TIB AV-Portal: Is it 0-day or 0-care?

Formal Metadata

Title
Is it 0-day or 0-care?
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2013
Language
English

Content Metadata

Subject Area
Abstract
Vulnerability Databases (VDBs) have provided information about security vulnerabilities for over 10 years. This has put VDBs in a unique position to understand and analyze vulnerability trends and changes in the security industry. This panel presentation will examine vulnerability information over the past several years with an emphasis on understanding security researchers, quality of research, vendors, disclosure trends and the value of security vulnerabilities. The emotional debate surrounding Full Disclosure has raged on for decades. This panel will use grounded data to discuss salient points of the debate to hopefully determine trends that may influence the debate. Maybe even in a positive fashion!
Multiplication sign Data conversion Flow separation Vulnerability (computing)
Logical constant Complex (psychology) Existential quantification Code Multiplication sign Source code Database Mathematics Semiconductor memory Error message Descriptive statistics Vulnerability (computing) Area Enterprise architecture Email Cloud computing Virtualization Bit Open set Web application Category of being Process (computing) Website Quicksort Point (geometry) Trail Service (economics) Overhead (computing) Real number Blind spot (vehicle) Motion capture Web browser Raw image format Twitter Number Goodness of fit Root Term (mathematics) Hacker (term) Energy level Traffic reporting Mobile Web Vulnerability (computing) Information Consistency Mathematical analysis Plastikkarte Planning Basis <Mathematik> Database Line (geometry) Cartesian coordinate system System call Online service provider Software Personal digital assistant Point cloud Buffer overflow
Slide rule Trail Mobile app Code Multiplication sign Patch (Unix) Source code 1 (number) Port scanner Database Rule of inference Software bug Number Twitter Revision control Goodness of fit Causality Root Different (Kate Ryan album) Intrusion detection system Core dump Energy level Information security Vulnerability (computing) Scripting language Injektivität Enterprise architecture Vulnerability (computing) Information Military base Mathematical analysis Database Bit Instance (computer science) Perturbation theory Exploit (computer security) Flow separation Type theory Web application Word Software Personal digital assistant Point cloud Right angle Whiteboard Quicksort Table (information)
Point (geometry) Slide rule Statistics Metric system Multiplication sign Patch (Unix) Combinational logic Database Online help Mereology Twitter Number Type theory Term (mathematics) Hypermedia Different (Kate Ryan album) Spacetime Utility software Information security Vulnerability (computing) Physical system Mobile Web Vulnerability (computing) Multiplication Channel capacity Volume (thermodynamics) Total S.A. Database Measurement Entire function Type theory Right angle Information security Metric system Discrepancy theory Volume
Context awareness Greatest element Group action Metric system Direction (geometry) Multiplication sign Source code ACID 1 (number) Mereology Computer programming Information technology consulting Subset Usability Software bug Medical imaging Type theory Hypermedia Endliche Modelltheorie Information security Vulnerability (computing) Injektivität Email Electronic mailing list Infinity Type theory Arithmetic mean Phase transition Website Right angle Figurate number Information security Metric system Volume Statistics Finitismus Flock (web browser) Divisor Civil engineering Patch (Unix) Barrelled space Number Product (business) Twitter Goodness of fit Spacetime Computer-assisted translation Hydraulic jump Computing platform Condition number Vulnerability (computing) Key (cryptography) Projective plane Mathematical analysis Total S.A. Database Denial-of-service attack Line (geometry) Axiom System call Cross-site scripting Sign (mathematics) Word Software Personal digital assistant Interpreter (computing) Game theory Local ring Window
Complex (psychology) Context awareness Code Multiplication sign Orientation (vector space) View (database) Decimal 1 (number) Mereology Perspective (visual) Computer programming Software bug Front and back ends Semiconductor memory Different (Kate Ryan album) Endliche Modelltheorie Extension (kinesiology) Information security Vulnerability (computing) Physical system Pattern recognition Electric generator Software developer Moment (mathematics) Data storage device Hecke operator Instance (computer science) Web application Type theory Vector space Buffer solution Website Right angle Quicksort Metric system Row (database) Point (geometry) Trail Service (economics) Pay television Patch (Unix) Transport Layer Security Real number Virtual machine Product (business) Number Internet forum Term (mathematics) Profil (magazine) Software testing Traffic reporting Computing platform Addition Vulnerability (computing) Information Surface Weight Consistency Computer program Counting Variance Line (geometry) System call Exploit (computer security) Peer-to-peer Software Personal digital assistant Blog Codec Freezing Window
Presentation of a group Confidence interval System administrator Disk read-and-write head Computer programming Software bug Medical imaging Different (Kate Ryan album) Information security Error message Social class Exception handling Personal identification number Arm Trail Electronic mailing list Coordinate system Bit Process (computing) Internet service provider Buffer solution Figurate number Writing Electronic data interchange Connectivity (graph theory) Drop (liquid) Product (business) Number Crash (computing) Bridging (networking) Term (mathematics) Intrusion detection system Energy level Traffic reporting Computing platform Information Line (geometry) Multilateration Cartesian coordinate system Exploit (computer security) Cross-site scripting Software Enumerated type Personal digital assistant Code Multiplication sign Direction (geometry) 1 (number) Set (mathematics) Parameter (computer programming) Mereology Mathematics Bit rate Semiconductor memory Endliche Modelltheorie Data conversion Extension (kinesiology) Logic gate Position operator Vulnerability (computing) Scripting language Injektivität Area Email Reflection (mathematics) Data storage device Measurement Entire function Proof theory Type theory Vector space Website Right angle Remote procedure call Metric system Freeware Resultant Reverse engineering Laptop Trail Dataflow Statistics Sweep line algorithm Real number Modulare Programmierung Twitter Software testing Integer Installation art Vulnerability (computing) Dependent and independent variables Multiplication Validity (statistics) Inheritance (object-oriented programming) Poisson-Klammer Gender Computer program Database Calculus Grand Unified Theory Calculation Local ring Buffer overflow
Group action Freeware Level of measurement Trail Multiplication sign Decimal Expert system Plastikkarte Price index Limit (category theory) Graph coloring Vector potential Revision control Mathematics Message passing Bit rate Universe (mathematics) Formal grammar Right angle Quicksort Vulnerability (computing) Physical system
thank you everyone for attending our panel today usually I find that I get these like 8am slots and it's really awful on so I'm very very happy that it's a late slot so thanks for everyone who has hung in there with us so over the last couple years several of us have been talking about vulnerabilities in depth and we started down this path of saying does anyone really care about vulnerabilities anymore and it turned into this conversation well it was zero day well then it's really important but otherwise maybe no one would care so we want to get some people here to get today got a pretty large panel of folks so I'm going to try to keep them reined in as best as possible and I want to get some people that had some you know good thoughts but we're really easy to manage that weren't that opinionated that wouldn't go off on rants and if you know anyone up here we'll see what happens right so yeah fail already thanks for that so anyways hopefully this be good panel for you I know a lot of times panels can be a real pain but we're going to try to make this interactive good some good topics that maybe aren't discussed as they should be so the first
thing with a good panel is getting the right people up here and I think that we have that hopefully you know all these people if you don't I'm not going to waste any time of the session introducing their BIOS we got Brian from 0 SPD be steve from CVE Carsten from secunia art from cert and Whitman dan from HP tipping point Katie from I don't know who's whistling for Katie or Dan Tom Katie for Microsoft and Alex I don't know where you are anymore so i'll just call you Alex all right so we're going to kick it off we're gonna try to do for questions about 10 minutes each and then at the end there's going to be some free for all if you really feel like you need to yell at one of the panelists there's a microphone come on up otherwise you can hold it again your call so the first
one we're going to start off with is does anyone that doesn't work on one of these things really care about vulnerability database is tracking or trending anymore and I'd like to start off with maybe Steve or Brian you guys want a popcorn people care but they don't know they care near as I can tell at least on the TV side we rarely get many many complaints about out about what's going on with us sometimes we have like blatant errors in cv descriptions that we never hear of from from anybody so in a sense that people don't necessarily care it seems like a really easy job to just like comb through thousands and thousands of vulnerability reports every day that always most of which have one of the following four properties which I call the four eyes vulnerability reports are either incomplete they're inaccurate they're inconsistent especially say between a vendor report and a researcher report not like we ever get those kind of inconsistencies incomprehensible some some of the stuff that we're dealing with comes in in a broken English from you know people who live in the Midwest poorly formatted advise advisories where the more the most serious vulnerability they don't even quite realize what it is and it's very even like a single sentence in a you know three-page spread about you know what they had for breakfast or something that's the kind of raw information that we have to deal with on every on a daily basis and unless you really deal with that kind of stuff on a daily basis it seems like really easy oh just scrape all these websites or taking all these emails and then just do a little bit of analysis in you know in two minutes and then push this thing out unfortunately there's a tendency that all of us in the vulnerability information industry have which is we kind of care about correctness and quality and in some ways this is why I think we're having this panel now because quality comes at a steep price so you want to talk about cve and are we still working on CVE these days just get right out and ask me is cve dead no prove it however we are going through we are going through a change so we're kind of in a cocoon and we'll kind of come out like beautiful butterflies or whatever but there's there are a couple realities okay first of all they're just the raw number of vulnerability reports coming out is increasing significantly the complexity of the vulnerabilities that come out are really difficult to sort of capture yeah we could go along the lines of other people and just call things memory corruption which is really code for some kind of buffer overflow that we don't really know how to describe exactly but at least on this CVE side what we've been doing is carrying a lot about that kind of the academic strengths aesthetics of what are the real root causes lying underneath these vulnerabilities so so terms such as memory corruption we actually try and dig a little bit deeper what that means though is that there's a lot bigger analytical overhead in our pursuit of correctness and those of you who follow cv on a regular basis you may see increasing levels of precision increasing levels of correctness but it's come at a pretty high price one of which is the the entry to have people on our team gets a little bit a little bit high in terms of the technical skills that are required and then and then the other price as well though is that you know we're at a constant level of funding and the numbers of vulnerabilities are getting reported are you know more vulnerabilities and more complex vulnerabilities something's gotta give and in this case recently what's been giving is the actual number of CDs that we've been that we've been publishing but this year especially we're working a lot on modifying our processes to change that ultimately Brian yeah real quick OSP TB has two different ways to handle this the first one we tried was crowdsourcing so we would put memory corruption in the title and we figure hey there's a lot of smart people out there there's the researcher whoever else could clear it up and of course no one did so after a while we went with our back a plan which is just let secunia do it all any comment I just wanted to comment on whether it does anybody actually care about volens kitchen it seems like everybody wants to talk about infrastructure you know virtualization and cloud and mobile and puke so at the end of the day all these berries infrastructures are all just delivering applications applications have vulnerabilities it's it's the same thing getting delivered it's still a web application it's still a web browser and they're full of bones and just because the infrastructure changes doesn't mean the vulnerabilities really change that much there's actually a blind spot that exists right now with respect to cloud-based services and services in general which is that us as vulnerability information sources don't cover those you know if there's a if there's a vulnerability and you know the Google you know Google search engine or something like that you know maybe it can be used to hack millions of people or whatever but that's a that's an online service that's not deployable software that goes into the enterprise so while I think we've been doing a good job in general tracking trends that that is one area that's a really big blind spot and it's going to get worse and worse as the adoption of services increases I completely agree because if
you look at the trends that have really occurred over the last five years or so so many of the things that have actually moved our web base whether it's browser app tracking a trend like SQL injection actually becomes quite fascinating because it actually moves as opposed to you know some other stuff from Microsoft for instance we've got when we got to the care I at least we hope that people care and that all the efforts that we make and in vain at least they are some people attending today so it's not completely cereal care but sometimes we see ourselves as making an elegy like we're providing electricity none of us are sitting in here now excited about their slide but we would be complaining if there wasn't and the same when we do the vulnerability databases when we correlate all the information when I advisories get out when they're correct when everything is as it should be it seems like no one's really paying attention it's not the type of thing where people say oh that's excellent that's good but if we send out an advisory with just like a small table we within five minutes actually have a haha may look send to us so at least with us we spend a lot of time making sure we get it correct everything from the analysis of the core problem in the vulnerability down to their spelling so I think we kind of live in this I mean security people kind of live in this world where vulnerabilities each vulnerability you know it's a beautiful snowflake right each vulnerability actually yeah they are each one unique but you know it matters a lot to us as security people and security people working inside of giant mega-corporations it matters to us each phone matters but to the broader world and you know maybe to some people who call themselves security people who don't actually care you know maybe they're talking apt in the cloud whatever you know if you because you said right bring gum you know maybe those people they are thinking of themselves well bones don't pone people exploits pone people you know so they really only care about exploitable volunteer about exploits so you know with all of these vulnerability tracking databases I think for a lot of the population what they really call out of those databases is what's exploitable or what has an exploit out there as in what do I need to get off my lazy butt and deal with right now I love that because I think I saw two tweets on the 0 dan WordPress and all of you people keep retweeting the metasploit for is out so thanks I know so it's cert we we had the same same feelings as Steven in Karsten we wanted to be correct and counted be vulnerable in the world but then maybe the last year or 2 i've come to the realization that people do care they do because they want to count a vulnerability call it something scan for it they want to be in compliance see if they can patch for it they need to name it something sadly they don't really care how accurate the advisory is to a large extent if you're really going to your dealings with something hot in its 0 day and they're special mitigation advice and you got to do something right maybe it matters but i think the big need is just having a label on the thing and being able to talk about that and it's the same label so that we're all talking you don't have to have eight different IDs you got one ID that rule them all which is probably should be see the places where accuracy has come into play at least what I've seen at CVE are you know all this root cause analysis i think is kind of cool and and and i've had kind of a mindset of well this may help influence how people think about vulnerabilities as these darling precious snowflakes that each and every one of them is but but when we do when we do get complaints on the CBE side of things in generally it's it's two things either the affected versions of the software we might be a little in precise about and then characterizations of the severity of the issue ultimately the CBS s4 and that kind of make that kind of makes sense right because that's ultimately what people seem to care about its how what is going to be the impact to my enterprise I don't care if this is some you know brand new really really cool attack that deserves a pony is it going to hurt me or not this thing is is a 9.8 I have to do something you know it was a lot of that mindset in the enterprise user of sort of community so can anyone up here does anyone know how many public vulnerabilities were disclosed any given year I think oh as VDV might be the closest these days well if you're talking about just the overall numbers one of the reasons that we say that we have an accurate numbers because we're the only ones that extract to the level we do where every single specific vulnerability in every script gets its own ID and everyone else across the board CBE secunia everyone says no we're going to lump them together and yeah that kind of moves into some of the stems fighting words yeah there are lots of different ways of there are lots of different ways of counting vulnerabilities I'm not going to dispute that you guys have a lot better coverage than and any of the other sources up here because you guys really try and track everything and you know you guys put in really long hours into it also your analytical overhead in general is very minimal you work goes into let's compose a title you know have fairly simple ways of like breaking things down that get a little bit more complex at least for at least four CVE when we're dealing with for example shared code bases and we're looking at two different bugs we may intentionally combine them in other cases we kind of have to look a little bit deeper to figure out if we need to split them or not because in some cases you know if we have to see bees that are out there that are duplicates that's kind of okay but if we have like one I hate duplicates by the way I don't get me wrong but if we have one CVE out
there that kind of combines multiple issues inadvertently then utility of the CVE goes down if people vendors are only fixing like one part of it and not not the other part so that said the way that you guys have structured things I think is really good because you can be the closest to counting the total number of vulnerabilities that are disclosed using the way that you count things well that let's let's move to the next world hey which taught us about the trends and the trick this is just a natural flow and plus you know tan so what are the mobility try it was average eight thousand year yeah I drop my point being who cares if you can't even can't even put a measurement on the number or how bad our vulnerabilities may we're just making stuff up which I'm gonna cover that okay all right is that the next question yeah yeah so what are some trends and vulnerabilities disclosures types volume that we're seeing and then are these security metrics even worth a damn so so what if you count this many what does that really mean so so the first question is who out there thinks that vulnerability statistics are helpful and useful and you actually do currently use them in any capacity anyone that's a lot of time sorry wrong I mean you are absolutely does this
slide help you yeah what's that we can see this one oh yeah my panel doesn't know the slide it's the the vulnerability counts for most b2b okay so one of the things I wanted to bring up about vulnerability stats just as a quick idea there were 8337 vulnerabilities in 2010 does that sound like a useful statistic only compared year-over-year it's it's over time that I think it does become useful okay well I mean like the media will call up and ask us you know how many bones were there last year as if they knew what was the previous here right well then they will get to that and I will do so you know I have to footnote well that's court in OS VDB what about other VD bees well there were 3648 according to secunia well wait why the discrepancy well now you have to get into different kinds of databases secunia database is geared for a very specific use they have an entire customer base that actually uses their database for day-to-day patching notification you know it's an entirely different system than 0 SPD be where we're looking for long term stats history and we abstract the way we do
yeah like our customers they would like flock me publicly if we started doing what I was video video and send out one advice with very single issue but they care about and how people like the idea of how to use our database is how many like axioms do I have to take so if there are 10 vulnerability is being fixed by one patch then they just want one advisory listing the 10 vulnerabilities and the patch they want to know which product is affected how many vulnerabilities how critical are they how do I fix it and then there's that subset of people that actually care about the call details of each vulnerability but they just want to know how do we fix it so we they don't want to have Tim advisories that tell them to install the same patch right and that's why goes back to that is that Oh sbtb would be horrible it would be useless in that situation because it would saturate them so after that the next question is that figure more or less than 2009 and so I say there's 7,000 678 vulnerabilities in 2009 less than two thousand and ten total according to sbtb next question we obviously get is is 2011 on par with the last well there's 3427 as of july twenty fifth this does not appear to be on par so now all of a sudden we have one number that we start out with that as soon as you start putting context around it and you even looking one year either direction the the stat starts to lose some of its meaning i'm also an assumption in those stats that your analytical capabilities are keeping on par with the publication sources that you're monitoring so for example a couple of years ago I think you guys started scraping almost the bottom of the barrel looking on various various sites and no one else was looking at right that effective your numbers for that year and then the year future for us I'll exactly people do a lot of CVE based analysis counting the number of vulnerabilities without recognizing that you know we don't have the complete coverage that we used to have to to some of our actors that I talked about earlier not not to be a pedantic ass but the stats don't lose the meaning stats are just numbers what loses meaning what your problem with meaning is actual the model you're inferring meaning in what you're doing there okay so it's not it's not the numbers it's your quest for knowledge out of the number right its interpretation then that's why I say it's all about context all right right exactly and I think that's what then I was just going to say Dan guidos been pointing out lately of course which vulnerabilities do actually need to worry about I think where the context and the numbers actually mean something is when you then pair them with attack data and then you can see are the trend similar completely different you know yeah activex disclosures going nuts and everybody's getting owned with activex same thing with SQL injection but not the same thing with pick whatever vulnerability type you know so that's that's where I think it actually you get more of a whole picture because people actually want to know how am I getting owned I don't care if there's a vulnerability out there that attackers don't use yeah in a sense in a sense it's it's it's too bad that the media can't ask you smart questions well that doesn't make the numbers any less useless well the funny part is is that they ask these questions are like what you know is there anything else you want to contribute to this and you know next morning they wake up and they have a 17-page mail from me explaining all this and they're like okay thanks for your time you know the hell if they're gonna write about any other I smaller so this is the kind of looking for it's the case of looking for the keys under the street light instead of where you think you kind of dropped them right this is the only data that's out there so people people are looking for it so real quick to jump back to my example you know if 2011 isn't on par with 2010 the question is why you know so all of us we know some of the reasons there's trends look at phases like cross-site scripting SQL I there's certain years where people jump on the bandwagon there was the DLL injection on Windows platform where everyone's finding software that did that you know lo couple years ago with all the image ones image 1 zip ones so that's low hanging fruit that will swing the totals there's change in desires disclosed you know a while back everyone's like well if I release an advisory that becomes free advertising for my company eventually these companies realize wait we're not getting business and then researchers are like oh zdi they'll get me all kinds of cash and hookers and blow for this you know so all of a sudden they have a very different desire to disclose in the way they do it um there's ones we know about the time we had to dig into it like Steve said there was a few years where I was a consultant you know I didn't work a whole lot enough to live and I spent all my other time on OS VTB and those years our numbers jumped dramatically because I was scraping changelogs I would go through like the Apache bug tracker and if you've ever been in that thing with all of their projects it's crazy and yeah I was the dumb acid action went and said okay I'm going to search for the word security and start reading every goddamn ticket Apaches ever written with the word security in it pull out every denial of service every stupid little bone every race condition local / missionary you name and we put in our database you still have to make a guess right because half the time it's like six words right some of the pics permission problem what does that mean that what does that imply security issues or usability issue yeah so not only do we have to write the entry then it was a techno disclaimer it says due to the vague wording of this we're not sure if it's a security issue you know you add this up and yeah also the numbers jump so then we get to what I call the security metrics factor who in here reads the security metrics mail list anyone yeah okay stay the hell off of it that is the biggest waste of time of academic masturbation you will ever see as soon as you get close to a real statistic these asshats like Fred CO and jump in what's a vulnerability wait what do you mean we have to define vulnerability now so he does he'll go down this path it well you don't know what a vulnerability is I say well I've got tonight me no one is and then he says well there's an infinite amount of vulnerabilities these stats don't mean anything I said what's not infinite in that game example I was like I have a 10 line program there is a finite amount of vulnerabilities in this he says no there's an infinite amount I was like ten lines it's not infinite dude trust me on that you know and so he will sit there and argue and this is just one example you know in one way or another they will figure out a way to make all the stats useless and you're wondering
kind of what's the purpose of this list again you know is it to get metrics or is it just kind of like you know have a civilized flame war so long story short you know we take all these factors in and we come to the conclusion of what I think jack daniel said that that original stat I gave you is about as meaningful as my cat weighs 134 miles per hour you know without context these stats mean nothing metrics aren't very helpful I mean how many of you like you said how many of you really cared that there was 8,000 some vulnerabilities last year you don't come on Ellie it's a perfect time for you all that software no one there's all sorry i don't i factor and I don't understand Brian why are you getting spun up because the numbers are just the numbers which really bitching about is the fact that you don't have a model right so propose a model and then I'll show you five assets it'll tear it down for stupid reasons and a bunch of panelists it'll tear it down for good reasons and that's that's what we call scientific method I call it academic masturbation so what one of the problems is also that people need to be aware of what they can interpret out of a given number like we have a lot of those cases like oh
there's ten vulnerabilities in product a 20 vulnerabilities in product B which one is the safest product yeah and then they even take the stairs patch even from from our side one of the problems with VD beasts also people take those metrics and then they just start interpreting out of it like oh they might even add because it's on securing site secunia says this is more vulnerable than this tonight no we don't we just tell you this 10 molar bilities net product is 20 in that product if you want to start evaluating more what if i add for instance that in the product with 10 vulnerabilities they all unpatched the product with 20 vulnerabilities they've all passed within a week so if we factor in time to patch which one is then the most saved the service product some of you may have changed your mind now about which one it is if I then go and add the one that has 10 vulnerabilities they've all classic stack-based buffer overflows the one that had to any issues there were more complex use after freeze which one is now the saves product which one would you prefer to use yeah I think that's a that's a really good point especially from you know a big bender perspective in terms of if we've actually put in the due diligence to you know when we get a vulnerability report we've actually put in the due diligence to look for variants of that and we fix all of those too you know for it to be bucketed as something like oh well you know they just they just fix they had more you know they had more bones but that that's not differentiating between vendors who actually do due diligence and find you no additional variance or additional vectors you know whatever with those who just kind of do the lazy thing and patch 1 vector in completely you know whatever and then it shows up in these in these counts as you know lazy vendor only had this one you know diligent vendor had a bunch more so it's there's no real way to differentiate you know the lazy from the diligent in this model real quick in case anyone's curious we're talking about Adobe I was going to say we we see it all the time through zdi the researchers are actually quite good about testing the vulnerabilities and I can't tell you how many times I come back and oh yeah it still works they didn't they didn't patch it you know and and or there's another vector whatever the case might be happens all the time oh I want to hear Katie about silent patchy hold on hold on one more get a mouthful of beer I want to claim that a metrics are totally fine if you understand it and it's your context and you wrote the metric start has this awesome metric that if anyone knows we still published vulnerabilities once in a while it goes from 0 to 180 two decimal points of precision so you can tell which fall is more important because there's a number and you can you can sort them which is totally worthless to everybody in the world except the people at cert and actually only was worth worthwhile to us years ago when we used it to decide whether or not to publish documenting or document be it was very worthwhile for that purpose at that time and that's it so II know it is context but right my point is is that you have that one line vulnerability and then you have a 87 line disclaimer you know rider saying this is what it really means it's subjective I mean all these a lot of the stuff is so that's kind of the trick if I might care about some vol you know you don't care about at all I might care about sakuni his balls or Microsoft I'm gonna I care about call a PHP includes you guys have maybe I do maybe I'm a PHP web app developer and I'm you're sick yeah that's my point though that's how people are getting owned PHP yeah then their metrics are bad because they're not telling them to PHP matters at the moment yeah that's what that's kind of everybody always asked about Oracle vulnerabilities unless you're litchfield nobody ever cared all right unless you're a pen tester pointing out how broken every deployment of Oracle ever is nobody cares but people get owned with cross-site and s cute you know everybody's PHP blog was going to know and left and right forward five years ago and so still are right exactly so that's my point I like the way osv TV does it because I actually know what is the attack surface available two attackers I think that's important that's entirely dependent on who the researchers are or conjugating on things and what they're concentrating on we if those of us have been in this industry for since about two thousand five or something like that remember a Latvian teenager age 14 or 15 who basically decided to spend ten minutes testing all the software that he could download and came up with 800 vulnerabilities within the course of like six weeks or something like that and just a couple years ago some guy for Debian basically used a a super-powerful vulnerability detection tool called gr EP I'm not sure what the acronym stands for and he found like five hundred vulnerabilities or something like that right and so we are still very much subject to pretty much the whims and the fads that researchers happen to go through and even one individual researcher can have a big impact on what these numbers are what were your four eyes again hmm the four eyes incomplete inaccurate inconsistent and incomprehensible and you need a fifth ignorant all right we're going to move on what are your thoughts are the value
of vulnerabilities bug bounty programs vulnerability buying and selling impact on disclosure and we're gonna give it over to Katie and you can give you a little spiel and I'll see what dad has to say then I have a question for Katie oh good go back to silent patching I believe in moderator has asked me a question so uh so the question about bug bounties and that type of thing so I don't know how many of you guys saw or heard about you know the talk that I gave a couple days ago at black hat okay all right well I'll just I'll fill you in as I go so um so I think that a lot of security researchers you know have varying motivations for what they do you know it's not all money how many of you out there you know who do this for a living I mean professionally have figured out ways to mint money on the back end of some financial system raise your hand liars come on there's more of you anyway if you want if you wanted it you know if money was it right there's a lot of unsavory ways that people with the dark arts know how to know how to get money now what you know what folks like you know tipping point do is is what we would consider the white market you know of vulnerability buying and it doesn't really the numbers don't really end up equaling anything close to what the gray and the black market will pay for right so there's a lot of researchers out there who you know think that it's a you know it's important to get recognition for you know either publicly or among their peers so when we looked at the possibility of doing some sort of a banty program a nominal fee for you know for vulnerabilities we looked at the motivations that were out there and we looked at the motivations for the researchers who are actually finding bones in our products because not every vendor has the same you know kind of profile of the researcher that looks at their code we're we're we're pretty popular popular target for research right partner right weird ya know what but we're a pretty popular target for research other vendors might have different you know different behaviors and different main motivators for you know the researchers who look at their products so we looked at what what researchers do with our you know why they do what they do with ours and what we found was this past year we've had about eighty percent of our vulnerabilities you know that were disclosed at all we're actually privately reported to us so eighty percent were privately reported you know let gave us time to fix the issues and the other twenty percent were dropped to zero day now in that eighty percent considering there are programs like you know Dan's over at a CDI that would offer you know a comparable price to a bug bounty like should we had decided to do so in that eighty percent ninety percent of those reports actually came directly to us so even though they could have made a small amount of money you know they actually the majority of the researchers you find moans in our products and want to give them to us to get fixed actually prefer to come directly to us so that's what we found when we took a look at that data now we absolutely you know are fine with the the researchers abilities to make money doing their vulnerability research and I think there's some great programs like zdi that are out there you know we love we actually talk about quality of reports actually the quality of reports that come from these guys is that is is really really good so thank you yeah no no problem but you know thank you but but hold on no no I agree with me later so but so that's what that's what we found when we looked at our data when we looked at our researchers right so instead of doing a bug bounty because it seemed like you know there's lots of ways for researchers to make that money we decided to do something different and that's what I talked about a couple days ago so if you go to www hat price calm and take a look we decided to offer over 250 thousand dollars in cash and prizes for mitigation research so we're looking for the next generation platform mitigations top prize gets two hundred thousand dollars so we're going to announce that the winners next year at black hat and the contest has already kicked off we've actually already gotten some entries to the contest and there are you know so top prize gets two hundred thousand second prize 50,000 third prize gets you know msdn subscription worth 10,000 you know and money fame I guess women as suppose if you would you know if money and fame bring women you know but that's what we decided was you know sort of the best way for us to encourage the research community to do what it does but figure out you know ways to mitigate exploitation because like I said you know bones don't pone people exploits do and we wanted we wanted to encourage the research community to work with us like that actually I'm I want to disagree because I think Microsoft is also setting a great precedent that they are rewarding not only badass exploits but the ones that are completely weaponized so your bug bounty does exist and it exists in the sense that I write an exploit it becomes really good it owns two hundred thousand machines and comes part of a botnet now you guys offer a reward for information on the botnet so my motivation is now not just to write a Microsoft exploit that was right of a badass one that's actually a different thing you're thinking about the the other reward that we have it has nothing to do with it yeah yeah you're in essence you were still offering money on what is fundamentally a very good working exploit against window systems nope that's not it so I think you're thinking about the the rustic but botnet bounty that's a completely different thing so that's a quarter million dollar bounty for info that leads to the incarceration of the people who wrote rostock totally different so what I'm talking about is this is a room I just announced this like two days ago i'm sorry you weren't looking but anyway listen so this is completely different so we're you know we're taking this approach where look there are open problems you know in modern exploitation that breaks our platform mitigations things that break a SLR and depth right so return oriented programming jet spray that kind of thing there are open problems there that we're working on mitigating so what we're actually rewarding are you know take one of those open problems right and these are for memory corruption vulnerabilities yes i know i said it and you don't like it but anyway take one of those open problems in the exploitation of memory corruption vulnerabilities and and come up with a with a novel mitigation so basically next generation aslr next-generation depth that kind of thing you know seh opie that type of research is what we're looking at and just just woke so she's asked to the question the question is will the research be made public so they can be used in other platforms the answer is it is up to the inventor the inventor retains IP ownership of that research we just get a license to use it so the inventor gets to choose what the heck they want to do with their research they want to port it to linux go for it my friend enjoy you know what i mean so yes if the researcher who wins chooses to make it public they can do so they own the IP a hundred percent from a vulnerability days point of view we can see though that to an extent bug bounties do matter and they do motivate people I've made a nice flight and and he killed it so now i'm just going to like describe it like this i made a case for instance with the see a bride stole had a fantastic track record two thousand four five six onwards they were like 80 vulnerabilities being reported one of their pride store solutions
laptops and desktops of thing is called in in 2007 and it actually triggered us and that was a time where said TI was was paying for CA reitst?lle issues and a lot of them actually came advice CDI and in the beginning of 2008 as part of our yearly report we actually then went out and said see a bright star is a solution we consider to be inherently insecure not only because of the vulnerabilities because we already talked about we can't look at that alone but we also found a lot of those vulnerabilities my research team and we could just see the code was terrible so we went out and said we consider this product inherently insecure while later ze i came up backed it up and also stated that they would no longer pay for vulnerabilities in invite store after that how many vulnerabilities have been reported so either they magically suddenly just up the quality of their product of people just stop giving her them and found other places and I think the adobe shockwave is an interesting one us because that has certainly received a lot of attention lately also and if I understand correctly you don't pay for shockwave anymore either well we had a presentation at Cannes a quest where we showed everybody have broken it wasn't yeah yeah so and I was also finding all those shockwave issues and they have some problems in some of their components so and it's it's quite realistic to also expect that since CD I won't pay for shockwave vulnerabilities anymore that we would like to see a drop in it because then people who find another target where they can get money so to a certain extent it definitely does motivate people to inches in which target they want to go for and this is one of the kind of metrics that's much more informative about the relative security of a software package then counting the wrong number of vulnerabilities that have been disclosed I can make this quite short I even questions EDI when it came out if you go back to 2005 this this room probably a little would have looked a lot different the whole industry was different the number of reverse engineers and researchers on the planet was far fewer but it was a very naive position to think that that number was not going to grow that a black market was not going to spring up and if any of you have ever read freakonomics that it pretty much proves that people there's very good positive response when there's is monetary reward and I think now is EDI is quite proven year-over-year it's more and more popular it's you know we I think we do a good job you know being responsible and being you know popular both vendors and researchers but it's you know if you look at everybody that's got their own velin programs now I think it's been I think it's been proven that it's a model that works well and for us you know the model that we chose for the blue hat prize was something where we were looking at as a platform provider we were looking at ways to to scale such that we were essentially blocking entire classes of vulnerabilities with some of the research that we hope to get out of this and you know certainly you know what simple was was hinting at is you know we're we going to share it with the community and quite frankly we got a SLR in depth from the community why why shouldn't we give back you know so absolutely i think the model that you know that we've chosen and i think there's room for lots of models here you know every vendor is not the same not every vendor is a platform provider you know what i mean so for other vendors other models might make sense but for us you know it makes sense to try and make these changes that won't that not only will impact our platform and our applications that run on it but these are platform level mitigations that will also help third-party applications on our platforms and mitigate some of those issues so for us we're looking at this you know in terms of sweeping or making much more difficult to exploit entire classes of vulnerabilities all right so a reflection of a growing trend in the area to move a bit more towards a not only defense like you're talking about with the blue hat prize but also prevention in the first place right there are entire classes of vulnerabilities we know about these in the common weakness enumeration we we document them but we still have like 800 different cwe IDs maybe 20 different ones for stuff that are related to buffer and memory corruption hers alright it myself sorry I want to get on to the next question come on this is a good one I want you guys to talk about being the people that track and deal with researchers as well as vendors name names tell us who they are how do you really feel about working with certain researchers and vendors and I know you guys are going to be you're not gonna be shy about this so he wants to talk about the research quality and gender response okay so you go first or last yeah so I've had a few problems with researchers and I I think I'm the only one out of any of us up here that will actually reply to bug track and full disclosure and call them out on it and part of that is you know yeah quit being a dick and sending this really worthless information and also just kind of teach a lesson that if anyone's reading these lists it'd strive for a little better accuracy in your reports because it's not just reflecting on you but it's causing a whole lot of headache on the part of everyone else involved you know if Microsoft gets a report and I know that they've gotten probably hundreds if not thousands of these where there's enough information and they're like wow this sounds like it may actually be an issue but the technical information isn't there and then all of a sudden they're in this like email back and forth and they spend two weeks all the figure out that well oh wait you have to have local admin privileges to do this you know so you know one of my today names you know one of the the most recent ones for me was HT bridge and I'm sure that one or two of you are in the audience hi I'll respond to your mail from three weeks ago when i get home you know they started releasing advisories and it's obvious they're using them as a way to promote their company and there's all kinds of really crappy stuff that they're releasing because they're going after beta products they're going after real low hanging fruit you know the lame yeah well no not only that but they'll find like oh here's to cross-site scripting in two different advisories oh and we've forgot your not forgot what we just kind of missed the remote code execution you know in the serious bugs in it and I don't know how many cross-site scripting issues I've seen reported that our error message that clearly indicate RFI or SQL injection yeah and they're missing these left and right and you're looking at it like you know if you guys would actually spend some time on this you would find some really neat stuff in your not and then they also have this habit of you know as an example it's like oh we're gonna contact the vendor and we're gonna give them two weeks and the fact that we type out on the email and the vendor never got it doesn't much matter you know we're gonna go and release in two weeks anyway bottom line is if you're discovering cross-site phones nobody thinks you're cool yeah yeah hey cross-site scripting is really old it's really kind of lame and it's one of those that's John over hide yeah well if you're going to do cross-site scripting just wait every 30 days and do one post with like all 750 of okay so if you can if you can own a mobile phone Matt pona own then your cross site is worth a crap otherwise disclose it to the vendor or the website or wherever the hell asked for some schwag and be done with it right and I'm fine with posting it to the list it's just don't think that it's anything other than you know a novelty for most of these and the other big pet peeve is like SQL injection it's like well here's cross-site scripting I think we'll actually include the script code to exploit it and you're like okay well this is valid and then when it comes to SQL injection they're like and the proof of concept is bracket sqli bracket wait a minute that's not proof of concept that's saying here's the script and here's the variable and wait a minute why couldn't they actually put escrow i exploit code in there is it because they're morons or do they actually think oh well if we do that bad things will happen to the 87 installs of this software that you've never heard of you know either way it's a cop-out and yeah it gets really tiresome and I want to be clear that HT bridge just kind of been my whipping boy for the past year but that's just the tip of the iceberg you know if I actually spent respond to all of these Liam advisories it would be more than a full-time job I gave up responding years ago just because of the amount of time it took to do that right so we spend time responding but it's to our researchers we don't do it publicly we do that you know we accept about thirty percent of what is what is submitted to zdi a lot of that is vulnerabilities that we're not necessarily interested in a lot of
that is crappy submissions and we want to work with the community and we've you know seen researchers come up through the years to make those submissions better that's obviously in our best interest but to call someone out i will i will call someone out and i will also give them kudos if any of you were aware of the policy change the only policy change we've ever had was EDI we now enforce a six-month deadline because there were some vendors that were kind of sitting on their hands and age gate and and that's absolutely correct yeah and so you know that's that's it's actually been phenomenal for HP because everyone decided you know what we're one of the culprits and we want to do this better when the older culprits was real networks if you go back to last year and you see how many real network vulnerability advisories we disclosed there were a lot and they took that policy change very seriously and look at how much better their software is so yes they were bad but now they're good so yes that's positive we generally experience that like in the past ten years I've been involved with VDB I actually all think that researchers are getting better they are getting better at providing the details we need don't get me wrong we're still killing about twenty-five percent of what is posted on the list but the level of quality seems to be being proving now katie has been baiting Stephen I for a while so let's go back to the memory corruption issue that is one trend that shows going the wrong way more and more people using the term memory corruption seriously if you're researcher then it's because you're damn lazy or you just don't really know what it is there are a couple of valid cases where it's perfectly fine case call it memory corruption but it's been like a thin covering image everything from a stack-based buffer flow to a use after free and he'll we've even sometimes seen it it's actually just the missing exception handling that just results in an application terminating it seems like being the standard thing oh I ran a fossa something crashed I don't really know what it is memory corruption John Sint kind of and went after the same from from vendors also in sight come on I mean the vendor should hopefully know what the whole problem is please tell us is it a stack-based possible flu is it an integer overflow is the use after free what is it I don't tell us it's a memory corruption so I'll also chime in because you know I mean obviously I'm here representing Microsoft a vendor but microsoft also you know we actually do vulnerability research on third-party products I i founded microsoft vulnerability research in 2008 to do this so we and we started releasing advisories on third-party products for vulnerabilities we found and work with the vendors to get fixed so we see it from both sides to you know we are both the researcher and the vendor and sometimes the coordinator will also mspr will step in and coordinate multi-vendor super nasty apocalypse kind of issues right and we'll try it will try and do our best to coordinate their so we feel the pain from all three roles and disclosure a lot of the time and yes some of you know some of the the researchers that we deal with are much you know much more able to articulate their issue than others you know but actually we have seen that you know same trend where they do actually get better over time and are you using it on the search side are we we stopped paying careful attention we stopped counting vulnerabilities we get maybe 30 direct reports about 30 a month so maybe one a day and we don't we don't run with all of them but probably half or more of those we go with that the only thing that really up you know really bugs us is that we get the the researcher who looking for some extra Fame and their company's not famous enough yet but maybe certain has an advisory that will help so there they'll be on us to make sure we publish something that has their name in there hasn't happened a lot in past couple years but that used to really underline you think that the quality of the incoming ports to you has improved I know it's tits all over the place there are great ones and there are horrible ones and I know I can't measure enough to really say there's a trend either direction but my you know gut feeling is it's about the same we actually see we actually see something really interesting to in that a lot of researchers are they only come to us with one vulnerability ever and they don't that they got lucky maybe or they didn't like you know doing vulnerability research anymore I mean you don't actually we don't really know what it is that you know made them come to us just one time and then disappear I think I think a lot of times it's pin testing yeah something or accidental discovery of me write something crash you bother checking it yeah I think a lot of researchers don't look for variants either I mean that was a major pain when PHP application vulnerabilities first started happening you'd have one reads or one researcher go oh I you know looked at this PHP golf application with 10 downloads in its entire history and i found this cross site scripting he needs 10 different vectors and then 10 different parameters razon like that and then like you know two days later some other person completely different reports 22 different vectors for the same vulnerability type and there's a little bit overlap but not all that overlap and it makes it very clear that you know the depth of the research is not necessarily there yeah and one of one of the last things I want to say about you know Microsoft and the fact that we are in all three roles you know of disclosure only really research you know both the finding coordinating and the fixing side but adds finders when we go to different vendors we've had to we've had to actually prove it just like any other researcher we've had to prove it to them sometimes by popping calc you know this is this has definitely happened in the course of my you know Microsoft vulnerability research where vendor just didn't believe us so we had to you know we had to show him so but part of that part of that mission for us is actually education for them right it's just like any other researcher it's education like no really this is exploitable I promise here you go and they're like what why why is this calculator showing up on my desktop I don't understand and then we use that as a way to start a conversation with them about secure development because we're saying we're saying to them look we've you know we've taken our lumps over the years we've learned our lessons in the following areas and we'd like to help you because you run on our platform we'd like to help you get better because that makes our platform more secure so we start talking to them about ways that they can catch these vulnerabilities earlier in the code but it's an educational process just like any other you know researcher who comes to a vendor you know and says hey your fly's down you might want to pull that up you know we not only say you know say that but we also you know we also definitely try to to make it so that they don't keep making the same mistakes over and over again all right so we're starting to get the hand signals but I want to ask you what this thing real quick real quick yeah just as a heads up there are multiple vulnerability databases that do this the data is not public when OS pdb has a data set we will make it public but one of the things that it's been fun tracking is what we call researcher confidence and os VTB is actually going to eventually track vendor confidence as well so researcher finds 50 vulnerabilities over the year and let's say 45 are accurate well that starts to give us a percentage you know of success rate and finding a vulnerability and at least one of the the VD bees represented here and it's not 0 SPD be tracks it even beyond that and when you start to look at these statistics you know Steve Chris and I we're looking at the data and we're like oh yep we know this guy yep that's accurate that's accurate and you know some of these is like it's amazing that some of these researchers that are well known and liked all of a sudden have a sixty or seventy percent success rate you know how many of you know that it's someone has a thirty or forty percent fill your right arm reporting a vulnerability that it's not accurate can't be reproduced or something else about it is wrong so down the road look look forward to that because I think it'll be very telling not only what we deal with but a lot of the big names that you guys recognize you know it becomes need all right so we're going to be going to the next room here he's telling me no but I want one comment from Alex and maybe our dough on what do you think about cvss that lee this into our room to thing to things with that are wonderful about cvss I all
rights it all back up my problem is CBS s is this it's a it's a attempt at formalization of something that doesn't exist I like the ratings there's nothing wrong with waiting and scoring and trying to figure out how smart something is but when you start multiplying ordinal values together you break the fundamental light that the universe works I you just can't do that and you end up with you know jet engine x peanut butter equals shiny and you're telling me that the result of shiny the second problem with it as decimals aren't magic they're not unicorn poop you can't just add them willy-nilly and suddenly it's a ratio scale it doesn't work that way the so the problem is that it may be right where you have a 15.4 is actually more severe than a 13.2 but when it is wrong because you're doing the wrong things with math it will be really wrong potentially and that that's dangerous i like it i wish they just wouldn't multiply things just give me a frickin baseball card scorecard like thing and let me look at it because i can look at that and digest it myself so there are two answers i have truly answers to that what color is that the most time in you know that was that was last night i have a message that's why I'm horse CBS s version 3 there are some rumblings within the special interest group without thinking about that for so for those of you who are stuck with see the SS version 2 with its the you know ugh warts and all if you have any concerns you can bring it up to bring it up to me or I'll named Katie as well or art because we're all one way or another kind of least indirectly involved on the sake the other thing is to address at least some of the limitations some of which you've alluded to Alex there's this thing called the common weakness scoring system which isn't at the vulnerabilities it's at the when you find a weakness indication of the potential for a vulnerability and still has multiplying ordinal values by ordinal values but it has built into it continuous values as well for those people are sort of the expert users I think we need to recognize that most people who are using CBS s right they they need a score one way or another they they all they care about is the score they don't necessarily care about a lot of the fancy math behind so my hope is that for CWS s some of our lessons learned can feed into the future CBS s alright with that thanks for your time appreciate it will be around find us for beverages and thanks again
Feedback