Are You In Yet? The CISO's View of Pentesting

Video thumbnail (Frame 0) Video thumbnail (Frame 8783) Video thumbnail (Frame 17566) Video thumbnail (Frame 20270) Video thumbnail (Frame 20720) Video thumbnail (Frame 25074) Video thumbnail (Frame 27026) Video thumbnail (Frame 29279) Video thumbnail (Frame 30029)
Video in TIB AV-Portal: Are You In Yet? The CISO's View of Pentesting

Formal Metadata

Are You In Yet? The CISO's View of Pentesting
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
When a CISO pays good money for a thorough pentesting, she wants results. Not necessarily the ones that the pentester had in mind, either. Whether the time allotted is too short, the pentester has to achieve multiple objectives, or they disagree on the severity of the findings, both the CISO and the pentester have to agree on both sides of the engagement. We discuss numerous aspects of voluntary pwnage: the differences between a security assessment and a penetration test, what color of box works best, tweaking the objectives for more targeted results, and ensuring a happy ending. @shrdlu has worked as a CISO since 25 years past the epoch, both in the public and private sectors, and has grown to enjoy the exquisite pain of being on the receiving end of a pentest. It should be noted that @shrdlu is not speaking on behalf of any employers, past, present or future, did not test the presentation on any live animals, and will not be dispensing any sort of legal or medical advice.
Slide rule Length Multiplication sign Modal logic View (database) Design by contract Black box Mereology Checklist Product (business) Different (Kate Ryan album) Analogy Cuboid Software testing Information security Vulnerability (computing) Physical system Standard deviation Electronic mailing list Library catalog System call Social engineering (security) Hand fan Data mining Word Message passing Process (computing) Personal digital assistant Self-organization Website Video game Escape character Quicksort
Mobile app Existence Multiplication sign System administrator Firewall (computing) Design by contract Mereology Proper map Vibration Energy level Software testing Traffic reporting Vulnerability (computing) God Software developer Electronic mailing list Staff (military) Bit Software maintenance System call Word Data management Process (computing) Integrated development environment Personal digital assistant Self-organization Figurate number Quicksort Window Reading (process)
Information Physical law Function (mathematics) Client (computing) Personal digital assistant Enumerated type Self-organization Software testing Quicksort Traffic reporting Information security Vulnerability (computing) God
Game controller Multiplication sign Connectivity (graph theory) Content (media) Counting Perspective (visual) Hand fan Software bug Revision control Data management Integrated development environment Personal digital assistant Internetworking Self-organization Software testing Quicksort Table (information) Traffic reporting Descriptive statistics Social class Vulnerability (computing) Physical system
Data management Information State of matter Object (grammar) Traffic reporting Equivalence relation
Presentation of a group Mereology Subset Product (business) Process (computing) Personal digital assistant Energy level Software testing Right angle Traffic reporting Position operator Vulnerability (computing) Physical system
I'm going to talk to you guys today about the scissors view of pentesting you know I never thought that I would actually be standing up in front of a room full of people talking about how I like to be penetrated so yeah but it's Vegas so I guess everything goes I think I have to define pentesting first because there are a couple of different ways that you can slice it first of all there's the do I get in you know guns a-blazing sort of pentest and then there's the horizontal sort of assessment where you're trying to catalogue all of the vulnerabilities at a given site instead of a binary do I get in or do I not get in and that's whether which one to use is actually a pretty big problem especially when it comes to a pen test that is supposed to happen for compliance or for due diligence so that when I when I was ordering up pen tests in my previous life as a CISO I would do either one depending on the requirements and one at one of the things that I heard about at shmoocon I think was last year there was a big debate about whether it made whether it counted if you didn't exploit a vulnerability when you did a pen test and so people were saying no no you you know you have to exploit it when I was asking for pen tests you know on production systems I would say do not write anything anywhere if you tell me how you're able to get in I'll take your word for it but there are some pen testers who felt that that was cheating and you know that was unsatisfying and so I just wanted to say that it's not about your satisfaction it's about mine especially since I'm paying the money so if I want you to go low and slow or really fast you know that's that that's my call not yours so you know there's a thing there's just so many analogies that I just can't escape here so I'm just going to relax and go with it so pen testing I have spent a lot of time trying to figure out what it was when I engaged a pen tester what they were going to do for me or to me or whatever and in some cases they were an external organization that was trying to verify my organization's compliance with some sort of security or simply to gauge whether we had security or not in other cases it was for our internal use and I simply wanted a list of everything that you know we might possibly need to fix in a lot of cases I'm really not a fan of a black box test because if I'm paying thousands of dollars you know between fifty and a hundred thousand dollars for a pen test I don't want that person to spend what short time they have looking at things that I already know about so what I would usually do is a sort of a semi crystal box test I would give them the list of all the things we already knew about and say okay the rest of the work is for you to tell me what I don't know and that could involve anything from social engineering to you know simply running necess now I have to give a shout out here to the pen text execution standard folks because I think that is such a useful standard that's coming up for those of us who need to order up a pen test and need to be able to specify exactly what it was it used to be that you know I couldn't list for a pen tester everything that I wanted them to do every tool I wanted them to use every technique because obviously it depends on what they found in the first pass so I would generally just have to talk to them and try to figure out from that if that they knew what they were talking about so then I could trust that they would actually you'd be able to do a good job for me and especially when you're putting it in a statement of work and you need to be able to enforce a contract that is a statement of work you need a lot you mean to a binary list where you can say you did this or you didn't do this because otherwise it's not enforceable unfortunately that turns into a checklist and I know people hate checklists but again for the purposes of a contract you have to be able to make sure it's enforced or not enforced in case you have to go to court so having something like the PT es as a list and being able to say okay you're going to do these parts these parts these parts and here's the time length that it will take will also help the pen testing organization price this appropriately because again when they come in with a bid it's very hard to figure out from the pricing what they're going to do how many people are going to put on it and how long it's going to take I have to apologize for not having any slides I just didn't feel like it but I would like to I would like to call up a stunt pen tester to the podium so this is mr. Joseph Sokoli who is who is just plain Austin see so if I start saying something stupid you can watch on his face as he starts reacting to what I'm saying the other problem that both scissors and pen
testers I think have in common is that there is never enough time to do a proper pen test I have talked to some folks at very very large MSSP organizations who the actual testers would be given maybe three days to pen test an app they would come at the last minute and say okay we just signed this contract you need to be done by Monday and of course that there's no way that you can do a proper job in that amount of time know that actually that happened to me once when my management came to me and said you know we need to do diligence on this product and by the way we're going to announce that we're purchasing it on Monday and this is Friday so Kay can you test it over the weekend and and my colleague said some really bad words for a very long time and and then we got to work and we worked you know all weekend now I can't resist as a SISO sometimes playing with my pen testers and for example there was one who gets when he's going to move there was somebody who was we knew I knew him pretty well he was going to do a pen test on us and we were going to do some maintenance on our firewalls during the the window of the engagement and we thought gosh how are we going to keep him from doing this and we knew that he tended to sleep in late so we figured what we would do is send somebody to get him drunk so that he would sleep in really late in the morning and that would give us time to do the firewall maintenance before he started testing another thing that we thought about doing was was what I call port flashing we would you know open something and then the next time he came back with another scan it'd be closed again so it'd be open and closed and open and closed and I really like to see that report you know when it came back I don't know it was there what yes it is horribly evil but but seriously you know in a dynamic environment you do have to deal with that sort of thing you know people are bringing things up and down and you just have no idea you can't have everybody stop while the pen test happens for two to four weeks or however long you're doing it so yeah there will be something that will be on a finding and you'll go back and go oh it's gone now so the are you in yet question because I can't tell if you're in yet that was that was the sort of that was the question I always asked you know and I would I would you know sleep with my blackberry at night waiting for it to vibrate to tell me that yeah me and my lonely I'm my loneliness and my blackberry waiting for them to tell me that they were in and you know when I got the call I would immediately have to you know get some people on whatever vulnerability it was that they found but yeah that was the the highlight of my existence is waiting for that waiting for that vibration waiting for that call now let me talk a little bit about reporting because that's where in my experience pen testers tend to fall down they want to write about the really sexy stuff and they want to do a couple of screenshots ago yeah you know this is where it was late wide-open I really love this part but that's not as helpful to me I mean first of all it makes very lurid reading for the auditor and they get very excited about that but but but it doesn't you know do a whole lot for me what I need for what I need a report to say is not only what what the vulnerability was how it was exploited what the potential impact was and we'll talk about impact in just a minute but how likely it would be that somebody would be able to do it what the skill level was that it would take to to exploit that vulnerability and then from there I would have to figure or who would want to do that because the other thing is when I would take the these lists of vulnerabilities these findings to assess admin or a developer they'd say well who would want to do that you know our users don't want to do that and I'd have to explain that it's not about our regular users so talking about the impact the other thing that I really hate is when I have a purist pentester who is like oh my god everything is so bad and this is not useful to me because if you list for example that we're still supporting we're still supporting SSL 1.0 that's the sort of thing that the auditor gets really excited about and I have to explain that this is not a big deal yeah yeah who cares debating criticality both with an auditor and with the staff who need to try to prioritize a fix for this is very frustrating first of all auditors generally don't tackle risk and I know that sounds kind of weird but if you've worked with an auditor they are not going to talk probabilities with you they are going to talk about you know it says here in my list that this is bad and you have this therefore you have to get rid of it so in some cases I will really prefer it if you tell me verbally what what the issue is and not write it down because I don't want to have to deal with arguing with an auditor over whether this is something that should be fixed or not especially if it's something that we can't fix or can't fix readily we have to support really ancient browsers out in the wilderness or something like that so you know I just don't want to go there so in in and
the other thing that really drives me crazy is if there is a finding that is that we're doing something on purpose so things like telling the user when they've gotten the username wrong you know everybody says oh my god that's terrible you can enumerate user names well I'm sorry but we have the kind of users who forget their user names and it is a huge support cost to my organization to have to talk with them on the phone to figure out what their user name is and them instead of just telling them that they got the username wrong and they think oh yeah I remember there was supposed to be a one on the end so I'm sorry if you tell me enumerated usernames is bad I'm going to say I'm not going to be really excited about that so there's a big trade-off between the things that we do on purpose and that we really don't consider to be high impact vulnerabilities and the things that really are you know oh my god you did what sort of things and I've had you know plenty of those too it's some pretty scary things coming in so by and large when I get a report from a pen tester I want it to be I only wanted to be the things I really care about and I'm sure there's some people who are going to argue with that and say you know if if they're gonna do their due diligence as a pen tester they need to list everything otherwise it'll look like they didn't find something and that reflects badly on them what do you think Joe yeah it comes down to what the client needs the the other thing is that in some cases particularly in the public sector the Public Information Act may allow citizens to request information that would really be sensitive such as the output of a pen test now in some
cases the law allows the the particular agency to say they're not going to disclose that information because it has to do with security of the infrastructure but in some cases that that law isn't there and so the only
thing that might protect the contents of that report are to call them auditors working papers or something like that something else that is protected from release otherwise they can if they're asked for they will be released they could be posted anywhere on the internet and I'm really not a fan of that and neither is my management so how am i doing on time I'm doing it okay so defining and time bounding pen tests or assessments and then figuring out exactly what you're going to do and what you're going to look at and how long that's going to take I know that on in some cases the scissors can be really unreasonable on that like here's a Class B I'll see you in two weeks that that's just that's not helpful to anybody although I like to do that too just to see their faces so working out that at that time to engagement and then and telling them everything I already know so that they don't go over that material again because the time is very precious and it's very expensive and then getting the report back which talks about the things that real I really do consider to be critical vulnerabilities the one other thing that I really appreciate from a pen tester is the description of how easy or hard they think it is to fix and that's another thing that really bugs me about some pen testers who have never actually been on the fixing side of the table where they say oh well there's just no excuse for that and they have no idea how difficult it is in an organization sometimes either to fix something in a legacy environment or an environment that includes third-party components that we have no control over so there are a lot of things that are just not plain not going to be fixed and everybody knows that everybody understands it but the pen tester you know who believes that everything should be fixed is not very helpful to me they need to be more realistic about the business impact of what they're finding and disc and be honest about how much work it's going to be from their perspective and then I take the organizational overlay and say well yeah yeah that that might sound really easy but for reasons of our own organization how its structured or how the the infrastructure is set up that's almost impossible to do so it may leave the pen tester really frustrated but I don't care because as long as I'm satisfied that's what counts because I paid for it so I wanted to ask for some
pen testers in the audience who disagree with anything I've said thus far and say yes and and tell me about it I'll repeat it after I after he says it yeah use your outdoor voice okay so if I understood that correctly that the question is that you don't feel you can leave off any sort of finding even if it is on a legacy system or a third party system that can't be fixed and so you're asking what what I think I should do about that in some cases I've seen two versions of reports one is the well one is the verbal report where they will tell me exactly what steps they took what the vulnerability was what they got and then there will be the for general publication report which will
simply say the equivalent of you know there was something really bad here and it might allow access to personal identifying information so I know in my working notes exactly what the problem
is and so I can follow up and see if it is really fixed but in any document that might have to be released either - you know senior management or to the public it just says something very vague and actually state auditors will do the same thing they will you know if they're looking at something and they they have a finding they would often call me up on the phone and say okay here's the finding here's what we have a problem with but then in their report they will just say oh there was some general badness here and that's the agreement that we have both to get it fixed and yet not to make anything too explicit and writing that might come back to bite us does that answer your answer your objection or is that still mm-hmm yeah yeah so you're already doing that one for the engineers one for the auditors and everything yes sir okay
okay okay good very good okay so what he said was he felt that my my requesting reports verbally without putting them in writing might be seen as a as a way to skirt regulatory requirements or reporting requirements and that strikes you as unethical that's very fair and I think that in some cases especially this is really sticky stuff especially when
you're talking about the public sector because there's the need to protect the infrastructure and that includes not publicizing vulnerabilities but there's also the requirement to fulfill any any reporting requirements that are there the thing is though that especially in legislation legislation is very very high-level it doesn't say what you have to report in a pen test so it's often up to my judgement to say here's what I think the public needs to know or deserves to know and here's the stuff that I think they don't need to know and yet it may be in an internal report that I give to my management and say you know here's here's what they really found and this is when it's going to be fixed but there's a there's a big you know gap in there yes sir you're in the back okay okay okay that's a really good question his position is that as a pen tester if he doesn't actually document and present everything it doesn't cover his ass sufficiently so if that system ever gets ever gets breached by something that was you know trivial to find and should have been fixed and everything that it might not he's afraid that it would come back to bite him and someone might think for example that either he didn't find it which would you know really reflect badly on him or he reported it but you know we chose to ignore it and you know you want to make it very clear which part of the risk acceptances ours and which part was your you doing your professional job right and making sure that that everything you found was communicated in a particular fashion and in some cases it just comes down to level of detail I mean there are a lot of things where I would say yeah you know we have to list that this was found but we don't have to say exactly what steps you took to exploit it now whether you feel that that's not being forthcoming enough with an internal organization or say with an external product that's something we can debate in the QA room because I have like one minute left so I will wrap this up but if you would like to talk about this more thank you very much and good luck to all of you out there