Panel discussion - Frontiers in Securing the Open Source Ecosystem
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 45 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/67276 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Foss Backstage 202120 / 45
6
10
15
18
28
38
40
41
00:00
Open sourceSoftware developer1 (number)Different (Kate Ryan album)Hacker (term)MathematicsCodeSoftware maintenanceVulnerability (computing)Right angleSpeech synthesisFormal languageCategory of beingEncryptionTraffic reportingGroup actionWeb 2.0Software bugPattern languageDataflowTelecommunicationAlgebraische K-TheorieBuildingPhysical systemPatch (Unix)AreaSemiconductor memoryExpert systemMilitary baseInformationExploit (computer security)Moment (mathematics)BitMultiplication signOnline helpProcess (computing)Connectivity (graph theory)Web pageShared memoryBelegleserMathematical analysisNeuroinformatikData miningPoint (geometry)Vector spaceWorkstation <Musikinstrument>Information securitySurfaceSource codeSoftwareCommitment schemeWave packetChainReverse engineeringQuantum computerCryptographyKnowledge baseWeb applicationAuthenticationSoftware testingGaussian eliminationGoodness of fitView (database)Perspective (visual)Computer hardwareNetwork topologyBranch (computer science)WhiteboardCore dumpMereologyTouchscreenOpen setProjective planeOffice suiteInternet forumIncidence algebraDependent and independent variablesPower (physics)Self-organizationComplex (psychology)Vector potentialLatent heat.NET FrameworkSequelServer (computing)Line (geometry)SpacetimeSoftware frameworkParameter (computer programming)Endliche ModelltheorieCycle (graph theory)LaptopFormal verificationEmailWindows RegistryLevel (video gaming)InternetworkingScaling (geometry)Presentation of a groupReading (process)Gastropod shellTerm (mathematics)Repository (publishing)Element (mathematics)Data managementRevision controlIntegrated development environmentLibrary (computing)Configuration spaceComputer fileBoss CorporationGame controllerLink (knot theory)Series (mathematics)Set (mathematics)RandomizationUltraviolet photoelectron spectroscopyProduct (business)Division (mathematics)Form (programming)Natural numberElectronic visual displayPattern recognitionMultiplicationEnterprise architectureComputer programmingMedical imagingType theoryLattice (order)CASE <Informatik>Roundness (object)Expected valueIdentity managementCuboidPeer-to-peerMalwareSoftware industryProgramming languageProgrammanalyseDefault (computer science)Machine learningQuery languageComputerData compressionFluid staticsInheritance (object-oriented programming)FreewareProfil (magazine)WebsiteNumberContext awarenessDenial-of-service attackSoftware repositorySound effectVideo gameBinary codeUniform resource locatorTrailIdentifiabilitySpectrum (functional analysis)Cartesian coordinate systemRegulator gene10 (number)StatisticsGame theoryFlow separationOrder (biology)Volume (thermodynamics)Service (economics)Point cloudCrash (computing)Electric generatorMobile WebChannel capacityDuality (mathematics)Client (computing)Stress (mechanics)Disk read-and-write headFuzzy logicInformation technology consultingWikiField (computer science)MathematicianSocial classSlide ruleAverageElectronic mailing listBlogInclusion mapRepresentation (politics)PropagatorData transmissionPhysical lawTwitterAbstractionProxy serverFunctional (mathematics)Information privacyMaterialization (paranormal)Bridging (networking)ChecklistCivil engineeringTheory of relativityInternet service providerOperations support systemIntelControl flowLocal ringGraph (mathematics)System callGoogolEntire functionMeeting/Interview
Transcript: English(auto-generated)
00:03
Okay, yeah, good morning, so we'll start off this morning I'd love to share with you some information about the open source security foundation the open source security foundation got started in August of last year and the reasons for our creating this foundation is that
00:21
We believe that open source software is a common good It powers the world's most critical infrastructure today There are security issues in open source software and we believe that the creating a secure ecosystem end-to-end including the supply chain all the way from open source components to you know
00:45
services that get delivered on cloud operating systems Requires all of us to participate and no one organization can do it alone So For that reason we got together with a bunch of other companies
01:01
So we've got I I'm from Microsoft. But so we've got Microsoft to IBM Google GitHub and a you know and a bunch more and Red Hat and some others that You know from the folks that are on the call today. We'll tell you more when we introduce ourselves. So So it's quite a group that's come together
01:22
the The mission for the foundation is to inspire enable the community to come together to Secure the open source that that we all depend on In our first few months we've been really growing a bunch we started off with six member organizations or maybe it was eight and
01:42
Now we are you know up to 36 we have 250 people who are active And you know We're getting I'll tell you some more about some some cool things that we're doing that are getting some good traction in the open source security foundation, we do have a number of working groups and
02:04
They work on different Aspects of security. So one deals with identifying security threats. We've got a group that's working on creating You know the best security tools and making them available to everyone We've got a group work focusing on best practices and vulnerability vulnerability is another
02:27
We're looking at how we ensure How we allow identities in the open source ecosystem to attest themselves and Then we've got another group that is looking at how we go about securing providing hands-on help for securing critical projects
02:45
So that's that's you know, some of the background for the Organization there are I wanted to share with you all some resources that are available to the community today that we provide through the open source security foundation and
03:02
those include an edX course, which is a training course that is free for anyone to take and It includes three sessions related to that and there's more information on edX and
03:23
You know, there are links to it in this slide. So we'll be sharing that later It focuses on practical steps that anyone can do even with limited resources Another thing another resource that we have available is our best practices badge So what this is is it allows
03:43
Open source up for maintainers To go through a set of best practices and indicate for each one of those if they're following those Some of those best practices might include things like having two reviewers using multi-factor authentication for code commits
04:01
setting up a CI CD system for ongoing testing things like that and At the end of going through those series of questions then there's a badge that That the project is able to qualify for anything from I forget what the starting level is
04:21
up to up to gold and silver so and then that badge is something that people can display on their website and also it's a great place where developers can go when they're trying to look at the maybe projects that they're Intending to use and incorporate in their own project to understand the security profile of those projects
04:44
Another Resource that we have is the security scorecard Scorecard and this is something that developers can run against their own projects or against other projects and get a quick look at this this Security profile for that project and the badge currently includes information like number of contributors
05:08
if there are CI tests being run against it if there are You know if there's a security policy file like etc So and that's something that you can learn more about from the OpenSSF GitHub location
05:26
Just a couple more one is we do partner with OpenSSF does partner with a number of other security related organizations and one of those is OWASP and An interesting project there that we're working on
05:43
Jointly working on is the ZAP project and this is something that developers who are creating web applications can use to So they connect through this They can connect to the ZAP proxy and then Have their you know a server
06:04
kind of Work against their application to identify security threats so that's great project and then the last one to share with you is another one that we're doing in cooperation with OWASP and There they have a security knowledge framework And so this is a place where developers can go to to get code examples for how to
06:27
To do certain tasks and implement them in a secure way and also checklists for developing Projects in a secure way and knowledge base for looking up Additional information. So those are a few of the resources that are currently available
06:45
We are encouraging people to get involved it's a great community it's an open and friendly community and We do have again from the slide If you or if you go to the openssf.org website, you can you can find out more
07:00
We've got a github mailing list slack channel blogs and and all those great things All right, so that's that's it about the open SSF back to you Jennifer Great Thank You Kay And for those that were unable to see the slides because we might have had some technical difficulties there
07:22
You can go to openssf.org and we will post the slides from today's presentation for everyone And there's a bunch of links to resources in there as well So for the remainder of the time that we have with you We wanted to do a panel where we were going to talk about Many interesting things as it relates to open source and security. So we want to talk about threat modeling We want to talk about historical examples where things have gone terribly wrong or sometimes, right?
07:44
But usually wrong we want to talk about the different ways of developers and maintainers the things they can do to make their projects more secure We're going to talk a little bit about coordinated disclosure because it's kind of mysterious and some other tips and tricks along the way So before before we get right into that
08:01
Maybe we'll give a quick introduction of each of us for about a minute or so to give you a bit of an idea of our backgrounds Start out. My name is Jennifer for Nick I am the global head of research at NCC group NCC group is the largest or one of the largest security testing companies in the world So I lead a team of a few hundred people that are both consultants and researchers and we just try to hack everything
08:22
We possibly can my background is I trained as a cryptographer and as a mathematician so I kind of grew up thinking mostly about the math of secret codes and Ultimately how quantum computers can break those But then I ran a big security team in a bank and now I'm at NCC group and really excited to be a part of Open SSF I could go next. Thanks, Jennifer. First of all, thank you everyone for tuning in to our panel discussion
08:52
Also, thanks fast backstage for hosting us Super glad to be here My name is Raul Akakula. I'm an executive director in JP Morgan Chase
09:03
I do run application security and mobile security groups there part of my responsibilities for that role is Securing the open source usage in the form and also securing the open source contributions back to the community So part of the job is also encouraging the teams to be more involved in open source security community and helping them
09:26
Do it properly, right? I've been using and developing open source security for almost like 20 to 23 years now Le versions of Linux PHP all kind of good stuff I was lucky to work on open source technology for my career to most of the time except I think couple of years I
09:46
Had to work in a hardcore dotnet and sequel server shop Hey after one year, I was successfully able to convince them move to my sequel and mono framework So I did my job evangelizing the open source software
10:02
After working for a 10 years in the development I after a few security mishaps Some are my own. I I got interested in security and last 10 years I've been mostly focusing on running the security engineering teams on the other side And how do we help the developer secure the software and code there, right?
10:24
So not as actively developing as much I used to be Because I'm running this teams, but I'm super interested still in the overall securing the ecosystem And I also serve on the open source security foundation as a governing board member and
10:43
That's how I got the privilege to work with these awesome guys on the screen Glad to be here. Okay, I can go next. So I'm Kay Williams. I Work at Microsoft. I'm in the Azure office of the CTO and
11:03
My role at Microsoft is to work on our industry Efforts related to supply chain security. So I am the chair of the governing board at the open SSF I also am the co-chair of an organization that's working on a software bill of materials
11:24
Specification and and then inside of Microsoft I work with a lot of Our teams looking at their practices for how they ingest Open source, but not just open source, you know ingest external artifacts
11:42
Scan them. We try to clone To clone everything so we've got a copy we can use in case something Goes down from an external partner and And we even try to rebuild this image Excuse me as much as we can. So so, you know that whole process of
12:02
supply chain and Consuming and using software are things that I look at for Azure but for for Microsoft And then I'm excited to be working at the industry level to partner with others And I guess I'll round up the introductions. Hi everybody. I'm krobe
12:21
Thanks again to the FOSS Backstage team for allowing us to have this opportunity to talk to you all and to my peers here from the open SSF in the industry I am the program architect for Red Hat product security Red Hat is a company that has been involved in the open source movement for well over 25 years
12:41
Product security has been securing open source for this is our 20th year. I Look so young I know right, we're responsible for the vulnerability management the governance and SDLC activities for all the products in the portfolio as well as overseeing governance for our supply chain, which
13:04
Absolutely is deeply embedded within the OSS community before that. I've worked in several fortune 500 companies I've been in banking medical and insurance, so I have a lot of experience with regulation and Looking forward to our exciting and engaging panel. I hope you all enjoy it
13:22
Awesome and on that note we may as well get started So let's just start off like why do we care? Why does securing the open source ecosystem matter and why should we care about security? I'll throw down some stats and then I'll let my esteemed friends here add on to that The Linux Foundation did a report last year
13:43
called the census of vulnerabilities in in the core and their analysis showed that approximately between 80 and 90 percent of all commercial software has leveraged open source components
14:01
So the exposure is very broad and then I went over to my pals at github and they are one of the largest Source code management tools on the web not the only but one of the large ones and just some stats There is last year alone. There were over 60 million new repositories added
14:20
They have they track about 56 million individual developers that contribute both directly to upstream or through we have private projects or corporate sponsored projects and Last year, they said they had about on the order of 1.9 billion contributions to code so from my perspective the volume and importance of open source is
14:41
Huge now yeah, and so I can speak to it for You know really why Microsoft is involved in this effort We you know, we just understand that it's not that you know in order to provide high-quality products to our end customers
15:02
we have to you know it we have to Help the help and be engaged in the ecosystem to you know To raise the quality of the products for of the software that that's produced for that everyone produces
15:20
Yeah, I could I could bring in the consumer perspective to here in my opinion the two most critical element for any company, however big or small to be succeed our innovation and
15:40
Open source software feels the innovation. There's no doubt about it. If you look at any modern application stack We have hundreds of open source component. It allows the teams to build on huge ecosystem already exists It allows the team the teams to focus on the business problem rather than the scaffolding, right?
16:01
So there's no doubt about another hand building customer. Trust is actually very hard Maintaining it is actually harder and losing it is actually a lot easier Simple data breach is what it takes, right? So in my for me Securing the open source ecosystem is very crucial to sustain the innovation for the companies to succeed
16:25
while maintaining the customer trust That they need and that's a mandatory so and as Krab mentioned like open source software development is Growing tremendously. I mean you see the numbers like how big that makes it a lot more important to secure it
16:47
to achieve these goals Yeah, I guess to build on that point. I mean we know that open source software is what underlies, you know Our core internet infrastructure all of the enterprises around the world are dependent upon open source
17:00
I imagine like a day without open source. I think every company would shut down, right? So it's foundational to everything that we do in a tech driven society and Technology increasingly takes over our lives So if we think about those ripple effects the security of these things really matters And it security is my life's work and will continue to be hopefully for a very long time
17:22
But there's a lot of I think unique problems that we can think about in the space of open source So ways that make it different from securing enterprise software There's many of these examples but a couple things would be like that. It's the obfuscated public facing source code So it's easier for attackers to take a look. They don't actually have to reverse engineer any binaries that reduces
17:44
You know the obstacles for an attacker to take a look at the source code and find vulnerabilities on top of that One of the other strengths of open source, that is actually one of the things we have to cope with differently in security Is that it's you've got this distributed community driven development. You don't necessarily know all of the other contributors
18:02
You don't necessarily know their motivations and we can't always assume that they are our friends often They are and they feel like our friends and so many of our best friends We've all met within open source But we also have to keep in mind that on a planet with billions of people Perhaps there will be malicious actors that want to introduce subtle dangerous things into our code bases and we need to be able to cope with
18:22
That beyond that. I think there's this tragedy of the comments Where we feel like just because people can look at the source code that they necessarily have looked at the source code and time And again, we've seen examples where there are code bases that support billions or trillions of dollars of industry activity that underlie
18:41
All kinds of important like public service and nonprofit projects that are just important across the board that at the end of the day were Sustained by a thousand dollars and a couple of volunteers and we'll even talk about some of those examples in this panel beyond that I mean when I think about why security matters security is a prerequisite to so many things that we value in our society
19:01
Security is a prerequisite to privacy. It's not a trade-off as is commonly said Without security you can't make assurances about what your system does and that includes how private it is and how the data is protected And privacy is important to all kinds of civil liberties civil liberties and other things that we care about within our society and wish to advance further
19:21
Security is also prerequisite to safety You can't guarantee that a system will behave in a way that is safe for the users or those impacted by the system if you Can't make assurances about what it actually does and you can't guarantee that the code isn't being played with to do dangerous things and Security is a prerequisite to ethical technology. So a popular topic in the last few years has been ethical AI and how
19:46
It's important to build Inclusive AI systems that you know lack bias and that are more Equitable and representative and that don't propagate discriminatory behavior, but we can't actually assure
20:01
Anything about an AI system or any computational system if we don't have control and assurance around that system So security is really a prerequisite to making any kind of guarantees that are so important in in society So maybe let's go into some of the historical examples we had mentioned I'm wondering if anyone wants to point us to some historical examples where
20:24
Vulnerabilities and open source components have had a large-scale impact and maybe croak. Do you answer? Absolutely, I can do that. I'll kick us off with the granddaddy of them all one of my least favorite topics are branded laws vanity or celebrity flaws and the
20:43
Incident that caused this trend within the industry was a heartbleed which was a flaw within open SSL and this technology helps secure and make Transmission between different elements on a network and between systems confidential and
21:03
It the technology was so good. It got embedded virtually everywhere in the networking world and on the internet and unfortunately, it was discovered back in 2012 or 2013 that there was a flaw that allowed The exposure of more data and as the world dug into the problem and was working to fix it
21:25
You've discovered there was really generally like two people that were maintaining the project that Millions on the globe depended on and they did not have any funding or any backing to really help them and that's where If organizations like the Linux Foundation out of their groups started to
21:42
Talk to fix this problem Yeah I heard something about heart believe that it was like they had a handful of volunteers and one person that worked full-time and like $2,000 in donations the year that the vulnerability was found and like this was for open SSL a crypto library that underlies all kinds of Like banking and secure communications and all kinds of things like that
22:02
Yep Then actually my former boss Mark Cox was on the security team for open SSL That's the whole reason I came back to Red Hat was because of this heart bleed was the incident that Made us understand we need to focus on these big things or more closely. That's awesome. I didn't know that
22:21
Yeah, another example is I believe it was the ESLint project and so there In this case, there was a project that had you know one maintainer and you know a lot of usage you know, I forget the exact numbers, but one maintainer and And he was you know, kind of he'd been doing up for a long time was moving off to something else
22:45
Someone who had been somewhat active in the project Volunteered to take over and become the maintainer and he said great. Yes. Thank you. And then it turned out that that person then committed code that you know that got distributed everywhere that
23:03
Used the you know set it set it up so that there was Bitcoin mining being done through our made available Through that code. So there's another example Yeah, the one interesting to me was the shell shock in 2014
23:21
It was that impacting bash obviously bash is everywhere on pretty much every mission You can get to write Was actually attacked as a privileged escalation attack attack. So the attacker can send a well crafted request modifying
23:41
The server to run the commands what they want to Which is it actually a high impacted attack. What's interesting about this attack was the original bug was introduced Accidentally unintentionally back in I think it's 1989 and
24:01
Shell shock was found in 2014 so you can see it took 25 years To find it and by the time you have millions and millions of the the host got impacted So that made this really what the attack is so simple anyone with like few lines of the command Can now bring down the host to distributed denial of service because you simply do sleep
24:26
hundreds sleep 200 commands with this parameters as the request and so within hours You see like thousands of service being attacked and I think the numbers were like within a week You see 1.5 million attacks per day
24:43
being carried out and this was like 2014 and sad thing is it's still relevant here because there are so so many hosts impacted there Quite a number of them unpatched it still so that makes this this open source security
25:00
it's kind of tricky is because it's so prevalent and it the patching is involved fixing the issue and also improving the awareness of is all the part of This kind of that makes it challenging and interesting Yeah, I think what's really wild about all of this when we talk through these examples is that if you look at the CVE databases
25:22
So the places where we track known vulnerabilities in known systems, there's tens of thousands Like I think there was 16,000 reported in 2018 alone and a thousand of those were critical like game over vulnerabilities So these examples that we're talking about now that is across proprietary and open source software
25:40
But still these these examples that we're talking about. They're not rare we can assume safely I think that there are many several dozens hundreds Maybe even thousands of vulnerabilities being committed into code bases today alone like just today and that's happening every day So finding scalable ways to deal with all this stuff and working closer with Maintainers with a respect for what maintainers are trying to do for their projects is absolutely essential for us from a security perspective
26:07
I'd love to talk a little bit more about threat modeling because like security is often thought of as Vulnerabilities in source code like we were just talking about but it's actually a much broader spectrum of things that we can think about You know in the software development lifecycle, so I would ask like what other end-to-end or like SDLC
26:27
Lifecycle things do open source maintainers need to think about to help secure their projects better Since I run that working group is part of the open SSF I can kick us off again And I know Rao has a lot of experience with this
26:41
From my perspective when you're talking about an open source maintainer This is they are men and women and they it's a broad spectrum of folks You'll have a one or two person project Maybe it's two ladies in Peoria, Illinois that are working on a graduate project And they push it up to a source code repo and it could be very large
27:03
Corporations like Microsoft or Intel or Red Hat that sponsor a project because it's critical to their core Portfolio, but it's just a broad spectrum and not all of these maintainers You know a have security training they might not be formally schooled and security defensive coding techniques
27:22
and they might not have access to tools or infrastructure to be able to Execute on their job yeah They have a lot of different motivations go into why someone Donates software to the world to make the world a better place and you know right off the bat some simple things you can do and again The Linux Foundation has a really good paper that kind of spoke to
27:43
Securing the open source supply chain and you can implement things like, you know, Val verifying your maintainers Using things like two-factor authentication Implementing automated testing into your CI CD pipeline, you know Just just doing very basic like dependency checks or known vulnerability checks will help eliminate broad swaths of problems
28:03
Before they get propagated downstream Yeah, I know Rao has some thoughts on this yeah, I think you covered actually really good points crap Yeah, the other is it's not just about the code. It has the vulnerabilities right like even the developer
28:21
Local Infrastructure itself is a attack vector your workstations your network your ID is Attackers can compromise them and get hold of Some of the source code which is not released to public yet They can get hold of the zero day one abilities researchers reported to you so that they could exploit them
28:42
They could even commit code on behalf of you without you realizing So there's a there's a lot of danger within that and also once the code is completed You're pushing to github Okay mentioned about multi-factor authentication for the source code repository attacker can actually target that and get hold of your credentials and then publishing to the the package managers
29:06
They can publish a different version of the package. Obviously get hold of the your credentials and then Interestingly The one other thing I forgot to mention is the develop our
29:20
Build NCI environment is another threat attacker attacker can get hold of your bill configuration They can modify for example and PMRC file and then point it to their Control registry and then could do damage, right? So I think monitoring all these Systems not just the looking for one of the built-in code is very important to keep your security
29:43
One of the things I would like to point is open SSF in the initial stages When we started we started with collecting all the threats possible in open source ecosystem and Ways to remediate so we published a paper Michael from Microsoft
30:02
Published a paper with which is like a 45 pages, but it's interesting read is published on the open SSF dot our website Please check it out. I mean we only touched a few of them. It goes over every step of this DLC Yeah, threats risks and vulnerabilities in the ecosystem and it's Michael Scavetta who wrote great paper
30:23
Really good paper and that term that thing that we're talking about right now for those that are not familiar is called threat modeling So when we're thinking about what are all the threats that are impacting this thing? Not just what are the potential vulnerabilities in the source code itself that we may have made mistakes that lead to exploitable vulnerabilities or that someone may have inserted malicious code into a specific code base when we think about that whole space of like
30:47
someone reading your emails where you might be receiving a Vulnerability report or someone taking over your account because you don't have to FA that's all when we think about our threat model we're trying to think about what are the things we actually need to worry about here and it's an important exercise because
31:04
Often we assume that people are thinking the exact same way we are or that they're motivated in the same ways that we are So we might neglect other motivations of these threat actors But also we have to think about like if you're using dodgy airport Wi-Fi or if you haven't patched the laptop you use to commit Code in a long time. Those things can be really dangerous. So we have to always think about those things
31:24
But let's get back to bones. I love talking about vulnerabilities. So let's do a little bit more of that What are the different ways of going about finding vulnerabilities in software? Okay. I know you think a lot about supply chain Can you talk to us like what that means in a security context and how we should think about that?
31:41
Yeah, so in the you know, when you think about supply chain, there are any piece of software that you Use or that you write has dependencies on other components and the dependencies You know spread out into a very large tree pretty quickly so You know when you're thinking about vulnerabilities
32:02
It's not just vulnerabilities in your code, but vulnerabilities and all the code that you bring in so, you know, there are some tools and a lot of the you know systems that the developers use for creating software now are starting to provide tools and some automatic ways to to let developers know about
32:21
About issues in their dependencies. So with github, for example, there are there's a dependency trackers Excuse me graph, I think is what it is which you know lets you see all of the dependencies dependencies that you're consuming There are a number of commercial resources that will
32:43
You know do scanning of your code and show you what your dependencies are and what CVEs exist in your code so, you know, so one aspect of Vulnerabilities is just understanding that you know the things that you bring in can bring vulnerabilities With them in the open SSF. We're also looking at doing
33:03
Some work where we're sharing And some analysis that different companies have done Microsoft and NCC group and others That talk about the quality of various components And so, you know one thing that's really good to do is just understanding what what components you're making bets on
33:23
and doing a little bit of research to make sure that the That there aren't serious issues with those Yeah, that's great. Thanks. Okay And in thinking about those vulnerabilities that can come up in our own code bases or in the dependencies that we're making use of Sometimes it's interesting to talk about like how do people find bugs in the first place?
33:43
And that's really what my team specializes in So, um, maybe I'll spend a few a few moments talking a bit about how do we actually find bugs? And there's a bunch of different ways of doing this. So we've talked a little bit about tooling and what do we mean by tooling? Some of the common categories of basic tools that people can use you can think of them as code scanners because basically they are
34:03
Static analysis is one where we look at a piece of or a computer looks at a piece of code for us and it looks for dangerous patterns within The this code that has been committed and this varies from programming language to programming language Because some programming languages are safer and they don't let you do dangerous things whereas other ones
34:21
C let you do very dangerous things So for example, if you were to talk to K's colleagues at Microsoft in the Security Response Center and you ask them Like what are these different categories of vulnerabilities which ones have the biggest impact on your systems and everyone else's systems They would probably say memory corruption where something weird goes wrong in the memory and you can actually have malicious behavior
34:45
That was not intended. That's a bit of a hand wavy explanation But I suppose like my overall point is that there's many different types of vulnerabilities We can categorize them and then when you have these tools they can look for how those vulnerabilities show up within Different programming languages so static analysis is really powerful and there's a lot of like free static analysis tools that people can use
35:06
There's also fuzzing. So this is a fun one You're basically throwing randomness at something until a program will crash and you're trying to just see where things don't exactly Behave as you would have expected and sometimes when you find those crashes
35:21
You can actually find vulnerabilities that are exploitable that an attacker could use to do dangerous things So there's a lot of ways that we can use tooling as researchers To find bugs and also as defenders or as maintainers can use tooling to get rid of bugs before People on my team can exploit them basically The the other more manual ways of thinking about this are doing things like code review
35:44
So in the same way that you would do a code review for quality within a project We would do it for security and we would look manually through it We might have tools that guide us as to where we should look more But it's often a manual kind of bespoke expert process The thing that NCC group specializes in is security testing or penetration testing
36:03
Which is basically you come to NCC group and you go I want to get hacked But I want the hackers that do it to be on my side So can you guys hack it and then please tell us how you hacked it And how we can fix the hacking and and that's what we we all do So in our client where we do this because companies hire us to do it in our research
36:24
We do it on both proprietary and open-source software. And then what we do is something called vulnerability disclosure So we find something dangerous in a piece of software We need to tell the person so that they can fix it before Someone that has maybe more evil intentions than we do finds that same vulnerability and maybe that's that's something we can talk about next
36:42
So there's this idea of coordinated disclosure or coordinated vulnerability disclosure and it's probably one of the least well-understood most confusing and contentious issues in Security people working with like developers. So I wanted to talk a little bit more about like what makes coordinated disclosure difficult
37:02
How does it work? How is it different or how does it work within? Open-source software specifically So maybe I'll just start with a couple of examples to get us on the same page and then we'll share our different perspectives on vulnerability disclosure So what do I mean when I say vulnerability? I mean that there's some flaw in software intended or unintended a lot of them are committed by mistake because they're really
37:25
Hard to spot even if you're an expert developer a vulnerability is some fun software that could be exploited or it could not an Exploit is when researchers for example hackers Malicious actors will write something that interacts with the vulnerability to take advantage of it to for example get more
37:45
permissions in a system or to do some other kind of like Not intended behavior on the behalf of the developers So to do something dangerous and when we talk about coordinated disclosure That is when the person who has found the vulnerability Doesn't want to exploit it and like mine Bitcoin or do something evil instead
38:03
We want to come back to the developer and say hey like security is my thing. I found this bug I want to help you fix this bug so coordinated disclosure is a really powerful kind of construct that we have in the security community where You know a vast majority of the people that do security and that find these exploitable vulnerabilities
38:22
Don't go and sell them on the dark web don't sell them to governments Don't personally exploit them for their own gain but instead they knock on the doors of whoever has been developing that software and they try to coordinate a way of Communicating about this and and of fixing the bug and ultimately reporting it to users So that typical flow is like we'll find a bug in something. We'll go to them and we'll say hey, we're researchers
38:45
We have this bug. We want to tell you about it We need a secure way of communicating with you so that you know an actual adversary can't see the report We're sending you and then exploit the vulnerability. We're trying to fix Often the recipient will say great like here's our secure communication. Maybe PGP encryption
39:02
sent us the report and then there's a back and forth explaining like how it works how to fix it all of that and The ultimate goal is that the developer and this can be proprietary software open source will patch the vulnerability issue an update and then Often both the developer and the researcher will issue something called a technical advisory that explains what happened
39:23
Why like what is the impact how we fixed it and all of that? But we only make it public after it's fixed so that people that have up-to-date software are safe And I guess from a researcher perspective, this is challenging with vendors and with open source projects But perhaps for different reasons
39:41
One of the things that we have to worry about is that Especially with small open source projects Not all projects have people that necessarily understand the bug reports that we're reporting because it is a super niche thing And it's not something we should expect all developers to know and they can't necessarily write the patches either If this just isn't an area they've thought about memory corruption isn't their thing, but they can build amazing systems at scale
40:04
They might struggle with like what do I do now? And how should I why should I trust this random hacker on the internet? That's telling me to push changes to my code right? Like why should anyone trust me or someone from my team? Um, so that's a huge challenge Sometimes as well maintainers won't see the danger in a particular vulnerability and they'll say oh that's not a security issue
40:23
And we'll say believe us it is So that's a challenge right because we're speaking different languages. We have different areas of expertise and We can't necessarily expect developers and maintainers to be able to understand all this nuanced security stuff right out of the box Sometimes patching isn't a priority
40:40
Open source is often done for free by people out of the goodness of their heart trying to build really important interesting innovations in software Maybe writing a patch to some obscure bug that we report to them is not a priority and I get that and also like projects can go unmaintained and sometimes we just can't even find anyone to Report the bug to and there isn't any clear way of getting it fixed
41:03
There's no way of necessarily getting a patch pushed out and those are all concerns So I think a common misconception that researchers get from open source projects, but also from some less Security mature enterprise software companies as well Is that vulnerability disclosure is adversarial like that?
41:22
We're coming to like prove that you did something bad or whatever and in fact It's it's pretty much never that that way if someone had a vulnerability and they wanted to be evil They would be exploiting it in the wild and not telling you the people that are knocking at your door Do you have your best intentions at heart in general and we're just trying to communicate in a way that works for everybody
41:42
So we're really just saying here's something dangerous and here's how we think you can fix it But anyway, I feel like I've been talking for a long time Rao probe I'd love to hear like a developer perspective and open source maintainer perspective on Vulnerability disclosure, right? I think you raised a really good point Jennifer and then I
42:02
in my opinion Wonderability disclosures are actually a good learning opportunities for open source developers and maintainers, right? So it's important I think when when the researcher reached out to them like to the Developer our maintainer is to provide more details on what is the problem provide more details on how it would exploit it
42:24
and also suggestion how to fix it and Maybe even offer The help to validate the fix right that because you know how this is exploited That's very important. That would help the developers to actually get interested in Security also, I mean, I'll tell a quick story like not gonna take much time
42:44
I was with one of the recent conferences and talking to this awesome security leader and she She is running a product security division for this security firm and few years back She was one of the open source maintainer for a package and no idea about security and one weekend
43:03
She got an email from this random guy on internet in a very little bit of threatening email saying Oh, you got a big problem in your package. You got to fix it immediately so she got ups like a little bit threat like frightened and was going through the code multiple times and
43:20
Couldn't find the problem. This looks good to me. So after a couple of hours, she end up reaching out to the Whoever contacted her and saying hey, actually I couldn't find the problem. Can you explain me? Where is the problem? So and Luckily, this guy was actually Gone through the code explained where the problem is and helped fix the issue and they were able to fix the issue that Saturday
43:45
And she was telling me this story. What did made was she got interested in security and realized? Oh, they actually can my quote do that, which is a kind of a relation, right? Like this is interesting and that led her interested in security learned more about it now
44:02
She's running a a big role in This security form so as you can see, I think that good nature of it Wonderful displays come made her career you could have been actually broke her kid and probably like Made her leave the open source because oh, this is too much work for me, right?
44:20
Because I'm not getting any recognition or not help. So I think you realize it a very good point I think that most of the one researchers actually are there to help they're not really finding problems in your code for sake of Blaming you are telling you there's something wrong instead. They're helping you. I think it's a good point Yeah, absolutely
44:40
and I mean like I think we're just really deeply curious about systems in in in a way that I think we share with developers and with maintainers but perhaps for a different angle and and there's so many ways of Specializing in computing that it makes sense that some people would be able to find these like super niche vulnerabilities in code But that that's not a universal skill set
45:00
I know that we are coming upon our time. We've got like four or five minutes left So I have two questions that I want to ask we're gonna try and fit them in So the next one I just love maybe crow But if you could just run through like a whirlwind like as an open source developer or maintainer What can you do to improve the security of your project? aside from listening to and participating in the open
45:23
Foundation openssf.org There's a couple really practical things you could take away and first off implementing things like multi-factor two-factor authentication in your project for commits Can greatly reduce the attack surface an attacker could come in and impersonate you Doing things like having implementing change control and change tracking in your project that as commits are made that's recorded
45:46
who what when where Doing things making sure that you have a unique identifier for each release that helps the downstream folks That come in and do the vulnerability response or the scanning of the pen testing later to identify This vulnerability was introduced at the X version X release
46:04
integrating security testing into your pipeline critical the more automation the better unfortunately Tooling is great But it's a lot of this is going to come down to you're gonna get the best Review by a manual peer review with a qualified security person, but tooling you can
46:23
Automate the thing to a point where you can pass off quality information to that person that might help you out Thinking about your dependencies, you know You write some software it has some functionality and you require other software outside of your scope understanding what dependencies you interact with that you require and
46:43
Listening to any vulnerabilities or reports or problems with those dependencies so you could potentially make changes if you need to You know doing things like cryptography where you're digitally signing And having be able to prove that a certain commit was made by a certain person a lot of different ways you can do that and then
47:02
Just paying attention just in general tracking or mediating vulnerabilities in your code. And again, you're your dependent projects That's important as both You know a good steward but also to protect your project your software is you need to understand kind of who else could Come in there and ruin your day because of a dependency you're using that is faulty
47:21
Thank you so much. Yeah, and I think I think these are some great suggestions and it's things we're trying to formalize and open It's a so if you are a maintainer developer listening to this talk right now And there's a gap between what we're saying and resources you need Let us know because we're trying to create the educational materials the courses all of that if you can identify gaps between what we're offering
47:41
Here and what we really need to build this bridge between these communities. Please do let us know To finish off and we might run like a couple of minutes over sorry I would love to talk about what happens if we don't intervene. So if we ignore security totally Will security and open source get better or worse? What will be the consequence and and maybe I'll kick off with my thoughts, but I'd love to hear from from the entire panel
48:05
From my view I guess from the research review There's a growing number of vulnerabilities in the wild right every day We can expect that the number of lines of code that's being committed on average is probably more than the previous day And we have no reason to believe that there's less vulnerabilities in that code Actually, we have good reasons to believe there's more vulnerabilities in that code as code bases get more and more complex
48:25
It gets harder and harder to filter out bones just through doing scanning So like as crow had mentioned It's not just about tooling if the tools can't find the bugs because they involve these magical abstractions that require some panelists Thinking about several different pieces of the code base at one time
48:41
Those are actually going to be even harder to find with just the basic tools So we're seeing an increased number of like lines of code You know open source projects every day and consequently a growing number of exploitable vulnerabilities that are likely in that code Security as its practice right now does not necessarily scale There are some scalable things but like in general it doesn't super scale
49:03
Well, and we think that it'll probably get harder and harder for defenders Especially as advancements in automated bug fund bug finding unfold So when we talk about tooling not being very good tooling is gonna get better over time And the real thing is who's using that tooling?
49:20
Is it the defenders and the maintainers finding the bugs before they commit stuff or or very early on or is it that you? Know people with malicious intent perhaps out in the wild on the internet are Making use of these tools and that's that's the big tension I think so some of the things that we worry about when we think about Attackers getting really good would be like large-scale fuzzing
49:42
Innovations in something called program analysis wiki that if you've never seen it. It's a ways of looking at code for vulnerabilities But we've also seen query languages like code QL where you can find classes of vulnerabilities across many code bases We've also seen some good work with machine learning over source code to identify certain patterns in which vulnerabilities tend to show up
50:03
There's even a field in security known as automated exploit generation AEG Which is all about if you find a bug how to change stuff together in an efficient way so that you can start exploiting it Quickly, so there's a lot of things that will scale up that are scary for defenders And I think that it's all it's all about
50:21
Understanding and making use of those in a defensive capacity over time and by having intervention interventions like what we're working on at open SSF we're trying to create a You know body of knowledge and body of work where we're able to scale with Lee make use of these Innovations to help mix it open source software more secure But it's always that tension between the dual use nature of using it for defense or using it for attack
50:45
But I think there's reasons for hope and and I think open SSF is trying to build those reasons Others what are your thoughts? What what will happen will security get better or worse? Well, I I think if you don't intervene, I mean, it's not gonna get better unless
51:02
You're talking about Bitcoin right then. Yeah, it always got better without Oh I think you well, I think you've raised a good point on Everyone writes good point not technically to I think as we all know The open source is tremendously growing
51:21
Wonderabilities are growing and also we're seeing highly complex not well-known Wonderabilities introduced every year So I think we all need to work as a group maintainers researchers and corporations To address the problem and it if we don't do it we lose this opportunity to fix in the beginning before it is
51:40
Uncomfortable, right? And I'll finish with one part It's not always technical though and most of the maintainers and developers Actually are employed by other companies and they have a day job So you have to work on other stuff other than these projects So we can't put a burden on these developed developers and contributors and maintainers to keep up the software
52:02
so for that I would say talk to your employer talk to your employer and convince them that why this software is helping the other corporations and why How we are actually taking out from the corporation so that you get a dedicated time to work on these open source projects I've been doing in the companies
52:21
I've been working and leading is all my developers get a certain time of Time of the week to work on contribute back to the open source projects I think every corporation like that because it's changing and I see the trends in they're realizing the value of the open source So they are okay to contribute back So I would say use that and that's the only way we can scale up. I mean researchers cannot help
52:47
Single-handedly our developers cannot have that that much burden working on multiple jobs. So that's my final thought Yeah, so I you know, I'm an optimist in that I think that there are you know changes coming along and you know things being automated
53:07
That that will help us. I don't think that this will happen on its own. I think that you know without concerted effort Security doesn't get better But I think that you know, we are putting that concerted effort in
53:21
I think that there's there's kind of a carrot and a stick aspect to this that will make it You know really better over time one is that developer tooling will get better about having Security by default requiring security as developers create new new projects providing information to developers about vulnerabilities and steps that they can take and best practices
53:46
Another piece I think that we'll see Happening is some automated infrastructure that allows Capturing information about what was done during the creation of software and then applying policy about what will you know? What can be accepted on the other end and with that cycle of somebody setting policy?
54:05
This is I I will only take software that's been Committed where the code commits are done using two-factor authentication then all of a sudden You know the system starts to build, you know, even more than it is already starts to build that in so with this
54:20
Automated system of providing information about what was done Having policy. This is what will be accepted and some verification of policy That's some some fun stuff that we're looking forward. Well from a corporate point of view that we're looking forward to in the future From my perspective we have a bottle where
54:43
The ecosystem is getting exponentially more complex You look at vulnerabilities like specter and meltdown where it was a incredibly technically difficult Problem to find and then solve and it involved a lot of different stakeholders across the industry ecosystem
55:01
It was a both a hardware and a software problem. It was a hardware problem that had The software was had the ability to help correct And so we're seeing more of these blended attacks things are more complex as that dependency tree branches out But I am very optimistic about it. We've been involved in the community
55:21
for decades and I think this is the most exciting passionate talented group of developers and practitioners that I've ever had the opportunity to work with and you know initiatives like The open SSF things like the forum of incident response and security teams we're all starting to talk to each other and the power going to solve this problem of this complexity and this
55:45
kind of snowballing of the potential for vulnerabilities and threats is Communication that having these good practices like coordinated vulnerability disclosure Having policies or setting expectations and be able to communicate between the different types of people involved in this process and a lot of the breakdown
56:04
When things go poorly, it's it's a mismatch of perception that the researcher doesn't understand the developer And the developer doesn't understand the researcher and the more we talk the more we have these forums and have Resources to help each other out and kind of educate each other the better off. We're all going to be
56:23
All right on that note, I think that mostly summarizes what we wanted to talk about today Thank you everyone so much for joining us for this panel for staying a little bit late Because we had more thoughts we wanted to share and especially to FOSS backstage for hosting us today We're happy to take questions here or in the question meeting room or something that I think exists. So thanks everyone