We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Ramping up Security at an Open Source startup Lessons learned

00:00

Formal Metadata

Title
Ramping up Security at an Open Source startup Lessons learned
Title of Series
Part Number
65
Number of Parts
110
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
19
20
Thumbnail
44:46
23
30
Thumbnail
25:53
69
Thumbnail
25:58
76
78
79
96
97
Open sourceInformation securityDiscrete element methodFreewareSynchronizationHill differential equationCodeProcess (computing)Formal grammarFluid staticsSource codeMathematical analysisSoftware testingRegular graphInsertion lossVideo gameWeb pageBit rateView (database)Water vaporVector potentialLattice (order)Physical lawMultiplication signSoftware developerMetrePhase transitionCategory of beingRight angleBitState of matterLevel (video gaming)Endliche ModelltheorieSoftware testingDemosceneMathematical analysisArithmetic meanWorkstation <Musikinstrument>Shared memoryInstance (computer science)Domain nameInformationTheoryClosed setAuthorizationMusical ensembleCondition numberMereologyMathematicsMedical imagingSystem callGroup actionWell-formed formulaVulnerability (computing)Programmer (hardware)Line (geometry)Process (computing)Connectivity (graph theory)StatisticsSoftwareCASE <Informatik>EmailGoodness of fitCartesian coordinate systemMessage passingSource codeProjective planeCodeNeuroinformatikAxiom of choicePoint cloudInformation privacyNumber2 (number)SynchronizationRegular graphClient (computing)Web 2.0Rotation1 (number)Information securityFunctional (mathematics)Fluid staticsData storage deviceSingle-precision floating-point formatDifferent (Kate Ryan album)Student's t-testSubsetOpen sourceComputer fileServer (computing)Universe (mathematics)Electronic mailing listTotal S.A.XMLComputer animation
Information securityRegular graphCodeMathematical analysisFluid staticsSoftware testingOpen sourceDiscrete element methodSoftware developerAuthenticationPhysical systemDirectory serviceServer (computing)Computer fileFunctional (mathematics)Open sourceTerm (mathematics)Exclusive orMultiplication signFluid staticsComa BerenicesWrapper (data mining)Medical imagingSource codeAddress spaceCode refactoringGame controllerDefault (computer science)Connectivity (graph theory)Vector potentialVulnerability (computing)Information securityTraffic reportingBitPoint cloudLink (knot theory)StatisticsStandard deviationProjective planeSoftwareMessage passingNumberFile systemDigital rights managementRootComputer programmingCausalitySoftware bugData storage deviceInternetworkingFlow separationAuthenticationSoftware testingMathematical analysisCodeType theorySoftware developerPatch (Unix)Different (Kate Ryan album)Process (computing)Video gameStudent's t-testUniverse (mathematics)Volume (thermodynamics)Right angleFood energyLine (geometry)VideoconferencingState of matterGroup actionMereology1 (number)Form (programming)Physical lawSystem callDataflowIntegrated development environmentView (database)CASE <Informatik>
Open sourceInformation securityDiscrete element methodImage resolutionData typeSoftware testingLinker (computing)Level (video gaming)AdditionInformation securityPasswordTraffic reportingRevision controlCoefficient of determinationProduct (business)Noise (electronics)Software bugComputing platformComputer programmingLimit (category theory)Projective planeWeb page1 (number)BitWebsiteSoftware developerInformationForm (programming)Multiplication signFormal languageMoment (mathematics)EmailPatch (Unix)Software testingFunctional (mathematics)AdditionHand fanEncryptionServer (computing)Computer fileCryptographyStatisticsMathematicsTranslation (relic)Open setSoftwareMountain passWordVideoconferencingEndliche ModelltheorieResultantImplementationState of matterPlanningMatching (graph theory)Video gameDataflowEvent horizonPhysical lawGroup actionGame controllerMathematical analysisFigurate numberUniform resource locatorLatin squareMereologyLevel (video gaming)Utility softwareLecture/Conference
GoogolCore dumpComputer animation
Transcript: English(auto-generated)
Hello, who am I, so if you ever went to who I made it with, this is my mail address,
I really love the new top level domain and in my actual life I'm a professional stock from the model, but Brunei needs more cybersecurity personnel but I didn't get any job offer from Brunei yet, so if anybody from Brunei is here, this is my mail address, right, so
who am I actually, I'm a contributor to own cloud, I'm contributing to it since 2012, I'm employed since 2014 and I'm the number 4 contributor meanwhile, which means I have fixed a lot of stuff, right, and before I go a little bit into what is own cloud actually and what we did at own cloud, let's first look a little bit about what is the
good side and the bad side of the cloud, so cloud sounds awesome, cloud is awesome, everybody loves cloud, so there is something called the awesomeness of cloud, if you talk about the cloud you have your data everywhere, you can go to other countries, use another laptop, have your data, right, your data is nearly back up online, you
don't need to do anything, that's pretty cool and it's all pretty much cheap, nobody is going to charge you for it, lots of money, right, so if you talk about cloud you always need to think about and display an image from the FSFE, there is no cloud, there are just other people's computers and that's fair enough, but do you really
know what other people do with your data on other people's computers, well they tell you they won't do anything bad with you and that's fair enough and you can believe it, that's fair enough, you may even live with the idea, you might even be okay with the idea, others control your data, that's the choice of everybody, but with oncloud you basically had the idea to take back your data, so take back your
data as store the data where you want, now a little sales pitch, what is oncloud, it's a single share solution and the elevator pitch is like, it's like Dropbox but open source and store it on your own server, we have meanwhile over 8 million users worldwide, we don't know the actual number because it's
just a software that you install, right, and it has pretty much all the functionality you know from other products, it has that synchronization client for Android, iOS, Mac, Linux, Windows, web interface, that sounds pretty good but that's not how we have started, so how we have started was pretty much different, so when we started it looked like this, right, and it's a
rotation interface, you were able to upload files, you were able to delete files, but that's pretty much it and we need to look back a little bit to reflect what we have done in security back then and what we have to do now, so the project has grown immensely over time, we have started with like
2 to 4 or 5 contributors, to nowadays we have over 500 contributors in total that has contributed, we have like, I think there are 70 different contributors a month, that's pretty okay-ish, companies actually started to use the solution because there was the idea, hmm, Dropbox is great, I love Dropbox
but it's not really compliant with any data protection laws in Europe, right, so that's actually why they started to use it, and meanwhile even universities started to use it, we have some prominent examples for example here, we have the Batikran and the Deutsche Bahn both, not really, well some people like them, some others are not right
and on the innovation side, we have for example the Seron and lots of universities worldwide, and they actually love the idea that you can share files over different instances, so you can share, as a student at ETH Zurich you can share data with somebody at the University of Vienna, pretty easy
and that sounds pretty cool, but we also need to look back at what was the security at the start, so when we started it was like one person who had the idea of I want to make the world a little bit better, everybody should install the Seron now, right and so the idea of the security at the start was basically
there was a good repository, everybody could push directly to it, you just needed push permissions which were pretty easy to get actually, there was no formal code reposess so everybody could push code as they like, there was no static source code analysis manual security testing or dedicated security personnel, and to say it a little bit evil
it's probably the same as still in many companies today, right we all know how it is in many companies, so that sounds actually pretty terrible I know this, and we actually have been thinking what do we need to do to make it better or at least a little bit better, right, the first thing that we actually have done
is introduce pull request reviews, and people are always like what, that's so that's a default, you need to do this, and right, you need to do this and we introduce it a little bit too late, but we have it nowadays, so we use GitHub and we have the police, we have component owners and two other persons need to review this
and I will show some statistics later how this has affected the amount of vulnerabilities tests gone down recently, and some nice quote about pull request reviews is the following one, as programmers to review ten lines of code they will find ten issues, ask them to do 500 lines of code, they will find 500
it's actually true because there are many pull requests which have like 1000 lines of code changes, and people after 20 seconds after it has been submitted there is a plus one, yeah, did we look at it so you really enforce it, right, tell the people hey maybe make 20 small ones
instead of one with 2 billion lines of code, yeah so what we also do now is regular code reviews for security issues that's something you can pretty easily integrate into any quality assurance process so what we have, we have for each component we have created a list of acceptance criteria
this application should probably check the permissions of the user before it allows editing the file, right, and then there is some state like pass or fail, and if it fails, it fails this is a pretty good thing to do, especially if you involve the developer itself because before developers don't really tend to think a lot about
what is required to make the software secure, what edge cases are there but if you force them to come up with a threat model, they actually think about what could go wrong that's good stuff automatic analysis is something that I also recommend, right, we use for example
dangerous usages of PHP functions, we have written in PHP in the server side and then we can see the stack trace and look whether this is a legitimate issue
or whether this is a false positive, this is pretty good to do it costs a lot of time, so if you do something like this on a regular base you need probably like one day per person to look at it for a week that's quite a time to do it and now comes some marketing lingo again, so customers to perform security tests
we have customers and we try to follow industry best practices I love this term because there is no actual definition of what is industry best practices and while there are some ISOs, these ISOs are like, yeah, you should have a security a company dot com address and it would be great if you also respond to these messages
but nobody knows this if they don't have read this pretty easy to fulfill we are very extensive about security issues we have found, we do a full blown adversary including a CVE, vulnerability description, root cause analysis and stuff like this
pretty much standard in many projects nowadays, not in all unfortunately so we also do lots of security hardenings, this is just one example what we have added in a recent release and why do I show this, it's a little bit because of a topic that I will show in later, so if you talk about on cloud
there are basically two kind of people, no three, there are the people that don't know on cloud then there are the people that like on cloud, I like them and then there are people that don't really like on cloud, let's say they hate on cloud and I don't hate them I just don't like them as much as I do so this is one example for haters gonna hate, right
let's go with somebody who has congratulated us for our fifth birthday, thank you very much Andreas happy fifth birthday on cloud, here is 96 reasons to avoid you with a link to the 96 security advice that we have published and that's fair enough, I mean you can create something like this but we also have found some statistics in the internet and this one is actually true as well
if you look for the internet is for our rights, nobody will be heard again so what I'm going to show you with this is if you take something like statistics always look at it at least three times so now I show my hopefully less statistics
here we go, these are the fixed vulnerabilities per year you see 2012 and 2013 is pretty much fucked up you have like, I'm allowed to say this right? but we pay for bugs later
so actually because we are becoming lazy we don't fix any bugs anymore and this actually looks pretty promising, it goes down and we can discuss the reasons later right? and if you look into who has found the vulnerabilities it even looks a little bit better for us
who has found the vulnerabilities and how are we doing this is the amount of security patches that we have made not necessarily the amount of security patches and who has reported them so we have three different types of vulnerabilities, low, medium, critical low is anything, you don't really have to worry about too much
medium is you should better patch it and critical is you better get up at night so as you can see most issues are nowadays found internally which means that actually having code review processes helps especially if you look into the data ideas so this is the severity by the reporter in 2012
you will see a lot of vulnerability reports and also externally twice reports if you go now into 2013 it looks a little bit less reports but still a shopping high amount of external reports right? now 2014 looks very better, there are still critical issues but they were all found internally fixed
so we have searched them on our own and fixed them 2015 again somewhat better right? how have we done this? actually it has involved quite some refactoring and rewrites so nowadays security checks have to be disabled by the developer right?
previously you always had to opt-in for a security check like this controller needs to perform this security check what we have nowadays is you need to opt-out of a security check so by default all security checks are performed and then you define something like this is a public page, make it public, no authentication checks
and there is no process you can virtually check your files at all and it's much harder to and much easier to forget something instead of if you have to write some exclusions way harder to fuck up we have done some more stuff so we use an internal file system
which is basically a wrapper around the storage we use and they are not vulnerable against direct introverses anymore let me show you an example so here you define the folder you want to change it into and then you say I want to fopen myfile.txt which will work and what won't work is something like dot dot slash dot dot slash myfile.txt
this makes it a little bit harder for developers because they need to come up with some workarounds around how can I go up a folder and stuff like this but it's actually quite an interesting approach if you prevent some critical use of vulnerabilities you just need to teach them how to use it and we also have enforced some security functionalities nowadays
like content security policing since 2013 which prevents the execution of inline JavaScript so something like XSS is not a problem if you don't use enterprise-grade software from the house of Microsoft because Internet Explorer doesn't perform any security checks here from CSB
we have also blacklisted potential dangerous PHP functions so we have written a static source code analyzer on our own it's not completely written on our own we have taken some open source components and then we look at the source code and be like, yeah, this is a potential insecure function
maybe you want to use the other one and this helps a lot because people don't know all the time what could be dangerous and what could not but we all know security is hard, right? so this is a nice image of Michael Pollock Michael Pollock is a great artist and he had the idea of I will give my images just a number
so this is number of I don't know, either nine, twelve or two and anyways he don't want to influence you with anything in advance so you should find on your own you should find on your own what it is so this could be like, for some it is a teapot for others it's an elephant
and for others it's just a few lights and for security it's the same if you have different people looking at your source code they will always look at it differently so that's why we have introduced a backbounded program so that others can do our work, right? because as we have heard before we have found less bugs before we use the HectorOne platform
HectorOne is a service-to-service platform you have basically the page online there and security reports there and it has some advantages it is used by other major vendors like Yahoo, Twitter, Adobe Dropbox so there is already a huge community of
let's call them security researchers and they actually are on this platform they can take a look at which company or which project is paying for security bugs just click on it, take a look at the projects and get paid there are quite some great fighting tools and support because if you launch a backbounded program you will get a huge amount of noise
some statistics later and the payments are actually processed by HectorOne so if you ever have the problem to find out how to pay money to Uganda you don't have to because they have to pay in this country, way easier this way the platform is one could call it a little bit a glorified issue to HectorOne
there are open issues and closed issues and this is a report and there is a fancy functionality it is called HackBot it basically looks at the security reports that have come into and correlates it with other security reports that other users of HectorOne have reported and so I can for example see somebody has reported something similar to Dropbox
let me see how they have responded to it which takes a lot of time if you know what it is about and others have done a lot of research big amount of the reports you are going to expect from a backbounded program you are going to get quite a lot of reports especially at the start we got 45 reports a day
which is quite a lot to handle and at the public lounge we got 30 reports and now it has gone down from 1 to 2 reports a day that is pretty much handleable so if anybody of you ever wants to start a backbounded program expect to have a huge amount of reports in advance
and then it will go down and if one looks at the amount of reports I have told you before there is a huge amount of noise if you look at the amount of noise we have 14% of valid issues and then we have 68% of issues which are actually not valid which is stuff like your user names do allow dogs in it
I don't know why this is an issue if anybody knows, tell me later please so we have researched 46 reports and we have paid out $700 in bounties so there were 3 reports in scope which means they were against the product and there were 42 reports against
our infrastructure like you disclose the server version in your Apache header that's not good, so fix it we fixed it and that's it then the lessons learned from a backbounded program is protect infrastructure against automated testing tools in advance, because people are going to test your website and we had something up there
which allowed you to contact our sales people and stuff like this and some of our business developers were not quite happy about 5,000 new customers called script.com so don't forget any contact form and stuff like this and the quality of reports differs hugely depending on the report I would say there are like 5 top performers
which I am super nice to and there are like 200 people who I don't really want to see reports from to be honest and just be nice to the nice ones and be a little bit less nice to the less good ones and there are likely no low hanging foods if you think there are some you can earn money but let's think a little bit about
what would have been better with regard to what we have done with the security changes first of all tool requests came really really late you should have done them before they would have prevented a lot of issues and sure they actually get reviewed
because if only people file issues and file tool requests then this is just going to get a little bit bigger this is cryptography I am not really a big fan of this topic, well I like cryptography but in the past we had some encryption functionality which was not really top notch
that's a nice word and some people were actually blaming us for it and fair enough so we are working actively on changing this we even paid out a recently $350 bounty for a small security issue openness in retrospective this is a little bit controversial
so if I would do it again publishing advice always I would do it first and the software is secure again enough so otherwise you will get 300 people who blame you based on the amount of reports let me go shortly through this you actually need to be aware
that external companies and people will look at your product and they will find security issues and the truth is best to be proactive and have stuff fixed before bug bounties are a good addition they are not a replacement I will start with higher rewards though and one small anecdote to the end here do not watch reviews without checking them in detail
so we had a big customer of ours which has performed a security check and then they called us like oh we have found one security issue so we actually had to roll that because they didn't want to send us the information so when we got there they actually told us hmm it's not one issue it's 500 different issues so they showed us the 500 different issues and it was 500 hard coded passwords
it turned out this was only the translation from the English-German password to the German password and stuff like this but because they are Americans they probably didn't know their other languages otherwise they couldn't explain this so always think twice about that that's it my time is over so if you want to contribute github.com flashcom.com
or we have stickers in building K thank you we have 5 minutes for questions
so we have some rewards and it's for all of you so we have said we support for all of these patches but we pay all for the data
we don't have any budget in it we have some additional bucks we pay $5 at the moment we are considering going up to $1 but we don't have any limitations like we have a song in this one which hopefully is not more questions?