Right hello everybody Sorry those of you who are expecting the psyllium and kubernetes talk The gentleman is a new father, and so this is this is a talk I had prepared for other conferences. I hope it will be of interest here today, so I'm Andy

I've done various different things you could call it dev sec ops, but it's a disgraceful term I think that is just an engineer position I Work at control plane a container and kubernetes security engineering firm

We find right now a lot of our work is educating people who've deployed containers So containers may not be entirely secured by default, but like everything else are as secure as they're configured to be So I want to talk about open source exploits web server exploitation and how containers have

Impacted the exploits we've seen over the last few years so open source exploits Vulnerabilities in open source code and insecure containers do they contain? What can they contain and how have they defended against major recent vulnerabilities?

Our users expect their data is safe when they hand it over to us, so is it possible to protect all data? Surely our users deserve some level of privacy and if we can't protect the data Code in the system Is it possible to protect all data? Well if we travel down and specified and untested code paths as bugs are then

We can't be sure that that's secure so is bug free code possible, and if not is anything completely secure Or to lower the bar is anything completely secure from teenagers So This talk is about security

The anatomy of some recent major open source vulnerabilities How containers affect security models and defense from future vulnerabilities right? We'll go through some internet melting open source vulnerabilities With the focus as I said on web servers and containers and see what part containers had to play

So heartbleed Catastrophic flaw in open ssl a commonly used encryption library referred to as a modest code base of outsized importance Considered theoretically an exploitable cloud flare open the competition and within nine hours people had retrieved their private keys from the shared memory of their servers

affected 25 to 50 percent of major sites at disclosure and impacted Apache and nginx email servers XM Pp ssl VPNs It's a problem in SSL heartbeat handling so a heartbeat is message used to keep a connection alive and in this case a user sends a question with an expected reply word and

Untrusted data containing the length of the word that they're sending as we know that should always be done server-side Never trust any client supplied data so in this case When the client says hat this is 500 letters long

then the server responds with hats and the next 500 bytes well the next 497 bytes of private memory memory that should never be leaked that may be containing encryption keys and Obviously breaks the security model of open ssl this attack can't be seen in the logs because it occurs at handshake time and

The poor guy involved missed validating a variable containing a length So it's a buffer overrun, and it was discovered by fuzz testing with American fuzzy lop of those you don't know it's a genetic algorithm based fuzzer and You provided the source code instruments recompiles and then

Significantly reduces the brute force problem space by intelligently attempting to make a program crash Mitigation updates open ssl So how could the containers help with a heartbleed scenario well? They don't really the hosts memory was protected, but this attack is only leaking process memory anyway

Heart bleed is essentially indefensible with a container But containers do help Hopefully having a mutable deployment artifacts in a swift ci pipeline Means that the time to patch is significantly reduced over traditional configuration managed systems

So in kernel and container terms hop leaders like building a castle securing the walls and then having the guards give away the secrets onto the next vulnerability Shell shock a bash vulnerability allowing local privilege escalation by adding code to a specially crafted environment variable containing a function declaration and

Executing code out of the calling context so heart bleed leaks data shell shock can be used for privilege escalation some impact CGI web apps Sshd the HTTP clients open VPN again and all Linux boxes all bsd boxes and all post

1991 Unix deploys and max Basically the world for most of us here although Debbie and Ubuntu with dashes the default shell were actually not vulnerable So Here's a shell shock call the payload is encode well is just sitting in the user agent field Obviously, that's just a bash function with a no op and then whatever you want on the end

Executed as I say out of context. That's obviously executed server-side when called remotely over HTTP At the last low half of this burp suite output demonstrates The contents of possible in fact the contents of the previous

Ping ID and password You only see this attack if you were logging all HTTP headers in your web server logs The bug was in the initial implementation of function exporting and importing on Which was written by Brian Fox on the 5th of August 1989? So this was before the web existed

HTTP existed or Linus had released Linux version 1 So it would be incredible if this bug wasn't being exploited in the wild at some stage during those intervening 25 years Loads of CVS all associated with this bug but of note is that it was

Eventually fixed by Florian Weimer red hat and he said he found the bug after fuzzing with American fuzzy lop So the mitigation again upgrade how to containers help against shell shock. Well, they provide process isolation So the expert the escalation of privilege or the running of that untrusted code is only within the containers PID namespace

So as opposed to heartbleeds We're able to cushion the impact of the host by segregating the process namespace second the process space That they're unable to run code in So back to the castle the perimeter is secure But this is like allowing a Trojan horse in through the front door, but locking it in the dungeons

It's safe, it's observable and Once it gets out and starts trying to do its thing. It can be effectively shut down and terminated so the next vulnerability drown TLS based attack decrypting RSA with obsolete and weakened encryption

It's a protocol attack again and similar to heartbleed can't be defended with containers The problem here is old US export grade cryptography and symmetric ciphers 33% of sites vulnerable at disclosure the mitigation disabled SL version 2 if that was being run for some unusual reason and upgrade open SSL

Rotating any secrets that may have leaked as a side note There are a lot of TLS attacks that containers simply can't help with and why so many and why the cluster after May 2013 our friend Edward Snowden a Renewed vigor was placed upon examining the protocols and algorithms used for encryption searching for backdoors or

intentionally introduced vulnerabilities so Was this more secure for being open source, but given the cavalcade of academics and researchers Required to find this particular attack. It was probably in the realm of nation-state exploitation

Back to the castle. This is like securing the entrance to the castle with an obstacle and only allowing people over a narrow bridge and Then turning a blind eye as the moat is drained and continuing to trust it provides security Can containers help? Well, this is a protocol attack

No amount of containers are going to protect us from broken protocol specifications The only benefit we reap is being able once again to redeploy quickly Onto the next vulnerability Dirty cow was a Linux kernel bug which allowed privilege escalation on every Linux kernel since July 2007

the URL dirty cow dot ninja potentially the worst of all logo vulnerability URLs and It's a race condition in the copy on write implementation in the kernel Again, this is on every Linux device and embedded devices probably have no roots to upgrade Exploitation of the bug doesn't leave anything unusual in the logs and the mitigation as with everything else is

Upgrade so of notes this bug was found in the wild by the researcher involved by packet capturing all traffic into his server One of his sites was compromised. He was running this rolling packet capture. He extracted the vulnerable pet of the payload

Reproduced it and submitted it back to the Linux main kernel mailing lists So how to contain us help well They couldn't contain this bug because the security subsystems that containers rely on are all kernel based subsystems containers rely on the kernel for protection Namespacing cgroups invocation of further extensions. So if it's in the kernel, that's almost nothing they can do

We'll look at exactly why that is and what we can do to prevent to protect ourselves against unknown vulnerabilities Later so back to the castle It's like building a castle on the biggest rock you can find and then being surprised when someone burrows in

Underneath and steals the crown jewels if the kernel lets the containers guard down. There's very little the container can do We'll demonstrate this exploits in a moment. So a couple more vulnerabilities honorable mentioned cloud bleed February 2017 an error in an HTML parser triggered the same error as heart bleed

Buffer over read private data leaked in HTML caches around the world it Probably could have been more secure of open source. There was a huge number of reads actually required to trigger this particular bug and potentially because cloud flow will never quite reveal the intricacies of their internal development processes will never know and

Because it's been slides already Apache struts and Equifax a simple example of failing to patch systems Vulnerabilities will always exist but timely patch hygiene is imperative Container networking policies may have helped here. We still don't know the full details, but the aqua sec security team

Suggest that the data that was exfiltrated from Equifax was all done on the basis of one RCE No pivots just straight through all the way to the database Obviously an application architected like that will be vulnerable almost no matter what IDS network policies

These sort of things are required So again without more information is difficult to say still speculation at this point. So what these vulnerabilities have in common Humans inevitably making mistakes and that will never cease The people reporting a lot of these bugs were not parts of the core project teams

So opening a project up and open-sourcing it opens a huge pool of resources with demonstrable advantages Fuzzing applications yields fantastic results. This should be performed whenever practical and Containers are not a panacea. There are plenty of security exploits and vulnerabilities and application level protocol problems

That cannot be fixed by name spacing and sandboxing application runtimes Okay a call to action for open source review fuzz other people's code Donate to open source projects open SSL is horrendously overfunded. That's sorry underfunded and

overused comparatively and as you've seen people migrating to other SSL libraries rather than trying to fix it, but We need some money in there. So finally major vulnerabilities Kernel TLS rotes execution for the last few years had no mitigation except upgrade

So let's demo a container breakout with Docker This is potentially the most insane attack to demo because it relies on a race condition And I'll demo inside a VM and my CPU stepping that or action battery happily but it's it's the most recent serious breakout and

potentially the most dangerous a dirty cow as I said was on all Linux versions since 2007 and All Docker versions Because those doc assist calls are going straight from the calling application through to the kernel Then there's nothing we can do inside the container itself. It's the container run to its container

demon and the kernel that we rely on So what this will do let's cross okay Yeah

Just wait for it to pop everything Is that all right? Well, well, there's gonna be a lot of data flowing so We will go through it once and then modify it if required so

What we've got here so this is Just a bit of bash to build a container image and we also popped some other windows So the other tmux tabs here, we're using Sysdig to trace dead beef, which is the exploit name

hilariously the file descriptor port one two, three four, that's because this it's a copy on right Copy on right bug exploitation. So The privilege of a piece of memory is changed and then ptrace is used to try and write to the original piece of memory exploiting a race condition so writing to a piece of memory that root owns and then

writing to it one in every n We will see crash triggered The ability to write some code is then written into the virtual dynamic shared object Which is mapped into all applications by the kernel that bit of code then opens TCP connection back on port one

two, three four, which is also running inside this container map to the host and That will be the the control channel Then the next window will be D message so we can see What the kernel thinks is happening as this goes through and then the final window at the bottom? Will be watching this file temp dot X which is essentially used as a lock because this is a massively parallel attack

we're going from Debian we're running dirty cow VDSO and At this point if we want to proceed yes, please This Is just to show that dirty cow is in fact matched to the kernel version do we want to run with that part?

Well, not in this case. I'm just gonna make this a little bit smaller because there is a lot of outputs Okay, so we can see the M advised call being filed off a lot of times You'll notice that that barely increments in terms of time ptrace call to attempt to write to that piece of memory

And actually take control and write to it while it's still owned by roots And then this will carry on for quite some time so at the bottom you see that we've got temp X currently owned by roots And there we go so the assisted call will continue for quite some time because of the volume of rights assist calls there, but if we now

Just check who we are up here Then we're roots, and if we have a look at the process tree we can see somewhere up there will be the docket demon We can see virtual books in fact so at this point we started off inside a container we ran an exploit that affected the kernel and

piece of shared code the virtual dynamic shared object use that to open a TCP connection as route back to our unproven container process and We are now We are now route inside that box so that's an example of The controls and measures the container puts around

Or that the uses name spaces and cgroups to put around a process being broken And we'll look at how to protect against that in a few minutes meantime Shut all this down Start it again

Okay, so we know we can't secure everything. We know there are problems in software development that will always generate bugs And We know that nothing is entirely secure the cost of formally specifying everything is very high and small formal specifications can still contain

bugs and From an application perspective security struggles to keep up with the cadence of delivery The DevOps revolution means shipping software fast Sacrifices security so unless we start to be penalized for insecure software as Bruce Schneier suggested in front of Congress

Then there is no incentive for the bottom line of these companies to prioritize security Speed of shipping features is a competitive advantage, and this is the case across IT phones cars wearables everything So we're not going to win by slowing down so

Back to software is it possible to write bug-free code? Well only with sufficient time and conformance specification does not guarantee the absence of bugs NIST has published an analysis of the state of the arts this subject is quite broad so we won't cover it now But it's in the slides for posterity

Essentially measure software quality efforts people be nice make sure people are well trained and If you are not measuring it you can't improve it There's also David Wheeler secure programming how to it's a wonderful primer So reducing bug count generally is one of the roots into this problem

But something that we can impact without the ability to change the application code itself is the architecture So let's examine vendor best practices with regards to deploying containers So where the containers excel we know about these things, but is security one of those

Well excluding the fact that integration with advanced kernel security features occurred rather late in the Docker lifecycle Are they bad at security well not everything is name-space we still don't have devices SIS bits of proc and dev The demon runs as roots so ultimately when we have one of these

Breakouts you saw similarly. We're escalating from a root based For a root owned process rootless run C is coming. That's an open sees a thing guy called Alexis Ris doing a lot of work. It's about a year's worth of PRs gone in or time to merge

But a lot of changes meaning that we can actually unshare namespaces without privilege and shed kernels, but we understand the compromises of isolation mechanisms cloud providers run stacked containers and VMs anyway So with the advent of things like Intel's clear containers, which is essentially a very thin

KVM wrapper or hypervisor wrapper, but able to plug into the OCI spec Sorry the container runtime interface spec Then and things like cryo in fact then we can choose our execution environment for the level of trust we have for our specific container

So the good Docker specifically is actually configured to prevent a large number of attacks by default configuration The reduction of attack surface is inside a container So the reduction of dependencies does minimize the attack surface per container it allows a more granular application of privilege to or in fact capability to

Be assigned per process Speed of deployment so running fast to CI pipelines immutably means that we can ship patches quickly Content trust signing this was touched upon earlier by Lenart But actually addressing things cryptographically means that we can take manifests and know what we have

It means that we can do things like scan everything we know in production offline. So for example union FS file systems Are able to scan the individual layers a new CVE feed comes out then rescan everything without having to go and Check exactly what we've got deployed and native log drivers. So post-mortem analysis is easy

These are a list of events This is a list of events that were prevented by default not comprehensive Of course, but you'll notice that the ones that are not mitigated are all beneath the container in the kernel So how does docker provide the security hardening? I'll run over these briefly

I'm sure we're mostly aware of these name spaces provide a different window into the kernel or whatever particular Entity we're looking at the process is running within the container cannot see processes without I've been in the kernel since July 2008 For example the PID namespace any time a process starts. It's assigned a PID in both the master namespace and its own

particular namespace So these exist for for a number of different things now The user namespace has been touched on already using namespaces v1 are as Lenart said yesterday incomplete I think maybe they're almost not fit for purpose from the perspective of

The way doc uses them and preaches about them as a security model There are very real dangers using a root user inside a container because you're mapped to the roots outside the container and That affects anything because the Linux security model is based on users

It affects lots of things all the way up and down the system now by extension This is why this is a hard problem because we want to be able to share bits of data between different containers We want to maintain the right permissions on files and processes As it stands there's probably still a few years work to go until this is a complete story

User namespaces are not on by default for Docker They have to be activated with command-line flag when you start demon and this is because they were only introduced around Docker 110 Which means if that was on by default suddenly a whole raft of containers prior to that would not function or container images rather

So Mapping the root user to a non root user outside the container really aids with mitigating the risk of container breakouts But it's something that is incumbent upon each administrator on his own system to actually configure correctly So control groups another key feature of isolation providing resource counting so things don't run away

Then that will mostly implemented in Docker. The store is not quite the same for kubernetes, which still doesn't have network IOPS or block IO properly accounted for but the story again is converging See groups are obviously fair sharing of resources and prevention of denial of service again

There are questions around how users actually operate here. So docker runtime There's various things you can set when you run the docker image. You can create a read-only file system So obviously an immutable container becomes more difficult to write to however This isn't mitigating all attacks because if you can pull code if you've got say if you've got bash and curl installed

Then you can just pipe to bash and you're still executing code And one in the mutual file system Pid limits and fork bomb proofing. Of course. This is now dependent upon the user namespace story, which isn't complete So without setting these things then you can effectively starve the resources of the host from inside a container

And security options. No, you know any privileges again it's a case of ensuring that your application doesn't require an escalation of privilege for a certain operation before you're able to To activate that option, but these are things that should be on by default

So kernel capabilities and process restrictions The capabilities give us a more fine-grained View into what root can do and avoids just a blanket application of all privilege But the kernel has over 600 system calls obviously a bug in any one of those could lead to process privilege escalation So only those necessary for containers do its job should be allowed

This can also protect us from dirty cow in this case dirty cow like the exploit demonstrated there relied on p-trace Now p-trace has a lot of valid applications Including just listing out processes So PS and s trace both rely on the system call

So again a blanket drop of that would be inappropriate for some operations, but we'll move on to how you can actually run debugging tools outside a container by mounting all the relevant namespaces and then perhaps We can drop all these debugging tools from our production Applications again reducing the attack surface that we ship inside a container

So Dockers app on a profile incidentally allows p-trace specifically for the reason of running PS inside the container Docker actually drops a huge number of capabilities the container bounding set on the left there is kind of going on for 40% Which is beneficial in some ways, but of course it's not a comprehensive

story Not many people really run their own hardened kernels in production, so that's more of a hat tip than a recommendation But security policies and whitelisting set comp and app armor are now both in and se Linux are all integrated with Docker and kubernetes

The problem there is one of utility It's not generally easy to statically analyze or dynamically analyze a program and see its entire range of system calls without Theoretically exposing it to all the potential actions it could perform So there's some benefit there, but again the the user experience is not quite what it should be for the average administrator

All those things said when surveyed and asked how many people use security policies Not really very many people Still less than 50% in production this is the only way to really be sure that you've locked down what you expect your containers to do and

should probably be higher, so Kind of a wall of text here again But I'll just go through some of them and then obviously these are in the slides for posterity Of note here is Dropping to an unprivileged user, but that's just not running as root, but debug by attaching to relevant namespaces

So this means for example when you're running from scratch And you've just got a statically compiled binary sitting in or if you're running a dynamic language You only you only put your interpreter in there you and whatever libraries you need don't install bash don't install your tier

OpenSL libraries if you don't need those things and then run in this case busy box But another container and attach into the namespaces of the containers that you're interested to So this gives you the observability that you require you can also run This debug docker image as privileged do whatever you need to do, but it's just going up one time

It's only running while your the namespaces open up for you to use it and you close it and everything shuts down again It's the most secure way to debug things in production, which invariably we need to do from time to time Obviously The Certain good binaries at the worst shouldn't use them privileged containers essentially act only as a bundling mechanism

They're just a way to ship tiles around and to run them because running is privileged Puts you in the same namespace as the host for almost all the operations that your container performs It also means fundamentally if you're in That privileged container as root you can remount the host file system and then you've got write access to the host

So it's really there is no protection at all And that should only be used in the knowledge that it's privileged as a way of shipping around Biners that may require that level of observability for example a lot of monitoring agents require Privileged sadly, so just good hygiene drop all capabilities and just add back in the ones that you need

running that through a CI pipeline where maybe Do your test suite once with and once without those? Capabilities is a good way of introducing that to a team who may not be Kernel competent if you like or aware of the

The range of capabilities because there's a huge confusing amount and they will use namespaces. Well, that's the demon again This is called whitelists So all these things are relatively obvious and securing a docker host that can be done with docker bench or cube bench has an equivalent Signing image is mandatory

So Moby here has Decided to try and have another pop at the way we interact with kernel capabilities By introducing entitlements which are loosely modeled on the iOS nature of entitlements that is now available actually So you can now run Moby from head and

Essentially what that will do is it'll bake in a set of required permissions and privileges capabilities at a Build-time and then when you run the container the image you have the option to give it a level of trust So if it attempts to for example Request privilege and you don't trust it and containers won't start

This is it looks like it's a nice way of bucketing up all the privileges that are normally just caps this admin But it's it's yet to land but it's it's something to keep an eye on Kubernetes actually supports set comp now so we can be far more granular FS sorry runners non-root is again. Just mapping to a non-root user outside

The pods that you're running in and then there's some extra se Linux bits and bobs in there Kubernetes has got some strange insecure defaults while we're on the subject at this point This was a default service count mounted into Every pod and up to one six with the introduction of our back that service token was administrator access to the API server

very bizarre default choice That is now fixed as in that service count has less privileges, but our back configuration is mandatory for Kubernetes I've got to keep the cube look ports locked down with TLS. Otherwise, you can also write in there. There are some crazy C advisor bugs Go back to the old adage that speed of delivery

affects the security right so going back quickly to the demo Just demonstrate what it looks like To run the same attack But this time with the application of an app on a profile will actually have a look at what that profile looks like

so Again just having a look It's noisy I apologize so run that Parma. Yes, please. So what does this look like? Well, this is default docker app on a profile

It's relatively sensible in most places, but This call here so p trace peer docker default That means if your docker you can run p trace in this case a quick fix for our problem It's to just disable that call and we'll have a look at how everything responds when we do that So we have just had a sick kill and D message has actually shown us what we want

But nevertheless that is a farmer kicking in and terminating the process and generating Anyway killing the process off So

This is how you would ultimately Configure all your applications minimum set of privilege and then anything that attempts to call system call It's not permitted to do segfaults and it's shut down So And how do we actually generate these security profiles they are really quite painful that requires intimate kernel level knowledge in order to identify

What your application needs the reference example is probably the sock shop demo down here It's got good defaults for databases web servers and microservices There's various different attempts to do this all the major container based IDS solutions will also do some sort of dynamic observation where they will

Whitelist these calls and generate a set comp profile for you under the hood Got some quick slides here, they're probably less relevant to the interests of this group, but I've got five minutes So I'll just whiz through them CI is probably the escalation point that people want to go forks it contains all your keys So there's some nifty tools you can use to for example prevent key leakage into git hub in this case

Get secrets is your actual repo the two of them search for entropy and strings and get secrets Fixed length strings actually AWS keys. So probably a better used as as a pair So container scanning this is focused on open source stuff

There's some vendors listed later, but broadly Claire is core OS is offering This is integrated into the their quay docker registry. It also runs open source And it will give you essentially the CVE numbers that you have Which types in which layers of the images that you post to it Linus gets a shout out for old-school gritty aesthetic

Docker scan is an interesting attempt to dynamically analyze who is running a container a run time Which is a less obvious problem that seems I can do some nasty trojaning and back dooring as well So demonstrating the fact that you shouldn't trust anything you pull off the docker hub

Honorable mentions open s cap was folded into red hat right now and banyan's older So docker bench security will essentially numerate the configuration options of docker on your host Make sure you haven't left any glaring wide open

Mistakes in there. So there's things like default you limit It could actually be more brutal and fail a lot harder, but it just operates more more of an advisory capacity at the moment Obviously your application dependencies are a likely point of escalation as well. There are some nice Plugins SNX really nice that will actually pull requests for vulnerable

Dependencies and transitive dependencies of github. It's currently only JS and Ruby right now Obviously AFL is it's the mother of all fuzzers Just gave you stuff there and also there's no point securing all these things if we allow our web applications to have our CEs and Open up an attack surface for people to get into our containers and actually do harm

So how do we stay secure while ultimately intrusion detection systems are mandatory. There's a whole host of container native stuff Available now, which is namespace aware. As I said, we'll monitor system calls either in Cystic's case they've just launched a cystic secure product which requires a kernel module to be compiled into all your systems

your security guys might not like that mine don't and There are a number of other options which sit in front of the dock socket and are able to proxy calls Falco has now actually been folded in cystic secure, but it's a source version and Twist lock and aqua come very highly recommended

So has anything inherently secure well, obviously not Is open source software more secure than proprietary? Well, no one really tells us any of these things open source vendors disclose everything close source vendors live through obscurity and tell us nothing But fundamentally open source is secure enough for our needs as demonstrated by its adoption in major enterprises and governments around the world

So this talk has been about exploits that are scary shiny Hopefully some more interesting, but they're not the only way that people get hacked The OWASP top 10. It's actually becoming a boss for 2017 right now as soon as they can stop in fighting is is the definitive list of

Various application level problems that we need to fix going forward Running consistent security scanning in the pipeline and assuring the base level that we set does not slip These are boring dull and omnipresent An application security is just as important as network security and user security. I was now recommend

Essentially pen testing is dead and that we should be constantly testing the security of our systems so In conclusion prepare for the unexpected secure your networks secure application code Secure users browsers and when all else fails run intrusion detection systems Thank you very much

So we have about one minute left any questions Okay, well then thanks again for your talk