More Keys than the Janitor: Hacking exposed AWS EBS Volumes
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Alternativer Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 335 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/48423 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
DEF CON 2737 / 335
8
9
12
14
32
38
41
58
60
61
72
75
83
87
92
96
108
115
128
132
143
152
158
159
191
193
218
230
268
271
273
276
278
295
310
320
321
335
00:00
HackerVolumenPhysikalischer EffektGeradeMetropolitan area networkMultiplikationsoperatorUnrundheitBitGüte der AnpassungComputeranimation
01:09
Mini-DiscMaschinenschreibenBildschirmfensterBitSoftwareUmwandlungsenthalpieExploitElastische DeformationInformationsspeicherungp-BlockProgramm/Quellcode
01:59
Mini-DiscFestplatteKartesische KoordinatenDefaultSoftwareschwachstelleVolumenMini-DiscEinsInformationsspeicherungInstantiierungp-BlockSchaltnetzComputersicherheitVirtuelle MaschineElastische DeformationSoftwareQuaderBildgebendes VerfahrenCodeMenütechnikComputeranimationBesprechung/Interview
04:15
SoftwareschwachstelleKartesische KoordinatenProgrammfehlerAusnahmebehandlungGamecontrollerInternetworkingSchlüsselverwaltungComputersicherheitDatenbankMini-DiscApp <Programm>Ganze FunktionClientSpielkonsoleVolumenWeb-ApplikationWeb logBenutzerbeteiligung
06:49
CAN-BusMIDI <Musikelektronik>ZahlenbereichPERM <Computer>VolumenInformationsspeicherungPunktSicherungskopieBitServerInformationTypentheorieCodeBenutzerbeteiligungQuaderSchnittmengeWort <Informatik>EindringerkennungProgrammfehlerSoftwareschwachstelleDefaultMini-DiscDatenmissbrauchGraphische BenutzeroberflächeElektronischer ProgrammführerMinimum
08:54
RoboterMini-DiscObjektverfolgungSoziale SoftwareSchlüsselverwaltungBinärdatenTotal <Mathematik>SoftwareRobotikTwitter <Softwareplattform>MAPMini-DiscCodeSingle Sign-OnSoftwareExogene VariableSchlüsselverwaltungDatensatzRoboterHypermediaSelbst organisierendes SystemDienst <Informatik>E-MailURLGamecontrollerPunktCASE <Informatik>Chatten <Kommunikation>Interface <Schaltung>ComputersicherheitRechter WinkelKette <Mathematik>SoftwareentwicklerFunktion <Mathematik>UmfangElektronische PublikationDomain <Netzwerk>Güte der AnpassungKonfigurationsraumEDV-BeratungSkriptspracheWeb SiteProgrammierungDomain-NameAdressraumDatenbankMailing-ListeSchnittmengeIdentitätsverwaltungTransitionssystemServerÄquivalenzklasseProzess <Informatik>Produkt <Mathematik>Kartesische KoordinatenVirtuelle MaschineEinsNatürliche ZahlQuellcodeProfil <Strömung>SystemverwaltungPasswortInstantiierungSoftwareindustrieWurzel <Mathematik>FIS Informationssysteme und Consulting GmbHIntegralVerzeichnisdienstSuperposition <Mathematik>MultiplikationsoperatorGebäude <Mathematik>GrenzschichtablösungGrundsätze ordnungsmäßiger DatenverarbeitungBesprechung/Interview
18:22
Virtuelles privates NetzwerkSchlüsselverwaltungPasswortToken-RingE-MailQuellcodeInstallation <Informatik>GoogolHill-DifferentialgleichungQuellcodeMini-DiscPasswortVirtuelles privates NetzwerkInstallation <Informatik>Metropolitan area networkSchlüsselverwaltungService providerSpieltheoriePunktSoftwareOffene MengeWeb-ApplikationSoftwareentwicklerMusterspracheElektronische PublikationCodeKartesische KoordinatenGüte der AnpassungDefaultSchnittmengeRechter WinkelInternetworkingServerWebcamStreaming <Kommunikationstechnik>App <Programm>VerzeichnisdienstRepository <Informatik>Token-RingE-MailBitNetzadresseClientPublic-Key-KryptosystemDigitales ZertifikatDatenbankProgrammierumgebungSpezielle unitäre GruppeSicherungskopieInformationHash-AlgorithmusProdukt <Mathematik>Wort <Informatik>InstantiierungZahlenbereichRohdatenTLSComputeranimation
24:59
FlächentheorieBitProzessautomationEinsProzess <Informatik>InstantiierungClientSoftwareschwachstelleBildschirmfensterSchnittmengeBesprechung/Interview
26:17
Mini-DiscProzess <Informatik>FlächentheorieVolumenInstantiierungProzess <Informatik>InstantiierungMini-DiscDialektComputeranimationBesprechung/Interview
26:57
Architektur <Informatik>Prozess <Informatik>VolumenInstantiierungMini-DiscAbfrageMAPComputerarchitekturInverser LimesMini-DiscKartesische KoordinatenDefaultCodePortscannerBitDatenbankProzess <Informatik>Message-PassingWarteschlangeEinsComputeranimationBesprechung/InterviewFlussdiagramm
28:30
Mini-DiscVolumenZahlenbereichDatensatzSpezielle unitäre GruppeEindringerkennungForcingUnternehmensarchitekturAusreißer <Statistik>FrequenzMultiplikationsoperatorLogistische VerteilungLokales MinimumDatenbankMetropolitan area networkProzess <Informatik>Elektronische PublikationBildgebendes VerfahrenBesprechung/Interview
30:58
PunktMetadatenURLDateisystemMini-DiscVolumenInstantiierungSoundverarbeitungCodeMini-DiscURLDateiverwaltungMetadatenPunktSoftwaretestFrequenzMultiplikationsoperatorProgrammfehlerSkriptspracheCASE <Informatik>Besprechung/InterviewComputeranimation
32:04
Mini-DiscElektronische PublikationBildgebendes VerfahrenAbschattungProzess <Informatik>Mini-DiscTypentheorieUmwandlungsenthalpieMailing-ListeDefaultPublic-Key-KryptosystemZahlenbereichGüte der AnpassungComputeranimationBesprechung/Interview
33:42
MultiplikationMetadatenURLFehlermeldungSoftwaretestCodeCodeURLSoundverarbeitungSoftwaretestMetadatenFehlermeldungEntscheidungstheorieBitDialektEindringerkennungSchlüsselverwaltungDatenbankMultiplikationsoperatorComputeranimationBesprechung/Interview
34:49
ProgrammierumgebungMini-DiscInformationCASE <Informatik>VerzeichnisdienstNP-hartes ProblemForcingDateiverwaltungHilfesystemBenutzerbeteiligungVarietät <Mathematik>Mini-DiscVerkehrsinformationSuite <Programmpaket>VolumenFlächentheorieProzess <Informatik>SoftwareschwachstelleSelbst organisierendes SystemSkriptspracheSoftwareentwicklerMultiplikationsoperatorLoginPhysikalisches SystemInformationFreewareFestplattePunktBesprechung/Interview
39:08
PasswortFahne <Mathematik>SchlüsselverwaltungWurzel <Mathematik>Mini-DiscSCI <Informatik>SchlüsselverwaltungPublic-Key-KryptosystemElektronische PublikationMini-DiscBildschirmfensterFlächentheorieVirtuelle MaschineLesezeichen <Internet>DatenbankFahne <Mathematik>InformationGrundsätze ordnungsmäßiger DatenverarbeitungMotion CapturingQuaderMultiplikationsoperatorDADSPasswortRSA-VerschlüsselungEinfache GenauigkeitBitrateProgrammfehlerBitZahlenbereichPhysikalischer EffektMailing-Liste
42:28
Total <Mathematik>SoftwareSchnittmengeMusterspracheExtrapolationMultiplikationsoperatorSchätzfunktionSoundverarbeitungSchlüsselverwaltungRandomisierungFestplatteWurzel <Mathematik>Mini-DiscDialektSoftware
43:53
Mini-DiscSoftwareentwicklerMultiplikationsoperatorDatenfeldComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:00
I'm really looking forward to this talk if uh if you've been to a lot of talks so far you know that they have all been pretty pretty technical. My understanding is we are going to get some entertainment and some jokes and some stories from uh from Ben here so let's get excited give Ben Morris a big round of applause who is going to talk to us about AWS.
00:21
Have a great time man. Thank you very much. Have fun. Really appreciate it. Hey everyone how's it going? You guys having a good time? Alright so am I so am I. Thanks for coming in today. I know you guys have probably been waiting in a lot of lines and uh yeah I just really appreciate you guys being so excited coming here to see this talk. Put a lot of
00:41
work into it so thank you. And uh we're going to be talking about a lot of cool stuff today. We're going to be stealing lots of secrets and we're going to be uh hacking AWS and showing how I did it all and then talking about how we can uh basically help fix the issue. And uh if you just uh give me one sec to actually get my timer started cause I am
01:03
okay perfect. So yeah thank you very much for coming in. Uh just a little bit of a disclaimer before we uh get started here. Uh please do not arrest me FBI. Uh no post exploitation was performed and everything I found was basically publicly available
01:23
already. Um I'm not going to be talking about any AWS zero days or any exploits in customers specific software. This is just a widespread misconfiguration issue with AWS and uh I'll talk about that a little bit more later but basically you know I was just
01:40
kinda driving down the road and I looked out the window and said to myself huh a lot of people's disks are on fire. I should probably you know just call the police and let them deal with it and I just kinda kept driving so didn't do a lot of post exploitation and I definitely stuck to you know the look but don't touch. But anyway um so what is EBS?
02:02
EBS stands for elastic block store and it's essentially a virtual hard disk that you can attach to a VM. So anytime you spin up a virtual machine inside of AWS it's going to have a disk that's automatically provisioned to it and that disk is an EBS volume. They can vary in
02:25
size and the default is like 8 gigabytes but basically anytime you start building an application using uh Amazon EC2 you're gonna be running one one of these EBS volumes so they contain your application code your data and everything else you would want to
02:42
deploy. So these volumes they can basically be detached and reattached to various machines you can move them around kind of like uh network attached storage in a way so um they can you can clone them you can delete them you can copy them you can do
03:00
everything with them that you would expect and um they come in generally for flavors for security purposes. They come in unencrypted, encrypted and public and private. So you can have a combination of them like uh public, unencrypted, private, encrypted, public,
03:21
encrypted and whatnot and we're gonna be looking at the uh public and unencrypted ones today. Those are the ones that are interesting and uh those are just the fun ones that have all the credentials in it. So if you have you know an encrypted uh or a private disk they're not really vulnerable and also um these disks by public or by default are
03:44
private so when you do spin up that instance it is going to be backed by a private volume which kind of made this uh vulnerability really interesting to me. I wanted to know you know who is out there exposing their disks to the public when AWS actually makes
04:00
it pretty tough. You know you have to go into a separate menu after you create the snapshot and after you create the image to actually go check that public box. So I was really curious to know you know who is out there doing this. So what could possibly go wrong with an unencrypted and public disk? Well basically in back in January I was at an
04:25
onsite for a client and you know I was just really jet lagged and not able to sleep I'm sure you guys have all been there it's like you know you're on the west coast and you gotta fly all the way to the east coast or you're on the east coast and you gotta fly to the west coast and you get to that hotel and you sit there and you're at the bar but
04:43
there's nothing to do you know you're just stuck there and then it's like 1 AM the bar's closed there's nothing there and you basically just say well I'll go sit on my computer fine. So I thought to myself well I'll look at uh this client's cloud security controls that'll definitely put me to sleep you know um except that it really
05:02
didn't uh I basically found an unencrypted EBS volume that was public on their account and I really wasn't familiar with this vulnerability so I just did what everyone else does and I just googled it you know and I found some blog posts but there were only a couple of
05:20
them and they didn't really talk about um this vulnerability very much they basically said oh well that's bad but just don't do it and it's fine which is you know always just peaks my interest I'm like okay well I gotta know more now um you know so I basically took this client's disk and I went through the 27 steps of attaching it to my VM through
05:44
the console and mounted it and I realized that this client had made a copy of their entire web application available to the public internet and this basically had everything to run the app including their AWS access keys, their AWS secret keys, API keys for third
06:05
parties, database credentials because you know of course and of course the database was exposed to the internet because why not of course you'd want to do that when you have AWS you know that's totally normal so after I discovered that disk and basically had this you
06:24
know incredibly critical finding um I knew I basically had to investigate more I needed to know you know how widespread was this bug because this bug was really powerful it had you know the keys to the kingdom for this whole application so you know I started doing
06:42
some digging and I wanted to ask myself well why does this happen you know how can this happen um and basically there's two screenshots here you want your disks to look like the top screenshot there it if you can see it says uh there's a this snapshot is currently
07:01
private and that's what you want it to look like by default that's what it will be but if you go into that tab and you change it to public or you use the API you have some kind of broken API code that ends up setting that snapshot to public what happens is it shows up in that nice search box down there at the bottom and this search box is wonderful because you
07:26
type in the word Jenkins and Jenkins servers come up and you type in the word backup and backup servers come up so basically at that point you know I kind of realized like whoa this is really cool got something here and um basically when you do set that
07:43
public tab and it shows up in that search box if it does have sensitive information in it you have to assume it's compromised at this point um because anyone can go search through it and I don't know if you guys have heard about um like the Capital One stuff that just happened recently or any kind of other like S3 bucket exposures this is kind
08:02
of a similar vulnerability to that um in that it's you know it's kind of going through someone's private data storage that they think is private but it's really not and one cool thing about this bug is all of the snapshots are queryable and you can pull all of the IDs back from the API it's not like an S3 bucket where you would have to start guessing
08:25
people's uh bucket names to try to find one and if they set something like a GUID for their bucket name you're not really going to be able to find it so this this made it really fascinating and really cool to me because you could basically just start going through all of their stuff um in a programmatic fashion very easy and you know even if you just want to
08:44
use the web GUI you just start you know typing in stuff in there so um at that point I was just like this is awesome um and so let's just talk a little bit about what I found because everyone likes loot and uh yeah that's what you're all here for so uh what did I
09:02
find on these buckets? I found a lot of stuff so I'm gonna give first I'm gonna give three examples of some critical exposures that I was able to find and then I'm gonna kind of talk at a higher level and talk about um overall trends and some more stuff that I did find um so the first example I'm going to talk about is about some robots and I like
09:28
robots robots are great they're our friends and if you think about robots and service accounts in your own organization you may be thinking about you know your slack bot or you know other chat bots you have and think about the what those robots can do they can
09:43
do things like push code or you know deploy new builds they can do a lot of stuff and have a lot of access but usually there's this you know interface between you and the robot like some kind of chat or something that lets you you know that delegates permissions so um this case I was able to find these credentials in this user data dot
10:05
config file on this random disk basically and I was a robot so what could I do with robot? Well I didn't have any permission restrictions and you know the ability to you know deploy stuff seems pretty great and um when I started looking at the disk um oh one
10:23
more thing uh this uh this output right here is basically the equivalent of who am I for AWS there's one command you want to run whenever you find a set of credentials it's called uh a STS get caller identity and that basically is like who am I anytime you have a set of any credentials you can pretty much always call this API endpoint um no
10:42
matter your region and it will come back with you know who you are so this is just a simple listing of you know uh who I am and uh so I started looking through this disk to try to find clues uh one thing that's interesting about this is you always gotta have like the scavenger hunt to figure out who owns the disk and who uh you know who who owns it so um
11:02
in some config files that these creds were near um there was some database configs with some internal URLs and uh some just some domain names and these domain names led me to a pretty cool company and uh the company ended up uh doing a lot of
11:20
really interesting things like uh tracking ISIL social media requests and uh posts and they did things like uh record border interdictions and they were basically uh software as a service company that sold pretty much exclusively to the government so they're just doing government stuff and uh their robots keys are just sitting out there for anyone to go
11:42
grab so if you guys wanted to you know read up on what ISIS is doing uh you know on social media these are the guys you want to talk to so you know at this point I basically like shit my pants and is like you know what am I in control of right now um so you know we reached out to this company and they were you know of course very grateful and they you know super positive response um and they gave us like some
12:04
remediation steps so uh they really liked this and um yeah this was this one just kind of highlights this problem entirely like it's just you know you could find anything out there and these accidental exposures could contain anything really um so it was really
12:20
interesting just kind of going through them all the uh the next set of credentials I want to highlight is uh something I just call woot woot and basically um there was a disk with a docker file and if you guys are familiar with docker it's just a way to manage your infrastructure basically and there was some other code around there there was like a
12:44
golang program uh that was compiled and then there was some kind of scripts and it looked like they were mostly for system administration it looked like this thing was responsible for uh you know spitting up infrastructure and the one thing uh that I
13:01
couldn't really figure out was who actually owned this disk um the config files didn't really have any clues there weren't any domains it was all just like internal ten dot addresses so it was just kind of like okay well I don't know who owns this disk but what I did know was you know which account I was and who I was and who I was was root so just out of thin air I was able to grab some root credentials for this account and if
13:26
you're not familiar with AWS root is is basically god permissions on an AWS account it has unrestricted access it's an administrative account and you're actually um not even really supposed to use them you're kind of supposed to delegate uh an account or
13:43
you're supposed to create an admin account and delegate admin permissions to it so that way you're not directly using the root account but these guys thought it was a bright idea to just start using that root account and they said oh well no one will ever find this disk you know it's just some internal thing that uh spins up infrastructure no
14:01
one even really interacts with it except me you know so the you know the the highlight here is just you know you could find everything this was actually the only set of root creds I found in the disks so that was kind of cool um I I honestly wasn't even expecting to find it um just because I think everyone kind of knows not to do this now but
14:20
um this just highlights again just the critical nature of these uh disks and what they contain because a lot of people just aren't expecting you to be able to get access to these ones so um the next one I want to talk about is a little it's about a little piece of software I love near and dear to my heart it's Jenkins and if any of you guys have uh you
14:45
know owned some Jenkins machines out there you know why I love it it's always full of credentials it like has access to production source code and it can like push builds people do all kinds of crazy stuff in their Jenkins jobs so you know anytime you come across a Jenkins server it's just great you know tons of stuff so um in this case I
15:05
found a Jenkins server and it was basically it looked like a developer instance it looked like to me some developer was trying to get an internal application to work with their um their own uh Jenkins set up so they kind of like spun up a copy of their Jenkins
15:22
server and were trying to you know get an application to work properly with it um so you know in the Jenkins server I found some AWS credentials and I popped them in to the STS get caller identity and I found out I was a dude named Kumar and I thought wow that's
15:41
great I uh I'm Kumar now sounds good so uh I looked in the users dot XML file which is if you're not familiar with Jenkins that is just the file that holds basically all of your usernames and passwords for users on the machine and that's kind of assuming there's no single sign on in place or no active directory integration but um so this users
16:05
dot XML file was kind of funny because um I looked at who made the server who had the admin account and their email address and it was definitely not anyone named Kumar so you know it was kind of funny some guy sat there and uh first they exposed their disk
16:23
publicly which was really bad and they exposed their AWS credentials but then they like also framed their co-worker somehow I don't know why maybe they maybe they were trying to like frame their co-worker I don't understand but um yeah so Kumar uh we got his keys and uh you know started looking around um trying to figure out uh who this was and from
16:44
the email address we were able to determine it was a uh software company and the software company um I can't name them but I can talk about you know who uh who they do business with and from their website you know these are the people they work with they work with you know Salesforce, Apple, FIS um a lot of other like fortune you know
17:03
fortune whatever end companies um so you know this is a large software kind of consultancy firm and uh they just did you know a lot of cool stuff but um you know these keys are just uh they're just sitting out there and they're keys that could potentially impact these other companies who you know I'm sure Salesforce and Apple and
17:22
all of them they have you know very good perimeter security and they're making sure that they have you know a tight a tight leash on their developers so they're not doing this kind of stuff but in this case you know you could almost have a compromise happen because of a contractor who maybe you know only has indirect access and you know are
17:41
are you are you really watching that whole supply chain of you know your contractors and who you're actually doing business with to make sure their security is also not weak um because in this case you know the compromise could you know could definitely lead to some pretty severe consequences um just with the amount of work this company does so you know overall these kind of three exposures highlight the
18:03
critical severity and and uh just the kind of stuff you're gonna find when you come across these disks and all of this just makes sense you know these are just like people's application servers it's just a lot of developers kind of uh you know uh doing whatever and trying to just make their stuff work um so you know overall we had uh these
18:24
are kind of the things I was looking for um when uh when you get into a disk you find a lot of leaked source code of course because they're you know mostly application servers so a lot of people are doing AWS right and they have a set of temporary credentials which allows their credentials to basically expire so you know if you
18:47
don't get access to those credentials within about 24 hours I think is the default um you know those credentials rotate out so you know that's good but even if those credentials rotate out you're still gonna have that source code laying around on someone's disk so
19:03
you know we found a source code for some government contractors um some large tech companies and a lot of these are just like boring internal applications but a lot of them also give really good insight into how these companies operate even just having their source code is really dangerous um you know we got like source code for a bunch of internal applications to um like host uh like huge databases for uh some like tech
19:25
companies and stuff so it's just like uh really really cool uh source code even if even if that's all you get and you know at the end of the day it's just a medium kind of like a medium risk finding but um another thing we got was tons of private keys I know it says SSH up there but just lots of uh private keys think about like TLS
19:42
certificates and whatnot uh we were we have just like tons of client and server keys and anytime you know you're using one of those exposed uh server keys you can be man in the middle and some of the client keys we got you know just allow SSH access you know you're like going through people's bash histories and trying to figure out well where are
20:01
these IPs you know who's who's running these servers try to figure it out um so you know that was a little bit harder to uh determine if those creds were valid so a lot of the SSH keys and whatnot we just directly handed over to AWS and just responsibly disclosed it with them to make sure they got you know word to their customers hey your keys are just sitting there uh you should probably do something about
20:23
that so um another thing we got a lot of which was kind of surprising to me was uh like SQL files that contained a lot of people's uh personal information and I think a lot of these came from developers they would um you know st- or borrow some data from
20:44
production move it down to a development environment and so they can play around debug their application do whatever they needed to do but then they kind of left their disk out there just sitting there and these SQL files contained like thousands of people's uh you know usernames, hash passwords, email addresses, phone numbers, all of
21:01
that stuff so just some like really nasty hygiene around uh like SQL files and and just kind of all of that and um another thing we got a lot of were like WordPress installations which are pretty cool um they if you get a WordPress uh backup um I should clarify that you know the WordPress like uh some of the some of the things we
21:23
found were WordPress backups actually so um in the backup you know you're gonna have like uh the database which would be basically a SQL file and that'll contain like all the password hashes and also like API tokens for third parties which are always great to find because they allow you to escalate further do uh you know more privilege
21:40
escalation kind of in their own environment you can potentially start taking over more and more resources so you know finding those API tokens was also really really great um and a lot of just kind of like off the shelf software it seemed like a common pattern would be like developers again kind of like uh just doing a bunch of dev debug work pulling down some stuff like a Drupal instance throwing it up there and then just
22:03
leaving it there so um a lot of those kind of uh credentials were laying around and also VPN credentials uh lots of open VPN uh creds and some of them were for legitimate companies who you know were using it to access their internal network so uh you know at that
22:21
point um you know it's pretty much game over because that's you know it's kind of one of the big goals of an external attacker is to get internal network access so uh that was really cool but also I found a lot of people had their like hide my ass creds and uh other like VPN providers just sitting out there on these disks and that made me really really
22:41
you know curious um just like what kind of attacks could you accomplish you know you could definitely just you know abuse them but could you also like maybe man in the middle of them when they think they're on their VPNs I'm not really sure but um I thought that was really cool to find and also really curious you know people are kind of automating their hide my ass setups um and then also just in general we just found lots of
23:04
AWS keys, Google OAuth tokens, you know third party API tokens, email passwords, think about like your SMTP creds, your web apps are using SMTP to send mail or mail gun um like one thing I one app I specifically remember uh very clearly is something called uh it
23:22
was just called surveillance app like that was the repo name I found there like dot get directory it's just like surveillance app okay and so I looked at it and it's just a bunch of code that just takes raw RTSP streams and just dumps them into S3 buckets so you know whoever or whatever this thing is surveilling I have all their keys now you know I
23:43
can go read their buckets I can go you know look at who's uh you know who's being watched and I could basically surveil their surveillers and uh that one was so hard to like not touch I did not you know like it's so hard to just not touch this stuff and and want to just explore so that one in particular you know you like want to see like
24:01
webcam roulette you know you always want to see what's behind that webcam or like uh you know one of those uh crappy cameras that uh you know is just on the internet so um that one in particular was just kind of funny to me it was really hard to just kind of bite my tongue there um so yeah so overall we just we just uh looked for you know a lot of easy wins um you know when I initially started off a lot of this research I kind of had
24:25
this dream of like oh I'll steal everything under the sun and then just deal with it later and you know kind of like go through it but it just turned out that uh kind of grepping for the common stuff you'd think uh would be good it was it was a good approach um you know just this just uh kind of goes to show like we had a lot of
24:41
success there so um yeah just a lot of uh great stuff and um you know sad I couldn't like do a lot more poking with uh what it actually was you know when you do find something but it was still really really cool to uh find all of this stuff so um
25:00
basically I want to talk a little bit about uh how I did find all of this um the the vulnerability and the misconfiguration on the surface is pretty easy to exploit and you know my uh you know that client back in January you know I was doing all of that manually and you can definitely do that it's uh it's totally possible but um I
25:25
basically wanted to uh kind of automate that process a little bit because um when you talk about temporary credentials and things like that it is uh a bit tricky to deal with those manually like if you if you have some set of temporary credentials you know those are
25:43
exposed for a good window of you know 24 hours and if you get those credentials you can basically endlessly refresh credentials I wasn't uh you know I didn't really like uh you know look at this too hard but um it's definitely a known technique like if you get those temporary IM credentials you can just refresh them endlessly until they
26:01
basically rotate them out from underneath you so it was really important to uh have some automation under your tool belt you can exploit some instances manually but a lot of the ones you're going to find are just unlabeled and uh difficult to detect so you definitely want to have some automation there and uh there's basically a really simple three step exploit process here um you know you're just going to pick a
26:25
snapshot you're going to attach that snapshot to uh your EC2 instance and then you're going to search it for secrets um but the problem is there's about 120,000 disks that are exposed across all regions and a lot of them are just um you're you're just not sure
26:41
what they're going to be because they're not really labeled and a lot of them are just garbage they're just totally legitimate disks um so you know each of each of these steps has some nuances that are uh you know kind of tricky so uh the first step uh clicker oh sorry uh clicker was malfunctioning uh so uh at a high level this is kind of the
27:04
architecture I used um there's nothing really crazy here I'm basically just using an asynchronous queue to uh send new snapshots to workers and then that master in the middle is just uh is just a little application that kind of coordinates everything so that
27:22
master he just pulls all the snapshots uh about every minute looking for new ones when a new one is detected it just it just puts it in the queue and then the worker in each region that um you know the message is destined for will just pick up that message and start the scraping process which uh you know extracts all the secrets and it just throws it into the
27:41
database so it's a pretty simple process and uh this this uh this asynchronous queue and worker setup gives us the ability to scale up and scale down the worker processes as we need so we can save a little bit of money too uh which is always nice so um and all of the all of the code and all of the um the scanning has is within the
28:05
default AWS API limits I didn't have to um like ask for you know well I need to scan lots of disks um they let you do like five disks concurrently uh across everything so you know within those limits within a default AWS account limit I was able to to scan
28:21
these disks so there wasn't anything uh any any like um you know any anything preventing me from doing this uh basically so uh with the architecture out of the way uh step one pick an exposed snapshot so uh what to read and there's kind of two ways you
28:41
can go about this uh you can do an exhaustive brute force over all of the disks and that's totally possible uh you can spend it you know a couple months doing that um or you can kind of do a more careful approach and I initially started off doing the brute force approach like I said I just wanted to steal everything under the sun you know um but that just didn't really uh work out for mostly a couple for basically like
29:04
three reasons um I was just fishing up a lot of garbage so uh the human genome project is on AWS and there's like you know genome sequencing happening so you get like 20 genome sequencing disks in a row and you're just like man I really want some AWS creds all I'm getting is a ton of garbage and there's no faster way to you know just uh thrash
29:20
your database than just filling it with worthless uh worthless files from that so um so there's that and then each disk also takes about 2 to 5 minutes to uh to scan so at a minimum just the logistics of of cloning the disk mounting it to your image and then detaching it and force detaching it um takes about 2 to 5 minutes so the more disks you have to
29:45
scan exponentially you know more time you you kind of spend um not exponentially but just a lot more time you spend uh doing it and also it just costs money so you know who likes spending the profits right? So um so basically I I kind of came up with uh uh a way to
30:03
filter out these disks uh if each disk has an owner ID and that owner ID basically uh just tells you who made the disk so I looked at the owner IDs I just counted them and I looked at the frequency that owners would publish disks and what I found was there were a couple of outliers that published about you know uh like 50 or 60 percent of the disks
30:25
there are about 4 to 5 of them uh one of them is Amazon themselves so uh these disks were basically just kind of worthless they're just deployments of like uh GitHub enterprise and just a bunch of stuff you don't care about so um I took those and I figured the smaller
30:40
owner IDs would have a better chance to kind of reel in those credentials and and get them going um so using that owner ID I was able to cut down the number of disks I had to scan to about 20,000 and uh that just made the whole process much faster and I was able to finish it in time um so uh the next step is attaching the volume and there's this like nice
31:01
AWS butterfly effect that happens where uh you end up wasting a lot of money because a tiny bug in your code ended up like breaking everything and um there's some really interesting failure points that I didn't realize exist like I didn't know the metadata URL could fail um and you know one day it failed it crashed my python script and I had to uh
31:22
you know just manually you know kill those disks those zombie disks that were laying around but it it kind of made me think about well if I'm testing for SSRF and I have a scanning you know some some scanner and it throws the metadata URL at the at the you know target and that metadata URL is had just happens to be broken during that time
31:40
period you know I just got a false negative and it kind of made me rethink some of the AWS testing um that I do myself um just in simple cases like that and then also you're gonna have like a ton of file system issues like LVM disks that just you know like for some reason it just needs a totally separate tool to unmount and mount I don't know why but you know that's just the way it is so um you're gonna run into a lot of those issues
32:02
and you kind of just want to make sure that you're uh you know taking care of them and then searching the disk for secrets um so one thing you can do is you can use something like DLP diggity or um like uh git rob or uh truffle hog to kind of go through like specific things and that's pretty much what I did I just uh I stole the greps from
32:20
truffle hog thank you and uh and then I kind of came up with some some of my own to just uh look for uh the private keys and and what not so um so this so this process uh was pretty pretty straight forward just mostly uh grepping for like really high signal stuff and also uh I did sniff the mime type for each file I would attempt to kind of
32:42
only scan files that didn't look like um you know uh binary or images or what not so uh so this kind of cut down on the number of files I had to scan for each disk and led to you know faster uh you know faster uh scanning so um and and another interesting thing I did
33:00
with this as well is uh because you have access to all the default disks on AWS I just spun up all of the default disks and made like a huge blacklist of every file that you don't want um and I then I manually like added some for etsy shadow we always want to steal shadow files so we kind of uh go through this uh this whitelisting and blacklisting process and
33:22
um all of this uh you know ended up uh kind of coming together to make it pretty quick to scan these disks you know each disk can kind of go go down in about like seven minutes so that's pretty good for uh you know for the purposes of my research and uh it ended up working pretty good um and yeah so we just uh end up grepping through
33:42
everything and you know just just uh just kind of like some lessons learned um like have tests for your code the AWS butterfly effect is definitely real and it's going to return things uh the AWS APIs will return errors you definitely don't expect like the metadata URL failing and also you definitely want to design for like multi-region up front um I made
34:03
some design decisions that ended up uh kind of you know um messing that up a bit because I didn't realize snapshot IDs are actually uh not um unique across regions so you could have two regions two different snapshots one snapshot ID and uh you know if the primary
34:21
key in your database is that ID you're kind of going to have a bad time so uh yeah just make sure you uh think about all of these things up front before you start uh you know kind of looking at it so we've kind of talked a little bit about um you know what this
34:41
is how to find it and uh we can talk about fixing this problem so remediation what does remediation look like in this case it is uh pretty easy but there's a couple of things that to keep in mind uh you definitely want to go and search for your through your AWS
35:03
account to find any public disks that are unencrypted um if you do find one that is sens- has sensitive information in it and is meant to be uh is not meant to be public you want to take down the snapshot first of all that should be obvious to everyone but
35:22
then you also want to rotate your credentials and I know a lot of people like to skip this step because it's a pain it's hard oh there's that one system you know that someone built five years ago we can't rotate the credits I'm sorry you know uh but you should definitely rotate the credentials because if you think about the vulnerability I'm scanning every single disk every minute basically and as soon as you
35:44
make that snapshot public I'm going to initiate a copy on it and once that copy is initiated it takes about two to three minutes to actually finish the copying process and from that point the copy is mine if that copy finishes you can't do anything about it and you're not even going to know honestly um and that's one cool thing I thought that
36:05
was uh interesting about this attack surface is unlike a lot of other attacks um with AWS if you're like you know sending a lot of web requests try brute force directories or something like that um you're going to uh you know show up in logs but in this case you're
36:23
not actually directly attacking a customer on AWS so you're actually not going to log anything and it's going to be really difficult for someone to actually detect that you're cloning their disks there's basically no way to do it so um at least that I could find you know in my in my uh research there might be some some you know logging setup somewhere
36:42
but I couldn't find it so it's kind of a sneaky attack you know you're gonna you're gonna have your creds stolen and you're going to uh not really even understand you know how it happened because there's no logs so if I'm stealing your disks every minute you're gonna want to make sure even if it's only a brief exposure of the disks you're going to want
37:02
to make sure you actually rotate those credentials because anyone performing this attack is almost certainly going to have a setup similar to mine where they're just kind of pulling it down all the time and uh and and not letting you you know kind of get away with uh without rotating those credentials so um and then the last step is of course to have a little post mortem you want to make sure that you understand how this
37:23
vulnerability happened in the first place you definitely don't want to uh just let your developers you know just kind of go and uh and just kind of have free reign um is this an SDLC problem or is this like a random script that we have that we occasionally use and just happens to create public snapshots um you just don't really know that so definitely
37:44
check it out and investigate how you got there and uh you're gonna want to uh you know uh go through this process uh you can go through it manually I'm going to uh delay the release of the tool just by a couple of weeks to uh coordinate with Amazon and get some of these uh disks you know taken down and give people an opportunity to go
38:05
through their own disks and just make sure there's no exposures um but after after a couple of weeks I'm going to release stuffle bag and you can you know just go download that and that will help you scan disks and I've written it in such a way so that um it will work on kind of any disk you have if it's uh private or public so it can help
38:25
in your own organization uh scan for disks you know because one thing uh that I find a lot is um you'll run a scout suite report and then you'll get a bunch of EBS volumes that are unencrypted and sometimes there's like a thousand of them and you're like how do I go
38:40
through all of these and figure out if any of these are actually if anyone should care about them so this tool should work with public and private disks as well as like a wide variety of file systems so you should be able to use this to kind of help uh sift through that pile of hard drives you do uh need to go through uh in your own organization so definitely check out that in a couple of weeks that will also help remediate it and um we
39:06
also uh just kind of wrapping up here uh some more loot that I thought was pretty funny um so I always like to look at my favorite password you know I was like when I crack passwords I always like to you know pull out my favorites so I found like nug lovers password and that was for like a docker account um so thank you nug lover and if that's your
39:26
password you should probably change it um yeah and another thing uh I was looking at some disks and I found some creds on there I was super excited I was you know going through everything and uh super super jazz yes creds and then I realized oh my god this is a
39:46
capture the flag box I just captured the flag on a disk I didn't even on a CTF I didn't even know I was playing you know it's just it's like yeah it was it was it was pretty good thank you thank you yeah so so that was uh another great find uh and uh one other
40:09
thing I found on there was wallet dot dats and uh if you don't know what that is that's your private key for your bitcoin wallet um and I found some like zcash wallets too and everything so basically when I found those I mean my heart rate went up so much you have no
40:24
idea I thought I was about to be crypto rich and you know like go live on my island yeah um it didn't turn out that way I'm here and not on an island so you can definitely know that uh I didn't really get any money from that but it was mostly just people toying with uh cryptocurrency which is great but um you know didn't get crypto
40:42
rich unfortunately with this bug um but that would have been nice so you know maybe in the future someone will throw their wallet dot dat up there with a couple hundred bitcoins whatever um and another thing I found a lot of was SSH keys on Windows machines and I was really curious about this because I thought a lot of people using AWS and and what not
41:03
would be um just using Linux or or some other disk and it just turned out there was a lot of Windows disks that I wasn't expecting and I had to go and make better blacklists and like every time I get an ID RSA off of a Windows box I just kinda smile a little bit cuz you know it's kinda like eh that's cool um so there was like a surprising number of Windows
41:23
machines uh out there and a lot of them were misconfigured in this manner to uh you know to to facilitate uh all the secret exposure so um lots of Windows disks just some really cool stuff there and uh yeah and you know I'm sure that there's a lot more out there you know I was only I was pretty time boxed on this research cuz you know it started in January and then I kinda put it off and you know uh you know just said oh well you
41:45
know no one no one is you know really gonna worry about this but then I realized like how widespread this is and wanted to start looking at it more and more in detail so you know there's still a lot out there I was only able to like look at text files and what not there still could be like database files and and lots of other interesting
42:00
information um and I also only you know I kind of limited myself to uh disks that were under about 100 gigabytes so there's still like more attack surface and the cool thing is new disks pop up every day there's about 5 or 10 disks that pop up every single day so it's like every day you get a nice little uh chance at like a present uh you know like a treasure hunt every single day um which is just fun you know uh you get you get those
42:24
uh emails back you know like ooh found some found some creds so um just kinda have some conclusions here um I manually validated about 50 you know sets of credentials and then after that I was like alright dude I can't do this anymore like my eyes are bleeding you know from like grepping out uh creds and testing them on like a million different
42:43
things so I would just kind of uh give them to Amazon and let them deal with it um I kind of estimate there's about uh 750 to 1250 you know high in critical exposures across all the regions um and this is just kind of a direct extrapolation from my uh you know my regions and the disks I was able to look at in this time and there wasn't
43:02
really a pattern for uh you know who was impacted it was just kind of like random um software uh government contractors healthcare everyone uh everyone just kind of uh was random so not really a whole lot of patterns there um and overall it would cost about $300 plus R&D time um so it's a very cost effective attack which is not my assumption it
43:26
would not it would not have been my assumption and it wasn't my assumption when I first started this um because I thought you know spinning up all these hard disks would actually be pretty expensive but it turns out if you destroy them pretty quick after you've scanned them it just doesn't cost that much so it's a very cost effective attack
43:42
and because you can kind of do it passively you know over time and when a new snapshot pops up you just go scan it really quick and turn it off um it's pretty cheap to do which I thought was great you know getting a set of root keys for 300 bucks it seems like a pretty good deal you know along with all the other stuff so um you know the research and development time of course and uh yeah and that was basically uh you know
44:02
just a really cool aspect of this that I really liked uh just super cheap pre-creds what could go wrong um yeah so that's pretty much it uh you know I just want to thank you guys again for coming and I want to thank these people as well and uh thank you so much for coming here today um that's pretty much it so yeah yeah enjoy the rest of your
44:31
stuff guys I'm gonna hit the bar and uh probably field some questions outside if you guys want to chat with me I would love to talk about this stuff and uh any ideas you
44:40
guys got um so yeah have a good one enjoy thank you thank you