"Get Off of My Cloud": Cloud Credential Compromise and Exposure

Video thumbnail (Frame 0) Video thumbnail (Frame 1141) Video thumbnail (Frame 4793) Video thumbnail (Frame 7289) Video thumbnail (Frame 8584) Video thumbnail (Frame 10903) Video thumbnail (Frame 16727) Video thumbnail (Frame 18766) Video thumbnail (Frame 19890) Video thumbnail (Frame 21604) Video thumbnail (Frame 22680) Video thumbnail (Frame 24306) Video thumbnail (Frame 26684) Video thumbnail (Frame 28045) Video thumbnail (Frame 29925) Video thumbnail (Frame 32388) Video thumbnail (Frame 36532) Video thumbnail (Frame 37716) Video thumbnail (Frame 39533) Video thumbnail (Frame 41337) Video thumbnail (Frame 50664) Video thumbnail (Frame 52249) Video thumbnail (Frame 57828) Video thumbnail (Frame 61307) Video thumbnail (Frame 62531) Video thumbnail (Frame 64301) Video thumbnail (Frame 65304) Video thumbnail (Frame 66699) Video thumbnail (Frame 67876) Video thumbnail (Frame 69300) Video thumbnail (Frame 70781) Video thumbnail (Frame 72096) Video thumbnail (Frame 74750)
Video in TIB AV-Portal: "Get Off of My Cloud": Cloud Credential Compromise and Exposure

Formal Metadata

"Get Off of My Cloud": Cloud Credential Compromise and Exposure
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
An Amazon Machine Image (AMI) is a virtual appliance container used to create virtual machines (VMs) within the Amazon Elastic Compute Cloud (EC2). EC2 instances typically interact with a variety of Amazon Web Services (AWS), and as such require access to AWS credentials and private key materials. In this presentation we will explore how AWS credentials and keys may end up being persisted within an AMI. If persisted within a public or shared AMI, these credentials and key materials may be unintentionally shared with 3rd parties. We will discuss the different types of AWS credentials and key materials, how they are used to access different Cloud services, and the risks and potential impacts of compromise of this sensitive information. A new tool, "AMIexposed" will be released that can check an AMI for the most common ways AWS credentials and keys are persisted within an AMI. The results of research using AMIexposed against public AMIs will be presented, helping to quantify the scope and prevalence of AWS credentials and keys exposed within public AMIs. We'll also discuss the risks inherent in trusting public AMIs to be free of backdoors, trojans, and other malicious hitchhikers. Results of an experiment demonstrating these risks will be presented. Finally, the talk will propose best practices for utilizing AMIs. These will include specific steps for ensuring you organization's AWS credentials and key materials are not unintentionally persisted within public or shared AMIs, and recommendations regarding usage of 3rd party public AMIs. Ben Feinstein is Director of CTU Operations & Analysis with the Dell SecureWorks Counter Threat Unit (CTU). Ben is an author of RFC 4765 and RFC 4767, and has over a decade of experience designing, implementing and operationalizing security-related information systems. His major areas of expertise include network IDS/IPS, digital forensics and incident response, and security operations. Ben has previously presented at Black Hat USA, DEF CON, ToorCon, DeepSec, the U.S. Department of Defense Cyber Crime Conference, and many other events. He is active in his local DEF CON group, DC404. Jeff Jarmoc: A first time DEF CON presenter, Jeff has been hacking most of his life. He got his start in the early days of the 312 BBS scene, moved on to IRC and USENET, and eventually pursued a career in enterprise infrastructure and security. His latest passion is abusing ubiquitous infrastructure devices and systems in an attempt to bring renewed focus on the security of these systems everyone has come to rely on. Jeff has previously spoken at Black Hat USA. When not abusing software and hardware he enjoys spending time with his wife and daughter.

Related Material

Video is accompanying material for the following resource
Execution unit Operator (mathematics) Execution unit Mathematical analysis Cloud computing Cloud computing Quicksort Information security
Purchasing Wechselseitige Information Game controller Server (computing) Multiplication sign Computer-generated imagery Virtual machine Analogy Cloud computing Set (mathematics) Mereology Vector potential Number Medical imaging Order (biology) Web service Type theory Term (mathematics) Different (Kate Ryan album) Computer hardware Backdoor (computing) Computing platform Scale (map) Scaling (geometry) Data storage device Cloud computing Instance (computer science) Term (mathematics) Scalability Vector potential Type theory Personal digital assistant Order (biology) Data center Website Self-organization Quicksort Reading (process)
Dialect Freeware NP-hard Virtual machine Cloud computing Plastikkarte Computer network Cloud computing Instance (computer science) Trojanisches Pferd <Informatik> Plastikkarte Scalability IP address Number Medical imaging Uniform resource locator Web service Computer crime Different (Kate Ryan album) Personal digital assistant Hacker (term) Traffic reporting Address space
Suite (music) Computer-generated imagery Virtual machine Cloud computing Data storage device Number Medical imaging Web service Different (Kate Ryan album) Term (mathematics) File system Elasticity (physics) Mutual information Physical system Addition Service (economics) Block (periodic table) File format Data storage device Cloud computing Virtualization Instance (computer science) Virtual machine Software Mutual information Quicksort Object (grammar) Block (periodic table) Elasticity (physics)
Run time (program lifecycle phase) System administrator Cloud computing Set (mathematics) Mereology Public key certificate Rotation Uniform resource locator Data management Medical imaging Web service Sign (mathematics) Mechanism design Different (Kate Ryan album) Query language Encryption Information security Physical system World Wide Web Consortium Electric generator Public key certificate Kolmogorov complexity Digitizing Structural load Web page Computer file System administrator Data storage device Airy function Bit Cloud computing Instance (computer science) Control flow Public-key cryptography Product (business) Annulus (mathematics) Category of being Root Interface (computing) Website Normal (geometry) Encryption Video game console Quicksort Remote procedure call Information security Content delivery network Slide rule Game controller Identifiability Computer file Numerical digit Authentication Virtual machine Password Floating point Content (media) Login Rule of inference Term (mathematics) Canonical ensemble String (computer science) Video game console Divisor Booting User interface Authentication Mobile app Key (cryptography) Uniqueness quantification Projective plane Content (media) Sign (mathematics) Inclusion map Uniform resource locator Software Password Internet forum Exception handling Window Address space
Authentication Purchasing Enterprise architecture Pay television Identifiability Digitizing Web page Authentication Token ring Code Cloud computing Data management Sign (mathematics) Medical imaging Fiber bundle Canonical ensemble Password String (computer science) Uniqueness quantification Self-organization Video game console Divisor Information security
Web page Point (geometry) Wechselseitige Information Group action Multiplication sign Virtual machine Cloud computing Set (mathematics) Number Medical imaging Mechanism design Endliche Modelltheorie Backdoor (computing) Vulnerability (computing) Physical system Scripting language Key (cryptography) Server (computing) Electronic mailing list Usability Cloud computing Instance (computer science) Performance appraisal Endliche Modelltheorie Game theory Information security Routing Resultant
Wechselseitige Information Group action Computer file File system Materialization (paranormal) Password Content (media) Public key certificate Number Medical imaging Web service Profil (magazine) Googol String (computer science) Shared memory File system Identity management Scripting language Stapeldatei Information Interactive television Data storage device Code Bit Instance (computer science) Variable (mathematics) Public-key cryptography Leak Internetworking Integrated development environment Oval Universe (mathematics) Key (cryptography) Quicksort Information security Laptop
Demon Wechselseitige Information Group action Code Firewall (computing) Time zone Local Group Medical imaging Sign (mathematics) Web service Kernel (computing) Formal verification Gastropod shell Information security Socket-Schnittstelle Dialect Key (cryptography) Firewall (computing) Binary code Projective plane Bit Instance (computer science) Trojanisches Pferd <Informatik> Open set Connected space Sign (mathematics) Rootkit Personal digital assistant Gastropod shell Key (cryptography) Quicksort Mutual information Information security Reverse engineering
Wechselseitige Information Manufacturing execution system Code Port scanner Database Client (computing) Variable (mathematics) Medical imaging Object model Row (database) Physical law Software framework Endliche Modelltheorie Social class Physical system Computer file Electronic mailing list Cloud computing Bit Instance (computer science) Regulärer Ausdruck <Textverarbeitung> Flow separation Vector space Software framework Website Configuration space Software testing Pattern language Modul <Datentyp> Quicksort Information security Automation Physical system Row (database) Dataflow Functional (mathematics) Computer file Authentication Password Similarity (geometry) Code Number Canonical ensemble String (computer science) Integrated development environment Software testing Default (computer science) Module (mathematics) Planning Database Personal digital assistant Information retrieval Key (cryptography) Backdoor (computing) Identity management
Wechselseitige Information Functional (mathematics) Thread (computing) Code Multiplication sign Software bug Revision control Medical imaging Web service Component-based software engineering Root Bit rate Object model Software testing Module (mathematics) Dependent and independent variables Dataflow Structural load Electronic mailing list Database Instance (computer science) Query language Phase transition Right angle Video game console Row (database)
Execution unit Touchscreen Moment (mathematics) Bit Group action Shareware
Execution unit Key (cryptography) State of matter Cloud computing Bit Instance (computer science) Avatar (2009 film) Shareware Backtracking Medical imaging Process (computing) Flag Hill differential equation Software testing Mutual information Video game console
Medical imaging Touchscreen Scaling (geometry) Thread (computing) Computer-generated imagery Cloud computing Software testing Bit Instance (computer science) Directory service Login Information security
Wechselseitige Information Context awareness Database Variable (mathematics) Medical imaging Object model Row (database) Information security Error message Recursive descent parser Social class Constraint (mathematics) Structural load Electronic mailing list Bit Instance (computer science) Software development kit Order (biology) Software framework Modul <Datentyp> Quicksort Wide area network Slide rule Inheritance (object-oriented programming) Module (mathematics) Computer file Computer-generated imagery Password Data storage device Frequency Voting Googol Implementation Booting Dataflow Key (cryptography) Interface (computing) Lemma (mathematics) Content (media) Code Volume (thermodynamics) Group action Coprocessor Cartesian coordinate system Vector potential Mathematics Function (mathematics) Social class Window Wavelet System call Modal logic Multiplication sign Materialization (paranormal) Instance (computer science) Parameter (computer programming) IP address Optical disc drive Hash function Cuboid Scripting language Computer file Attribute grammar Flow separation Connected space Software testing Right angle Automation Information security Physical system Resultant Laptop Row (database) Multitier architecture Server (computing) Game controller Functional (mathematics) Regulärer Ausdruck <Textverarbeitung> Authentication MIDI Content (media) Plastikkarte Metadata Root Integrated development environment Software testing MiniDisc Gamma function Software development kit Task (computing) Default (computer science) Module (mathematics) Inheritance (object-oriented programming) Interactive television Database Key (cryptography) Momentum Object (grammar) Identity management
Sensitivity analysis Statistics Existence Computer file Materialization (paranormal) Password Maxima and minima Axiom Variable (mathematics) Number Fraction (mathematics) Medical imaging Goodness of fit Software testing Information security Identity management Social class Authentication Module (mathematics) Key (cryptography) Information Sine Uniqueness quantification Keyboard shortcut Projective plane Cloud computing Bit Total S.A. Variable (mathematics) Flow separation Public-key cryptography Open set Numeral (linguistics) Integrated development environment Oval Personal digital assistant Password Normed vector space Configuration space Mutual information Remote procedure call Resultant
Wechselseitige Information Server (computing) Computer file State of matter Virtual machine Control flow Total S.A. Number Medical imaging Web service Software testing Booting Position operator Task (computing) World Wide Web Consortium Stapeldatei Validity (statistics) Projective plane Volume (thermodynamics) Bit Instance (computer science) Group action Game theory Quicksort Information security Booting
Mathematics Thread (computing) Computer file Software testing Directory service
Medical imaging Computer file Sequel Authorization Software testing Database Instance (computer science) Quicksort Public key certificate Flow separation Number Task (computing)
Medical imaging Execution unit Bit rate MIDI Virtual machine Selectivity (electronic) Database Login Row (database) Shareware
Slide rule Group action Key (cryptography) Firewall (computing) Multiplication sign Cloud computing Cloud computing Instance (computer science) Instance (computer science) Group action Blog Configuration space Software testing Information security Table (information) Exception handling Information security
Scripting language Email Dialect Statistics Group action Code Sampling (statistics) Twitter Local Group Backtracking Backtracking Different (Kate Ryan album) Internet forum Cuboid Software testing Information security Information security Traffic reporting Backdoor (computing)
Web page Meta element Trail Wechselseitige Information Scripting language Computer-generated imagery Time zone Instance (computer science) Function (mathematics) Mereology Root Cuboid Information Gamma function Local ring Information security Scripting language Addition Slide rule Feedback Sound effect Cloud computing Instance (computer science) Statistics Backtracking Inclusion map Process (computing) Personal digital assistant Quicksort Information security Booting
Slide rule Wechselseitige Information Open source Link (knot theory) Computer-generated imagery Electronic program guide Projective plane Open source Sampling (statistics) Bit rate Coma Berenices Software maintenance Shape (magazine) Number Medical imaging Web service Kernel (computing) Internet service provider Elasticity (physics) Website Information security Information security
Wechselseitige Information Pay television Link (knot theory) Code Computer-generated imagery Number Product (business) Medical imaging Web service Computer configuration Software suite Damping Information security Traffic reporting Address space Form (programming) Enterprise architecture Pay television Execution unit Email Coma Berenices Cloud computing Cartesian coordinate system Product (business) Confluence (abstract rewriting) Repository (publishing) Royal Navy Self-organization Information security
Boss Corporation Information Multiplication sign Queue (abstract data type) Cloud computing Bit Information security
I thanks for joining us today and my name is Ben Feinstein I'm the director of operations and analysis for the Dale SecureWorks counter threat unit to my left is my colleague Jeff charmot if we get his mic hot as well can we get that mic on so I'm Jeff Jarmo comma a security researcher with the Dell SecureWorks counter threat unit so today
will be given a talk we call get off of my cloud just to start with a little thought or a pictograph if you will this
is one conception of multi-tenancy in the cloud sort of a crazy bizarre of different customers all sitting next to each other on the same shared infrastructure this is one way you can concede with that so what are we going
to be talking about today well when we set out to start this work some months ago we wanted to first understand more about amazon's cloud platform we didn't set out to do this to pick on amazon by any chance but really they're they're the big gorilla in the room when it comes to public clouds and infrastructures of service clouds so if you're going to be doing research as it relates to public cloud and infrastructure cloud really Amazon is is the biggest player right now so we want to understand all the different types of credentials that you use when you're using Amazon's web services and we really want to understand sort of the order of precedence of all those different credentials so if you were to have one type of credential what could you do with it and what other types of credentials could you control or manipulate also understanding common mistakes and pitfalls a people or organizations that are using Amazon's Cloud Services so part of this is looking at best practices and the guidance that's published but also understanding what are some really easy ways to make some bad mistakes out there and then with that in mind that we set off to develop a set of tools to detect instances of these problems basically cases where credentials could be exposed with in amazon's virtual machine images or rather virtual machine images that are published within amazon's cloud by third parties and also these tools would detect malicious images or backdoored images that could be out there in the public the public set of images also we performed to better quantify the scale of potential victims if you were to release a malicious machine image and we'll define some of these terms so I apologize if this is a terminology upfront but if you were to release a malicious virtual machine image and publish it in amazon's cloud we did an experiment to get a sense of how many victims you could you could count on receiving and how many of those victims you might actually be able to take control of their their virtual machine and also throughout this work we maintained our work was consistent with our reading of Amazon Web Services customer agreement and their terms of service that are published on their website so people are you know organizations are moving all their infrastructure into the cloud at a rapid pace there's a number of reasons why they're doing that but really it's it's you shouldn't fear the cloud or you shouldn't blindly embrace the cloud it's really just a tool like any other tool there's good uses there's bad uses and there's really suicidal uses so you could think of it sort of like a knife you know there's obviously some good
uses for knives here in this case Crocodile Dundee he has a knife there's lower costs in the cloud there's decreased time to market which is very attractive for organizations today you can rapidly scale about your infrastructure without having to purchase data centers and hardware and servers and storage and also you could inexpensively get geographically diverse infrastructure by using cloud services such as Amazon's there's obviously some
bad uses for the cloud just like bad uses for a knife there's relative anonymity to be found basically you can spin up instances in different regions of the world it's very difficult for a third party to determine who's the actual actor that's behind that virtual machine image or behind that IP address that's sitting in amazon's cloud it's it's a in expensive or if you perhaps have stolen credit cards or you've stolen credentials to other cloud users it's free it's very hard for to finish the blacklist cloud infrastructure the IP addresses are ephemeral they change rapidly you're going to you're going to do a tremendous impact to your business if you just blacklist like large swathes of amazon's cloud because there's a lot of legitimate uses and legitimate services that are running their it makes jus geoip address blocking or geographical blocking much more difficult because you can just spin up a virtual instance in any number of places around the globe to kind of hide that we're where the actual location of the attacker is and again it's highly scalable that plays for both good and bad if a malicious party which is to have a highly scalable infrastructure to support cybercrime or some other aspect well the cloud will offer that as well you can you can find any number of press
reports about you know spammers using cloud services even malware is now starting to attack cloud services there was a case of a speii trojan that had been modified to access amazon s3 and compromised some buckets and that and what we're really talking about also is
the suicidal uses of the cloud here inadvertently we found that many many publishers of third-party images but also many users of the cloud that are using third-party images are putting themselves a great risk so don't be this guy you need to you know look at the cloud and but do it with open eyes and do it you know by considering the risks and looking at the best practices and making sure that you're really adhering to the published guidance out there so that you can use these cloud services but using in a safe way and take take assessment of the risk so some
terminology i apologize for those of you that are already this is already old hat to you but when we gave some dry runs of this and many many folks weren't really intimately familiar with amazon's web services yet and there's a whole lot of acronyms out there that will be using so AWS that's Amazon Web Services that's sort of the overarching suite of different services they offer for their cloud service delivery ec2 you'll hear us use that term that's their Elastic Compute cloud it's essentially a infrastructure as a service you can get shared compute storage and network through the ec2 an ami ami i've heard it called as well is Amazon machine image it's essentially what it is is a virtual appliance container format so you would pick an Amazon machine image and you could launch any number of instances of that image and those actually become virtual machines that we running in the ec2 cloud s3 or simple storage service that's object storage that amazon offers its customers you basically use what's known as buckets and you read and write data into those buckets as objects and then a more more recent addition though it's it's probably not for a few years now is the elastic block store and that is a it's a virtualized block device just like on a unix or linux system you can mount a block device on that system and read and write to it just like a filesystem EBS is a cloud file system a cloud block device that you can mount on your images there so when we first set
out to do this project we wanted to understand all the different kinds of credentials out there and what they're different what the different uses of those credentials are and then which one's control which other credentials so there's really three broad categories of credentials for Amazon Web Services there's access credentials there's sign-in credentials and then there's a set of different account identifiers that you have to use in terms of access credentials you've got access keys and access key is merely a long unique string of digits there's a public and a secret part of that it's a little bit analogous to certificates but there are certificates in play as well we'll talk about those but your access key ID also has a secret access key and what you use that for our authenticated web services API is like soap api is to use simple storage service or amazon's mechanical turk which is essentially a service where you can get real human beings to do your bidding for for pennies per request there's a screenshot on the next slide where basically you manage all these different credentials with in amazon's web console and these keys are also the access keys are also used for cloud front which is Amazon's own content distribution network that they offer as part of their web services sweet Amazon's recommendation here rotate those keys at least every 90 days here's a quick screenshot of the nifty web interface where you can you can manage these access keys both the secret key and the private public key part of it enough are important credential is x.509 certificates so many parts of amazon web services you use a certificate in private key to access or to sign your images to bundle images these are again they're managed through Amazon's web services console but you also can do some of this with AP is you can generate your own certificate or private key or you can provide your own certificate this is sort of a trade-off of convenience for security either you let Amazon generate the secret key for you or you do that on a system you know it's obviously recommended that you probably want to generate your secret key yourself and then provide up to Amazon use it to bundle your am Isaac cryptographically signs and encrypts am is that our private or just cryptographically signs the machine images that you're going to make public again Amazon's recommendation rotate these things every every 90 days so go issue new certificate in private key at least every 90 days for your infrastructure another webs another screenshot here easy to keep airs this is a this is one of the biggest findings that we'll get into later of what we actually found when we scanned the u.s. East cloud with our tools that were we're releasing today their SSH key pairs for all intensive purposes when you spin up when you when you launch an instance of a machine image you specify an SSH key pair for that image to load and on bootup it essentially loads the private excuse me loads the public key into the SSH authorized keys file and then you use your private key to access it to ssh and get a console on that virtual machine so it's very convenient way to to get at get secure access to a virtual machine you don't need to bake an authorized key file into the image itself you specify it at runtime when you spin up a virtual machine there's no explicit security recommendations I've found from Amazon around these key pairs or rotating them or such and there's also interestingly there's windows virtual machines that you could run in the AC two and these key pairs play a role there where literally the administrative RDP or remote desktop password is encrypted with the private key and you would decrypt it and then access the RDP port on the windows image so it plays a role in accessing windows as well and you can login to the web console and play with these things and allocate your keys and such cloud front
key pairs yet another set of credentials what we found is there's there's so many different you know access keys private keys key pairs cloud front key pairs that it's quite confusing and really the first part of our research we're just figuring out and identifying and enumerate all the different credentials that you have to use when you use Amazon's web services this is the key you use when you're using their CDN network to generate sign URLs essentially way of offering private private content with in amazon's content distribution network so pretty much the one key to rule them all is the sign-in credential this is the actual login that you log into their administrative web interface and control all aspects of your amazon web services account so this is if you're going to defend any of these and secure them strongly this is probably the most important set of credentials there are what it is if you if you ever bought a book from amazon or a CD it's your username and password all you do is take a normal amazon account you may have purchased the book and you activate web services on that account and that is now that your log in amazon's web service is now the credential website is now the credential to your web services account they've just also offered multi-factor authentication it's not RSA securid so you can protect this with multi-factor
off you do a purchase at 12 or 13 token online you activate your account with this token and then now you've got multi-factor authentication to add to just username and password to protect this this is really important for enterprises or larger organizations that are going to be using cloud services so that you're not your whole kingdom isn't just relying on one user name and password here they haven't rolled out the kit and retinal scan just yet
account identifiers there's there's a two-count identifiers there's a canonical one which you use for certain AP is and there's just another idea that you use for other AP is and it's not really a secret but it's a long string of digits that you could inadvertently exposed in your images
so as we're going through this we found some prior research that we thought was interesting we wanted to call out so a few years ago back at black hat 09 Def Con 17 a group of researchers including at Alex Stamos delivered a talk called cloud computing models and vulnerabilities raining on the trendy new parade and they showed to among other things they show two interesting techniques one was a way to essentially game the system to get prime placement within the list of public machine images of their own machine image so literally there's there's hundreds of different say fedora images that are available in the public cloud and the key is if you're not really on that first page of search results people probably aren't going to pick your machine image so they were doing this so that they could get a set of victim users that would run I wouldn't say a backdoor machine image but a machine had a phone home mechanism in it and so they also they gained the system they got prime placement for their virtual machine image and then they had a phone home script that was running every time it you spun it up and they could measure how many people how many people did this there's some
interesting precedent that was published earlier this year so in tipping points blog published this a copy of this letter that amazon sent to their customers back in april and essentially someone had found that an image had been shared with the public that had authorized ssh keys baked into it so the publisher of this image could potentially have full access as route to anybody that that chose this machine image and spun up instances of it so this also reinforced to us that this is something we wanted to look at more closely and this is why we built these tools to go ahead and evaluate large numbers of machine images to look for things such as SSH authorized keys that aren't really shouldn't be there Jeff
you want to take over from here okay so there was after we were accepted for this talk there was a group in Germany some postdoc researchers at a university there that released some researchers it's very similar to ours they also released a tool that scans am eyes in a very similar fashion to what we do their tool is a Python script that essentially you have to run on the on the instance itself whereas our tool interacts with the with the Amazon api's to spin up instances SSH to them scan them remotely and turn them down so they're they're similar but the approaches are slightly different and we've scanned more images and and you know have a little bit larger data set from our tool but I just want to highlight that you know there's other people working on this and it's possible that there's that there's others that are publishing it so you know this is probably something that's being done in
the wild so when creating a public am i or am i that you share with other users there's a number of ways that you can that you can you can accidentally leaked your own credential material you know on the file system itself you may have your you are amazon web services can private key you might have ssh key pairs either you know an authorised key that been mentioned or you know an actual identity key you know you might have ssl certificates and private keys that are on the image you need to be conscience as a publisher of an image what you're putting out there because you know once it's public it is truly public and you know anybody can can access these images and search for these sorts of credential materials and that's essentially what we've done there's also other ways to leak information bash history files is a big one if you run the run the image as an instance yourself and done any sort of work on it and and then made a public image out of it that bash history might expose that and it might also expose some of these these credential materials that you've used files like batch RC bash profile often contain you know environment variables that are set to these values for scripts to access and things like that so we look for these as well and then also vim info files you know occasionally there's a chance someone might be trying to clean up and go in and edit a file to delete it but they search for that and there they're vim info stores that search for that string you know so it could be there as well and then we try to try to
do a little bit looking for signs of a of a malicious am I this is primarily the SSH authorized keys there's possibilities for you know rootkits trojan binaries you know reverse shells connect backs those sorts of things you know with the pv grub kernels that they support now you can run your own kernel on on ec2 so you could you could you know back to where the colonel directly if you wanted to there was a talk today in sky talks on CH with 303 demonstrated a nami image that you know essentially would when it was launched would would phone back download a piece of code executes it and hit it he'd get a meterpreter shell from it anytime somebody spun that instance up so that's also you know related work to this so
there's been other work on malicious am is been mentioned some of the work from 2009 from Def Con 17 this is something you know anybody who's looked at amazon web services you know thinks of this that that you know these images republic and there's not much much verification of who's who publishes are and what their intentions are and in a lot of cases is really difficult to detect you no malicious intent and malicious behavior so we we undertook a project to try and gather some data surrounding this you know how easy is it to find victims what kind of instances are they running it on we're like regions are most popular and kind of most importantly you know with their security groups which is a similar to a firewall policy for an ec2 instance you know that would those be configured sanely don't allow the world to connect out how your ssh ports for example you know which would be effective at you know being at being somewhat of a hedge against against an ami that has you know a trojan ssh daemon or a pre-existing authorized key and then
we also released our tool well developed our tool that we're releasing to perform large-scale scans of images within the cloud so the name of the tool is am i exposed so it's kind of a pun on am I and you know asking a question and the tools seeks to find the answer so it's a it's a framework essentially for scanning am eyes for these credential leakage patterns I tried to follow a sort of similar model to what metasploit does although my codes nowhere near as elegant and an over near as capable as the stuff they've got but in a similar you know in a similar vein there's a there's test modules that you can plug in to add new functionality with very little code and leverage the existing classes the existing object model to to add new tests so at its heart it uses Amazon's API to automate you know the retrieval of a list of images defining your scope of your tests you know i'd imagine common use cases would be testing all images that i own you know but maybe before i make them public if i'm planning to do that you know as a penetration tester you might be interested in you know testing all images that are owned by a specific client you know to see if you can find any other credentials being stored publicly so once the scopes generated we basically iterate through all the images launch an instance run tests over ssh and record findings to a database the tools released on our website SecureWorks calm / research / tools there's currently a little bit of a problem with the file that we're going to get corrected probably Monday morning so if you have any trouble download again the meantime I apologize for that
as I said there's there's several test modules included a release there tend to be precise you know they check for the things that we've been discussing and I'm not going to keep repeating you know all these these different vectors it also tests within the system files so it'll search for a number of files and then search within them for for specific strings these strings are all definable by a configuration file you can just pop a regex in there and it'll it'll search for that regex and you know you don't you don't necessarily even need to write it module 2 to add a new string that you're interested in so here's kind of the flow
of how it executes up at the top there we've got our scope definition phase you can either do this manually just by just by going into the AWS console and putting tags on images that you're interested in or there's a tool with it that's included called tagged images Darby that allows you to define larger scopes you know more easily you can can't go and click through you know thousands of images you know again query for the tag damages iterate over them and obviously we wanted to thread it oh I should mention this is all in Ruby I didn't even say that but down the right here we have some of the gems that are used so we're using the threaded collections gem for for managing threads and then we're using the well the official amazon web service SDK for Ruby which was released about three weeks ago so I actually kind of wrote two versions of this tool I spent a lot of time writing one that interacted directly with the soap api is over X you know parsing the XML to my own object model I was just about done with that and about to do our large-scale scan when Amazon released their own SDK it was much more elegant much cleaner much more efficient so i ported i ported my code over to adopt that rather than duplicating that functionality they've also been improving it at a pretty rapid pace so there's been a couple of releases that you know made minor adjustments and bug fixes is you know just in the past few weeks so once we launched the instance we have to you know we have to figure out what account we are going to use so that's down there on determining us as h username so we try and login as root if that Bales we you know try and parse any response that comes back a lot of times you login as root and you get a response that says you know please use ABC instead of root so we try and parse that from the banner and then if that fails we just did rate through a list of usernames that we've discovered or commonly used out there so you know generally we're pretty capable of you know being able to determine the username click and then you know then we run our tests
so we go through each of the test modules you know load the module will run it and record all the findings in a database so we're using net SSH for the ssh components and active record for the the database component so I'm going to
demo it it's going to take a little while to run so we'll talk a little bit more about the internals while it's running give me a moment here
apologize for the delay
okay you see my screen yet okay so out
here this is the day AWS console I'm showing three images these are all images that I've created i'm not i'm not going to show anybody else's data but my own so two of these this first one is a public image that i created that's an ami that allows you to run backtrack 5 within within Amazon's Cloud and we'll talk about that a little bit more later the second one is from when I was developing that image this is a private instance or a private image but you'll notice i scanned it previously and it's marked as failed you know as I was iterating through the process I broke this as h de menthe on this one so it starts but i can ssh to it so i just wanted to show that it flags that has failed and then the other one down there is one that I that I made just for the you know for the demo here that demonstrates some of these key material leakages so I've got I've got things there that look like real keys they are real keys but i'm not using them anywhere and you know it's not like data it's just for demonstration purposes but you'll notice there's a tag on these and two of them are in pending state and that's how we define our scope so the
scope of this test is those two images
so just do this full screen whoops ties
that's still there alright so this is actually I'm SSH tintu into a running instance on Amazon's Cloud that's where I did all my tests from so it's kind of interesting that I was using the cloud to scan the cloud for security problems so it was all taking place in hamazon the infrastructure so this is a directory where I've got the tool here you'll see this tag images file that we that we talked about that that allows
you know defining the scope and then
scan dot RB is really the main you know the main heart of of the you know of the tool so I'm going to go ahead and launch scan dot RB it looks a little bit weird here my ascii art didn't scale to this this font size so you see there it says it's storing logs for you know each am i under a logs am I subdirectory starting a test of two am I I've got to configure for 13 threads but it's not going to run 13 it's only got two images that just timestamps and it's scanning so this is probably take about 10 or 15 minutes so we will go back and come check on this
keep doing that come check on this in a
little while
okay we got through this sorry i lost my
spot okay so while that's running we
talked about some of the things I did to expand the Amazon Web Services SDK I basically just opened their their classes and defined some additional methods to give me the functionality I needed to do this that they didn't provide you know through the through the SDK so we modify the instance added some methods to interact with instances run a command over ssh either via ssh exact or PTY you know and then it's gotta run test method that basically loads a test module runs it against that instance I'm going to go through these kind of fast the slides are a little bit wordy but that's mostly mine as a reference i modified the image class to add a you know another method there added some places to store parameters we discovered that ssh user and i need to store it somewhere so i just store it as a tag also that test status you know updates that tag that that defined our scope and defines the results of the test created a finding class this really simple class it's just bass but backed by active record so when you when you instantiate a finding object it just logs it in the database and then there's a test RB is our base test class so this is the parent class of all the test modules it has some some common functionality that's that's used across test modules so you don't fear I tit on each one and then it implements the file helper module that that allows us to test within files so we're not we're not redefining that everywhere you know any any test that needs to look within a file for contents can just call this method and you know it's already taken care of and then the individual tests themselves derived from the base test they have an init method that just basically defines you know some metadata the name of the tasks you know the severity of various findings etc and then a run method that runs the test so we're trying to keep things simple bye-bye you know by having a sound object model and keeping all well not all but keeping a lot of the you know the heavy lifting out of the tests so here's an example of a test file at the end of the day this is a pretty simple one just checks for Bosch history file if it exists it looks within that Bosch history file for you know things that we've defined as you know as potential key materials or credentials this one's similar here although I got the subject the title of the slide wrong this one's SSH authorized keys so this is where we're looking for any authorized key file looking within that file to see if there's any keys present other than the one that is provided through you know through the Amazon interface so this will identify unknown keys this one's a little bit more complicated it was an attempt to try and catch any you know any sort of persistent connections so you know we're looking at a netstat and seeing if there's active connections from the host excluding a few IPS you know not not going to look at loopback connections I'm not gonna look at connections are coming from my test box etc this isn't terribly effective because it has to be a really long live connection in order to to be caught if you're doing something like you know hit neces H and download in the script or I'm sorry hitting you know an HTTP server download the script that's good be a really quick connection and the odds that we would catch that or pretty slim but you know if somebody had an ssh session persistent for a long period of time and and we happen to test the right time there's there's a chance so let's talk about the scope of our testing we started with the you know the idea of looking at all US East images we removed Windows images from the scope they're generally paid images because of licensing constraints and they're more difficult to interact with remotely because you have to do it all over rdp it's a little harder to programmatically access so that left us with 5515 non-windows images this was the list of images that we attempted to process with the tool there was 771 images that were paid so you need to have an additional license pay an additional fee to use those and we just kind of skipped those because we were trying to keep you know keep costs under control there was 2767 that bailed a test and this is really kind of interesting it seems like almost all of these were due to some sort of error with the image itself so you saw the one that I had where you know my sshd was broken that was a private image you know it's broken why would I make it public but it seems like a lot of people make these public anyway so you know you can't I couldn't act interact with them over ssh because they won't start you know get Colonel Maddox on boot you know you can't find the the root volume on boot sshd doesn't work etc so we ended up with 19 well 1977 images that were that were scanned by our tool we had 580 unique images that had one finding or another so that's 29.3 percent of the images that completed all the tests had at least at least one security problem we shared these findings with Amazon security team we didn't verify any of the you know any of the credentials we didn't attempt to use any of this stuff so it's possible some of these had you know been revoked or something like that I have no way of knowing you know in amazon's been responsive about about trying to you know contact the the image owners and the users if it's applicable and and you know make everyone aware and remove images if necessary etc so i really want to applaud amazon security team they you know they take this threat seriously and they've they've been you know tremendously cooperative with us so the
largest finding was the problem of ssh authorized keys i suspect that many of these are you know people that just didn't didn't realize they were there they're probably not you know it's hard to say how many of them are malicious but the fact is when you download an image that has a key file whoever has that private key if they can reach the ssh port can authenticate so there's no real way to separate what's accidental and what's malicious you know it's a pretty severe finding so 19.5 two percent of the a.m is we tested nearly twenty percent had an unknown authorized key file so you know by that statistic if you randomly launched five you know public ami is there's a good chance that somebody else has access to one of them this represented 44.9 six or our findings so it's it's far and away the the most prevalent problem that we've encountered this chart here breaks down
all the different findings that are various test modules you know encountered the number of occurrences of each the number of unique images that was affected by that finding the percentage of our of our total total number of findings that were represented by that class and the percentage of a mis that had that you know had that problem or I'm sorry the percentage of the percentage of a ok so that's the percentage of all am is with bindings so if we take the number of the number of images that had on a heist ssh-keys as the numerator and the number of images that had some problem you know that we discovered as the denominator those that's what that percentage represents and you know then we have the percentage of our of our test case so we see there again unauthorized ssh-keys existed in you know nearly twenty percent i believe the German German researchers their number was thirty percent but I'm not quite sure what does go they tested a thousand images we tested nearly 2,000 so that's going to vary a little bit and I'm not quite sure what region they tested I sort of suspect it might be you just you know because they're from Germany but I'm really not sure so I don't know how directly comparable are our findings are with each other but it seems we're both in the same ballpark our second highest finding was just the existence of history file so you know that in and of itself may or may not be a you know significant finding if that history file contains you know sensitive environment variables key materials etc we would log another finding but in some cases the the history file might have you know have some sensitive info that we weren't standing for you know the other tops their environment variables that are commonly used to point to to Amazon Web Services key materials the access keys themselves you know that's that's a significant one it's not a huge number of images but you know with those keys essentially if we'd wanted to if we were so inclined we could have scanned until we found one of these and then continue the project you know by using that credential and you know reduced our cost quite a bit but you know we're not we're not trying to be criminals here we're just trying to gather the scope of the problem other issues SSH identity keys so that's the actual private key which is kind of kind of unusual in at least one case I was able to find the you know the identity key file and then go back to the history file and find a remote host name that they've logged into with that identity key so chances are good if I'd used that same key against that same host I probably would have gotten that host you know which isn't isn't on Amazon's Cloud but you know may have had its security compromised as a result of this you know this lacks practice of keeping the key there and there's various other findings you know much lower proportion so I'm not going to discuss them all the ssh password authentication enabled we actually added that test as we were like you know most of the way done with our with our scan so those numbers are really incomplete but that's a module that's looking for an sshd config that has password often abled you know Amazon Web Services you too is really built strongly around key authentication so if you have a password authentication now the you know the strength of the passwords didn't accounts that are configured on the on the image comes into play so we didn't find all that but you know as I said we didn't we didn't test all these with that with that test
so let's talk a little bit about the costs excuse me the my amazon web servers bill last month was 1333 dollars you know i got completed all these tests within the month and the vast majority of that is due to you know it's due to this project it's a little too bad three dollars in 84 cents more and I would have been at 1337 so that would've been kind of cool and yeah I wish I would have figured that out I wonder if they'll let me over pay the bill you know like just here's 384 more like anyhow if that breaks down to 67 cents per image scan 98 cents per finding 15 dot sixty-eight cents per amazon web service credential so that seems like a bargain you know with a little bit more manual review we might find that some of these things are fine are you know that are you know for example a batch history file if you were to look at you might be able to find other things and have more findings kind of conversely you know you might find that someone more false positives so our numbers are obviously rough and then the cost per finding or you know a little bit rough there as well we could have leverage spot instances to lower costs so spot instances you zoc peak pricing you know if you don't care when you run a task you can run it whenever amazon it has more capacity available at a lower cost it would have made this grips a little bit more complex to you know to keep state of what machines are running it not and there's also the thought of analyzing the EBS boot images themselves without you know without launching the instance this might reduce costs it might also increase the amount of images were like we're able to test some of those public game eyes that when it boot properly if they have a valid EBS volume that volume might still have data on it you know the fact that the machine doesn't boot doesn't mean there's you know no take out there and i sort of suspected you know if someone's going to go to the trouble of making a broken image public they're probably not going to be following some of the best practices in securing their you know their credentials as well so that might be interesting and again as I said before you know if we were malicious about this you know we could have reduced our costs pretty greatly let me
go back to our test there that should be just about complete now
10 minutes oh wow ok I'm going to have to pick up the pace here okay so keep changes okay so this is our
test here and you can see those two threads completed those tests completed by going through the log am eyes directory this might be a little hard to read at this at this size but let me look at the log file for the first one in here Oh for BT so yeah it doesn't
wrap around too well here with this with the size is that maybe I can shrink it awhile anyway you can see here you know it starts up the beginning launches an instance it's got the instance ID number there times justice h2 it ssh hadn't started yet so it sleeps for a little while discovers the ssh username and then starts running tests this particular image didn't have any any issues so runs a test completes a tas runs a task completes a test etc not a
whole lot of interesting there it eventually terminates and shuts down if we look at the other the other tests this one's a little more interesting so
same sort of things garden instance discover ssh now there you can see we found an unauthorized ssh key we call that high severity and there's the key you know we we checked for x.509 certificates found a certificate file there's the file name and path it just kinda continues on like that with with the various findings these are also all stored in a database if we go back to using sequel light here but it's active
record so you can use whatever adapter you want select star from findings and
you'll see we've got our database there you know that logs all our findings where things were found in the details that go along with them so that was a you know that was a demo showing to two machines being scanned in real time at
our peak we were we were scanning about
110 images an hour and when I say scanning that includes identifying images as paid images which obviously doesn't take very long um but yeah that seemed like a pretty decent decent rate took about two days
you know too well it was two and a half three days to scan you know the entire scope of our our tests but the question then kind of becomes you know the ssh keys was our with our biggest finding
don't people use security groups to protect their instances if I've and ssh key it shouldn't matter if I can't reach the you know the associated port so
here's this you know just a slide kind of explaining you see two security groups it's just like a firewall policy there's screenshot there the interesting thing is they're inbound only unless you're in a virtual private cloud so alcoholic from a potentially malicious am I there's really not a strong way to filter that I could use IP tables or something on the instance but you know when it's someone else's provided am I at least the first time you run it you're running their configuration so we
did some testing of security group practices I released these am is across all regions to allow running backtrack 5 in AC to june twenty third just kind of announced it publicly in a couple of different places you know i thought this was an interesting sample because it's something that you know people would that are more security minded might be interested in has anybody here actually used these at all i'm just curious no okay well it was also useful to me for gathering data and statistics there was
a phone home script in it now this wasn't a true backdoor it wasn't anything that i could log into i couldn't execute code in the boxes but it did reports in data back to me when it was launched here's the script oh
it's on the next page so there was a script that was part of a you know part of the startup collected some metadata from from the easy to metadata API and sent it back to me here's the script itself and you'll notice there there's a comments kind of explaining like you know what we're trying to do and trying to calm people down that it's not real malicious notice the curl directs the entire output to Devon also I can't save it as a script and run it and you know it's got a website there if you have questions visit this the website explained it and kind of solicited feedback to say how did you find this you know I'd like to kind of gather processes that have been have been helpful in discovering these sorts of things and I was planning on crediting people who had found it but as you can see a nobody contacted me nobody hit that web page so it doesn't seem like too many people found it so we received
95 phone homes as of July thirty-first this was 69 unique instances and when we received a phone home we would try and connect back on 22 in poland ssh banner we were successful in 71.5 percent of cases seventy two percent if you encountered it by unique instances rather than all instances to get rid of the effects of people rebooting an existing instance so yeah that that's kind of concerning that tells me that you know seventy percent of of people who are presumably somewhat security minded because they're downloading backtrack after all are not downloading other you know the running back track in the cloud didn't bother to properly firewall off the rest is h port if instead of put in this phone home you know I put the phone home in addition to ssh key which you'll remember was our most common finding it's a pretty safe bet that i would have had you know had root access to you know most if not all of these boxes so our lessons here we
kind of summarize them already one interesting thing I think is that you know it doesn't seem like too much of a stretch to say that the average user is probably were soft and you know then our sample again you know more sophisticated back doors would be harder to detect than this and nobody detected this so that doesn't bode well for for you know using militias am eyes and actually finding them when you when you use a public am i you're putting a lot of trust in the publisher so at the very least consider the source of the image you know who that person is if you want to be super safe build your own images but you know at the very least consider that images from amazon or another trustworthy provider you no official open-source maintainer 'he's maybe more safe than you know trust in me so as we're in the midst of this
project amazon published a number of new documents given some security guidance and best practices we wanted to offer some links in our slide deck to these so if you're if you're using their web services we encourage you to go check out all the guidance that amazon's published out there about sharing am is how to do it safely how to use public am is in a safe fashion and there's a whole bunch of articles out there on this elastic com site from Eric Hammond so we would encourage you to check those out there's a lot of good material out there and Jeff mentioned this earlier so where
would you obtain trustworthy third-party am eyes besides you you know building them from scratch yourself well amazon themselves offers some supported and maintained images you can follow the link up there or find it yourself and they've got a number of images out there they have their own yum repository security updates you know a product lifecycle if you want to pay for their premium support service these are the the images they're going to support so this is this makes a good option to enterprises or other organizations that are using their services there there's a number of third-party vendors that also provide their own a my images right scale cloudera a number of other vendors and one of the best practices we've talked to some larger organizations that heavily leveraged web services from amazon is they many of them take Amazon's own images as a base and then build a customized am i for their own application suite on top of it and then they can leverage amazon's lifecycle and support and package updates to keep their base up to date and then just all they have to worry about its updating whatever the application code or other sweet they add on top of that was if you
find a issues in the cloud we had a really good experience working with amazon security team they're very responsive they're easy to get a hold of they investigate every report so you can can find them online they have a form PGP keys email addresses it's not hard to get ahold of them and we just wanted
to give thanks to beetle from the Amazon Web Services security team and the rest of his team for working with us as as we did this project and for working you're creating work for them as they're doing customer notifications as we as we fed these findings to them and also think our bosses at Dell SecureWorks for authorizing us to do this project in and giving us a little bit of budget to make it happen we'll be moving over to the Q&A room here shortly across the hall I would like to meet any of y'all that are you know using cloud services have questions or have some information to share with us so I hope something y'all will join us in the queue a room across the hall thank you appreciate your time