AI VILLAGE - Automated Planning for the Automated Red Team

Video thumbnail (Frame 0) Video thumbnail (Frame 3461) Video thumbnail (Frame 4684) Video thumbnail (Frame 6735) Video thumbnail (Frame 8015) Video thumbnail (Frame 10164) Video thumbnail (Frame 11365) Video thumbnail (Frame 12052) Video thumbnail (Frame 13503) Video thumbnail (Frame 14336) Video thumbnail (Frame 15405) Video thumbnail (Frame 16939) Video thumbnail (Frame 18040) Video thumbnail (Frame 19346) Video thumbnail (Frame 20897) Video thumbnail (Frame 22572) Video thumbnail (Frame 24712) Video thumbnail (Frame 27123) Video thumbnail (Frame 28945) Video thumbnail (Frame 29821) Video thumbnail (Frame 32184) Video thumbnail (Frame 33630) Video thumbnail (Frame 34779) Video thumbnail (Frame 38674) Video thumbnail (Frame 40203) Video thumbnail (Frame 40989)
Video in TIB AV-Portal: AI VILLAGE - Automated Planning for the Automated Red Team

Formal Metadata

AI VILLAGE - Automated Planning for the Automated Red Team
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Offensive assessments – i.e., penetration testing, adversary emulation, red teaming – have become a key component of maintaining a secure network. Unfortunately, offensive assessments require significant resources, and can vary in quality and structure based on who specifically is conducting the assessment. In the past few years, we’ve seen people try to remedy this problem by creating automated offensive assessment tools, but the capabilities and goals of these tools are highly variable, and many either require personnel to manage them or lack the ability to conduct dynamic or end-to-end tests. We believe that automated offensive assessments can be done better using automated planning. One of the older branches of AI, automated planning seeks to solve problems where an autonomous agent must determine how to compose a sequence of actions together to achieve an objective. Problems in this space can range from constructing offline deterministic plans, to planning under probabilistic conditions, or to planning in scenarios where the world and underlying model are un- or partially-known. Planning techniques have been applied to solve problems in a variety of domains, including controlling unmanned vehicles and designing intelligent agents in computer games. In this talk, we’ll describe how we’ve leveraged concepts from the automated planning community to help us design CALDERA, a free, open source automated adversary emulation system. Using these concepts, CALDERA dynamically strings techniques – taken from MITRE ATT&CK™ – together to achieve objectives and conduct end-to-end tests. In addition to describing CALDERA itself, we’ll also discuss more generally some of the challenges and advantages of deploying automated planning to automated offensive assessments, discussing alternate approaches that we as well as others have considered in tackling this problem. Attendees should walk away with both an understanding of how they can use CALDERA as well as how planning can be used for automated offensive assessments.
Statistical hypothesis testing Multiplication sign Control flow Stress (mechanics) Disk read-and-write head Statistical hypothesis testing Planning Measurement Data model Emulator Latent heat Term (mathematics) Different (Kate Ryan album) Endliche Modelltheorie Information security Menu (computing) Computer network Bit Statistical hypothesis testing Statistical hypothesis testing Software Integrated development environment Interface (computing) Negative number Automation Resultant
Statistical hypothesis testing Meta element Group action Open source Link (knot theory) State of matter Set (mathematics) Parameter (computer programming) Statistical hypothesis testing Emulator Latent heat Analogy Software Program slicing Core dump Integrated development environment Circle Vulnerability (computing) Vulnerability (computing) Electric generator Projective plane Directory service Group action Power (physics) Statistical hypothesis testing Statistical hypothesis testing Degree (graph theory) Category of being Emulator Software Atomic number Gastropod shell Game theory Object (grammar)
Series (mathematics) Model theory Set (mathematics) Group action Variable (mathematics) Hypothesis Chaining Emulator Emulator Software Endliche Modelltheorie Bounded variation Address space Form (programming)
Slide rule Enterprise architecture Multiplication sign Real number System administrator Public domain Perspective (visual) Graph coloring Bookmark (World Wide Web) Emulator Goodness of fit Latent heat Malware String (computer science) Matrix (mathematics) Software framework Traffic reporting Descriptive statistics Cybersex Electronic mailing list Computer network Multilateration Incidence algebra Group action Complex number Local Group Software Enumerated type output Configuration space Normal (geometry) Pattern language Physical system Window
Coefficient of determination Group action State of matter Set (mathematics) Bit Automation Group action Planning
Group action State of matter Set (mathematics) Dynamic random-access memory Graph coloring Field (computer science) Number Planning Sequence Atomic number Different (Kate Ryan album) Representation (politics) Initial value problem Descriptive statistics Condition number Boolean algebra Block (periodic table) Classical physics State of matter Group action Variable (mathematics) Sequence Preconditioner Uniform resource locator Logic Representation (politics) Block (periodic table) Table (information)
Point (geometry) State observer Group action Distribution (mathematics) State of matter Graph (mathematics) Decision theory Set (mathematics) Branch (computer science) Mereology Field (computer science) Planning Operator (mathematics) Core dump Resultant Energy level Analytic continuation Form (programming) Distribution (mathematics) Graph (mathematics) Decision theory Classical physics State of matter Core dump Bit Contingency table Group action Continuous function Cartesian coordinate system Markov chain Heuristic Extension (kinesiology)
Slide rule Group action Graph (mathematics) Model theory Constructor (object-oriented programming) Graph (mathematics) Model theory Exploit (computer security) Computer network Entire function Tracing (software) Graph coloring Planning Chaining Software Logic Graph (mathematics) Logic Utility software Information security Information security Vulnerability (computing)
Statistical hypothesis testing Classical physics Group action State of matter Graph (mathematics) System administrator Latent heat Type theory Authorization Data structure Endliche Modelltheorie Information security Vulnerability (computing) Numerical taxonomy Area Distribution (mathematics) Graph (mathematics) Mapping Knowledge base Military base Interactive television Computer simulation Computer network Line (geometry) Group action Connected space Software Penetrationstest Normal (geometry) Right angle
Group action Computer file Open source Decision theory Computer file Open source Shared memory Pivot element Preconditioner Sequence Emulator Emulator Filesharing-System Surjective function Condition number
Information management Group action Functional (mathematics) Computer file Building Shared memory Line (geometry) Group action Reinforcement learning Number Emulator Moment of inertia Hardy space Bit rate Core dump Right angle Address space Weight function Vulnerability (computing)
Vulnerability (computing) Group action Core dump Binary file Perspective (visual) 2 (number) Planning Chaining Emulator Extension (kinesiology) Descriptive statistics Vulnerability (computing) Condition number
Statistical hypothesis testing Group action Server (computing) Algorithm State of matter Set (mathematics) Planning Data model Emulator Component-based software engineering Heuristic Endliche Modelltheorie Descriptive statistics Enterprise architecture Algorithm State of matter Group action Preconditioner Statistical hypothesis testing Chaining Numeral (linguistics) Exterior algebra Emulator Personal digital assistant Enumerated type
Statistical hypothesis testing Group action Server (computing) Game controller Scripting language Open source State of matter System administrator Determinism Control flow Mathematical analysis Computer font Rule of inference Perspective (visual) Web 2.0 Architecture Emulator Latent heat Component-based software engineering Logic Heuristic Spacetime Codierung <Programmierung> Endliche Modelltheorie Extension (kinesiology) Tunis User interface Simulation Algorithm Server (computing) Java applet Database Bit Group action Chaining Emulator Integrated development environment Software Telecommunication Revision control Interface (computing) Quicksort Virtual reality Simulation Physical system Spacetime Extension (kinesiology)
Windows Registry Game controller Overhead (computing) Structural load Closed set Computer network Software Military operation Operator (mathematics) Quicksort Traffic reporting Identity management Surjective function
NP-hard Complex (psychology) Group action Public domain Replication (computing) Mereology Data model Emulator Different (Kate Ryan album) Machine learning Electric generator Trail File format Kolmogorov complexity Open source Mereology Instance (computer science) Formal language Statistical hypothesis testing Replication (computing) Emulator Software framework Data structure Point (geometry) Vacuum Slide rule Trail Open source Link (knot theory) Disintegration Virtual machine Wave packet Planning Operator (mathematics) Data structure Model theory Analytic set Usability Public domain Continuous function Group action System call Vector potential Shareware Personal digital assistant Logic Password Blog Recursive descent parser Vacuum
Cybersex Emulator Context awareness Link (knot theory) Emulator Repository (publishing) Cybersex Repository (publishing) Analytic set Analytic set
Link (knot theory) Emulator Link (knot theory) Cybersex Repository (publishing) Analytic set
so I'm gonna be talking about how we can use automated planning for kind of automated red teaming my name is Andy I am a researcher at mitre he is familiar with mitre see a lot of head nods and I like that I'm also working attack and caldera which is kind of a lot of where this this talk came from and the way I
like to kind of bring this up and talk about this problem is that as a defender it's really hard to assess what we're missing in our networks it's easy to Tom this like castle mentality where our networks are castles adversaries we're never gonna break in and well mouse isn't working they are gonna break in they're gonna find ways to get around your walls and as defenders if we're not really like assessing our own networks we're not gonna know where those holes might be and where we should actually be looking to improve our defenses so to try to solve this problem we can run offensive assessments and that's a catch-all term I'm gonna go into it and you know a little more specifics in a bit but the idea is to stress test your network by executing a real attack and seeing what actually happens you know did you attack did you detect the attack did you see how can you see how far they got did they make it you know into the walls that they go to the treasure or how what do they actually do and then try to understand how you can improve your detection and your prevention and so offensive testing is great you know penetration testing red teaming adversary emulation all these things are wonderful however they're hard they cost a lot of money to run they require a very significant time investment results are dependent on the capabilities of the personnel who are executing the attack if you run two different you know red team's at different different times a different personnel you're gonna get very different results exercises can also be hard to repeat this is a problem because you're gonna want to see how your environment has changed over time how your security has change what defenses have changed if you can't repeat your exercises you're gonna have a problem and then designing designing red team exercises is also if it's is a kind of like time consuming and haven't heavily invested process and well automation can solve a lot of these things if you can automate your offensive tests you will lower the cost to run an exercise you just need a tool to run it they're less time intensive you just push a button and it goes and it does its thing and now you're dependent on an attacker model as opposed to the actual personnel so things are more consistent and you can repeat that does at the push of a button and best of all designs could be can be saved then you just again hit a button run the same test you ran before and
indeed a lot of people in the community have realized that automation is good and this is kind of a snapshot of only a small slice of guys don't like these links a small slice of some of the kind of open-source projects out there that do automated offensive testing at various degrees I come from the mitre side that's caldera at the top caldera is open-source you have the link there but there's also atomic red team which aligns to attack there's meta I think it's by uber red team automate automation I think is the name by end game infection monkey if that wasn't that's my garden core and then on the right those are a lot of tools that kind of circle around bloodhound which does automated attack path generation kind of you know pluming active directory to figure out like what the trust relationships are then a couple of projects that can kind of build off that automation so anyway automation is
fantastic and I keep saying offensive testing but what do I actually mean by that I'm gonna kind of talk about three different categories the first is pen testing that's kind of what we talk about a lot you know penetration tests they're helpful to determine the state of a vulnerability you test methods of gaining access you look for weaknesses you know this is really it just really tends to be like exploit heavy and from a castle analogy this is you know really kind of you know going up at the walls of a network red teaming is a little more expansive now red team to actually simulate the goals of an adversary you know this is much more end-to-end you're gonna gain your network access you're gonna move about the network and you're gonna pursue an objective and here as opposed to just kind of targeting the walls now a lot more is in scope we have kind of the holes and all even the catapult off to the side which might be you know the red team's custom set of techniques that they like to use then over here this is adversary emulation and here this is like red teaming but you want to simulate the actions of a specific adversary we can call this kind of threat based red teaming and unlike red teaming this is a lot more constrained we're going to talk about one specific path and one specific threat actor and so I'm gonna talk mainly throughout this talk about adversary emulation I think adversarial relationships here and adversary emulation and red teaming those generalized to pen testing or kind of a they all generalize around argue maybe that some of the ideas for pentesting don't necessarily generalize to adversary emulation but that's a different argument so if we want to kind
of do successful automated adversary emulation we we need to kind of have a set of goals and the first is to make it real you don't want to just kind of run a notional like hypothetical address or emulation exercise you want actually use the same techniques the same tools and the methods and goals of a real attacker you also want to do this end to end you don't want to look for one offs you want to actually compose a text because that's what we only have a series do they do a they do B they do C they don't just do a and they stop they go throughout your network repeatability is important I mentioned at the beginning and then this last one is extensible we don't necessarily just want to come up with a model where everything is hard-coded it's going to do the same thing over and over again you want some variability you want to extend the TTP's that aren't that are in your tool you want to kind of grow as needed and if we really want to realize these goals we're gonna need some form of advanced decision-making to kind of chain things together and figure out what we should be doing so I'm gonna talk about can the
adversary model that I used throughout this talk it's the miter attack
framework out of curiosity how many people here are familiar with attack yes this is awesome a good amount of people so so attack is an awesome framework that kind of talks about what adversaries tend to do after they compromise networks it breaks things down into the adversaries tactical goals these are the things like you know persistence privilege escalation lateral movement exfiltration and then the techniques as to how the adversary achieves those goals this is more focused on kind of enumerated adversary behaviors as opposed to a specific like attack patterns or iOS C's if you're familiar with a pyramid of pain this is really targeting that top of the pain the TTP style you know enumeration of what an adversary is doing and in the attack framework we not just enumerate them as a nice little picture in a matrix we have a description of what the technique is as well as a list of examples of adversary usage of each technique as well as software that executes that technique and I'm gonna try not to spend too much time talking about attack apparently Orange is a bad color here some of the nice things about attack are that's grounded in real data from from cyber incidents all the TTP's we have an attack all the techniques are backed up by real publicly available threat intelligence and my favorites probably the third one attack decouples the problem of understanding what adversaries really do when they attack networks from the defense solution you want to deploy so we can he was attack in any way we want to this is just an enumeration you know from the adversaries perspective this gives us a lot of flexibility so just give an
example this is a slide I stole from a colleague where he just went through and he analyzed a threat report and he found kind of some of the the strings in a piece of malware and you'll note here that a lot of these things are they all map to attack and that's the enumeration on the left but a lot of these things are really just like you know normal windows things there is IP config at local group administrators that local group administrators domain netstat ping nbtstat is in there as well der all these things that are really kind of these normally benign things then adversaries used when they actually compromised network so I bring this up to say you know mainly when we talk about adversary emulation we don't talk about things from a pure exploit driven perspective this is more the end-to-end thing where we're talking about also like living off the land and that's gonna play an important role kind of later in the talk I'm gonna give it a
little bit and talk about automated planning I think I have to it's in the
title automated planning is kind of old school it's relatively simple to describe basically given the state of the world an end goal in a set of actions how do you compose those actions together to achieve your end goal and here's just an example that's totally contrived at waking up and I'm going to eat breakfast so I'm going to get my bowl get my milk and my cereal then you might breakfast I'm not gonna go get the leash and walk the dog if I want to go eat breakfast I tried that it doesn't
work as I said automated planning is kind of old school one of the biggest kind of solvers the most well known in the planning field is strips the Stanford Research Institute problem solver that's from 1971 so again old school it's a relatively straightforward representation you have basically a set of atoms and boolean variables a set of actions an initial situation and a goal situation and the challenge is to figure out you know what sequence of action starting from the the initial condition achieves the eat goal situation this is this is pretty straightforward and the thing about kind of the strip's formulation is that for each action we define the name the preconditions that must be true for us to execute that technique the atoms are the kind of the the post conditions that we'll be true after we execute the technique and then the atoms are things that will be false at when we after we execute the aurochs evening out the technique the action jumping ahead
here's a simple example that a lot of people talk about the planning community it's the blocks world the colors aren't showing well but the basic idea is you know we have a bunch of blocks on a table you know B is on a C is on spot number two is on spot number three we want to achieve this end state where a is on one B is on two CS m three relatively straightforward we have an action that moves one to the other we have some kind of logic description of you know how we can move blocks between different locations and a plan just kind of says okay B is gonna go and C is gonna go on one be an AC on three be on two relatively straightforward and you
know at its core you know really planning is really just a pathfinding problem over a graph and the states are nodes and the actions are the edges in it that this is a very like horrible oversimplification but this is a big part of what planning is it's really trying to figure out how you navigate that that graph and there's a lot of research on doing this more efficiently some kind of three buckets down here one of them is heuristics trying to analyze a node in this graph and figuring out okay should I actually keep exploring this branch landmarks trying to find those landmark actions that oh I know I need to execute this action at some point so I'm going to throw it into my plan and then helpful actions which kind of helped you along along the way and there's a ton more stuff here I have a reference down there that is probably illegible so classical planning is fun
but it doesn't handle uncertainty at all so the planning field is really exploded there's like lots and lots of research on planning this is just only a snapshot of that at one level you have conformant planning this is a little bit more in-depth than classical planning here your initial state is unknown instead your different kind of a set of possible initial States contingent planning is another kind of more advanced form of dealing with uncertainty now you have actions that can have non-deterministic outcomes this is like probabilistic planning even more confusing is partially observable Markov decision processes Pugh MVPs here actions yield observations which inform our belief about the resultant state and then we always kind of maintain a belief distribution of what state we think we're in and then the last one I'll put down there is offline versus replanting and continuous planning in part of the planning you kind of come up with an operator plan you figure it out and then just go execute an ABCD and E and you kind of keep going in many applications you find that you're not going to come up with an operator plan you know from the get-go you're actually going to execute a few few actions reevaluate execute some more
actions then reevaluate some more and then lose your slides Green's a good
color so yeah that's illegible but so planning and security has kind of a you know again an old-school approach really traces it back to kind of attack graphs you can't see the graph but the basic idea is we can use planning to figure out like how to change vulnerability together and move about your network it's relatively straightforward its older there's a lot of logical models a lot of research and you know kind of the late 90s and early mid-2000s which talk about how you can construct attack graphs how you can optimize them they tend to be kind of exploit heavy and and there there's a lot of utility but it's hard to see when you're just looking at
that more interesting approaches to using using planning and security are kind of automated pen testing using P um DPS there's a line of a line of papers that took about this approach which was pretty interesting and and the reason is that you know assuming an attacker has full network knowledge that's that's kind of unreasonable if I'm an attacker and I'm even if I'm a red teamer and I'm going at a network I'm not always gonna know exactly what the network map looks like or what you know vulnerabilities exist whose admin where you know the whole ad structure I don't really know that so instead you know they take this P MVP approach where you know it's relatively you know straightforward where they have sensing actions in action acting action is you know the sensing action is something like running an map and scanning for vulnerabilities acting actions you know launching an exploit relatively straightforward but this is resulted a much more robust solution that's actually able to work in the presence of uncertainty or attack graph approaches didn't weren't able to
shine and then probably can't read this either it's just a taxonomy from a researcher in the air in the planning community who's been working on using planning for kind of pen testing and he kind of bases it based on what the action model is so do you even have actions are you just kind of drawing a network around give monotonic connections where each action increases your knowledge base you know delete free or you have general actions which is a normal planning thing and then you can balance that versus uncertainty where you have a you know no uncertainty we have uncertainty interaction outcome so you have uncertainty and whether or not something like succeed knee of uncertainty regarding your state distribution and what state you're even in and depending on the specific model you're going for there's a few things in the literature where you might use you know just just a graph or classical planning or an MVP or pew MVP and so this is a great paper I'd highly recommend it if you're interested in this area it's called if you can't read it simulated penetration testing from Dijkstra deterring test your Hoffman is the author it's a great paper all right
so I'm gonna pivot back it's kind of talking less about kind of you know planning and more about an adversary
emulation and so here's an example to motivate it let's say you've got this host 1 and you have a foothold there and you've seen this host 2 and you want to copy a file from host 1 over to host 2 you might say ok what do I need to do to copy a wrap file over you know working rat on the source host on host 1 I need the mounted file sure access from the target onto the source and then I need write access to that file share well these look a lot like preconditions and then you can say oh what happens after copying a rat file over well there will be a new new file on the target host that file will contain the wrap and those are the consequences or post conditions and what you can do is you can say okay if I want to make a plan to copy a file I'm gonna I need to get that file in the target well to run that copy final action I need a mountain sure okay I'm gonna run the mount share command I'm gonna get the mounted sure but I need credentials okay I wanna dump credentials get those credentials and as long as I start with an elevated for at I can go about and do this and this looks a lot like planning so this oh you get a fun graphic - um so yeah this looks a lot like planning and just to
give you a few examples you know suppose you want to go from host 1 a host to an excellent rate data from host to one potential plans I'm going to dump credentials I'm gonna mount the share copy of file remotely execute that file in the next let's create the data another plan might be exploiting a vulnerability and the next we're trading the data then another plan might be dumping credentials using RDP and then Excel trading the data so so relatively simple examples but again you can see how we Cheney's together one of the questions we might want to ask is you know suppose we're doing this chaining we're coming up with these plans well how do I select the right plan and if you have an explicit goal you've enumerated it beforehand that's straightforward you kind of enumerated all the plans and go towards a goal but you're trying to do an address or emulation you're trying to kind of show these repeat behaviors you know what's the right plan to execute or even if you haven't have a goal and you have multiple plans how do you choose the right one and something that that we're we've done or the line of resource set that we've taken about is to basically just kind of assign each action or assign each plan of score based on kind of a heuristic function we have basically just a decreased weighted sum over each action it's it's very straightforward but the idea is you know if we assign each action a reward function or a individual reward and then sum up the rewards kind of decreasing as it you know executes further in the plan then we're gonna execute that plan it's straightforward and here I'm not going through the example in detail but you know if you follow what's on on the lower right-hand corner then then plan number three is the best plan so that
seems really easy and it's not that easy
there's a lot of uncertainty that comes about when you're doing adversary emulation and so let's consider kind of you know to kind of walk through that let's consider two plans the first is you know that dumping credentials plan and the second is exploiting a vulnerability we consider the
uncertainty for exploiting a vulnerability it's relatively straightforward first is a target susceptible to the exploit that's a binary that's yes or no and that's someone you can scan beforehand and then the second one as well was the exploit technique executed successfully it's also a binary yes or no and you could scan for that after the fact it's relatively straightforward how you do that and and this is straightforward but it still leads to very interesting scenarios for planners but it's kind of straightforward from our perspective now consider dumping credentials this is really easy to describe when I dump credentials I need a rat with elevated access that's straightforward after dumping credentials I'm gonna get credentials for all accounts that have active sessions on the host I just dump credentials on again that's also straightforward however in practice while the description is easy what we're gonna get back our consequences here the post conditions for this action are kind of complex realistically running credentials might fail entirely that's one way it can work I might not get any credentials okay that's that's certainly possible I might get credentials for accounts that I've never even heard of so I can't do any reasoning over them I've never heard of them I'm I think financial is that for accounts that I've heard of but I just can't use and then if I'm lucky I'm gonna get credentials that I can use and the problem here is that while these these scenarios are somewhat innumerable to an extent they lead to non-deterministic outcomes that explode in difficulty as we try to kind of chain forward and plan for the future and now
this is totally illegible so a lot of the stuff in the literature the planning problems that that we see they tend to have bounded uncertainty where you know at kind of the biggest oversimplification possible they are kind of explore you know heuristic lee an optin and they optimize it but they explore all possible states for each unknown and this is a snapshot of a paper where they use partially observable contingent planning as an alternative to solving a you know fully described PMD P and what they do is that in their problem description they actually enumerate they say hey you know host 0 that can either be win NT you know the server edition or win NT the Enterprise Edition and if i'ma numerating that beforehand then I can use all these techniques if I don't have that enumeration beforehand it makes it a lot more difficult so you know in our case a lot of the actions we deal with kind of you know using attack as this threat model trying to do full adversary emulation these actions have unbounded uncertainty it's really not possible to plan over all of the states we just we just can't de New Marais that and instead you know we've kind of driven us to this conclusion that you know continuous automated offensive test and you my adversary emulation it's not just planning there's also a big acting component so some of the things one of
the things we did is came up with kind of a planning algorithm it's relatively straightforward we want to update the world state and then after we do that we figure out what valid actions there are that I can execute right now that's just looking at my preconditions he's straightforward I'm gonna construct all plants that lead off with that with with those actions chain the actions together by leveraging the model run the heuristic over those plans once I have that set I can just use a heuristic execute the first action in the best plan and then repeat relatively straightforward but the problem here is really well how do i construct plans if I can't enumerate all the outcomes on the ax that I have in my model and so we've
taken a few approaches one of them that we did kind of on a more research perspective we just kind of ran this in a little simulation environment we basically kind of just guessed what the world looked like and we use deterministic techniques to identify plans very straightforward it worked well in the simulation plate simulation space but even there it was computationally sluggish and very heuristic I mean we have a paper there two intelligent automated Red Team emulation that walks through some of the stuff we did there we extended that by using kind of a small world extension idea this is I call it the light planner you basically basically would have guessed small extensions about the unknowns we have in our environment so if I see a host and I've never probed the admins on that host I might guess okay this accounts an admin and this account isn't an admin and then we can use a rule-based approach to kind of you know tune what the probabilities are that we're gonna guess like I'm gonna guess there was an admin there that I have creds for or an atom in there that I don't have creds for and this works well in simulation but again it's hard to implement in practice and it's hard to come up with a good like rule set that you actually want what to want to leverage in a real environment well we ended up doing is I'm calling optimistic best best guest this is basically just guessing this is a simple approach where for each action we have in our model we're gonna encode specific hints that talk about like what the OP what the kind of best-case scenario is so if I'm dumping credentials my hint is gonna be I'm gonna get credentials for an account that's an admin somewhere and so this this works reasonably well in practice there's a lot of room for improvement here it's something we're working on I want to invite everybody to work on calderas open source but it works well enough in practice so I'm gonna talk a
little bit about caldera it's written in Python 3 the rat is in c-sharp we use MongoDB have all sorts of fun stuff but there's three main components we have an admin web UI that that you can use to kind of control operations control what what it's doing we have this server that kind of controls everything you know the human interfaces with the HTTP server and then the server has a database of all the things that are true an execution engine that talks about how you actually do things and then the planner which leverages the attacker model in the world state to forward trying to kind of use the the algorithms I've been talking about each host that you're testing in the environment you need to have a kind of shim agent this just facilitates communications between the HTTP server and the agent this guy just facilitates that communication though there's also a rack component that actually does kind of move around the network so there is a real rat it is really executing stuff the agent is there it kind of just kind of delineate which hosts are in scope or out of scope conducting in this
assessment is very straightforward and easy this is one of the nice things about using planning you load the calderas shim on to the network host you create an adversary you identify which which hosts are actually in scope for your assessment and then you just launch the operation it's really nice because you it's very kind of low overhead for an operator or someone wanting to run an exercise during the operation caldera will record its activities any artifacts that creates you know anything it does and then it'll automatically stop if it can't do anything further after the operation we get a report of everything that it did you can kind of go in in detail and kind of scope scope it out and then and then it'll automatically have reset the infected host removing any artifacts it created and you can kind of control that and so this will stop it from dropping rats all over your network and modifying registry keys and doing all sorts of other stuff and I am
NOT going to try to run the demo because that this is not working well so I'm just going to jump ahead to some closing thoughts using it automated adversary
emulation is fun there's a lot of cool things you can do with it testing analytics and sensors you know that can help you see ok does it do my defenses actually work I'm gonna go you know I think I'm going to detect credential dumping I'm gonna go throw an analytic and into my Splunk instance and run caldera and I can see okay did that analytic fire correctly or not data generation is also is also good I don't know if we can produce at this point enough very data to like train models off of but that's one of the use cases we want to do is to be able to kind of generate generate data to kind of train people and train anything losing the number-3 red and blue team training you know you can tell your red teamers hey here's caldera go learn it's TTP is go follow what it's doing it can be like a first nice introduction of some to some potential attack paths you can also teacher your blue team you know hey here's what an adversary did continuous testing and straight forward and then see structure and replications another big big use kiss some notes for the future automation is a hard problem we've encoded some straightforward techniques but we're getting to the point where I think we have like 30 or 40 different attack techniques encoded in caldera and managing the complexity if you want to use each one of those techniques that gets kind of hard because you have all these different atomic things that you can do and chaining them together is more and more difficult then some techniques are harder to execute RDP is a common one for red teamers but finding like a way to automate RDP sessions is not particularly straightforward because we don't have good api's for that key logging is another one where now we have to do like asynchronous operations because we're just waiting for someone to enter in a password right now a lot of this stuff we do is just we can do it just go execute it and go forth but we want to kind of get to the point where we can do with that kind of you know interactive operation a big question is how we can do this more intelligently we're kind of working on that stuff on our own but there's a big machine learning story here again calderas open-source you know optimistic best guess is fun to say but that's just kind of guessing there's a lot of cool things that you could potentially do a lot of things we've looked into a lot of things we've heard you know there's a huge and awesome story here and then the last point I want to make is that we aren't automating offense in a vacuum you know their caldera we have a rat and Symantec now picks it up which is yeah we're not automating offensive action yeah if you
want to be involved with caldera or anything we're doing there's a few things you know we have the github we accept call requests anything you want to do my favorite and I pulled some of these slides up sizing to do the demo is that you can actually create your own techniques and put them into attack or into caldera and there was actually some some guys who they wrote a blog post of how they wrote their own techniques and they put in the logic they put in the execution and they threw it into the into caldera and it was able to string it together as part of a larger attack path so that was really cool but anyway you can get involved there we have a slack Channel please reach out to us if you're interested you know we've got miter people everywhere and we have military were too and then on the AI side if anyone's interested in automated planning we've just released a or we were involved in the you know the International planning competition so we submitted a caldera domain that you know kind of got rid of some of the uncertainty in in a way and that for the deterministic track but it standardizes some of the actions and kind of the the pddl format which is what a lot of automated planners do lots of links lots
of context if anyone's interested I'm Andy I mentioned attack one thing I'll mention is we're running at at calm we're running a conference dedicated specifically to attack the CFP is open right now it closes on August 15th submit stuff it's it'll be fun during 10 minutes Oxford M&S talks anything anything involving attack please let us know we have something called the cyber analytic repository that has some analytics adversary emulation plans is a new one that we've recently released this is kind of an in-depth look at kind of how you can go about emulating an
adversary and it's basically done
anyways yeah I'll leave the links up here