Finding Xori: Malware Analysis Triage with Automated Disassembly

Video thumbnail (Frame 0) Video thumbnail (Frame 481) Video thumbnail (Frame 2002) Video thumbnail (Frame 3686) Video thumbnail (Frame 4643) Video thumbnail (Frame 14805) Video thumbnail (Frame 16741) Video thumbnail (Frame 17443) Video thumbnail (Frame 18356) Video thumbnail (Frame 20849) Video thumbnail (Frame 22132) Video thumbnail (Frame 25992) Video thumbnail (Frame 27106) Video thumbnail (Frame 27817) Video thumbnail (Frame 28422) Video thumbnail (Frame 30216)
Video in TIB AV-Portal: Finding Xori: Malware Analysis Triage with Automated Disassembly

Formal Metadata

Title
Finding Xori: Malware Analysis Triage with Automated Disassembly
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English

Content Metadata

Subject Area
Abstract
In a world of high volume malware and limited researchers we need a dramatic improvement in our ability to process and analyze new and old malware at scale. Unfortunately what is currently available to the community is incredibly cost prohibitive or does not rise to the challenge. As malware authors and distributors share code and prepackaged tool kits, the corporate sponsored research community is dominated by solutions aimed at profit as opposed to augmenting capabilities available to the broader community. With that in mind, we are introducing our library for malware disassembly called Xori as an open source project. Xori is focused on helping reverse engineers analyze binaries, optimizing for time and effort spent per sample. Xori is an automation-ready disassembly and static analysis library that consumes shellcode or PE binaries and provides triage analysis data. This Rust library emulates the stack, register states, and reference tables to identify suspicious functionality for manual analysis. Xori extracts structured data from binaries to use in machine learning and data science pipelines. We will go over the pain-points of conventional open source disassemblers that Xori solves, examples of identifying suspicious functionality, and some of the interesting things we've done with the library. We invite everyone in the community to use it, help contribute and make it an increasingly valuable tool for researchers alike.
Malware Mathematical analysis Software bug
Ocean current Slide rule Execution unit Distribution (mathematics) Touchscreen Assembly language Demo (music) Code Projective plane Binary code Sampling (statistics) Bit Line (geometry) Funktionalanalysis Web browser Twitter Malware Hacker (term) Computing platform Window Disassembler
Boss Corporation Presentation of a group Sampling (statistics) Automation Disassembler Software bug
Email Dynamical system Thread (computing) State of matter Code Multiplication sign WebDAV Sheaf (mathematics) Combinational logic Function (mathematics) Stack (abstract data type) Disk read-and-write head Software bug Emulator Medical imaging Mathematics Semiconductor memory Different (Kate Ryan album) Befehlsprozessor Core dump Encryption Electronic visual display Automation Flag Software framework Endliche Modelltheorie Sample (statistics) Stability theory Graphics tablet Programming language Email Touchscreen Cross-platform File format Block (periodic table) Structural load Software developer Binary code Bit Funktionalanalysis Opcode Parsing Type theory Process (computing) Repository (publishing) Right angle Stability theory Spacetime Asynchronous Transfer Mode Booting Point (geometry) Asynchronous Transfer Mode Server (computing) Existence Service (economics) Open source Virtual machine Mathematical analysis Wave packet Revision control Programmschleife Read-only memory String (computer science) Booting Loop (music) Plug-in (computing) Computer architecture Information Projective plane State of matter Mathematical analysis Memory management Density of states System call Compiler Performance appraisal Integrated development environment Logic Personal digital assistant Data acquisition Object (grammar) Window Disassembler Library (computing)
Trail Medical imaging Uniform resource locator Personal identification number Computer file Semiconductor memory Structural load Sheaf (mathematics) Configuration space Address space Window Speicheradresse
Email System call Structural load Ferry Corsten Code Function (mathematics) Exponential function Host Identity Protocol Revision control Emulator Semiconductor memory Different (Kate Ryan album) String (computer science) Gastropod shell Aerodynamics Conditional-access module Error message Address space Time zone Email Structural load Electronic mailing list Memory management Word Process (computing) Pointer (computer programming) Integrated development environment Vector space Order (biology) Right angle Window Library (computing) Address space
Greatest element Code Direction (geometry) Multiplication sign Set (mathematics) Mereology Electronic signature Emulator Different (Kate Ryan album) Pattern language Cuboid Chi-squared distribution File format Binary code Sampling (statistics) Parallel port Infinity Bit Funktionalanalysis Parsing Electronic signature Befehlsprozessor Hexagon Sample (statistics) Hash function Order (biology) Configuration space Hill differential equation Right angle Reading (process) Reverse engineering Asynchronous Transfer Mode Artificial neural network Control flow Online help Branch (computer science) Graph coloring 2 (number) Goodness of fit Queue (abstract data type) Right angle Graph (mathematics) Matching (graph theory) Information Mathematical analysis System call Loop (music) Integrated development environment Personal digital assistant Function (mathematics) Cube Video game
Dialect Building Mathematical analysis Funktionalanalysis Function (mathematics) Mereology Computer programming Inclusion map String (computer science) Data mining Convex hull Hill differential equation Resultant Disassembler
Execution unit Computer file Parity (mathematics) Software developer Parameter (computer programming) Function (mathematics) Funktionalanalysis Mereology Regular graph System call Radical (chemistry) Enumerated type Personal digital assistant String (computer science)
Graph (mathematics) Demo (music) View (database) Multiplication sign Sampling (statistics) Mathematical analysis Funktionalanalysis Mereology Binary file Computer programming Moment of inertia Different (Kate Ryan album) String (computer science) Normal (geometry) Automation Information Musical ensemble Hydraulic jump Library (computing) Disassembler
Code Game theory
please join me in welcoming Amanda Russo and Lich Seymour hello everyone I'm here to talk to you about finding sorry it's a malware analysis triage with automated disassembly a little bit about
ourselves my name is Amanda Russo also known as Maori nakorn on Twitter I'm currently I'm our researcher at endgame my day-to-day activities is looking at malware everyday writing detections for them teaching my intern something and then hey what's up my name is Richard Seymour also known by my hacker handle are Seymour on Twitter I'm a data scientist at endgame and I got chance to work with Amanda on this project I'm really psyched about it we started in January and we're about six months later it's about 36,000 lines of code and we're gonna tell you about it all right thanks so a little bit before we begin I kind of want to go over a quick overview of current assemblers of today talk about what they do for PE right now the functionality and features and I wanted to go over a usage and demo but first I actually want to show you the tool before we get into it so you know what the hell I'm talking about awesome
so that loaded up pretty quick so if you don't know what a disassembler is already it's kind of taking the binary code and turning into assembly for you so you can analyze this the Mauer if you're looking at malware so I'm just gonna go ahead and drop this wanna cry sample in here really quickly and it will disassemble it really quickly and put it to the screen which is browser base so it's cross platform for you to run on any Linux OS X or Windows distribution and you can play around with it you can see here's the short circuit saying that our tech registered and as well as linear disassembly let me make this a little bit bigger for you so it's all there for you to play with in the U I know crazy buttons is really easy to mess with it has a click-and-drag we made this in two weeks for you we know we use it for automated disassembly but we wanted to have a tool that you could play with in the browser on any platform and get used to looking at disassembly so I'm gonna switch back to my slides and continue
presentation here so now that you've
seen what the hell it is now we can talk about what it does and how it compares to other disassemblers so what is the
problem my boss told me look at a thousand samples in three days and I'm like there's no way I can look at a thousand samples quick enough and triage it and sit there and copy one sample into a VM opening up an Ida and analyze it so what did we do we had to think of a better way to scale our automation so we wanted to basically be able to take
our pipeline right now which handles training our models and also add to it disassembly which is something we've wanted to do endgame since before I was an endgame so it it allows us now to basically take our sample set and turned it into disassembly features that we can read because it throws out right now JSON but you could basically serialize it to any format you can integrate into your pipeline so well we thought about doing automation we wanted to try some disassemblers that were already out there open source if it's a library and try to compare them across what we needed so we were thinking about how much how big the size of the code was how big it was in memory if it was stable how much the price was it was a cross-platform was it easy to use as output accurate and can we integrate it into our services already so I kind of have this little chart that it you know we have good ok and not so good comparing capstone Rodari to Ida Pro hopper and binary ninja which are pretty much the common you know disassembly libraries out there so requirements we needed fast development we needed stability and resilience because if we're running through budget services we'll need it to spin up again it had to be cross-platform because we're not just using in a Windows or is it Noah sex and we're also running it on a server output can be easily integrated into our services which you know JSON is really useful for that it had to be really easy to use and had to have a core feature set and output how to be accurate alright so the first step we did was diving into the code of all of these disassemblers and trying to pick and choose like what are the pros and cons of each and kind of build it into something that we needed in our own internal framework so we adopted a different different aspects of what capstone does since it's based on the LLVM and gdb repositories also a little bit of emulation that you know Khemu does as well as understand how to put that logic using rust and then fix some bugs along the way so evaluating example say you have an x86 32-bit instruction here so typically the 66 represents the opt size of the instruction which tells you is the operand bigger or larger than it should be and then you have the opcode which tells you you know which instruction actually is so how many of you are familiar with the opcode 90 yeah so it's not right nom-nom-nom so really this is actually exchanged ax X in object dump and LLVM because of that OP size is there so I did it correctly capstone doesn't do it correctly and this storm doesn't do it correctly so some of these like minor changes between different disassemblers is going to give you a different output when they disassemble the instruction but exchange axis actually is a knop but it technically is not based off of the CP right so here's where I asked who here knows what rest is okay and who here is programmed in rust okay that looks like mm 12 people maybe that sounds great I'm glad everyone's gonna be able to contribute to this project I think the thing that we want to emphasize besides the bullet points on the screen is that I didn't know rust really I mean I'd been tracking its existence but that's a big difference from like diving into writing a disassembler in a few months so Russ gives you all these nice things like it protects you from stomping on the OS from your stock it handles memory much more cleanly than any programming language I've used it's really fast and the oddest thing about it is that it's actually it was really fast for us to develop in because all of the really dumb bugs that bog you down for a week and then it's another week and then it's the next month and you're like why can't I get this to work basically get caught either by the compiler not letting you do it or well yeah by the compiler basically so it's really helpful and I suggest you learn it right so current features yes it's open source I was really like hell-bent on making open source you can contribute now we already had like how many contributions last night and it's only been live for two days so it's really bumpin and grindin yeah currently it supports x86 and 64-bit architecture I only had time for that since my job requires PE executables it disability displays strings from reference memory if you're familiar with the flareon plugins they have worried if you're pushing strings to the stack and it emulates it and produces a string for you that's already built in it manages memory it outputs JSON for your service consumption it has two modes light emulation which means it does handle register stack in some instructions as well as full emulation which I don't recommend is really slow right now but you can do it if you want to get through a packer or some type of encryption routine I also was able to simulate the thread environment block as well as the peb are the process environment block so that you can do dynamic API call loading as well as evaluate functions from DLL exports so you can populate those in the code itself all right so the design so we have a PE loader similar to what Windows does for people owning a PE it's very basic loads to temp m as well as dll's from the image the PE image header and also a memory manager that manages all those memory spaces so that in case the Maur is trying to access an address outside of those memory spaces you it won't allow you to as well as the analysis struct which contains all the functions of disassembly and all the imports and then finally when you're analyzing through all of the different sections of code it has a state which tracks to CPU registers and all the flags a new version of the stack as well as tracking loops which I'll get into later so if
you're wondering how a chump like me contributed to this project I decided to play not invented here syndrome on a PE parser I use this great library called Nam which I think originated in France it's a parser Combinator framework and allowed us to parse the header PD headers rather safely if you've ever messed who here has ever looked at a PE header you know MZ was a real person and the thing about headers like if you've looked at core commies excellent work I'm like describing what this head stuff is to people there's a lot that's useful to the machine to run like getting the imports correct and the tab and the pad and entry point all that there's also a lot of stuff there's just kind of like retro garbage from the 8-bit das days 16-bit windows days the 32-bit when those days and the 64-bit windows days so the cool thing about this parser is it basically takes everything so we can data science the heck out of it later if we want to figure out that that corrupted da stub means that it's some thread actor we've never heard of then we can do that now so I'm pretty excited about it and yeah that's my little thing so the information that we take for the
Pieta is used to build the impe image in memory so that means building all the sections making sure they're read/write executable and tracking that as well building the stack building the DLL into memory and the right locations you can actually configure that we have a little JSON config file that you can set whatever address offset you want for the dll's or the Potemkin peb as well as building the tab and PETMAN memory so
what I meant by loading the DLL exports we have this giant JSON of all the DLL exported from like Windows 7 and it pulls all of the API calls there so you don't have to keep using the DLL you can just use this giant JSON and it'll load it up and rebuild all of the addresses as it would in the PE image and fill all those addresses for you so if it's doing any dynamic API calling and it moves an address into a register or an address into a new memory location that address will actually be there simulated alright
so like I said dealing with dynamic API calls you need the tab and PEM so we kind of recreate that in the window for the windows 7 environment it handles memory so anytime an access is one of those it can you can choose to allow it or not allow it as well as managing all of those addresses on the stack which is nice and let me explain what I mean by this so we have the shellcode doing some interesting stuff in the header imports the only API calls that's actually using is exit process get last error get local time and get module handle a which we don't actually see load library a which will protect or shell execute a right so here you can see it's taking this string called the library a and pushing it onto the stack from there it dysfunction right below here it's going to call this function and get that address off of the pen so that means it traverses all of the dll's as well as the names from the PE headers in order to collect that address offset once it's done it's going to put the address into e^x and then move into four zero one zero zero for that pointer and then down here it's actually going to call that newly moved address to load shell 30 shell 32 DLL so it can also load shell execute a afterwards so in reality the dynamic imports are low library a virtual protect and shell execute a so this is actually the output from zouri and you'll be able to see all of that happen in there with the light emulation all right so the Tevon peb I
didn't do anything really crazy there's so many resources out there to look at the whole struct for the tab in PEM some versions of Windows are gonna be different but I just used you know whatever was available out there you don't need the whole like hidden features of the struct at all here's just like the top of it for example so I recreate the nice thing about breasts is that you can take a struct and serialize that struct into a binary array or a binary vector and you can have that vector being accessed by the memory manager as if the assembly instructions word access those addresses as if they was actually on the OS which is really nice and so I had to figure out how to do a linked list in bytes serialized bytes and put that all together into a structure so that the shell code could use it alright and so another thing to
help with the analysis is that we have this thing called analysis queue where it will take to the if a branch instruction if like if else and there's a call you'll have two different directions in order to take for the call so the right hand side of the branch will go to the top of the cube well the other part of the branch will goes to the bottom of the queue which will help the color analysis follow the actual control flow for you so it follows both directions and I also introduced looping for the emulation so say you run into an infinite loop and you know it's an infinite loop but you in a short-circuit that loop case there is a configuration there for you to short-circuit all over the looping in there say you only want to do 4,000 times you can set that information I don't recommend using the emulation mode but if you want to you can play with it I really need to improve it a little bit when you read a couple of research papers and then so signature analysis is a way of adding annotations to your code so that you don't have to be able to read hex like commanda you can just get a nice signature to tell you what the function was I spent the last week and a half of my life trying to reverse a cig parser from from IDE up format so we could use it I failed catastrophically in that those formats are always changing it's very hard to keep up with it that said we do have signature matching in zouri right now but we're always looking for for help to make it better and luckily some folks are already stepping up some yeah oh no fines me do oh my goodness we're prepared so this is the cool part for me as the data scientist not a reverser we took a hundred ember samples which is a data set we our company released of hashes we took the actual binary x' ran them on this old dell box that was under my desk the only new thing new about it was SSD and it could get through all these samples in in 20 minutes so that was a thousand samples twenty minutes about 1.25 seconds the sample running in parallel you know maxing out all the CPUs and it just dumps a huge amount of data and sometimes too much data if the binary is really big um and then you can do whatever you want you can grep through it you can find function names you can find imports you can do graph analysis over in AI villagers a bunch of people who think they can do a lot of cool things so I'm really psyched about this
example so it's really easy to build
zari all you do is make sure you install rust cargo build release of target really sorry - f want to cry a THC and it'll produce some
results so if you just want if you don't want to use an ounce that's part of zouri you can just pop in the disassembler part of it and throw it like this nice chunk of bytes and whole disassemble this HelloWorld program but isn't really showing you any strings it just gives you the disassembly output so if you want to use the analysis part it will actually populate all the strings when it's like popped off the stack and show you when a function ends and annotates all of those nice lovely comments for you when comparing zouri
with Ida Pro well we have the wanna cry ransomware file you could see we don't have all of the nice argument enumerations yet but you know six months as what we could do we could also see the strip the short circuit string like I showed you in the beginning this is just the normal output from the terminal but you know we have the UI for you to play with so you don't really have to use this part of it but for development maybe if you want to use it from the terminal you can
here's your diary to comparing the two different functions here it's kind of weird because Rodari does this thing where if it identifies API call within like the first couple instructions of a function that's what it names a function in this case we don't try to do that it's just a regular function call but it's got parity between the two and then
I showed you the demo earlier but that I could show it to you again with a different sample I'll just do a normal bin file so it does PE
and binary so this is the same hello world program it's really fast you just want to see the linear disassembly you can see it populates hello world and you can see the function in the graph view as well or if you want to do something more fun with a bunch of jumps oh actually run one can see how fast it is this one was run pertick it's actually part of my re 102 course a lot of the longest loading time is actually the libraries there but you can see this was a fun function and it populates everything there for you I don't have any string analysis since this was built for automation but hopefully soon it will have all that in there for you so I made it look really cool and spent the whole weekend on that so hope you guys like it [Music]
if you want to go download the code it's already on github.com and gaming / sorry people are already playing with it there for you to play with thank you [Applause]
Feedback