Web application analysis with Owasp Hatkit

Video thumbnail (Frame 0) Video thumbnail (Frame 5145) Video thumbnail (Frame 6490) Video thumbnail (Frame 8856) Video thumbnail (Frame 10102) Video thumbnail (Frame 11727) Video thumbnail (Frame 13362) Video thumbnail (Frame 14797) Video thumbnail (Frame 15871) Video thumbnail (Frame 16852) Video thumbnail (Frame 18110) Video thumbnail (Frame 19955) Video thumbnail (Frame 20841) Video thumbnail (Frame 22146) Video thumbnail (Frame 25633) Video thumbnail (Frame 29395) Video thumbnail (Frame 31048) Video thumbnail (Frame 32758) Video thumbnail (Frame 34940) Video thumbnail (Frame 36540) Video thumbnail (Frame 38534) Video thumbnail (Frame 39552) Video thumbnail (Frame 41026) Video thumbnail (Frame 43187) Video thumbnail (Frame 44692) Video thumbnail (Frame 45996) Video thumbnail (Frame 46914) Video thumbnail (Frame 47846) Video thumbnail (Frame 49039) Video thumbnail (Frame 50000) Video thumbnail (Frame 51620) Video thumbnail (Frame 55054) Video thumbnail (Frame 56273) Video thumbnail (Frame 57563) Video thumbnail (Frame 58490) Video thumbnail (Frame 60065) Video thumbnail (Frame 61427) Video thumbnail (Frame 66979) Video thumbnail (Frame 68422)
Video in TIB AV-Portal: Web application analysis with Owasp Hatkit

Formal Metadata

Web application analysis with Owasp Hatkit
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Martin Holst Swende, Patrik Karlsson - Web Application Analysis With Owasp Hatkit https://www.defcon.org/images/defcon-19/dc-19-presentations/Swende-Karlsson/DEFCON-19-Swende-Karlsson-Owasp-Hatkit.pdf The presentation will take a deep dive into two newly released Owasp tools; the Owasp Hatkit Proxy and the Owasp Hatkit Datafiddler. The name Hatkit is an acronym (of sorts) for Http Analysis Toolkit and are tools mainly for people who analyse (hack!) web applications. The tools make extensive use of MongoDB, in particular the advanced querying facilities in available in this database. Prior knowledge of Javascript and Python is an advantage, but absolutely no requirement. Martin Holst Swende is a Senior Security Consultant at 2Secure AB, working with application security- and penetration testing. Martin has a background as a Java/J2me-programmer but nowadays finds more joy in Python and Javascript. Martin is project leader for the Owasp Hatkit Proxy/Datafiddler projects and contributor to open source security projects such as Webscarab and Nmap. Twitter: @mhswende Patrik Karlsson is a Senior Security Expert with over ten years of experience in the field of IT- and Information Security. Patrik's area of expertise includes security penetration testing, vulnerability assessments, software application security audits and incident investigations. Patrik is an active Nmap contributer and runs the security related web site www.cqure.net. Twitter: @nevdull77

Related Material

Video is accompanying material for the following resource
Server (computing) Musical ensemble Proxy server Euclidean vector Sequel Open source Software developer Java applet Multiplication sign Port scanner Mathematical analysis Information technology consulting Number Web 2.0 Programmer (hardware) Different (Kate Ryan album) Software Software testing Software framework Traffic reporting Information security Proxy server Task (computing) Vulnerability (computing) Software development kit Injektivität Scripting language Area Presentation of a group Email Validity (statistics) Projective plane Moment (mathematics) Transport Layer Security Bit Cartesian coordinate system Twitter Cross-site scripting Web application Data management Telecommunication Order (biology) Configuration space output Speech synthesis Software testing Quicksort Information security
Presentation of a group Proxy server Sequel 1 (number) Database Streaming media Content (media) Mereology Graphical user interface Read-only memory Different (Kate Ryan album) Internetworking Software testing Interrupt <Informatik> Process (computing) Fuzzy logic Proxy server Vulnerability (computing) Physical system Injektivität Socket-Schnittstelle Online help View (database) Data storage device Bit Funktionalanalysis Regulärer Ausdruck <Textverarbeitung> Inclusion map Graphical user interface Process (computing) Fluid statics Personal digital assistant Telecommunication Quicksort Information security Table (information) Routing
Slide rule Asynchronous Transfer Mode Server (computing) Proxy server View (database) Tape drive Set (mathematics) Real-time operating system Mereology Writing Query language Telnet Row (database) Reverse engineering Proxy server Beta function Projective plane Database Bit Chaining Process (computing) Telecommunication Intercept theorem Communications protocol Electric generator Row (database) Asynchronous Transfer Mode
Windows Registry Server (computing) Scripting language Sequel Java applet State of matter INTEGRAL Code Demo (music) Instance (computer science) Client (computing) Mereology Coprocessor Latent heat Befehlsprozessor Gastropod shell Information Scripting language Touchscreen Key (cryptography) Demo (music) Projective plane Database Coprocessor Cartesian coordinate system Windows Registry Pseudopotenzial Password Order (biology) Interpreter (computing) Text editor Intercept theorem Table (information)
User interface Execution unit Server (computing) Sequel Interface (computing) Address space
Scripting language Point (geometry) Authentication Server (computing) Sequel Connected space Type theory Process (computing) Different (Kate Ryan album) Password Gastropod shell Encryption Proxy server
Scripting language Enterprise architecture Password Encryption Gastropod shell Video game console Client (computing) Cartesian coordinate system Login Connected space Product (business)
Scripting language Point (geometry) Aliasing Authentication Execution unit Server (computing) Sequel Database Cartesian coordinate system Login Coprocessor Query language Password Convex hull Selectivity (electronic) Proxy server Form (programming)
Demo (music) Query language Password Gastropod shell Cartesian coordinate system Coprocessor Proxy server
Scripting language Presentation of a group Password Demo (music) Moment (mathematics) Pattern language Cartesian coordinate system Coprocessor
Filter <Stochastik> Digital filter Proxy server Sequel View (database) Connectivity (graph theory) 1 (number) Mathematical analysis Mereology Disk read-and-write head Web 2.0 Cache (computing) Different (Kate Ryan album) Computing platform Pattern language Software framework Information Proxy server Plug-in (computing) Alpha (investment) Programming language Beta function Demo (music) Information Software developer Projective plane Data storage device Bit Database Funktionalanalysis Cartesian coordinate system Type theory Function (mathematics) Order (biology) Software framework Software testing Pattern language Computing platform Electronic visual display Table (information) Window
Email Dataflow Server (computing) Group action Table (information) Dependent and independent variables Multiplication sign Demo (music) Password Database Mathematical analysis Data storage device Parameter (computer programming) Web browser Content (media) Mereology Field (computer science) Object (grammar) Core dump Selectivity (electronic) Data structure Extension (kinesiology) Proxy server HTTP cookie Exception handling Email Information Demo (music) View (database) Cross-platform Server (computing) Interactive television Code Parameter (computer programming) Bit Database Binary file Cartesian coordinate system Web browser Cross-site scripting Vector potential HTTP cookie Object (grammar)
Identifiability Oval Structural load Set (mathematics) Database Bit Object (grammar)
Dependent and independent variables Email Touchscreen Key (cryptography) Real number View (database) Letterpress printing Set (mathematics) Database Variable (mathematics) Data dictionary Graph coloring Unicode Attribute grammar Application service provider Type theory Hash function String (computer science) Representation (politics) Object (grammar) HTTP cookie Table (information)
Code String (computer science) Right angle Funktionalanalysis Object (grammar) Parameter (computer programming) Table (information) Variable (mathematics) System call
Type theory Web application Execution unit Empennage Dependent and independent variables Uniform resource locator View (database) Bit Funktionalanalysis Object (grammar) Cartesian coordinate system Summierbarkeit
Type theory Dependent and independent variables View (database) Cartesian coordinate system
Filter <Stochastik> Dataflow Dependent and independent variables View (database) Expression Electronic mailing list Bit Funktionalanalysis Cartesian coordinate system Twitter Attribute grammar Element (mathematics) Type theory Query language Order (biology) Object (grammar) Typprüfung
Email Dependent and independent variables Java applet Mountain pass Codierung <Programmierung> View (database) Demo (music) Database Field (computer science) Differenz <Mathematik> Computing platform Uniqueness quantification Website Summierbarkeit Scripting language Default (computer science) Dependent and independent variables Key (cryptography) File format Content (media) Parameter (computer programming) Database Cartesian coordinate system Software framework Text editor Key (cryptography) Table (information) Extension (kinesiology)
Mapping Key (cryptography) Multiplication sign Demo (music) 1 (number) Combinational logic Website Funktionalanalysis Quicksort
Message passing Server (computing) Bit Twitter
Server (computing) Dependent and independent variables Email Key (cryptography) Multiplication sign
Server (computing) Inclusion map Computer file Direction (geometry) Quicksort Parameter (computer programming) Remote procedure call
Mapping Network topology
Plug-in (computing) Proxy server Structural load Multiplication sign Demo (music) Mathematical analysis Bit Database Mathematical analysis Funktionalanalysis Generic programming Mechanism design Mechanism design Process (computing) Function (mathematics) Order (biology) Cuboid Software framework Process (computing) Proxy server Plug-in (computing)
Webcam Proxy server Code Demo (music) Mathematical analysis Generic programming Mereology Tracing (software) IP address Field (computer science) Web 2.0 Medical imaging Square number Process (computing) Proxy server Personal identification number (Denmark) Plug-in (computing) Exception handling Plug-in (computing) Dependent and independent variables Scaling (geometry) File format Generic programming Funktionalanalysis Vector potential Cross-site scripting Mechanism design Web application Word Process (computing) Function (mathematics)
Execution unit Wechselseitige Information Proxy server Demo (music) Demo (music) Binary code Quilt Mathematical analysis Cartesian coordinate system Proxy server Plug-in (computing)
Empennage Code Patch (Unix) Lemma (mathematics) Bit Function (mathematics) Proxy server Row (database)
INTEGRAL Code Projective plane Virtual machine Resultant
Web 2.0 Process (computing) Personal digital assistant Right angle Proxy server Plug-in (computing) Field (computer science)
Web 2.0 Process (computing) Funktionalanalysis Window Plug-in (computing) Vector potential Cross-site scripting
Axiom of choice Email Intel Keyboard shortcut Euclidean vector INTEGRAL Multiplication sign Demo (music) Database Graphical user interface Videoconferencing Process (computing) Information HTTP cookie Beta function Software developer Binary code Keyboard shortcut Transport Layer Security Open source Parameter (computer programming) Bit Instance (computer science) Chaining Fluid statics Order (biology) Software framework Software testing Right angle Computing platform Information security Asynchronous Transfer Mode Implementation Proxy server Table (information) Open source Dependent and independent variables Codierung <Programmierung> Disintegration Password Hidden Markov model Mathematical analysis Generic programming Cache (computing) Read-only memory Hacker (term) Telnet Uniqueness quantification Reverse engineering Summierbarkeit Proxy server Alpha (investment) Complex analysis Metre Matching (graph theory) Information Server (computing) Content (media) Database Binary file Web browser Function (mathematics) Key (cryptography) Videoconferencing Electronic visual display Window Extension (kinesiology)
Scale (map) Game controller Server (computing) Email Proxy server Server (computing) System administrator Projective plane System administrator Interactive television Electronic mailing list Mass Electronic mailing list Cartesian coordinate system Perspective (visual) Front and back ends Angle Software testing Reverse engineering Proxy server Sinc function Modem Reverse engineering
hello everybody and welcome and first of all I'd like to thank you all for taking the time to be here and I hope to make it worthwhile for you my name is Martin
holst's vanda I work for Swedish company called to secure as senior security consultant and I've been working the IT security for a couple of years before which she worked as a programmer mostly with Java and nowadays I mostly do web app security and pen testing in working hours programming Emma spare hours and I really like the open source movement and try to participate in contributes wherever I can and as I contributed the few scripts to end up and I see some features in web scribed and some bits to Mallory and w 3f I also think you with a lot of different tools to automate and make my workflow more efficient and sometimes it ends up in usable applications and had kids hat kit is a primary example that which became an OS project a few months ago and with me today i have a friend and colleague Patrick yes oh hi my name is Patrick Olsen and also work for the for the same company as Martin I'm also working with the IT security focusing on web application security and databases I've been a speaker here before at the DEF CON 15 were presented a speech on sequel injection and other band channeling and as well as Martin I'm trying to contribute as much as I can to the open source community in mainly to to end up for the moment where I've committed the script or two yeah so what we're going to do today is talk about the Hat kit project I'm going to start off talk a bit broader general about what the project is all about and then mark going to go on and talk a bit more more in detail about the the tools so as some of you know testing web applications is a complex task I mean looking at methodologists like the ovas for example / testing methodology as a tester you have to cover a few different areas or a number of different areas including like web server configuration with education session management input validation and so on and in order to do so you usually need some sort of tool or framework to work with and there's something there's a whole bunch of tools ranging from fully automated scanners we just click about them and it will spit out a report with all the vulnerabilities oh and you also have a mean different smaller tool specializing and looking for for example sequel injection a cross-site scripting and then of course you have the the proxies which a Didion doing I mean the manual tests where you can intercept the communication you can modify stuff you can view all the headers and so on
so I mean a typical typical part of the intercepting proxy is obviously the intercepting part but if you look at the process available today they come with a bunch of different features which we can see in the in this table here what we try to identify is I mean the different components present in these proxies and whether they actually have to be in the proxy or not and in most cases I mean most of the features you could you could use them you can leverage these features from different tools and at the intercepting proxy as long as you have access to the to the collected data ordered store the data within the within the communication obviously you can't intercept you have to have the intercepting functionality within the proxy itself
so I mean looking at these typical proxies in my experience I've had I've tried a few different ones and I've they all have their strengths and weaknesses obviously but I mean one thing that bothers me quite a lot is that they're usually very resource-intensive I mean especially like when you do testing with with like Internet Explorer or Safari or Chrome I mean we have to set the system setting for the proxy you route all the traffic through that through that proxy of intercepting proxy I mean you'll have OS updates coming in you have video streams and so on and after a few hundred megabytes of data the proxy tends to get a bit slow so that's one of the challenges also when you look at the data collected by the proxy the available tools usually are pretty static you can't you can view a table above the all the collected data but you usually can't modify the log and also when you want to to work with the collected data like analyze it you usually have the possibility to do searches or regular expressions in some cases but you have to have limited possibility to actually work with the data once it's collected and also you have limited post processing capabilities like for example if we want to use the collected data to actually do some sort of run it through another tool to do with sequel injection test for example in most cases is not really obvious how to actually extract the data from the from the intercepting proxy so
what we've done with this project is try to address these these few drawbacks that we actually listed here the project is actually two different tools it's an intercepting proxy with a very very lightweight feature set and lightweight footprint it's a recording process so it records all the data it sees into a MongoDB database also it comes with another tool which which you can later on use to look at the collected data which is which is stored in the database where you could wear Martin is going to talk a bit more and it's going to how you could use this tool to actually have a very dynamic view of the collected data and do post processing
so the arcade proxy it's based on the August proxy actually which is written by Rogan Rogan tapes it comes with all the usual stuff like intercepting it does reverse proxy support it has syntax highlighting it has a fully qualified and non fully qualified mode which allows you to modify stuff within the HTTP protocol which you were which usually can't modify through through other proxies it also has something called TCP I mean it has TTP interception in the early early better stage which allows you to to intercept TCP communication and actually modify it in real time and then just let it send it through to the server and that's what I'm going to focus on these last few slides on the TCP intercepting part and then I'm going to hand over to Martin who's going to talk a bit more about the about the data fiddler which is the other part of the health care proxies how to get the project so TCP
interception we provided the possibility to intercept ecp traffic within this project we provide two ways of doing it either through manual interception where I intercept the department the packets and you get an editor where you can edit the packet we also have something called a scripted scripted possibility where you have different processors so we have you think you can write your own code in Java through the bean shell integration and basically each TCP session gets its own own bean shell interpreter so you have the possibility to to keep state within a TCP session I mean if you collect some interesting data in the first two packets you can keep them in a registry and then just put them up in the sixth or seventh packet where you actually need them and the registry is basically just a hashmap with the screen key where you can store whatever they they want
so I thought I'd show you some some demos on the TCP intercepting part what I've done is I've used them a small ERP or quite RG or pure application which consists of a client part in a server part the server part is actually I mean it's just a database that it's a stick client so it connects to database there's no application server in this particular example so what we're going to look at is is how the thick client connects to sequel server directly using them using a common application account and once you're connected to the database and you try to login to the European it will query a table for a username and a specific password in order to log on the user so going to see some different scripts that that can be used to analyze this traffic and to actually manipulate this traffic too so
what I'm hopefully going to show you now if everything works
so this is what the user interface looks like what we're doing here first is actually we're setting up a forwarding address so what it essentially means is that we listen to the to deal to our interface on port 1433 and all packets that come in on that interface or then forwarded to the to the above one to the 101 1435 it's it's a sequel server instant that listens on that particular port so what we need to do is that we need to say that want to process all
those packets using a bean shell script and at this point i'm specifying a
script called marks of sequel server downgrade so what we'll do it will attempt to downgrade the authentication process of their sequel server connection and I don't know if you're familiar with it but basically you can you have the different different types of authentication a sequel server what I'm going to do is try to downgrade to the weakest one where basically the password is just X soared over the using your ex or encryption and sent over the wire allowing us to to decode it instantly so we start up the proxy then
we see in the console I'm switching to
the application starting it up we see a
connection getting in we see that the
the bean shell script actually successfully downgraded the encryption and we can see that you have a login
from a client using an account which redacted away since it's the name of the product and using the password enterprise 123 so with access to that account you can pretty much do anything in this CRP application since it says the most it has the highest privilege just in the application so you could it could do that with the script we're going to look at the few other
script as well here's the same
application again with the logo that discreetly removed we start up the proxy
and at this point we choose another processor called the MS sequel query
sniffer we try to authenticate to the
application and we see that there's a
bunch of sequel queries running over the wire connecting to the server and what we can see that there's actually specific where is saying select something password alias from G underscore users where the ls equals the one that we put into the authentication form so what the application is doing is trying to retrieve the encrypted password from the database and once you login through the through the login form it compares your encrypted password to the one retrieved from the server does anyone see a problem with this so what we do is that we open yet another bean
shell processor what it does it actually
looks at the it tries to match the query that we just saw in the sniffer and simply replace it with an empty password
so what I do in the proxy I just specify
I want to use a different processor and then i use the DEF CON demo at the top
and we apply it and then we go back to the application once again and we login
using a blank password and we can see
found a replaced pattern and will bring
us right into the ERP application without knowing the users password so those are the kinds of things you can do
with the with the processors automatically so you don't have to
modify each and every packet the scripts
have just shown will be are included in the in the release of the tool that we're doing now for the moment there will be some updates also after this presentation to these scripts and I'm going to end here and pass the mic over to Martin
alright so now I'm going to focus a bit more on the second part of that project which is the head keep data peddler and I'll try to answer the answer these questions the what and the wise in the house and I'll make some demos also so
what is it well as we've already mentioned it's a tool or framework to analyse web traffic and it go a bit more into details we could describe it more like a platform where several applications have been implemented on top of some common components UI components and filters and window and Lane and shared database layer the database layer itself is based on MongoDB which is a so-called no sequel type of document storage database and one idea which has been important during the development has been to reuse existing tools as much as possible not just rewrite the same old tools in a different programming language you're surfing so the aim is to make it a platform which can use existing to some pre-recorded data as much as possible and what does it do well as of the DEF CON release there are four such applications implemented and the ones that exist today our table view which is China like the first tab of web scam where you see you get some arikil information shown in the table in a highly flexible manner and it can be tailored by the user the something called the aggregator which does traffic and pattern aggregation and then there's a third party plugin which among other things it can utilize w 3f a red proxy to analyze traffic pre-recorded traffic and it can also be used to export data to other proxies it also contains some common functionality to filter data in order to work on with the parts of the data that is relevant so you can basically unchanged your data if you happen to catch some oohs updates whatever there's also cash proxy which is still an alpha and table view so it
gives you a highly customizable way to get an overview of an application flow and it's very simple to write reuse the kind of you that you need for your particular scenario and what I mean by scenario will some such scenarios are that for example you might be interested in analyzing user interactions so you use two different browsers may be on the same target and try to see if one one user can access the information belonging to the other and that's one scenario and there you might be interested in being able to differentiate based on user agents and now in another scenario might be a more interested in analyzing the server infrastructure so you're more interested maybe in the server banner or the headers and another time you might be in studying analyzing for encoding escaped mistake soon and cross-site scripting potential cross-site scripting issues stuff like that I still make a demo but before I'll do that I'd like to spend a little bit more time on the core of the data field which is the data that has been stored inside MongoDB by the proxy so the traffic is stored is parsed object in the database and what does it mean for us what it means that when we do selection what we want to load from the database up to our application we can we can specifies criteria things which are deep inside the objects themselves so we can say for example that we only want to work now with objects which have something some request headers set cookie and certain cookie or where there is a certain parameter of Jason and it also means that we can when we load the data up we can we don't have to love the entire objects we can just load the parts for example if we just won't work on the server infrastructure we just have to load the headers and also when we get it into the application it still retains the same structure so to some extent the MongoDB is very similar to an old-school object database except for this platform independent okay so let's move away from the nitty gritty details and see some actions
so what you see here this is the data fiddler and this is not how it normally looks sorry it's not how it normally
looks but I made it cleaned it out a bit so that i'm going to show you now how we can populate this with something that's interesting because currently is just showing the the database identifiers of each object in the database so i open the these settings for this and i'm also going to double click on one of these objects by doing so i get the something called the object inspector which loads the complete object from the database so we can see what how it is structured and
they see this for example the response and the response contain headers each header is I stored it is an attribute in the headers dictionary basically and the has called to sub cookie and it contains an asp cookie with value and their path attribute and we go back to the settings so we want it we're interested in between there in the response object that was re the request object so we love that into on the left side here we define variables that's what we load from the database so in variable V one with willow the request object or node and then we add a column which is on the right side we just we define what we want what do you want to see in the table so I can I can just type in variable V one there optionally add a title and if you have your your coder
goggles on you might see that what appeared on the second column is the string representation of a Python dictionary unicode keys and values everybody see that but that's not very user-friendly is it so we can install here in the column definition reach into the method fly it's a method that would give us the request method and then we have all of a sudden you can see that there are get some posts you might also see that there is coloring enabling the coloring it's just the hash of the text value so it can be good to have if you want to see where it changes I'm not may be interested in the particular value because I wants to save some screen real estate so I can disable the actual print out of gift and just see the color value and we can say this view fellated wants so we don't have to redefine it now the current definitions are really just Python it's a pure python mature is
evaluated and since it is piped them we
can write any kind of Python code there and there are some helper functions we can help us produce some nice use readable strengths for our tables for example we're going to use a helper function called fq host it takes the request object and produces the fully
qualified hostname so we can just write the fq hosts on variable V one there add the third column and now you have the fully qualified house in third column and in a similar manner we can decide we want to see the parameters also and there's the param string and which is that in your call right param string and you can write arbitrary Python there
or as what I just showed you is how you can start populates and write your own view definitions so you can get exactly what you want now I'm going to show you some bit more advanced usage where where
we see we're going to use the
functionality to reach deep into objects and the example scenario here will be that I might be analyzing a web application which is based on ajax and the the workings of the application that aren't visible in the URL it's only visible deep inside the the request and response bodies so I need to reach into it to get to visualize it okay and what you're seeing now that's the normal view of the data fiddler with it we don't do and it changes to it so we enter the set up again and going to the type of data
based filtering and we start on the only the sub tab called native clauses and here we can specify some claw-some some clauses which says which arbic to load and we type responds to Jason that node has to exist its exist we get it into our application and we can test it and there's 113 of these objects okay that
sounds good we apply it and now the only thing in our view there are the JSON responses now to do we also want to
reach into the JSON for our viewing so we need to modify this view a bit and I prepare this bit so there's a JSON there let's say I'm going to load it and what appeared now as you can see in column 2 is that I reach into this jay responds to Jason the trends which is in v6 and from that I pick out the attribute which is a date which and in that list I take a seraph element and the query attribute so it's an object women list within or between in objects within Jason and yeah we'll show you here also what I mean so we're trying to reach one of these queer objects inside of that we want to follow
that through this application flow through this request so I just apply that now you see that I several places there are type errors and that's because even if all the responses had Jason they didn't all contain these objects so in order to fix that the final details we can go into to a bit more advanced usage where we're going to the JavaScript expressions if you type JavaScript expressions in filters these are actually surpassed down into MongoDB and evaluated on the fly by MongoDB so write a function there which returns true if you want to send this object back to the application and the one I'm loading here is basically checks if if there is Jason and if it contains the trans object and it contains the list blah blah blah if so return true and apply that and there we have the flow of the application it was all the same but what you've seen is how can reach deep into the JSON and see defined use that let us see on exactly what we're interested in
and I forgot to mention that but when you're in table view also integrated that with with the request you can see you can see the diff using kd3 or whatever differ you like you can view the response content with your do with your platform default editor for java script or HTML or you can override it and use your eclipse or whatever another
data field application is the aggregator so if the table use the way of representing data in a one-to-one format the aggregator instead walks through the data on the database side and collects the interesting pieces which are sorted by specified key this is a feature of MongoDB it's very similar to MapReduce if any of you are familiar with that and
so this is the aggregator are on a treeview and I open the setup here and
the one that was pretty final read it was the aggregate path and we're on the basic time now and here you can just load some predefined combination so reduce functions and sorting keys and the aggregate paths ones that you're seeing it just aggravates the path sort by the host name which what do we end up with wind up with the site map if you do that there a lot of predefined ones here
we can just play along a bit with them there's the aggregate pass by there HTTP
status there's the you aggregate you can
a great server banos by host for example so you can see here all all the server banners that Twitter and Google and whatever this is just some random collected data used
he released the response service that is listing all the other unique is keys that we have in our collection and also counting how many times way so all them and this is a bit interesting if we go
into the Advanced tab we can see that the key is a static kee called one which means that everything will just be sorted on the same on the same entry we can change that now save yourself were right to request header host and bam all
of a sudden we have the same thing that we sorted it by by host so this is for example if you want to analyze the infrastructure and you're suspecting that for certain paths they use different servers then you can do the same thing but you you sort of just bypass some other examples scenarios that you can some other baseball's that are really useful can be to check all
the parameter names that are used for example if you're suspecting that there's remote file inclusion possibilities or direct arbic reference it can be pretty useful to look at the names of the parameters and if that seems interesting you can also add
another node to that tree we look at their each individual value also just aggregate everything and there we have it with the values maps
and as we mentioned several time one base caddy and Hackett frameworks is to use reuse existing tools as much as possible because functionality is an asset but Cody's liability and there is a mechanism inside the framework which we call the third-party plugin and what it does is that it loads data from the database and one by one it's it lets the plug and process that data and such plugins can do a lot of interesting stuff and so far we've implemented for one is the the plugin for red proxy analysis perhaps you're already familiar with Michael Psilakis rat proxy it's mostly passive proxy which analyzes the data on the fly as it goes through the proxy and in order for us to to make use of red proxy we had to trick red box a little bit so we do it by skip here oh
that image didn't came out well I'm just
explaining words we do that by the data Peter starts listening to a porch then it starts to read proxy process and tells the red proxy process to use that port is forwarding proxy now we start sending feeding red rocks to traffic come when we once we get the request on the data fiddler back and we can send response back and we can collect the output and and this method of export has also been generalized in a generic proxy exporter is basically worked exactly the same way except that we need to to to manually open up your proxy whichever you want to use web square burps up and configured to use data field as a forwarding proxy for it to work and there's AK that it doesn't have let us out properly right now unfortunately this was a webcam exporter which exports data in the format that web skype can read unfortunately when you do it that way web scale doesn't process the data so you reduce the generic exported for that and finally we have w 3f i guess most of you have at least heard of it but it's a web application and audit tool no web application ordered an attack tool and it contains a lot of functionality and one of the things i like about it most is that it contains something called the grippers the grabbers is just Python code which takes through customs bones and the searches for stuff it can be different things that can be in such respect traces your internal IP address or social security number disclosures or potential cross-site scripting issues and the third part plugin that this hasn't been released that it was released earlier this week and the DEF CON release so I'm going to make some
demos these plugins
so with a rat proxy exporter or I need
to do is basically tell the proxy where the binary is and then you run it the
thing with the rat proxy is that it is
kind of slow because the application doesn't know how many requests rest Brooks will make sometimes it makes several requests from one inbound so we have to wait for it to time out before we can move on to the next there we go
so I will gather the output and there you have it and that's that's basically the row output from from red proxy and then I'll show you the w 3f now w three
AF that's all Python code so it's very
it's very much more efficient where we we can interact with that we don't have to trick it anything we can just reach into the code and monkey patch it a bit and just use the grippers so you need to just enable it sell it with the code is
with your purse or and bam there are the results and I like to mention also that
the integration with both of these two pop rocks and w 3f it requires the third party to actually be installed at least accessible on your machine so we didn't just take the code from w 3f and put it in our project we try to give some added value to both projects by doing it this way
let me see that was wrong
and the final then i'm going to show you
the generic generic export to demo right
so we set here that we want to use the
proxy export the plug-in and tell the plugin where where that proxy is listening and define where we want data pitot tube listing okay and then we
start our proxy in this case its web
scam sickly we doesn't have intercept on because if we do it to disrupt the process and the the data field with timeout and that's bad things will happen sky become tumbling down we check that it is configured to use a forwarding proxy which is the same as our data fiddler 9999 okay and click the summary tab so we can see what's happening
and then we it run and as you can see and now we start populating web-scale with data you might notice also that the the plugin window has a filter tab like most windows in the data filler so you can actually fill if you want to send some stuff to web scare but not everything perfectly doable I'm tainting your data and yeah of course this is a fully functional web scarab when you import it this way so there you have some processing you can see that the web scarab has detected some potential cross-site scripting issues okay
I'm going unbeaten
it's not
alright so there are some upcoming
features there is a cash proxy kaspersky starts a little HTTP forwarding proxy and it matches the requests against what is stored in the database and it returns the best match and they can be configured either to just to be closed so that just returns for four if it doesn't match or it can be open so it goes goes ahead and fetch it's your moat content if it can't find it in the cash so what can you do with this well you can do for example you can resume an ecto scam and you can also use it to if you want to gather screenshots or videos outlining what you did on a particular assignment after this time this completely maybe don't any longer have access to the target and this feature is already in alpha stage implementation wise and if you're a pipe from hacker you can definitely get it working without any magic problems the thing is there's no you i implemented for it yet so unless you want to hack you gonna have to wait a little bit faster integration we hope to to integrate directly with j brief us so that if you have a requesting you want to want to do a manual request you can just send to debrief us and it pops up currently we only implement so you can like send it to browser and and yeah you can also do right clicking at copy fully qualified request which you can paste into the manual request of proxy of your choice planning to add some some more advanced text search capabilities either biopiracy in a whoosh and that's what i mentioned there was new release earlier this week both of the proxy on the data fiddler containing all this stuff talking about so what would you use it well to better be able to make sense of large parties of complex information and get all the information you want after your body of data so you can download the source from bitbucket you can download the release binary from bid packet there is documentation on the ovas website unfortunately during the summer there's been a lot of development and everything hasn't been documented on OS yet in order to get it running a you need Python cutie cutie for bindings and mongodb driver and of course access to a MongoDB instance with pre-recorded data w 3f in red rocks are optional we've got it working on linux and mac OS x I think you're out of luck if you're running Windows so who is this meant for well
obviously for application testers that's the perspective I've been giving here today and it's it's not true it's not the test is to just want to point a click tool it's with someone who really wants to take control of the data and really fill it with it basically but it's also it's also another angle on this and that's with server administrators who can use the proxy as the reverse proxy and use it log all incoming traffic they can then use the data pillar to analyze user interaction for example detect malicious activity or perform post modem post-mortem analysis and since we're using MongoDB one bonus feature we got was that we have a backend which can scale massively and potentially handled very large amounts of data we love to get some feedback and
new members to the project this is still very much on the development and please don't hesitate to join the mailing lists and I like to thank you all again for listening we want to do in Q&A and this through but if you join us in the Q&A room wherever that is we can show you some more hands on and answer all your questions thank you very much