Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-timelog analysis by processing and filtering them, and finally it stores the logsor routes them for further analysis. This session focuses on how syslog-ngparses important information from incoming messages, enriches them withadditional contextual information, and concludes with demonstrating how all ofthis can be used for alerting or for dashboards. Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-timelog analysis by processing and filtering them, and finally it stores the logsor routes them for further analysis. In an ideal world, all log messages comein a structured format, ready to be used for log analysis, alerting ordashboards. But in a real world only part of the logs belong to this category.Traditionally, most of the log messages come as free format text messages.These are easy to be read by humans, which was the original use of logmessages. However, today logs are rarely processed by the human eye.Fortunately syslog-ng has several tools to turn unstructured and many of thestructured message formats into name-value pairs, and thus delivers thebenefits of structured log messages. Once you have name-value pairs, logmessages can be further enriched with additional information in real-time,which helps responding to security events faster. One way is adding geo-location based on IP addresses. Another way is adding contextual data fromexternal files, like the role of a server based on the IP address or the roleof the user based on the name. Data from external files can also be used tofilter messages, for example to check firewall logs to determine whethercertain IP addresses are contained in various black lists for malware commandcenters, spammers, and so on. Logging is subject to an increasing number ofcompliance regulations. PCI-DSS or many European privacy laws require removingsensitive data from log messages. I will demonstrate how logs can beanonymized in a way that they are still useful for security analitics. At theend I would like to introduce you to the basics of syslog-ng configuration,and demonstrate how the collected logs can be used for alerting or visualizedon a dashboard. |