We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Extending syslog-ng in Python: Best of both worlds

Formale Metadaten

Titel
Extending syslog-ng in Python: Best of both worlds
Serientitel
Anzahl der Teile
561
Autor
Lizenz
CC-Namensnennung 2.0 Belgien:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs, or forwards them for further analysis. It was originally developed in C due to its low footprint and high performance. Recently Python language bindings were added, so you can collect, process, and store log messages using embedded Python code. Learn how to customize your log processing infrastructure in Python using syslog-ng, integrate your logs to local and cloud services, and the implementation details and benefits of such solutions. The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally it stores the logs, or forwards them for further analysis. It was originally developed in C due to its low footprint and high performance. Recently Python language bindings were added, so you can collect, process and store log messages using embedded Python code. Python bindings always have two parts: a configuration part – where you configure a source, parser, or destination – and the actual Python code part. You can store the code in-line in the syslog-ng configuration file, or in external files, if you have a larger project. The first Python binding added to syslog-ng was the Python destination. Many databases and cloud services have an easy to use Python API, while a C implementation of the API is either missing or difficult to implement. The syslog-ng application handles log messages and parsed data as name-value pairs. You can pass name-values to the Python destination code in mulitple ways: an object which contains all of the name-value pairs, or as a dict, which contains only name-value pairs configured in syslog-ng.conf. Recent releases also support the use of templates. Next the Python parser was implemented. Originally it was intended as an easy-to-code parser for messages not supported by the built-in parsers, like PatternDB or csv-parser. But you can also use it to enrich log messages from external databases. For example, you can use it to resolve host names from IP addresses. Recently support for Python source was added. You can use it to fetch data or implement a server in Python, for example, to collect messages over HTTP or Kafka protocols. Obviously the Python code is slower than native C code. On the other hand, extending syslog-ng in Python offers a lot more flexibility than development in C. You do not need a development environment other than a text editor and the Python module installed. Neither do you need you to compile the code. Just write it and use it. It is easy to get started, as there is no need for extra setup. You can get proof-of-concept minimal setup working with a single method quickly. You can extend it later to have proper initialization and connection control for added robustness. Learn how to customize your log processing infrastructure in Python using syslog-ng, integrate your logs to local and cloud services, and the implementation details and benefits of such solutions.