Get the most out of your security logs using syslog-ng
Czanik, Peter
Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-timelog analysis by processing and filtering them, and finally it stores the logsor routes them for further analysis. This session focuses on how syslog-ngparses important information from incoming messages, enriches them withadditional contextual information, and concludes with demonstrating how all ofthis can be used for alerting or for dashboards. Event logging is a central source of information for IT security. The syslog-ng application collects logs from many different sources, performs real-timelog analysis by processing and filtering them, and finally it stores the logsor routes them for further analysis. In an ideal world, all log messages comein a structured format, ready to be used for log analysis, alerting ordashboards. But in a real world only part of the logs belong to this category.Traditionally, most of the log messages come as free format text messages.These are easy to be read by humans, which was the original use of logmessages. However, today logs are rarely processed by the human eye.Fortunately syslog-ng has several tools to turn unstructured and many of thestructured message formats into name-value pairs, and thus delivers thebenefits of structured log messages. Once you have name-value pairs, logmessages can be further enriched with additional information in real-time,which helps responding to security events faster. One way is adding geo-location based on IP addresses. Another way is adding contextual data fromexternal files, like the role of a server based on the IP address or the roleof the user based on the name. Data from external files can also be used tofilter messages, for example to check firewall logs to determine whethercertain IP addresses are contained in various black lists for malware commandcenters, spammers, and so on. Logging is subject to an increasing number ofcompliance regulations. PCI-DSS or many European privacy laws require removingsensitive data from log messages. I will demonstrate how logs can beanonymized in a way that they are still useful for security analitics. At theend I would like to introduce you to the basics of syslog-ng configuration,and demonstrate how the collected logs can be used for alerting or visualizedon a dashboard.
 Hello, welcome to the final session of the Security Dev Room. Let's welcome Peter, who will talk about Syslog NG. Thank you. First a few words about me. Sorry for those who were there yesterday in my talk. The first few slides will sound quite familiar. I'm Peter Tzenik from Hungary, community manager at Bellabit. Syslog NG is upstream developer. I'm working on packaging supporting Syslog NG. First of all, what is logging? It's a recording of events on a computer, for example, an SSH login message under Warlog messages in a file. You can see here Syslog NG itself is an enhanced login with a strong focus on high performance central log collection. Why central logging? First of all, it's ease of use. One place to check for your log messages instead of money. It's also availability. So even if the sender machine is down, you can read the logs at the central location. And last but not least, it's also security. That's the first thing in hacking a machine is to remove traces of what you have done. But if you have central logging, then all of the traces are forwarded in real time to a central location where you can check it what happened to the machine. There are four main roles of Syslog NG. First of all, it's collecting messages. It also can process log messages, filter them, and at the end, either store them locally or forward somewhere. The main role, the first role is data collection. And I say data and not log collection. Syslog NG can collect practically any kind of data. System messages, application logs together. And they can provide quite useful contextual data for either side. It can collect messages from a wide variety of platform specific sources like Devlog or Jornal or Sunstreams as many different platforms are supported. As a central log collector, it knows both the legacy and the new Syslog protocol and can use UDP, TCP, TLS, in any cases. It can also collect logs or any kind of data from applications through five socket pipes. And if the application is started by Syslog NG, then even application output. The next and in my opinion, the most important role, at least in a security environment, is log processing. Using Syslog NG, you can classify, normalize, and structure log messages. I will talk about this later. You can even rewrite log messages. And you don't have to think about falsifying messages here. But for example, in many cases, you need to analyze log messages due to compliance requirements. Messages can also be reformatted using templates. When log analysis software needs a specific format like a specific date format or have logs in JSON format and so on. And you can also enrich data. Using GIP or adding fields based on the message content. The next role is filtering. It has two main uses. First of all, discarding surplus logs like debug level messages or the other one is message routing. So you can make sure that the right messages are reached the right destinations. For example, you only send authentication-related messages to a SIEM system. There are many possibilities for filtering. It can be based on message content or parameters. Using comparisons, wildcards, and many different filtering functions. And best of all, any of these can be combined using Boolean operators. At the end, you can either store your messages locally or forward to many different destinations. Here comes a tricky question for those who don't check my slides right now. I mean, you're on a phone or so. Which Cyslogangy version is the most used? Do we have any idea? Project started a long time ago. Red Hat has version 3.5. The latest stable version, oh, it's actually 3.9. What do you think? Which is the most used version? Any guesses? You are the closest. It's version 1.6. There are over 100 million Kindle eBook readers and all of those run Cyslogangy for local action. Now, back to more serious topics. We have a lot of questions. What are the most used versions? You can run Cyslogangy for local action. If you take a look under Varlog Messages, you will see that most of the messages have the date, host name, and some kind of text at the end. The text part is an English sentence with some variable parts in it. It's quite easy to read by a human, but if you have a busy server with a thousand messages a second, it's quite difficult to deal with it. You don't have the time, and the machine cannot really understand it. There is a solution for this program. It's called Structured Logging. In this case, you answer presented as name value pairs. Back to my favorite SSH login example, you can describe the event as application, SSHD user root and IP address. The good news is that Cyslogangy has name value pairs inside right from the beginning. Facility, priority, and so on are all saved as name value pairs. This is the way how you could use them in filters or in templates. There are parsers in Cyslogangy which can turn unstructured and some structured data into name value pairs as well. Let's talk about these. The first one is the JSON parser. If you couldn't parse JSON with Cyslogangy, then you can see inside the message just simply store the whole JSON message somewhere. But as Cyslogangy can parse JSON messages, you can use the values inside the JSON message for filtering, for reformatting, for other destinations. So data is much more valued in this way. CSV parser can parse columnar data into fields. The most popular example is Apache Access Log. In this configuration snippet, you can see the column numbers. If you take a look here at the bottom of the screen, you will see that the username is used in a filename template. So each user has its own log file in this case. The key value parser was added just recently to Cyslogangy and it can parse key, equals, something like data. The most popular example for this is firewall logs. For example, IP tables, but this one is from Zorb, another firewall. The most interesting parser in Cyslogangy is patternDB, which can extract information from unstructured messages into name value pars. It can also add status fields based on the message content and do some message classification like log check does. The downside of this parser is that it needs an XML database describing the log messages. So it can only parse messages for which you have a description. Back to my favorite SSH example, here are a few fields. The first line has some data which is directly extracted from the log message, the application name, the username, source IP, but based on the message content, you can also add the action of the log was login and the status was a failure. And based on this, you can classify this log message as a violation. Cyslogangy can enrich log messages and create additional name value pars based on the message content. I already mentioned patternDB on the previous slide. You can also use a GUIP parser to add geolocation based on IP addresses, for example, a country name or longitude, latitude information. It can be used in security as well to detect anomalies, as it's not a real-world situation that someone logs in from Europe and from Australia at the same time. And you can also use it to display locations on a map. Recently, we added another parser which can add metadata from CSV files to log messages. For example, add the host role or contact person, which can considerably speed up login analysis and also helps to create more accurate alerts or dashboards. Next, I would like to talk a few words about configuring Cyslogangy. My first advice is don't panic. I often get the feedback that Cyslogangy configuration is very difficult. Yes, it looks difficult at first and sometimes even at second site, but once you take a deeper look at it, you will see that it's simple and logical. It follows a pipeline model where you have many different building blocks like sources, destinations, filters, and so on. And all of these can be connected into a pipeline using log statements. Cyslogangy configuration always starts with a version number for which version it was created and some global options. Most of these can be overridden later in the configuration. You can define sources. Here, as you can see, this is a single source definition, but it combines two different sources, internal messages. So those generated by Cyslogangy itself and system is for platform-specific log sources. This helps to hide the way platform-specific differences. So you can use the same configuration on Linux, FreeBSD, Solaris, and it will find log sources. The other one is a network source, in this case UDP. Here are a few destinations, the typical VAR log messages. It's a flat file. The other one is more interesting. It's an elastic search destination where you specify the index name, the type of the message cluster name, and here you can see that it's using JSON formatting and selecting different fields what to forward. Here you can see some filter definitions and loading a database for pattern-db parsing, and here is the part of the configuration, the log statements, where the upper one is for VAR log messages, local sources, related filter, and the destination. The other one is more interesting with multiple sources, filtering pattern-db, and sending the results to elastic search. And here is a nice dashboard from elastic search. It's a world map where the log source was IP tables, and the attack sources are visualized on screen. Here is the configuration, how you can do this. First of all, you need a key value parser to extract the different fields from the message. In this case, the source IP, for the source IP, the geolocation is looked up. As Kibana needs a specific format for it, it's rewritten to that format, and here in the log statement, all of these building blocks are combined. PEG is a software in heavy development. It was formerly known as ELSA. It's a web application for security analysts, and it's built on syslogang, and pattern-db message parsing. The results are sent to elastic search, and it has its own web interface. It's not Kibana. In this case, there are some nice graphs coming from an intrusion detection system. In a few words, what are new in syslogang 3.8? It has this base buffering, a new parser which is easier to use. You can also write parsers in Rust. It has support for all of the latest elastic search versions, and many, many performance improvements. Finally, a summary of what syslogang can help with security logs. First of all, it's high-performance and reliable log collection. You can parse and enrich log messages, so you can do basic log analysis inside syslogang, and you can also efficiently forward log messages to other log analysis tools. If you want to know more about syslogang, our central hub of information is syslogang.org, or our source code is on GitHub, where we also have an issue-tracking system. If you need help, we have a mailing list, or for near real-time help, we have IRC or Guitar Genos. Do you have any questions? Thank you, Peter. We have an applause. We have five minutes for questions, so raise your hand if you want to ask a question. We have the first one here. Hi. What about support for Docker containers or other containers? Instead of having LogStash or Fluent.de getting the container logs put into this, and then enriching the log format with container-relevant information? Has that supported yet? Yes. We have a Docker image, and I just checked it and was quite surprised as the development version of the Docker image had just a few hundred downloads, but the regular image had over 100,000 downloads. So people are using it in Docker. Sorry. I wasn't referring about running syslogang inside a container, but rather on a host or a VM where you have syslogang, and it has a Docker daemon, or it has Rocket or whatever, and you're running, say, 10 containers, and you'd like the logs from those containers to go back to the host base syslogang to enrich, to then send them on to some other location. So in order to do that, it would be useful for it to get some information from, say, Docker daemon to understand the name of the container and some other attributes. So as far as I know, Docker is logging in JSON format, and we can process that. I know that it's used, but personally, I don't have much experience with it. Okay, there's one more question here. Raise your hand if you want to ask a question. Hello. So my question, it gets more into, I suppose, the legal aspects, which I saw mentioned at the start of the talk in your brief. I suppose, are there ways which syslogang can support compliance to EU data protection law for not storing IP addresses or storing them in particular ways, which better support, I suppose, minimal standards in what should be captured and not captured? What we can do is, using pattern DB or using regular expressions, you can find IP addresses or credit card numbers in log messages. Both have advantages and disadvantages, and you can either replace with a static text the IP address or you can use hashes, and if you use a hash of the original, then you can use it in further analysis of the message. So you can follow sessions, what the given user did, even if you don't know the name of the user, but you can follow where it is coming from, what it did. We have one more here. We're trying to just have a comment or reflection or criticism or discussion. So most of the new assistants are running JournalD. What is your recommendation? Where is the work split between JournalD and syslogang? JournalD is for local log messages. Of course, you can pull log messages from it periodically, but it's still not something for central local action. With syslogang, you can constantly read log messages from the journal and forward it to a central location in real time. So it's better suited for security purposes, I think. Thank you. Final question? No. Okay, let's thank Peter. APPLAUSE And thanks for coming here. This was the final presentation for the day. APPLAUSE Thank you.
 Hello, welcome to the final session of the Security Dev Room.  Let's welcome Peter, who will talk about Syslog NG.  Thank you.  First a few words about me.  Sorry for those who were there yesterday in my talk.  The first few slides will sound quite familiar.  I'm Peter Tzenik from Hungary, community manager at Bellabit.  Syslog NG is upstream developer.  I'm working on packaging supporting Syslog NG.  First of all, what is logging?  It's a recording of events on a computer, for example,  an SSH login message under Warlog messages in a file.  You can see here Syslog NG itself is an enhanced login  with a strong focus on high performance central log  collection.  Why central logging?  First of all, it's ease of use.  One place to check for your log messages instead of money.  It's also availability.  So even if the sender machine is down,  you can read the logs at the central location.  And last but not least, it's also security.  That's the first thing in hacking a machine  is to remove traces of what you have done.  But if you have central logging, then all of the traces  are forwarded in real time to a central location  where you can check it what happened to the machine.  There are four main roles of Syslog NG.  First of all, it's collecting messages.  It also can process log messages, filter them,  and at the end, either store them locally or forward  somewhere.  The main role, the first role is data collection.  And I say data and not log collection.  Syslog NG can collect practically  any kind of data.  System messages, application logs together.  And they can provide quite useful contextual data  for either side.  It can collect messages from a wide variety of platform  specific sources like Devlog or Jornal or Sunstreams  as many different platforms are supported.  As a central log collector, it knows both the legacy  and the new Syslog protocol and can use UDP, TCP, TLS,  in any cases.  It can also collect logs or any kind of data  from applications through five socket pipes.  And if the application is started by Syslog NG,  then even application output.  The next and in my opinion, the most important role,  at least in a security environment, is log processing.  Using Syslog NG, you can classify, normalize,  and structure log messages.  I will talk about this later.  You can even rewrite log messages.  And you don't have to think about falsifying messages here.  But for example, in many cases, you  need to analyze log messages due to compliance requirements.  Messages can also be reformatted using templates.  When log analysis software needs a specific format  like a specific date format or have logs in JSON format  and so on.  And you can also enrich data.  Using GIP or adding fields based on the message content.  The next role is filtering.  It has two main uses.  First of all, discarding surplus logs like debug level  messages or the other one is message routing.  So you can make sure that the right messages are  reached the right destinations.  For example, you only send authentication-related messages  to a SIEM system.  There are many possibilities for filtering.  It can be based on message content or parameters.  Using comparisons, wildcards, and many different filtering  functions.  And best of all, any of these can be combined  using Boolean operators.  At the end, you can either store your messages  locally or forward to many different destinations.  Here comes a tricky question for those who don't  check my slides right now.  I mean, you're on a phone or so.  Which Cyslogangy version is the most used?  Do we have any idea?  Project started a long time ago.  Red Hat has version 3.5.  The latest stable version, oh, it's actually 3.9.  What do you think?  Which is the most used version?  Any guesses?  You are the closest.  It's version 1.6.  There are over 100 million Kindle eBook readers  and all of those run Cyslogangy for local action.  Now, back to more serious topics.  We have a lot of questions.  What are the most used versions?  You can run Cyslogangy for local action.  If you take a look under Varlog Messages,  you will see that most of the messages have the date,  host name, and some kind of text at the end.  The text part is an English sentence with some variable parts in it.  It's quite easy to read by a human,  but if you have a busy server with a thousand messages a second,  it's quite difficult to deal with it.  You don't have the time, and the machine cannot really understand it.  There is a solution for this program.  It's called Structured Logging.  In this case, you answer presented as name value pairs.  Back to my favorite SSH login example,  you can describe the event as application,  SSHD user root and IP address.  The good news is that Cyslogangy has name value pairs inside right from the beginning.  Facility, priority, and so on are all saved as name value pairs.  This is the way how you could use them in filters or in templates.  There are parsers in Cyslogangy which can turn unstructured  and some structured data into name value pairs as well.  Let's talk about these.  The first one is the JSON parser.  If you couldn't parse JSON with Cyslogangy,  then you can see inside the message just simply store the whole JSON message somewhere.  But as Cyslogangy can parse JSON messages,  you can use the values inside the JSON message for filtering,  for reformatting, for other destinations.  So data is much more valued in this way.  CSV parser can parse columnar data into fields.  The most popular example is Apache Access Log.  In this configuration snippet, you can see the column numbers.  If you take a look here at the bottom of the screen,  you will see that the username is used in a filename template.  So each user has its own log file in this case.  The key value parser was added just recently to Cyslogangy  and it can parse key, equals, something like data.  The most popular example for this is firewall logs.  For example, IP tables, but this one is from Zorb, another firewall.  The most interesting parser in Cyslogangy is patternDB,  which can extract information from unstructured messages  into name value pars.  It can also add status fields based on the message content  and do some message classification like log check does.  The downside of this parser is that it needs an XML database  describing the log messages.  So it can only parse messages for which you have a description.  Back to my favorite SSH example, here are a few fields.  The first line has some data which is directly extracted  from the log message, the application name, the username, source IP,  but based on the message content,  you can also add the action of the log was login  and the status was a failure.  And based on this, you can classify this log message as a violation.  Cyslogangy can enrich log messages  and create additional name value pars based on the message content.  I already mentioned patternDB on the previous slide.  You can also use a GUIP parser to add geolocation based on IP addresses,  for example, a country name or longitude, latitude information.  It can be used in security as well to detect anomalies,  as it's not a real-world situation that someone logs in from Europe  and from Australia at the same time.  And you can also use it to display locations on a map.  Recently, we added another parser which can add metadata from CSV files to log messages.  For example, add the host role or contact person,  which can considerably speed up login analysis  and also helps to create more accurate alerts or dashboards.  Next, I would like to talk a few words about configuring Cyslogangy.  My first advice is don't panic.  I often get the feedback that Cyslogangy configuration is very difficult.  Yes, it looks difficult at first and sometimes even at second site,  but once you take a deeper look at it, you will see that it's simple and logical.  It follows a pipeline model where you have many different building blocks  like sources, destinations, filters, and so on.  And all of these can be connected into a pipeline using log statements.  Cyslogangy configuration always starts with a version number  for which version it was created and some global options.  Most of these can be overridden later in the configuration.  You can define sources.  Here, as you can see, this is a single source definition,  but it combines two different sources, internal messages.  So those generated by Cyslogangy itself and system is for platform-specific log sources.  This helps to hide the way platform-specific differences.  So you can use the same configuration on Linux, FreeBSD, Solaris,  and it will find log sources.  The other one is a network source, in this case UDP.  Here are a few destinations, the typical VAR log messages.  It's a flat file. The other one is more interesting.  It's an elastic search destination where you specify the index name,  the type of the message cluster name, and here you can see that it's using JSON formatting  and selecting different fields what to forward.  Here you can see some filter definitions and loading a database for pattern-db parsing,  and here is the part of the configuration, the log statements,  where the upper one is for VAR log messages, local sources, related filter, and the destination.  The other one is more interesting with multiple sources, filtering pattern-db,  and sending the results to elastic search.  And here is a nice dashboard from elastic search.  It's a world map where the log source was IP tables,  and the attack sources are visualized on screen.  Here is the configuration, how you can do this.  First of all, you need a key value parser to extract the different fields from the message.  In this case, the source IP, for the source IP, the geolocation is looked up.  As Kibana needs a specific format for it, it's rewritten to that format,  and here in the log statement, all of these building blocks are combined.  PEG is a software in heavy development.  It was formerly known as ELSA.  It's a web application for security analysts,  and it's built on syslogang, and pattern-db message parsing.  The results are sent to elastic search, and it has its own web interface.  It's not Kibana.  In this case, there are some nice graphs coming from an intrusion detection system.  In a few words, what are new in syslogang 3.8?  It has this base buffering, a new parser which is easier to use.  You can also write parsers in Rust.  It has support for all of the latest elastic search versions,  and many, many performance improvements.  Finally, a summary of what syslogang can help with security logs.  First of all, it's high-performance and reliable log collection.  You can parse and enrich log messages, so you can do basic log analysis inside syslogang,  and you can also efficiently forward log messages to other log analysis tools.  If you want to know more about syslogang, our central hub of information is syslogang.org,  or our source code is on GitHub, where we also have an issue-tracking system.  If you need help, we have a mailing list, or for near real-time help, we have IRC or Guitar Genos.  Do you have any questions?  Thank you, Peter. We have an applause.  We have five minutes for questions, so raise your hand if you want to ask a question.  We have the first one here.  Hi. What about support for Docker containers or other containers?  Instead of having LogStash or Fluent.de getting the container logs put into this,  and then enriching the log format with container-relevant information?  Has that supported yet?  Yes. We have a Docker image,  and I just checked it and was quite surprised as the development version of the Docker image  had just a few hundred downloads, but the regular image had over 100,000 downloads.  So people are using it in Docker.  Sorry. I wasn't referring about running syslogang inside a container,  but rather on a host or a VM where you have syslogang, and it has a Docker daemon,  or it has Rocket or whatever, and you're running, say, 10 containers,  and you'd like the logs from those containers to go back to the host base syslogang to enrich,  to then send them on to some other location.  So in order to do that, it would be useful for it to get some information from, say, Docker daemon  to understand the name of the container and some other attributes.  So as far as I know, Docker is logging in JSON format, and we can process that.  I know that it's used, but personally, I don't have much experience with it.  Okay, there's one more question here.  Raise your hand if you want to ask a question.  Hello. So my question, it gets more into, I suppose, the legal aspects,  which I saw mentioned at the start of the talk in your brief.  I suppose, are there ways which syslogang can support compliance to EU data protection law  for not storing IP addresses or storing them in particular ways,  which better support, I suppose, minimal standards in what should be captured and not captured?  What we can do is, using pattern DB or using regular expressions,  you can find IP addresses or credit card numbers in log messages.  Both have advantages and disadvantages, and you can either replace with a static text the IP address  or you can use hashes, and if you use a hash of the original,  then you can use it in further analysis of the message.  So you can follow sessions, what the given user did,  even if you don't know the name of the user, but you can follow where it is coming from, what it did.  We have one more here.  We're trying to just have a comment or reflection or criticism or discussion.  So most of the new assistants are running JournalD.  What is your recommendation?  Where is the work split between JournalD and syslogang?  JournalD is for local log messages.  Of course, you can pull log messages from it periodically,  but it's still not something for central local action.  With syslogang, you can constantly read log messages from the journal  and forward it to a central location in real time.  So it's better suited for security purposes, I think.  Thank you.  Final question?  No.  Okay, let's thank Peter.  APPLAUSE  And thanks for coming here.  This was the final presentation for the day.  APPLAUSE  Thank you.
Get_the_most_out_of_your_security_logs_using_syslogng_0_47.png Get_the_most_out_of_your_security_logs_using_syslogng_48_131.png Get_the_most_out_of_your_security_logs_using_syslogng_132_623.png Get_the_most_out_of_your_security_logs_using_syslogng_624_1271.png Get_the_most_out_of_your_security_logs_using_syslogng_1272_1943.png Get_the_most_out_of_your_security_logs_using_syslogng_1944_3131.png Get_the_most_out_of_your_security_logs_using_syslogng_3132_3515.png Get_the_most_out_of_your_security_logs_using_syslogng_3516_5135.png Get_the_most_out_of_your_security_logs_using_syslogng_5136_6767.png Get_the_most_out_of_your_security_logs_using_syslogng_6768_8015.png Get_the_most_out_of_your_security_logs_using_syslogng_8016_8351.png Get_the_most_out_of_your_security_logs_using_syslogng_8352_9479.png Get_the_most_out_of_your_security_logs_using_syslogng_9480_9803.png Get_the_most_out_of_your_security_logs_using_syslogng_9804_10787.png Get_the_most_out_of_your_security_logs_using_syslogng_10788_12191.png Get_the_most_out_of_your_security_logs_using_syslogng_12192_13163.png Get_the_most_out_of_your_security_logs_using_syslogng_13164_14075.png Get_the_most_out_of_your_security_logs_using_syslogng_14076_14771.png Get_the_most_out_of_your_security_logs_using_syslogng_14772_16667.png Get_the_most_out_of_your_security_logs_using_syslogng_16668_18935.png Get_the_most_out_of_your_security_logs_using_syslogng_18936_20099.png Get_the_most_out_of_your_security_logs_using_syslogng_20100_20663.png Get_the_most_out_of_your_security_logs_using_syslogng_20664_21947.png Get_the_most_out_of_your_security_logs_using_syslogng_21948_22739.png Get_the_most_out_of_your_security_logs_using_syslogng_22740_23063.png Get_the_most_out_of_your_security_logs_using_syslogng_23064_23795.png Get_the_most_out_of_your_security_logs_using_syslogng_23796_24347.png Get_the_most_out_of_your_security_logs_using_syslogng_24348_25247.png Get_the_most_out_of_your_security_logs_using_syslogng_25248_26147.png Get_the_most_out_of_your_security_logs_using_syslogng_26148_26459.png Get_the_most_out_of_your_security_logs_using_syslogng_26460_27083.png Get_the_most_out_of_your_security_logs_using_syslogng_27084_27911.png Get_the_most_out_of_your_security_logs_using_syslogng_27912_28559.png Get_the_most_out_of_your_security_logs_using_syslogng_28560_35639.png Get_the_most_out_of_your_security_logs_using_syslogng_35640_35711.png Get_the_most_out_of_your_security_logs_using_syslogng_35712_35761.png
Hello, welcome to the final session of the Security Dev Room. Let's welcome Peter, who will talk about Syslog NG. Thank you. First a few words about me. Sorry for those who were there yesterday in my talk. The first few slides will sound quite familiar. I'm Peter Tzenik from Hungary, community manager at Bellabit. Syslog NG is upstream developer. I'm working on packaging supporting Syslog NG. First of all, what is logging? It's a recording of events on a computer, for example, an SSH login message under Warlog messages in a file. You can see here Syslog NG itself is an enhanced login with a strong focus on high performance central log collection. Why central logging? First of all, it's ease of use. One place to check for your log messages instead of money. It's also availability. So even if the sender machine is down, you can read the logs at the central location. And last but not least, it's also security. That's the first thing in hacking a machine is to remove traces of what you have done. But if you have central logging, then all of the traces are forwarded in real time to a central location where you can check it what happened to the machine. There are four main roles of Syslog NG. First of all, it's collecting messages. It also can process log messages, filter them, and at the end, either store them locally or forward somewhere. The main role, the first role is data collection. And I say data and not log collection. Syslog NG can collect practically any kind of data. System messages, application logs together. And they can provide quite useful contextual data for either side. It can collect messages from a wide variety of platform specific sources like Devlog or Jornal or Sunstreams as many different platforms are supported. As a central log collector, it knows both the legacy and the new Syslog protocol and can use UDP, TCP, TLS, in any cases. It can also collect logs or any kind of data from applications through five socket pipes. And if the application is started by Syslog NG, then even application output. The next and in my opinion, the most important role, at least in a security environment, is log processing. Using Syslog NG, you can classify, normalize, and structure log messages. I will talk about this later. You can even rewrite log messages. And you don't have to think about falsifying messages here. But for example, in many cases, you need to analyze log messages due to compliance requirements. Messages can also be reformatted using templates. When log analysis software needs a specific format like a specific date format or have logs in JSON format and so on. And you can also enrich data. Using GIP or adding fields based on the message content. The next role is filtering. It has two main uses. First of all, discarding surplus logs like debug level messages or the other one is message routing. So you can make sure that the right messages are reached the right destinations. For example, you only send authentication-related messages to a SIEM system. There are many possibilities for filtering. It can be based on message content or parameters. Using comparisons, wildcards, and many different filtering functions. And best of all, any of these can be combined using Boolean operators. At the end, you can either store your messages locally or forward to many different destinations. Here comes a tricky question for those who don't check my slides right now. I mean, you're on a phone or so. Which Cyslogangy version is the most used? Do we have any idea? Project started a long time ago. Red Hat has version 3.5. The latest stable version, oh, it's actually 3.9. What do you think? Which is the most used version? Any guesses? You are the closest. It's version 1.6. There are over 100 million Kindle eBook readers and all of those run Cyslogangy for local action. Now, back to more serious topics. We have a lot of questions. What are the most used versions? You can run Cyslogangy for local action. If you take a look under Varlog Messages, you will see that most of the messages have the date, host name, and some kind of text at the end. The text part is an English sentence with some variable parts in it. It's quite easy to read by a human, but if you have a busy server with a thousand messages a second, it's quite difficult to deal with it. You don't have the time, and the machine cannot really understand it. There is a solution for this program. It's called Structured Logging. In this case, you answer presented as name value pairs. Back to my favorite SSH login example, you can describe the event as application, SSHD user root and IP address. The good news is that Cyslogangy has name value pairs inside right from the beginning. Facility, priority, and so on are all saved as name value pairs. This is the way how you could use them in filters or in templates. There are parsers in Cyslogangy which can turn unstructured and some structured data into name value pairs as well. Let's talk about these. The first one is the JSON parser. If you couldn't parse JSON with Cyslogangy, then you can see inside the message just simply store the whole JSON message somewhere. But as Cyslogangy can parse JSON messages, you can use the values inside the JSON message for filtering, for reformatting, for other destinations. So data is much more valued in this way. CSV parser can parse columnar data into fields. The most popular example is Apache Access Log. In this configuration snippet, you can see the column numbers. If you take a look here at the bottom of the screen, you will see that the username is used in a filename template. So each user has its own log file in this case. The key value parser was added just recently to Cyslogangy and it can parse key, equals, something like data. The most popular example for this is firewall logs. For example, IP tables, but this one is from Zorb, another firewall. The most interesting parser in Cyslogangy is patternDB, which can extract information from unstructured messages into name value pars. It can also add status fields based on the message content and do some message classification like log check does. The downside of this parser is that it needs an XML database describing the log messages. So it can only parse messages for which you have a description. Back to my favorite SSH example, here are a few fields. The first line has some data which is directly extracted from the log message, the application name, the username, source IP, but based on the message content, you can also add the action of the log was login and the status was a failure. And based on this, you can classify this log message as a violation. Cyslogangy can enrich log messages and create additional name value pars based on the message content. I already mentioned patternDB on the previous slide. You can also use a GUIP parser to add geolocation based on IP addresses, for example, a country name or longitude, latitude information. It can be used in security as well to detect anomalies, as it's not a real-world situation that someone logs in from Europe and from Australia at the same time. And you can also use it to display locations on a map. Recently, we added another parser which can add metadata from CSV files to log messages. For example, add the host role or contact person, which can considerably speed up login analysis and also helps to create more accurate alerts or dashboards. Next, I would like to talk a few words about configuring Cyslogangy. My first advice is don't panic. I often get the feedback that Cyslogangy configuration is very difficult. Yes, it looks difficult at first and sometimes even at second site, but once you take a deeper look at it, you will see that it's simple and logical. It follows a pipeline model where you have many different building blocks like sources, destinations, filters, and so on. And all of these can be connected into a pipeline using log statements. Cyslogangy configuration always starts with a version number for which version it was created and some global options. Most of these can be overridden later in the configuration. You can define sources. Here, as you can see, this is a single source definition, but it combines two different sources, internal messages. So those generated by Cyslogangy itself and system is for platform-specific log sources. This helps to hide the way platform-specific differences. So you can use the same configuration on Linux, FreeBSD, Solaris, and it will find log sources. The other one is a network source, in this case UDP. Here are a few destinations, the typical VAR log messages. It's a flat file. The other one is more interesting. It's an elastic search destination where you specify the index name, the type of the message cluster name, and here you can see that it's using JSON formatting and selecting different fields what to forward. Here you can see some filter definitions and loading a database for pattern-db parsing, and here is the part of the configuration, the log statements, where the upper one is for VAR log messages, local sources, related filter, and the destination. The other one is more interesting with multiple sources, filtering pattern-db, and sending the results to elastic search. And here is a nice dashboard from elastic search. It's a world map where the log source was IP tables, and the attack sources are visualized on screen. Here is the configuration, how you can do this. First of all, you need a key value parser to extract the different fields from the message. In this case, the source IP, for the source IP, the geolocation is looked up. As Kibana needs a specific format for it, it's rewritten to that format, and here in the log statement, all of these building blocks are combined. PEG is a software in heavy development. It was formerly known as ELSA. It's a web application for security analysts, and it's built on syslogang, and pattern-db message parsing. The results are sent to elastic search, and it has its own web interface. It's not Kibana. In this case, there are some nice graphs coming from an intrusion detection system. In a few words, what are new in syslogang 3.8? It has this base buffering, a new parser which is easier to use. You can also write parsers in Rust. It has support for all of the latest elastic search versions, and many, many performance improvements. Finally, a summary of what syslogang can help with security logs. First of all, it's high-performance and reliable log collection. You can parse and enrich log messages, so you can do basic log analysis inside syslogang, and you can also efficiently forward log messages to other log analysis tools. If you want to know more about syslogang, our central hub of information is syslogang.org, or our source code is on GitHub, where we also have an issue-tracking system. If you need help, we have a mailing list, or for near real-time help, we have IRC or Guitar Genos. Do you have any questions? Thank you, Peter. We have an applause. We have five minutes for questions, so raise your hand if you want to ask a question. We have the first one here. Hi. What about support for Docker containers or other containers? Instead of having LogStash or Fluent.de getting the container logs put into this, and then enriching the log format with container-relevant information? Has that supported yet? Yes. We have a Docker image, and I just checked it and was quite surprised as the development version of the Docker image had just a few hundred downloads, but the regular image had over 100,000 downloads. So people are using it in Docker. Sorry. I wasn't referring about running syslogang inside a container, but rather on a host or a VM where you have syslogang, and it has a Docker daemon, or it has Rocket or whatever, and you're running, say, 10 containers, and you'd like the logs from those containers to go back to the host base syslogang to enrich, to then send them on to some other location. So in order to do that, it would be useful for it to get some information from, say, Docker daemon to understand the name of the container and some other attributes. So as far as I know, Docker is logging in JSON format, and we can process that. I know that it's used, but personally, I don't have much experience with it. Okay, there's one more question here. Raise your hand if you want to ask a question. Hello. So my question, it gets more into, I suppose, the legal aspects, which I saw mentioned at the start of the talk in your brief. I suppose, are there ways which syslogang can support compliance to EU data protection law for not storing IP addresses or storing them in particular ways, which better support, I suppose, minimal standards in what should be captured and not captured? What we can do is, using pattern DB or using regular expressions, you can find IP addresses or credit card numbers in log messages. Both have advantages and disadvantages, and you can either replace with a static text the IP address or you can use hashes, and if you use a hash of the original, then you can use it in further analysis of the message. So you can follow sessions, what the given user did, even if you don't know the name of the user, but you can follow where it is coming from, what it did. We have one more here. We're trying to just have a comment or reflection or criticism or discussion. So most of the new assistants are running JournalD. What is your recommendation? Where is the work split between JournalD and syslogang? JournalD is for local log messages. Of course, you can pull log messages from it periodically, but it's still not something for central local action. With syslogang, you can constantly read log messages from the journal and forward it to a central location in real time. So it's better suited for security purposes, I think. Thank you. Final question? No. Okay, let's thank Peter. APPLAUSE And thanks for coming here. This was the final presentation for the day. APPLAUSE Thank you.
Hello, welcome to the final session of the logging? It's a recording of events on a it's ease of use. One place to check for your log done. But if you have central logging, then all filter them, and at the end, either store them application logs together. And they can provide quite useful contextual data for either side. It or Sunstreams as many different platforms are opinion, the most important role, at least in a structure log messages. I will talk about this have logs in JSON format and so on. And you can kind of text at the end. The text part is an It's quite easy to read by a human, but if you this program. It's called Structured Logging. In root and IP address. The good news is that Cyslogangy has name value pairs inside right from can turn unstructured and some structured data inside the JSON message for filtering, for popular example for this is firewall logs. For information from unstructured messages into name describing the log messages. So it can only parse based on the message content, you can also add additional name value pars based on the message add geolocation based on IP addresses, for information. It can be used in security as well from CSV files to log messages. For example, add dashboards. Next, I would like to talk a few starts with a version number for which version it This helps to hide the way platform-specific source was IP tables, and the attack sources are log messages to other log analysis tools. If you and you'd like the logs from those containers to address or you can use hashes, and if you use a hash of the original, then you can use it in further analysis of the message. So you can you don't know the name of the user, but you can