Deep Dive How To Debug Syslog Ingestion for Sentinel and Log Analytics

Hello everybody, Simone here to tell you about a situation that happened many times to my customers: understanding how the syslog ingestion works.

To make subject clear make sure you have clear in mind the below references:

Most of the time nobody knows what needs to be collected and how hence, with this article, I just want to make some clarification on what is behind the scenes.

Starting from RFC, it is mentioned that we have a list of “Facility” like in the screenshot below:

sifriger_0-1608569858351.png

And for each of them we could have a specific “Severity” (see the corresponding picture below):

sifriger_1-1608569858358.png

Back to the situation, the natural question that comes up is: how can we clearly understand who is using who if we have no information about facilities and severities about related products we are using?

To find the information we need, we must capture some /UDP packets from the syslog server and rebuild the packets in wireshark and then analyze the results.

Let's start with first step: packets capture. Below you have the macro steps to be followed:

  • From the syslog server (in this case a server) we will use the tcpdump command,
    if not available follow this link on setup
    https://opensource.com/article/18/10/introduction-tcpdump
  • the command could be for example
    tcpdump -i any -c50 -nn src xxx.xxx.xxx.xxx (replace with source IPADDRESS under analysis)
  • the results after the rebuilt with wireshark, should be something similar the following image:sifriger_2-1608569858371.png

The header of every row contains exactly the information that we are looking for; deal with this piece of info? Easy; use the formula contained in the following part directly taken from RFC:

“The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity.  For example, a kernel message (Facility=0) with a Severity of Emergency (Severity=0) would have a Priority value of 0.  Also, a “local use 4” message (Facility=20) with a Severity of Notice (Severity=5) would have a Priority value of 165.  In the PRI of a syslog message, these values would be placed between the angle brackets as <0> and <165> respectively.
The only time a value of “0” follows the “<” is for the Priority value of “0”.  Otherwise, leading “0”s MUST NOT be used.”

In the example above, we have the value of <46>. According to the above-mentioned RFC, the formula used to translate that number into something human readable is the following:

8 x facility + severity

We now must look for the formula result in the following matrix:

EmergencyAlertCriticalErrorWarningNoticeInformationalDebug
Kernel01234567
user-level89101112131415
mail1617181920212223
system2425262728293031
security/auth3233343536373839
message4041424344454647
printer4849505152535455
news5657585960616263
UUCP6465666768697071
clock7273747576777879
security/auth8081828384858687
FTP deamon8889909192939495
NTP96979899100101102103
Log Audit104105106107108109110111
Log Alert112113114115116117118119
Clock120121122123124125126127
local0128129130131132133134135
local1136137138139140141142143
local2144145146147148149150151
local3152153154155156157158159
local4160161162163164165166167
local5168169170171172173174175
local6176177178179180181182183
local7184185186187188189190191

So now, let's make one step back to customer' question and “guess” what the “Facility” and the “Severity” are in the provided example.

Since header was 46, the result was:

  • Facility = message
  • Severity = Informational

Once we understood what to deal with, it's time to configure Log Analytics / Sentinel enabling the Syslog data sources in Azure Monitor.

All we have to do is to:

  • add the facilities (by entering its name and leveraging the intellisense) to the workspace.
    sifriger_3-1608569858377.png
  • select what severity(ies) to import.
    sifriger_4-1608569858407.png
  •  and click Save.
    sifriger_5-1608569858411.png

Using some real-life example, if we want to collect the logs for FTP, the corresponding facility to be entered is “ftp” and the following logs will be imported:

Syslog fileLog Path
ftp.info; ftp.notice/log/ftplog/ftplog.info
ftp.warning/log/ftplog/ftplog.warning
ftp.debug/log/ftplog/ftplog.debug
ftp.err; ftp.crit; ftp.emerg/log/ftplog/ftplog.err

Differently, talking about Users, the facility is “user” and the imported logs will be:

Syslog fileLog Path
user.info;user.notice/log/user/user.info
user.warning/log/user/user.warning
user.debug/log/user/user.debug
user.err;user.crit;user.emerg/log/user/user.err

Another one: for Apache, the facility is “local0” and the logs will be:

Syslog fileLog Path
local0.info;local0.notice/log/httpd/httpd.
local0.warning/log/httpd/httpd.warning
local0.debug/log/httpd/httpd.debug
local0.err; local0.crit;local0.emerg/log/httpd/httpd.err

We have everything in place, but are we really sure that info is produced?
What if you would like to effectively test that data is flowing in the corresponding facility?
We can leverage the following sample commands for CEF & Syslog using the logger built-in utility:


logger -p auth.notice “Some message for the auth.log file”

logger -p local0.info “Some message for the local0.log file”

logger “CEF:0|Microsoft|MOCK|1.9.0.0|SuspiciousActivity|Demo suspicious activity|5|start=2020-12-12T18:52:58.0000000Z app=mock suser=simo msg=Demo suspicious activity externalId=2024 cs1Label=tag cs1=my test”

Note pay attention to time when you query for this result!!! ;)

That's it from my side, thank you for reading my article till the end.

Special thanks go to Bruno Gabrielli for review

Simone

 

This article was originally published by Microsoft's Azure Blog. You can find the original article here.