Blog

For my last blog we discussed a Splunk topic geared towards the Windows side of the shop (Splunking Microsoft Windows Firewalls). So now it’s time to show some love to the Linux admins out there. More specifically, in today’s blog we will explore some tips for gaining insight into Linux audit logs using Splunk.

A little background on the Linux Audit System

The Linux Audit system provides a way to track security-relevant information on your system. Based on pre-configured rules, Audit generates log entries to record as much information about the events that are happening on your system as possible. This information is crucial for mission-critical environments to determine the violator of the security policy and the actions they performed.

Clearly the Audit system is a very powerful and versatile tool and the data that it provides can be of great value for reporting and alerting within Splunk. Some of the more common use cases for Audit include:

• Watching file access, execution and modification
• Monitoring system calls
• Recording commands run by a user
• Monitoring network access
• Recording security events

Here is a great resource for information regarding Linux Audit.*

On-Boarding Audit logs

Linux audit logs should be ingested into Splunk using the Splunk Add-on for Unix and Linux. This TA should be installed and enabled to run on any Linux forwarder that you wish to collect audit logs for. In a distributed environment the Nix TA will also need to be installed on all Splunk search heads and indexers.

The input for the Nix TA that should be used is rlog.sh. This script tells the Splunk forwarder to monitor /var/log/audit/audit.log and to pass the events through the ausearch utility. The ausearch utility parses the event and provides field values in a more easily understandable format. The output of the script will then be forwarded to Splunk to be indexed.

Splunk_TA_nix/local/inputs.conf
[script://./bin/rlog.sh]
disabled = false

Searching for the data in Splunk

Once you have the data within Splunk you can easily search for the audit data via the following search. Of course you should adjust accordingly if the index has been customized.

Simple Search:

index=os sourcetype=auditd

Because Linux audit events can span multiple lines you may want a search that can group multi-line events together for the full context of the audit event. In order to do this we need a field within the events that we can transact on. The msg field is the best to use for this purpose.

msg=audit(time_stamp:ID)

The msg field provides a time stamp and unique ID of the record. Multiple records can share the same time stamp and ID if they were generated as part of the same Audit event.

Transaction Search:

index=os sourcetype=auditd | transaction msg maxspan=5s

Creating targeted Audit rules

Of course the true value of Splunking Linux Audit logs comes in creating targeted searches, alerts and reports based on specific security use cases.

As an example we will explore how to monitor for any writes and attribute changes to the file /etc/rsyslog.conf.

Step 1: Create the audit rule

In order for the Linux Audit System to keep track of edits to rsyslog.conf we need to provide an audit rule. This rule can be entered via edits to /etc/audit/audit.rules or via use of the auditctl command.

-w /etc/rsyslog.conf -p wa -k rsyslog_changes

Important: Notice the use of a custom key (“-k rsyslog_changes”) in the rule. This will allow us an easy field that we can use to search for these exact events within Splunk.

Step 2: Create the specific search

Now that we have added the audit rule we can create a targeted search that will provide us the full context for any time rsyslog.conf is modified as follows:

index=os sourcetype=auditd | transaction msg maxspan=5s | search key=rsyslog_changes

And here are the results of the search:

Screen Shot 2015-07-12 at 3.17.26 PM.png

At this point you could save this as a report, alert or scheduled search.

Conclusion

Hopefully this blog has peaked an interest from those Linux admins out there that were looking for ways to gain operational and analytical value from their audit logs using Splunk.

Thanks for reading and please leave a comment or email us at info@function1.com with any questions!
 
Link Credits

* https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6...