In my last post entitled Forwarding Syslog to Azure Log Analytics we setup our Linux VMs to send Syslog data for centralized collection to Azure Log Analytics. This allowed us to capture all our Syslog data as well as setup alerts for anomalous behavior in our logs. In this post we want to take it a step further and add auditd to our Linux VMs and setup some very specific auditing rules around file access and modifications and send those to Log Analytics as well.
Recently I was asked to design a solution that would monitor accesses to sensitive files and directories in Linux and capture that activity in Log Analytics. Linux doesn’t, by default, report this type of access but with a little help from the Linux Audit system and a few packages we can start setup rules to monitor file accesses and lots of other factoids from a Linux system.
Linux Audit System
The Linux Audit System can be enabled to expose detailed audit information for just about everything on a Linux system. You can track security related events, detect misuse or unauthorized activities and even drill into process details if you so desire. However, I will caution that the volume of data produced by the audit system can be quite a large just from the default audit rules. Logrotate is enabled by default for auditd but just be aware you should always put /var/log on a separate mountpoint just in case…. filling up a root filesystem is always a bummer of a day.
In this post we will focus on 3 key components of the Linux Audit System:
- auditd: User-space daemon which gathers information based on the defined rules and generates the audit.log file with the resulting information.
- audispd: Dispatcher daemon which is used to sent events to other programs on the system like Syslog
- auditctl: Utility from controlling and interacting with the auditd daemon
There are a few other components that comprise the Audit system but we won’t go into any detail here on ausearch or aureport since we’ll use Log Analytics for searching and reporting.
In this post I’m using CentOS 7.5 so the installation steps will be yum based but similar steps are available for Debian based systems like Ubuntu.
To install the audit system packages and enable the default rules you can execute
>> sudo yum install -y audit audit-libs
This will also install the auditctl program that allows you to read and manage the audit system rules. Let’s take a look at what rules are enabled by default. Let’s also look at the layout of the configuration directory for auditd.
>> auditctl -l No rules >> cd /etc/audit >> find . . ./audit-stop.rules ./auditd.conf ./rules.d ./rules.d/audit.rules ./audit.rules >> cat audit.rules ## This file is automatically generated from /etc/audit/rules.d -D -b 8192 -f 1 >> cat ./rules.d/audit.rules ## First rule - delete all -D ## Increase the buffers to survive stress events. ## Make this bigger for busy systems -b 8192 ## Set failure mode to syslog -f 1 >>
From this we see that there are currently no rules defined for auditd. We also looked at the /etc/audit directory. In this directory you have the audit.rules file. This is the file that contains the system’s persistent audit rules. However, you shouldn’t modify /etc/audit/audit.rules directly as this file is dynamically created by aggregating all the rules that are defined in files that end with the .rules extension in the /etc/audit/rules.d directory. Currently there is only one file in /etc/audit/rules.d directory and it doesn’t contain any rules currently.
Note above that I said persistent rules are in the /etc/audit/audit.rules file. That’s because you can add rules at any time using auditctl utility, however those rules are not persisted unless you also add them to /etc/audit/rules.d
Let’s go ahead and create some simple rules. The first rule will watch for any changes to the /etc/hosts file of our system. Our second rule will watch all the files in the /etc/syconfig/ directory for any changes or additions to the directory.
>> auditctl -w /etc/hosts -p wa -k hosts_file_change >> auditctl -w /etc/sysconfig/ -p wa -k sysconfig_file_change >> auditctl -l -w /etc/hosts -p wa -k hosts_file_change -w /etc/syconfig/ -p wa -k syconfig_file_change
So now when we execute auditctl -l we see that our audit system has a rule that is listed. If you notice we also passed a -k option to auditctl. The -k option is a key that will be used in the log file to allow us to easily search the audit.log for changes related to this rule.
Let’s make a change to the /etc/hosts file and see what gets logged in our /var/log/audit/audit.log file.
>> tail -f /var/log/audit/audit.log type=SYSCALL msg=audit(1579470434.614:79433): arch=c000003e syscall=82 success=yes exit=0 a0=15f9710 a1=161df60 a2=fffffffffffffe80 a3=7fff74e8e760 items=4 ppid=104824 pid=108103 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=6098 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="hosts_file_change"
There are a few more lines that were added to the log but this is the one that provides the most information. If we break this down you’ll see that hosts_file_change was used as the key in the log so we could easily identify that this SYSCALL was related to the rule we previously added. We also can see that the executable used to change the file was /usr/bin/vi. Additionally, we see that the root user was the one that made the change thanks to the uid=0 and gid=0 fields. One of the REALLY useful pieces of information in this line though is the auid field. This field is the uid of the original user that logged in to the system. So even though the root user made the change to the file, we know that before the user was root they logged into the system using the user associated with uid 1000.
So now that we’ve tested our rule and know we are getting the audit logs we want, We can persist the rule so that it’s enabled on reboots. We do this by adding the rule to the /etc/audit/rules.d/audit.rules file like so…
>> cat /etc/audit/rules.d/audit.rules ## First rule - delete all -D ## Increase the buffers to survive stress events. ## Make this bigger for busy systems -b 8192 ## Set failure mode to syslog -f 1 ## Custom rules added here -w /etc/hosts -p wa -k hosts_file_change -w /etc/syconfig/ -p wa -k sysconfig_file_change
Audit Logs to Syslog
We’ve added a custom audit rule to our system and have it logging to our audit.log. Now we need to send our audit logs to Log Analytics so we have a central repository for all our auditing needs.
The Linux Audit System comes with audispd which is the audit log dispatcher. This dispatcher can be configured to dispatch logs to Syslog using one of the custom Syslog facilities (ie local0 - local6). We’ll use local0 in this example. In order to start sending logs to Syslog we’ll need to modify the configuration for the audit dispatcher at /etc/audispd/plugins.d/syslog.conf. The key fields to modify are active and args.
>> cat /etc/audisp/plugins.d/syslog.conf # This file controls the configuration of the syslog plugin. # It simply takes events and writes them to syslog. The # arguments provided can be the default priority that you # want the events written with. And optionally, you can give # a second argument indicating the facility that you want events # logged to. Valid options are LOG_LOCAL0 through 7, LOG_AUTH, # LOG_AUTHPRIV, LOG_DAEMON, LOG_SYSLOG, and LOG_USER. active = yes direction = out path = builtin_syslog type = builtin args = LOG_INFO LOG_LOCAL0 format = string
At this point you’ll need to restart the auditd subsystem and then you’ll be streaming audit logs to Syslog. Once the audit logs are streaming to Syslog you’ll need to reconfigure the omsagent on your Linux VMs to forward the local0 facility of Syslog to your Log Analytics workspace. You can reference Forwarding Syslog to Azure Log Analytics if you aren’t familiar with how this can be done.
At this point we now have a central location for analysis of our audit logs in Log Analytics. We can run reports and setup alerts based on the incoming Syslog records. To validate that our logs our accessible we can modify our /etc/hosts file on our Linux VM and then query the Log Analytics workspace as shown. You should find records similar to what you saw in /var/log/audit/audit.log.