There are not enough human hands and eyes to monitor what is happening in an operating system, and even less for a whole corporate network where the flow of events can exceed 3000/sec, even more so. Many operating systems already have standard logging utilities pre-installed that record what is happening in the system, for Linux it is syslog and any kind of its implementation, and for Windows it is Windows Event Logs.
But often the functionality of standard solutions is not enough, for example in Linux syslog there are no audit policies that would determine what of all the information should be collected and what should be ignored. There are alternative solutions just for that!
How does Auditd work?
Auditd is a logging subsystem that allows you to use audit policies, kernel module and other software to log events on the OS. The composition of such a subsystem is a combination of:
- auditctl client utility, which passes to the configuration daemon, new rules;
- auditd daemon, which occupies file sockets to listen to auditctl calls and logs from the kernel;
- ausearch utility to search the logs for parameters;
- /etc/audit configuration files + plugins + audit policies or rules.
Since the execution of almost any action on the OS is accompanied by a reference to the kernel through system calls, the process of logging the actions of subobjects in the OS is as follows.
For example, a process with SSH server has decided to occupy a network socket and informs the kernel about it through the socket() system call. The latter, in its turn, processes its request and generates a record specifying all the main parameters of the call, after which it saves it in the audit buffer. From there the logs are sent via IPC or Inter Process Communication algorithm to the file socket, which is occupied by the auditd daemon. Inside it the process of filtering by rules already takes place and it selects what should be saved from the received data.
If necessary, the filtered packets can be sent to the collection servers, which will pass the data to SIEM or other protection systems. But in this article we are going to look exclusively at the subsystem on the local host.
How to install and configure Auditd?
To install it is enough to use the base repository, where the auditd binary is already included, the installation for different distributions will be different. For deb-like distributions you need to use the command:
For rpm-like distributions, commands like:
After that, the auditd daemon for listening to logs and processing them will be automatically raised:
The whole subsystem is deployed automatically and independently of the system administrator. When we are sure that the required service has been successfully launched, let's move on to configuring the policies that will determine what we will store from the mass of existing events. To do this, let's open the rules/policies directory, in your case you can name the rule by any convenient name, but only with the .rules file format:
Consider each of the lines presented in the default config:
- -D option removes previously added rules to the service memory, to avoid conflicts;
- -b option sets the size of the audit buffer for events before writing a file to the log;
- -backlog_wait_time 60000 option sets the maximum time (in milliseconds) that the audit daemon will wait for events to be sent if the buffer is full. In this case, it waits up to 60 seconds before forcibly clearing the log;
- -f 0 - use minimum message level (silent);
- -f 1 - recording only critical events (compressed output);
- -f 2 - more detailed messages (default).
And also the audit policy rule itself, which will filter out the necessary events from a heterogeneous array of events.
How to write rules for auditd?
There are two types of rule writing in auditd, the first type is for monitoring system calls to files and directories:
This rule allows registering requests to work with a file system object in write and attribute modification mode. The -w option specifies the path to the monitored file, the -p option specifies the rwx file attributes to monitor, and the -F option filters. If all these options are true, the event is written to the log file of the service!
Or a more extended format of the second kind, where the system call and the filter applied to the event will be explicitly specified:
Where the -a (append) option will add a new rule to the end of the list, the always/never action determines whether to log the event or not. The -S determines the type of system call, here are the main ones:
- open - accessing a file system object;execve - loads the application into the process memory. It can be used to track software startup, because the first fork() system call creates a
- process, and execve loads files into the memory of the new process;
- socket - call to create a socket for inter-network/inter-process communication between applications;
- listen - call to start listening for incoming requests on the socket;
- setreuid/setregid - call implements privilege escalation/reduction for a process.
Real-world use cases
For tools such as SIEM/EDR/Zabbix, etc. that allow monitoring the performance and security of the existing infrastructure, logs from machines are required. Consider a task where you need to track who ran an infected Docker container by elevating permissions from normal user to root.
Let's trace the system calls to start the container with the command sudo strace -o ss.txt <malicious_command>:
The output is quite large, using the grep utility you can filter by key system calls from the ss.txt file:
As we can see, two processes with two PIDs 2545, 2547 were started, which loaded executables from the /usr/bin/sudo|docker path. Let's write a rule for the found event:
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/sudo -F auid!=0
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/docker -F auid!=0" | tee -a /etc/audit/rules.d/audit.rules && auditctl -R /etc/audit/rules.d/audit.rules
After executing the command to load rules into the logging subsystem kernel module, it is necessary to check the correctness of the performed operation:
At the end we can see our loaded rules, so the operation was successful! Let's start docker from sudo and commit the logs in the standard file:
After filtering through the many recorded events in the saved logs folder, we can see system calls to execve with an AUID or real UID other than root. This event may indicate an IS incident that violated the policy, and therefore may be an attack! Logs are used for proactive detection and mitigation of threats, as well as monitoring the state of services and the system as a whole.
Generalising the algorithm of service operation, let's specify that when a process is running in the OS, it generates a lot of requests to the kernel, which can be tracked. The audit policy allows selecting the most necessary and important events out of millions of events, which can then be transferred to analytical tools or used for debugging the system!