You’ve likely been there: it’s 2:00 AM, an alert triggers indicating a potential lateral movement within your production environment, and you realize with a sinking feeling that your standard syslog output is functionally useless. You can see that a user logged in, but you have no record of which files they touched, which privileged binaries they executed, or whether they tampered with the system clock to mask their tracks. This “visibility gap” is the primary reason security professionals lose sleep during an incident. We rely on logs to tell a story, but without a properly tuned Linux Audit Daemon (auditd), that story is missing its most critical chapters. The frustration isn’t just about the lack of data; it’s about the overwhelming complexity of the tool itself. Most practitioners look at a CIS (Center for Internet Security) Benchmark for Linux and see a wall of cryptic -a always,exit syntax that looks more like assembly code than a configuration file. There is a very real fear that turning on deep auditing will “break” the server, causing CPU spikes or filling up the disk in minutes. I’ve spent years tuning these systems, and I can tell you that this anxiety is valid—but it’s also solvable. We’re going to move past the generic “log everything” advice and build a surgical, high-performance auditing engine that satisfies the most stringent compliance auditors while keeping your SREs happy.

How Does the Linux Kernel Audit Subsystem Actually Work?

Before we drop a single rule into /etc/audit/rules.d/, we need to understand the plumbing. Unlike standard logging which often relies on applications voluntarily reporting their state, auditd hooks directly into the kernel. When a process makes a system call (syscall)—like open(), execve(), or unlink()—the kernel’s audit subsystem intercepts that call. This interception happens at the entry and exit points of the syscall. The rules you define via auditctl are loaded into a kernel-space matching engine. If a syscall matches your criteria (e.g., “someone is trying to write to /etc/shadow“), the kernel generates an audit record and sends it to the auditd daemon in user-space via a netlink socket. The beauty of this architecture is its integrity. Even if a root-level attacker modifies your shell history or deletes /var/log/auth.log, the kernel-level audit events have already been dispatched. However, the danger lies in the “context switch.” If you write a rule that monitors every single read() syscall on a busy database server, the kernel will spend more time generating audit events than actually serving data. This is why we focus on high-value targets: file system integrity, identity management, and privilege escalation.

Why Do Most Auditd Implementations Fail in Production?

The most common mistake I see is the “Compliance Copy-Paste.” A team takes a 500-line rule file from a GitHub repository, drops it onto a high-traffic web server, and wonders why the load average doubles. They are falling into the trap of auditing the noise rather than the signal. For example, auditing every file access (-a always,exit -F arch=b64 -S open) is a death sentence for performance. A CIS-compliant configuration is not about logging every action; it’s about logging every significant action. We need to distinguish between routine system operations and the specific syscalls that indicate a breach.

What Are the High-Priority Rules for CIS Compliance?

To build a gold-standard configuration, we focus on four critical pillars of system activity. These rules provide the highest ROI for security monitoring.

1. Monitoring System Identity and Authentication

The first thing an attacker does after gaining a foothold is attempt to create a backdoor user or modify existing permissions. We must watch the files that define who can access the system. -w /etc/group -p wa -k identity -w /etc/passwd -p wa -k identity -w /etc/gshadow -p wa -k identity -w /etc/shadow -p wa -k identity -w /etc/security/opasswd -p wa -k identity The -p wa flag tells auditd to watch for “write” and “attribute” changes. The -k identity is a key—a label that allows you to quickly search for these events later.

2. Tracking Kernel Module Loading

Rootkits often hide by inserting malicious code directly into the kernel. In a hardened production environment, kernel modules should rarely change after boot. -w /sbin/insmod -p x -k modules -w /sbin/rmmod -p x -k modules -w /sbin/modprobe -p x -k modules -a always,exit -F arch=b64 -S init_module -S delete_module -k modules By monitoring both the binaries (insmod) and the underlying syscalls (init_module), we catch both standard administrative actions and manual exploits.

3. Detecting Unauthorized Time Changes

Why monitor the clock? Because many security protocols (like Kerberos or TOTP) and log correlation engines rely on precise time. An attacker who can shift the system time can bypass authentication or make their logs appear to have occurred in the past or future, complicating the forensic timeline. -a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change -a always,exit -F arch=b64 -S clock_settime -k time-change -w /etc/localtime -p wa -k time-change

4. Privileged Command Execution (Setuid/Setgid)

Attackers love binaries with the setuid bit set (like sudo or passwd) because they run with root privileges. We need to log whenever these are executed by a non-privileged user. -a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k setuid -a always,exit -F arch=b64 -S execve -C gid!=egid -F egid=0 -k setgid

How Do You Stop Auditd from Killing Your Server Performance?

This is where the “Masterclass” level of configuration happens. Most people focus on the rules, but the directives at the top of the file are what keep the server alive. The -b (backlog) setting is critical. This is the number of audit records that can be queued in the kernel before they are passed to the user-space daemon. On a modern production server, the default (often 64 or 128) is laughably low. If the buffer fills up, the kernel has to decide what to do based on the -f (failure) flag. If -f 2 is set (panic), the entire system will crash to ensure no un-audited event occurs. This is great for high-security military systems, but disastrous for a SaaS platform. I recommend -f 1, which sends a warning to printk but keeps the system running. Finally, the -e 2 directive makes the configuration immutable. Once this is set, the rules cannot be changed without a reboot. This prevents an attacker who gains root from simply turning off the audit trail.

The Actionable Playbook: Implementing CIS Auditd Today

Follow these steps to deploy a production-ready audit configuration that balances security and performance. 1. Baseline the Current State: Check if auditd is running and see the current ruleset. auditctl -l 2. Configure the Daemon: Edit /etc/audit/auditd.conf. Ensure max_log_file is set to at least 100 (MB) and num_logs is sufficient for your retention policy (e.g., 10). Set flush = INCREMENTAL_ASYNC to minimize disk I/O impact. 3. Deploy the Rule Template: Create a new file at /etc/audit/rules.d/audit.rules. Use the following optimized structure: # Delete all previous rules -D # Set buffer size -b 8192 # Failure mode: printk -f 1 # Monitor identity changes -w /etc/passwd -p wa -k identity -w /etc/shadow -p wa -k identity -w /etc/sudoers -p wa -k identity # Monitor network config -a always,exit -F arch=b64 -S sethostname -S setdomainname -k network -w /etc/issue -p wa -k network -w /etc/hostname -p wa -k network # Monitor privileged executions -a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k priv_esc # Make the configuration immutable -e 2 4. Load and Verify: Restart the service. service auditd restart Verify the rules are active with auditctl -l. 5. Analyze the Output: Use aureport -s to get a summary of syscalls and ausearch -k identity to find specific events related to your keys.

How Do You Turn Raw Audit Logs into Actionable Intelligence?

Collecting logs is only half the battle. If you don’t have a protocol for analyzing them, you’re just burning disk space. I use a 3-step protocol for audit log analysis: Step 1: The Daily Summary (aureport) Run aureport --summary every morning. This gives you a high-level view of failed logins, executable events, and modifications. Look for anomalies—if you see 5,000 failed syscalls from a service account that usually has zero, you have a starting point. Step 2: The Keyed Investigation (ausearch) When an anomaly is found, use the keys we defined in our rules. If you suspect someone tampered with the network, run: ausearch -k network -i The -i flag is crucial—it interprets numeric values (like UIDs and GIDs) into human-readable names. Step 3: The Correlation Audit logs don’t live in a vacuum. Correlate the auditd event (which tells you what happened at the syscall level) with your application logs (which tell you who the application thought the user was). This is how you prove that a specific web request led to a specific file being modified on the OS.

Frequently Asked Questions

Does auditd affect system performance? Yes, but the impact is manageable. With a CIS-optimized ruleset and proper backlog buffering (-b 8192), the CPU overhead is typically under 3%. The primary bottleneck is disk I/O, which can be mitigated by using INCREMENTAL_ASYNC flushing and SSD storage. What is the difference between auditctl and audit.rules? auditctl is the command-line utility used to interact with the kernel’s audit subsystem in real-time. audit.rules is the persistent configuration file that auditd loads during boot to ensure rules survive a restart. How do I rotate audit logs to prevent disk exhaustion? Log rotation is handled in /etc/audit/auditd.conf. Parameters like max_log_file (size in MB) and num_logs (number of files to keep) control rotation. You should also set max_log_file_action = ROTATE to ensure the daemon doesn’t stop when the limit is reached.