auditd

Why Normal Logs Miss It

A symlink your service needs is gone. The service is down. Who
deleted it, and when?

The logs cannot say. syslog records what applications write. The
journal records what systemd units emit. Neither records that a
process called unlink on a path, as which user, from which parent.
That happens in the kernel, in the syscall path, below every
application log. Nothing watched, so nothing was recorded.

auditd watches at that level. It does not log what programs say. It
logs what the kernel does.

How It Works

The kernel has an audit subsystem. It records events at the syscall:
a file opened, written, renamed, unlinked, a process executed, a
permission changed. It records them whether or not the program wanted
to be logged, because the hook is in the syscall, not in the process.

auditd is the daemon that pulls these events from the kernel and
writes them to disk. auditctl loads the rules that tell the kernel
what to watch. A watch on a path fires an event every time a syscall
touches it, and the event carries the syscall, the pid, the uid, the
parent pid, the working directory, and the timestamp.

That is the difference from application logging. A program can lie or
crash before it logs. It cannot opt out of the syscall record. The
kernel saw it happen and wrote it down from underneath.

The Symlink Case

Symlinks disappear for dull reasons. A deploy script runs cleanup but
skips create. A package upgrade drops a link and never replaces it. A
cleanup job with a wide glob catches one it should not. Someone runs
rm in the wrong directory.

The symptom is identical every time. The link is gone, the service is
broken, no application log explains it, because from the app side
nothing happened. The path was just missing.

Without a watch you recreate the link and move on. You never learn
what removed it, so the cause survives and does it again next month.

With a watch, the log has the process that called unlink, the user,
the parent, the timestamp. One line. You stop recreating the link and
kill the script that keeps eating it.

When To Watch

The cost scales with what you watch, so watch little.

Watch a path when something keeps changing or deleting it and you need
the responsible process. Watch sensitive files, credential stores,
auth config, sudoers, cron dirs, because those are what attackers
modify and misconfigurations corrupt. Watch privileged operations
when you need a record of who ran them.

Do not watch everything. The kernel can audit every syscall on the
box. Do that and you fill the disk, add latency to every operation,
and bury the five events you wanted under millions you did not. A log
too noisy to read is worse than none. It looks like monitoring and
monitors nothing.

Keep it off hot paths. A rule that fires on every write to a busy
directory adds overhead to every write. auditd belongs on paths that
are sensitive, not paths that are busy.

The Catch

It is forensic. The value is entirely past tense, and you collect the
evidence before you know you need it. The log that solves the symlink
mystery only exists if the watch was live when the deletion happened.
Turning auditd on afterward tells you nothing about what already
occurred. It only helps with the next one.

So decide before the incident. Pick the few paths and operations worth
a permanent record and watch those. On most boxes that is a short
list. Write it anyway. The alternative is finding out mid-outage that
the one record that would have explained everything was never kept.

Normal logs tell you what your programs said. auditd tells you what
the kernel saw. It is not a tool for the crisis. It is a tool you
arm so the crisis is not a mystery.