perf Beyond top

This assumes you already know which process is the problem. top,
htop, and ip -br addr got you that far. The question now is why
that process is slow, and the answer lives below the level those
tools can see. Here is the layer I actually reach for.

CPU: Where the Cycles Go

perf top is the live view. It samples the running system and shows
which functions are consuming CPU right now, kernel and userspace
together, updated continuously. Unlike top, the unit is the
function, not the process.

perf top -g              # with call graphs
perf top -e cache-misses # sample on a different event

For a recorded profile you can analyze rather than watch, perf record followed by perf report:

perf record -F 997 -g -p <pid> -- sleep 30
perf report --stdio

The -F 997 is deliberate: a prime sampling frequency near 1000 Hz
avoids lockstep aliasing with periodic workloads that a round 1000
would beat against. -g captures call graphs so you see not just the
hot function but the path that reached it.

The output that actually communicates is the flame graph. Brendan
Gregg's toolchain turns a perf record into an SVG where width is
time spent and the vertical axis is stack depth:

perf record -F 997 -g -p <pid> -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg

One glance at the widths tells you where the time went. This is the
single highest-value CPU profiling artifact and it takes three
commands to produce.

The counter that separates a real diagnosis from a guess is
Instructions Per Cycle. perf stat reports it:

perf stat -p <pid> -- sleep 10

An IPC below roughly 1.0 on a modern core means the CPU is stalling,
usually on memory. The core is busy waiting, not computing. A high
cache-misses rate alongside low IPC is the signature of a workload
that is memory-bound wearing the costume of a CPU-bound one. This
distinction changes the fix entirely: optimizing the algorithm does
nothing if the problem is cache locality.

Memory: Beyond RSS

perf stat with memory events surfaces the cache hierarchy behavior
that RSS numbers hide:

perf stat -e cache-references,cache-misses,LLC-load-misses,\
dTLB-load-misses -p <pid> -- sleep 10

LLC (last-level cache) misses are the expensive ones: they go to
main memory, hundreds of cycles each. dTLB misses mean the page
table walker is working, which points at a working set too scattered
for the TLB to cover, often fixable with huge pages.

For allocation-level insight, the eBPF tools from bcc are the modern
answer. memleak attaches to the allocator and reports stacks that
allocate without freeing:

memleak-bpfcc -p <pid> 10

It shows the call stacks responsible for outstanding allocations,
which is the actual leak location, not just the fact that RSS is
climbing.

I/O: Per-Request Latency, Not Averages

iostat -xz 1 is the starting point, and the column that matters is
not throughput but await (average request latency) and %util.
But averages lie about I/O, because a device with a good average and
a terrible tail is a device that produces user-visible stalls the
average hides.

biolatency from bcc gives you the histogram instead of the average:

biolatency-bpfcc -D 10

The output is a latency distribution per disk. A bimodal
distribution, most requests fast with a cluster of slow outliers, is
the pattern that averages erase and that users feel. This is the
single most useful I/O diagnostic that the traditional tools cannot
produce.

biosnoop gives you the per-request trace, each I/O with its issuing
process, sector, size, and latency:

biosnoop-bpfcc

When you need to know which process issued the slow request and to
what offset, this is the tool. iostat aggregates that away;
biosnoop keeps it.

For the mapping from a filesystem operation back to the process and
file, ext4slower (or xfsslower, btrfsslower) traces filesystem
operations slower than a threshold:

ext4slower-bpfcc 10

Ten milliseconds and up, with the process, the operation, and the
filename. This is the tool that answers "the application is slow and
I think it is disk but I cannot prove it" with a definitive yes or
no.

The Practical Order

When a process is slow and you have already identified it:

Run perf stat first. The IPC and cache-miss numbers tell you in ten
seconds whether you are looking at a CPU-bound, memory-bound, or
stalled-on-something-else problem, before you invest in deeper
profiling.

If CPU-bound with good IPC, flame-graph it and read the widths.

If IPC is low with high cache misses, it is memory locality, and the
fix is in the data layout, not the algorithm.

If perf stat shows the process is not CPU-bound at all, it is
waiting, and the question moves to I/O: biolatency for the
distribution, then biosnoop or ext4slower to attribute the slow
requests to a process and a file.

The rule that saves the most time: measure which resource is the
bottleneck before optimizing any of them. perf stat for ten seconds
prevents the classic waste of optimizing CPU on a workload that was
blocked on disk the entire time. The traditional tools tell you a
process is busy. This layer tells you what it is busy with, which
is the only thing that leads to a fix.