eBPF - The Kernel Stopped Being a Black Box

For most of its history the Linux kernel was a black box you observed
from the outside. You could read what it chose to expose through /proc
and a handful of interfaces, and if you wanted it to do something it did
not already do, you had two options, both bad. Write a kernel module,
which runs with full privileges and can crash the entire machine if you
get a pointer wrong. Or get your change accepted upstream and wait years
for it to land in the kernel people actually run. The kernel was a wall.
You worked around it.

eBPF turns that wall into a door. It lets you load small programs into
the running kernel, attach them to events as they happen, and have them
run safely, without a module, without a patch, without a reboot. That
one sentence is the whole story, and it is why a technology most people
cannot quite define is simultaneously eating Kubernetes networking,
performance tracing, and runtime security. Those look like three
unrelated fields. They are three consequences of the same capability.

What It Actually Is

eBPF is a small virtual machine inside the kernel. You write a tiny
program, it gets compiled to eBPF bytecode, and you attach it to a hook:
a syscall entering or leaving, a network packet arriving, a kernel
function being called, a tracepoint firing. When the event happens, your
program runs, right there in kernel context, and can observe the event,
record it, or in some cases change what happens next.

The part that makes this safe, and the part worth actually
understanding, is the verifier. Before the kernel loads your program, it
runs a static analysis that proves the program cannot do the dangerous
things. It cannot loop forever, because the verifier requires that it
provably terminates. It cannot read memory it should not, follow a bad
pointer, or crash the kernel, because the verifier rejects any program
whose safety it cannot prove. If the analysis does not come out clean,
the program never loads at all. That is the trade that makes the whole
thing possible: you get to run your code in the most privileged place in
the system, in exchange for proving to the kernel's satisfaction, ahead
of time, that it cannot misbehave.

This is the difference from a kernel module in one line. A module you
trust because you wrote it carefully. An eBPF program the kernel trusts
because it checked. One of those scales to running untrusted,
dynamically loaded code in production; the other is how you take the
whole box down at 3 a.m.

Why That Is a Big Deal

Make the kernel safely programmable at runtime and a lot of problems
that used to require heroics become ordinary. You can ask the running
kernel a question it was never instrumented to answer, get the answer,
and detach, all without restarting a single process. You can insert
logic into the network path without routing traffic through a userspace
proxy. You can watch every process creation on a busy host and act on
the suspicious ones as they happen rather than reading about them in a
log afterward.

The clearest way to see the shift is through the three worlds it is
taking over, because in each one eBPF is replacing an older approach
that everyone had simply accepted as how things were.

Networking: Cilium

In Kubernetes, the classic way a Service routed traffic to Pods was
kube-proxy, which programmed a growing thicket of iptables rules on
every node. It worked, and it scaled badly: thousands of services meant
thousands of linear rules, re-evaluated constantly.

Cilium, the eBPF-based networking layer that graduated from the CNCF
back in 2023 and is now the default dataplane under several managed
Kubernetes offerings, throws that model out. It does the load balancing
and policy enforcement in eBPF programs attached directly to the network
path, using hash-table lookups instead of a linear rule walk, and it can
replace kube-proxy entirely. Connecting this to the last post: where
nftables filters packets as they traverse the stack, Cilium reroutes and
load-balances them below that, in eBPF, often before they ever reach the
parts of the stack a traditional firewall sits in. Its observability
companion, Hubble, then shows you the actual flows and which policy
allowed or denied each one, which is visibility the iptables era never
had. The networking did not get a new feature. It got a new foundation.

Observability: bpftrace

The second world is tracing, and it is the one you can try this
afternoon. bpftrace is a high-level language over eBPF for asking the
live kernel questions. A one-liner like

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

counts, live, which programs are opening files and how often, on a
running production box, with no code changes and nothing restarted. When
you detach, the instrumentation is gone and the overhead with it. This
is the dynamic cousin of the perf work I wrote about earlier: perf
samples what is running, bpftrace lets you attach precise logic to exact
events and aggregate in the kernel. The reason this matters is that the
hardest production questions are the ones you did not instrument for in
advance, and eBPF is the first tool that lets you answer those without
shipping a new build first.

Runtime Security: Falco and Tetragon

The third world is the one closest to my day job. Back in the auditd
post, the whole point was that the kernel sees every syscall and auditd
writes down what happened so you can read it after an incident. eBPF
takes that from past tense to present tense: you can watch the syscalls
as they happen and act.

Two tools lead here and they make a clean contrast. Falco observes
syscalls, through eBPF probes, and evaluates them against a mature
library of YAML rules with readable, semantic conditions. It is
detection-first and has the broader set of prebuilt rules and the more
mature alerting and SIEM integrations, so it is the easier on-ramp.
Tetragon, built on eBPF and part of the Cilium stack, works closer to
the raw kernel hooks and can do something Falco's model does not emphasize:
block a syscall in real time, inside the same eBPF program that detected
it, before it completes. That is the jump from "you will find out a
malicious exec happened" to "the malicious exec does not happen." The
cost is that its policies sit nearer the kernel level and are harder to
write than Falco's named rules. Detection with a gentle on-ramp, or
in-kernel enforcement with a steeper one, is a real and useful choice to
have.

The Honest Limits

Three things keep eBPF from being magic, and the honest post names them.

You almost never write it by hand. The verifier that makes eBPF safe is
also strict and unforgiving, and raw eBPF development is genuinely hard,
full of programs rejected for reasons that take experience to read. For
nearly everyone, eBPF is something you consume through Cilium, Falco,
bpftrace, and their peers. The tooling is the product; the raw layer is
plumbing most people should not touch.

It is tied to the kernel. eBPF programs hook into kernel internals, so
portability across kernel versions was historically painful. The CO-RE
approach, backed by BTF type information, made writing-once-running-many
workable, but it remains a real constraint: an older kernel gives you
less, and some capabilities simply are not there before a given version.
eBPF rewards a current kernel.

And it cuts both ways. A technology that can observe every syscall and
sit invisibly in the network path is, from another angle, an excellent
place to hide. eBPF-based malware and rootkits are a real and growing
category, precisely because the same power that lets a security tool
watch everything lets an attacker's program watch everything and stay
below the tools looking for it. A door in the wall is a door for whoever
reaches it. That is not a reason to avoid eBPF; it is a reason to watch
what loads eBPF programs on your hosts as carefully as you watch anything
else privileged.

The Takeaway

You do not need to write eBPF to benefit from it, and most people never
will. What you need is to recognize that it is the common foundation
under a surprising amount of your modern stack: the CNI moving your
Kubernetes traffic, the tool answering your hardest production question
live, the agent deciding whether a syscall completes. The kernel stopped
being a black box you work around and became a platform you can program,
safely, at runtime. Know what is standing on that door, pick tools that
use it well, and keep an eye on what else might walk through it.