kubectl debug

Distroless images are right for production: your app and its runtime,
nothing else. No package manager, no curl, no shell. Then something
breaks, you type kubectl exec -it pod -- sh, and it fails, because
there is no sh. kubectl debug is the answer — it attaches a
throwaway container with real tools to the running pod without
touching the image or restarting anything.

Run It Yourself

Deploy a distroless nginx. This is Google's distroless variant with
no shell inside:

1kubectl run web --image=gcr.io/distroless/base-debian12 \
2  --command -- /nginx -g "daemon off;"
3# In practice you'd use a real distroless nginx image; the point is:
4# no shell in it. Confirm exec fails:
5kubectl exec -it web -- sh
6# error: exec: "sh": executable file not found in $PATH

There it is — the failure that sends people down the wrong path. Now
attach an ephemeral debug container that shares the app's process
namespace:

1kubectl debug -it web --image=busybox:1.36 --target=web -- sh

You get a busybox shell running beside nginx, in the same pod,
sharing its network and process namespaces. nginx never restarted.

Inside that shell, the useful part is reading the distroless
container's world through /proc:

1ps aux                        # PID 1 is the nginx binary
2ls   /proc/1/root/            # nginx's root filesystem
3cat  /proc/1/root/etc/nginx/nginx.conf   # its actual running config
4ls   /proc/1/fd               # its open file descriptors

/proc/1/root/ is the app container's filesystem, seen from your
debug container. The image with no shell cannot browse its own files —
you browse them from next door.

Network Issues: Use netshoot

Swap busybox for nicolaka/netshoot, which carries curl, dig,
ss, tcpdump:

1kubectl debug -it web --image=nicolaka/netshoot --target=web

1curl -v localhost:80          # you share nginx's netns: localhost IS nginx
2ss -tulnp                     # confirm it's actually listening on :80
3dig kubernetes.default.svc.cluster.local

When the Pod Is Crash-Looping

Ephemeral containers need a running pod. A CrashLoopBackOff pod dies
too fast to attach. Copy it and override the entrypoint so the copy
stays up:

1kubectl debug web -it --copy-to=web-debug --container=web -- sh

You get web-debug with a shell instead of the failing command. The
original is untouched for its logs and events; the copy is a stable
place to run the entrypoint by hand and watch where it breaks.

Two Gotchas

Ephemeral containers are permanent — you cannot remove one, it stays
until the pod restarts. Exiting the shell ends the session, not the
container. That is fine; it costs nothing idle.

Set --profile explicitly on mixed-version clusters. If strace
fails with a permission error, you need a profile that grants
SYS_PTRACE: add --profile=general (or netadmin for packet
capture).

The Takeaway

Distroless in prod, tools on demand. Never ship a debug build, never
bake busybox into a production image. kubectl debug --target
attaches the tools when you need them, /proc/1/root gives you the
app's filesystem, --copy-to handles the crash-loopers, and the
ephemeral container evaporates on the next restart.