Recovering Deleted Binaries from /proc

A surprising number of senior Linux engineers, including some who
have run production fleets for a decade, do not know this one. A
binary that has been deleted from disk while a process is still
running can be recovered byte for byte. The kernel keeps the inode
alive until the last reference closes. The /proc/PID/exe symlink
is a real, readable, copyable handle to it.

The scenario is more common than you would think. A botched
deployment removes the binary before the service is restarted. A
package upgrade replaces a library while a long-running process
holds the old version. A piece of malware deletes itself after
launch to make analysis harder. In every one of these cases, the
process is still alive and the bytes you want are still in the
kernel.

The recovery is a single command.

1$ ps aux | grep myservice
2me   12847  ...  /opt/myservice/bin/myservice
3 
4$ ls -la /opt/myservice/bin/myservice
5ls: cannot access ... : No such file or directory
6 
7$ cp /proc/12847/exe /tmp/recovered-myservice
8$ file /tmp/recovered-myservice
9/tmp/recovered-myservice: ELF 64-bit LSB executable, dynamically linked, ...

The file is gone from disk. The process is still running. The copy
out of /proc/12847/exe works because the kernel resolves the
symlink to the actual inode, which is still present in memory and
on the underlying filesystem until the last open file descriptor
closes.

The same trick works for any open file descriptor, not just exe.
/proc/PID/fd/N is a symlink to whatever the process has open at
descriptor N. A configuration file that was overwritten can be
read from there. A log file that was deleted can be recovered if
the process still has the descriptor open.

The next time someone tells you that a deleted file is gone, point
them at /proc. The kernel is more forgiving than the filesystem
suggests.