Linux From Scratch Again

bash-5.3# cat /etc/os-release
NAME="Linux From Scratch"
VERSION="13.0-systemd"
ID=lfs
PRETTY_NAME="Linux From Scratch 13.0 (systemd)"

That is a machine I built by hand, from source, one package at a
time. The last time I did this was seven years ago. Coming back to
it after that long is its own kind of measurement.

The Return

The first time I built Linux From Scratch, I was following the book
the way you follow a recipe in a language you half-speak. I got to
the end, I had a working system, and I understood maybe sixty
percent of what I had done and why. The other forty percent was
"the book said to, and it worked." That is a legitimate way to
learn, and it taught me a great deal, but there were steps I
executed on faith.

Seven years later I decided to do it again, on 13.0 with systemd,
and the experience was completely different. Not because LFS
changed, though it has evolved. Because I changed. The steps I once
executed on faith, I now execute with understanding. The toolchain
bootstrap that was once a mysterious sequence of "compile this,
then use it to compile that" is now a thing whose every stage I can
explain: why the cross-toolchain is isolated from the host, why the
temporary tools exist, why the final system has to be built against
itself and nothing else. Seven years of compiling kernels, debugging
init systems, and maintaining real infrastructure filled in the
forty percent.

What Seven Years Changes

The value of doing LFS a second time, well apart from the first, is
that it is a benchmark you run against yourself. The book is roughly
the same. The task is roughly the same. What is different is
everything you bring to it, and the difference is legible in a way
that daily work never quite is.

The first time, the C library build was a step. This time, I
recognized what glibc's configure flags were choosing and why, and
when the test suite reported the handful of expected failures, I
knew which ones were normal instead of trusting the book that they
were. The first time, the kernel .config was mostly defaults with
a few changes the book suggested. This time it was a set of
deliberate choices, because I have spent seven years learning which
options matter for the hardware I run. The first time, systemd
coming up was a relief. This time it was unremarkable, because I
have debugged enough systemd boots to know exactly what has to be
in place for it to work.

None of that recognition existed seven years ago. It accumulated
quietly, one real problem at a time, and I had no clean way to see
how much of it there was until I ran the same task again and felt
how much less of it required faith.

The One Lesson Worth Passing On

There is a single practical habit from this build that I would give
to anyone attempting LFS, and it is the one the book does not
emphasize enough: back up your working build before you touch it.

An LFS system reaches a state where it boots and works, and then you
start making changes, adding packages, rebuilding the kernel with
new options, reconfiguring the init. Any one of those changes can
leave you with a system that no longer boots, and unlike a
distribution, there is no rescue ISO that knows how to repair an LFS
box. The system is yours in the most complete sense, which means the
recovery is yours too. The only recovery that reliably works is
restoring the last state that booted.

So before I make any change to a working build, I image the root
partition:

1DATE=$(date -u +%Y%m%d-%H%M%S)
2sudo dd if=/dev/nvme0n1p5 bs=4M status=progress \
3  | gzip -c > "/home/chris/lfs-root-backup_${DATE}.img.gz"

A block-level image of the whole partition, compressed, timestamped.
If the next change breaks the boot, the recovery is to gunzip and
dd the image back onto the partition, and the system is exactly
where it was before I broke it.

The honest caveat, because it matters: imaging a mounted, live root
partition captures an inconsistent snapshot. Open files, dirty
buffers, and a journal mid-transaction are all frozen in whatever
state they happened to be in. The restored image will almost always
come back and replay its journal cleanly, but "almost always" is not
"always." The correct version of this backup is taken from outside
the running system, booted from a live USB with the LFS partition
unmounted, which guarantees a consistent image. The live-system
version above is the pragmatic one I use for quick checkpoints
between changes, and it has never failed me, but I keep the
unmounted version for anything I actually cannot afford to lose.

The principle is the one that runs through everything I have written
about backups: the recovery path has to exist before you need it,
and a change to a working system is exactly the moment you are most
likely to need it. An LFS build is fragile by nature and
unrecoverable by any tool but your own foresight. Image it before you
touch it.

Why Do It At All, Twice

LFS was never about the artifact. The system it produces is a
minimal Linux that does less than a fresh Debian install, and nobody
keeps an LFS box as a daily driver for the capability. I did not
build this to use it. I built it the second time for the same reason
I built it the first: because assembling the whole stack from source,
by hand, is the most direct way to keep the abstractions honest.

Every layer I stand on daily, Debian's toolchain, its libc, its
kernel packaging, its init, is built by people who understood it so
I would not have to. That is the entire value of a distribution, and
I am not romantic about giving it up. But there is a specific kind of
understanding that only comes from doing the bootstrap yourself, and
it decays if you never refresh it. Seven years is long enough that a
refresh was overdue. Doing it again reground the things that had
slowly become abstract again through disuse.

Closing

The system in the screenshot will sit in the homelab as a reference,
the way the last one did before it. My actual machines keep running
Debian, because Debian is the right tool and LFS never pretended to
be. The artifact is not the point, this time any more than last time.

The point is what the second build measured. Seven years ago I
followed the book and trusted it. This time I followed the book and
understood it, and the gap between those two experiences is the
clearest picture I have of what seven years of doing this work
actually added up to. You cannot see that kind of growth in the
day-to-day. It hides inside problems you solve without noticing they
would have stopped you cold years earlier. Running the same hard
task twice, far enough apart, is one of the few ways to make it
visible.

I will probably not wait another seven years for the next one. Now
that I have felt how much the second build revealed, the benchmark
itself is worth running more often than that.