Backup at 64 KB/s

When the Backup Takes Longer Than the Data Retention

Some backup configurations are slow. Some backup configurations are so
slow they are technically not backups, because the rate at which they
move data is lower than the rate at which the data being backed up
changes. A weekly snapshot that takes ten days to upload is not a
weekly snapshot. It is a never-completing transfer that the operator
periodically restarts and hopes will catch up.

A chunk-store offsite backup in the homelab spent some weeks in
exactly this state earlier this year. The local store was
approximately sixty-six gigabytes of deduplicated chunks. The
destination was a commodity SFTP-capable storage box at a major
European provider, accessed at the time via a CIFS mount. The
transport on top of CIFS was rsync, scheduled nightly with a
generous timeout.

The measured throughput was approximately sixty four kilobytes per
second. Sustained. Not "burst", not "during the initial scan", not
"on Sundays when the link is busy." Sixty four kilobytes per second.
The arithmetic: sixty-six gigabytes at sixty four kilobytes per
second is roughly twelve days. The nightly timeout was eight hours.
Most nights the backup made progress somewhere in the high tens of
megabytes before being cut off. The estimated date of completion
for the first full sync, calculated honestly, was approximately
never.

The replacement, deployed two weeks ago, runs at approximately
twenty four megabytes per second sustained on the same data, the
same network, the same off-site provider. The full first sync of
the sixty-six gigabytes took forty-seven minutes. Subsequent
incremental nightly runs complete in approximately five minutes,
most of which is spent enumerating what is already on the remote.

The ratio is roughly two hundred eighty. The change was the
protocol.

What CIFS Is Actually Doing

CIFS, also known as SMB, is a network filesystem protocol that
was designed for office file sharing on local area networks in the
late 1980s, refined through the 1990s, and adopted by Linux clients
in the early 2000s through Samba. It is excellent at what it was
designed for. A user opening a Word document from a file server is
the canonical workload, and CIFS handles it competently.

The way CIFS handles file operations is the reason it is the wrong
choice for chunk-store backup. Every file operation in CIFS
requires a request-response exchange between the client and the
server. Opening a file is CREATE. Writing to it is WRITE. Closing
it is CLOSE. Each of these is a separate network round trip. The
SMB2 protocol introduced compound requests that can bundle several
operations together, but in practice the kernel CIFS client does
not aggressively compound, and many operations remain serialized.

For a single large file, this overhead is invisible. CIFS can
saturate a gigabit link uploading a multi-gigabyte ISO because the
WRITE operations are large and the per-file overhead is amortized
across millions of bytes. For many small files, the overhead
dominates. Each file costs at minimum one CREATE round trip, one
or more WRITE round trips, and one CLOSE round trip, regardless of
how small the file is.

The network in question has a round trip time of approximately
twenty milliseconds to the off-site provider. At twenty
milliseconds per CIFS operation, a single rsync stream that
processes one file at a time can complete approximately sixteen
files per second after accounting for stat calls, open, write,
and close. If the files are small, this is the bottleneck. A
chunk store with files averaging four kilobytes hits exactly this
case. Sixteen files per second times four kilobytes is sixty four
kilobytes per second, which is where the measured throughput came
from.

This is not a configuration problem. It is not "CIFS tuning."
There is no SMB option that fixes this within the protocol's
design constraints. The protocol assumes a small number of large
files, and any chunk store violates that assumption.

Why SFTP Is Different

SFTP is the file transfer protocol that runs over SSH. It was
designed for general-purpose file transfer over wide-area
networks, with explicit attention to high latency. The design
choice that matters most for backup workloads is that SFTP
supports request pipelining. A client can send multiple read or
write requests without waiting for each individual response. The
server processes them in order and returns responses as they
complete. The round trip cost is paid once across many operations
rather than once per operation.

For the same network and the same files, SFTP can move data at
the rate the underlying bandwidth allows rather than at the rate
the round trip latency caps. A 4 KB file no longer requires a 20
ms round trip. It shares the round trip with the next several
files queued behind it.

This is the protocol-level difference. The choice of client tool
on top of SFTP then adds another dimension of speed: parallelism
across files. rclone with --transfers 32 opens thirty-two
concurrent SFTP sessions and uploads thirty-two files in parallel.
Each session benefits from SFTP pipelining within itself. The
total throughput is the product of pipelining within each
connection and parallelism across connections.

The result on the homelab workload, measured carefully on the
same hardware, was twenty four megabytes per second sustained for
small-file workloads. Bandwidth-bound rather than protocol-bound.
The actual ceiling is the remote storage's ability to accept
parallel writes, not the network's ability to carry them.

Why SSHFS Is Not the Answer

The intermediate choice is SSHFS, which mounts an SSH endpoint as
a local filesystem. SSHFS would have been roughly an order of
magnitude faster than CIFS for the same rsync workload, because
SFTP under SSHFS does support pipelining within a session.

SSHFS does not solve the actual problem, for two reasons. The
first is that SSHFS is still a single connection. rsync running
on top of SSHFS issues operations serially through the mount.
There is no parallelism across files. The pipelining inside one
session helps, but it does not approach the throughput that
multiple concurrent sessions achieve.

The second is that SSHFS introduces its own caching layer that
interacts oddly with rsync's metadata operations. Attribute
caching, kernel page cache, and SSHFS's own buffering combine to
produce inconsistent performance characteristics that are
difficult to reason about. The direct-rclone path bypasses the
mount entirely. Operations go straight from the local filesystem
through SFTP to the remote. The filesystem semantics that SSHFS
is faithfully implementing are simply not needed for this
workload.

The rclone Configuration That Matters

The specific rclone invocation for chunk-store sync was a result
of measurement, not first principles. The flags that mattered:

--transfers 32 is the parallelism knob. The right value is the
number of concurrent uploads the remote can sustain without
contention. For commodity SFTP storage, thirty-two is usually
safe. The provider rarely complains. A higher value can produce
diminishing returns or rate-limit responses depending on the
provider's policy.

--checkers 16 is the number of concurrent file-existence
checks. For an incremental run, most of the wall-clock time is
spent checking which files are already on the remote. Sixteen
concurrent checkers move this phase from minutes to seconds.

--multi-thread-streams 4 is per-file parallelism. For a few
large files in the chunk store, splitting each into four
parallel streams adds throughput without contending with the
small-file transfers.

--size-only is safe for content-addressed chunk stores. The
files are named by SHA-256 hash. Two files with the same name
and the same size have identical content by construction. There
is no need to compare modification times, no need to compare
checksums beyond the implicit hash in the filename. The
comparison phase becomes a directory listing, which is fast.

--no-update-modtime and --sftp-set-modtime=false skip a
post-upload SFTP round trip per file that would otherwise be
spent setting modification time. For small-file workloads, this
is most of the per-file overhead that remains after the
pipelining and parallelism. Disabling it adds another two to
three times on top of the underlying speedup.

The combination produces the twenty-four megabytes per second
sustained number. Each flag is worth measuring on its own to
understand which contributes how much, and the answer varies by
workload, but the combination is the right starting point for
any chunk-store sync over commodity SFTP.

When CIFS Is Still Right

The point of this post is not that CIFS is bad. CIFS is excellent
for what it was designed for. The cases where CIFS is still the
right protocol include desktop file sharing on a local network,
single-large-file workloads where per-file overhead is irrelevant,
Windows interoperability where SMB is the native protocol,
authenticated access by domain users, and any case where the
filesystem semantics CIFS provides (locking, change
notifications, ACLs) are actually needed.

What CIFS is not good for is chunk-store backup. Or any backup
workload dominated by file count rather than file size. Or any
workload where the round trip latency to the remote is more than
a few milliseconds and the file count is more than a few
thousand. Each of these conditions independently produces the
sixty-four-kilobytes-per-second failure mode. All three together
produce the can-never-finish failure mode.

The diagnostic is simple. Look at the average file size of what
you are backing up. If it is less than approximately one
megabyte, and the file count is greater than approximately ten
thousand, and the round trip is more than approximately ten
milliseconds, CIFS is the wrong protocol and you will discover
this only when the backup window starts overlapping with the
next backup.

Closing

The replacement of CIFS with rclone-over-SFTP for chunk-store
backup is the kind of operational decision that does not show up
in any vendor's comparison matrix. CIFS and SFTP are both
file-transfer protocols. Both work over the same SSH or SMB
service. Both are widely supported. The difference is in how each
protocol handles many small operations across a high-latency
link, and that difference is what determines whether the backup
finishes.

The lesson is to test the backup at the file count and file size
it will actually face, on the actual network, with the actual
tools you intend to use. The two hundred eighty times speedup
that the rclone path produced was sitting on the table the entire
time the CIFS path was failing. No new hardware. No new bandwidth.
No new storage. Just a protocol that pipelines.