Stop Shipping Lucky Training Runs

Train the same small classifier twice, same data, same
hyperparameters, and you will get two different models. CUDA
nondeterminism, cuDNN kernel selection, the order the dataloader
happens to shuffle in: none of it is controlled unless you go out of
your way to control it, and even then not fully. On the small
focused classifiers I have been training, run-to-run variance of
several points is normal.

Most people train once. Whatever the run produced is what ships. This
is not a decision, it is a coin flip you did not notice you were
making.

The obvious fix is to train K seeds and pick the best one. That is
correct, and it is also where the interesting failure happens,
because "the best one" is where people quietly cheat.

The Temptation

You train ten seeds. You have ten accuracy numbers. You take the
highest one. Done.

Except accuracy is almost never the thing you actually care about. In
any classification task where one class of error is unacceptable —
fraud you must not miss, a malignancy you must not call benign, spam
that hides a real email in the junk folder — accuracy is a weighted
average that will happily trade away the errors that matter for the
errors that do not. A seed that is 96% accurate by being slightly
better on the easy majority class, while missing one more of the
thing you must never miss, is worse. Accuracy says it is better. It
is not.

If you select on accuracy, you will eventually select a seed that
bought its accuracy with exactly the failure you cannot afford. And
you will not notice, because the number went up.

The Fix: A Gate, Not a Ranking

The discipline is to name the one metric you refuse to compromise on
and make it a hard gate. Everything else is a tiebreaker.

For a spam classifier where the unforgivable error is a legitimate
email landing in the junk folder, the gate is recall on the
legitimate class. It must be 100%. Not 99.4%. Every seed that misses
even one real email is disqualified, no matter what its accuracy
looks like. Among the seeds that hold the gate, then and only then,
take the most accurate.

That is a lexicographic sort, and it is four lines of Python:

 1def score(rows) -> tuple[float, float]:
 2    """(recall on the must-not-miss class, overall accuracy) for one
 3    model's held-out rows."""
 4    must_keep = [r for r in rows if r.is_legitimate]
 5    recall = (
 6        sum(not r.junked for r in must_keep) / len(must_keep)
 7        if must_keep else 1.0
 8    )
 9    accuracy = (
10        sum(r.pred == r.gold for r in rows) / len(rows)
11        if rows else 0.0
12    )
13    return recall, accuracy
14 
15 
16def pick_best(results: list[dict]) -> dict:
17    """Recall is the hard gate (maximise first), accuracy only breaks
18    ties. Pure and testable, so the selection policy cannot silently
19    regress."""
20    return max(
21        results,
22        key=lambda r: (round(r["recall"], 4), round(r["accuracy"], 4)),
23    )

The tuple comparison does the work. Python compares the first element
first; accuracy is consulted only when recall is identical. The
rounding is deliberate: it stops a recall difference in the eighth
decimal place, which is noise, from overriding a real accuracy
difference.

The reason pick_best is a plain function that takes a list and
returns an element, rather than a method buried in a training loop
with a config object and a logger and a checkpoint writer, is that a
plain function can be unit tested. The selection policy is the thing
most likely to silently rot — someone adds a metric, someone flips a
comparison, someone "improves" the ranking — and if it is a pure
function, a test catches it.

 1def test_recall_beats_accuracy():
 2    lucky = {"seed": 1, "recall": 0.98, "accuracy": 0.97}
 3    correct = {"seed": 2, "recall": 1.00, "accuracy": 0.94}
 4    assert pick_best([lucky, correct])["seed"] == 2
 5 
 6 
 7def test_accuracy_breaks_ties():
 8    a = {"seed": 1, "recall": 1.00, "accuracy": 0.94}
 9    b = {"seed": 2, "recall": 1.00, "accuracy": 0.96}
10    assert pick_best([a, b])["seed"] == 2

Two tests. They encode the entire policy, and they will fail loudly
the day someone decides accuracy should come first "because the
recall numbers are all basically the same anyway."

Why the Gate Is a Minimum, Not an Average

Here is the part people get wrong even after they get the gate right.

You do not have one eval set. You have several: the held-out split,
the adversarial set someone hand-built, the slice from a different
source than your training data, the regression set of cases that
broke in production. If you average recall across them, a seed that
scores 1.00 on three sets and 0.95 on the fourth averages to 0.9875,
which looks fine, and it is not fine. It missed real emails on the
fourth set. The average hid it.

The gate has to be the minimum across every eval set. Every set
must hold the line independently. A seed that aces one slice and dips
on another has not held the gate, it has just found a slice it likes.

1def evaluate(model, eval_sets) -> tuple[float, float]:
2    """Recall is the MIN across every eval set (each set must hold the
3    gate on its own). Accuracy is the mean. A seed that wins on one set
4    but dips on another does not get selected."""
5    scores = [score(run(s, model)) for s in eval_sets]
6    recalls = [r for r, _ in scores]
7    accs = [a for _, a in scores]
8    return min(recalls), sum(accs) / len(accs)

Minimum for the gate, mean for the tiebreaker. The asymmetry is the
point. The gate is a claim about the worst case, so it takes the
worst number. Accuracy is a claim about typical quality, so the mean
is fine.

This one line is the difference between "this model never dropped a
real email on anything we tested" and "this model averaged pretty
well." Only the first is a statement you can put in front of someone
who has to trust the system.

An Aside: The Base Model Beat the Large One

While running this, the thing that surprised me was that the smaller
base model consistently beat the larger one on the task. Not
marginally. The large model overfit the training set, and the extra
capacity bought nothing on a task that was narrow and well-defined.

This is worth saying out loud because the reflex in 2026 is to scale
the model when the results disappoint. On a small, focused
classification task with a few thousand examples, the extra
parameters are capacity to memorize, not capacity to generalize. The
lever that actually moved the numbers was the data: more examples of
the confusing cases, cleaner labels on the boundary, and a couple of
adversarial slices added to the eval sets so the gate had something
real to hold against.

Scale the data before you scale the model. On a narrow task, the
model was never the bottleneck.

The Takeaway

Train K seeds, because one run is a coin flip. Name the one error you
cannot afford and make it a hard gate, because accuracy will trade it
away without telling you. Compute the gate as the minimum across
every eval set, because an average hides the slice where it failed.
Keep the selection policy a pure function with unit tests, because it
is the thing that will silently rot first.

And if the numbers disappoint, look at the data before you reach for
a bigger model.