The ratchet

In this chapter, we will learn how the keep-or-discard rule turns a night of mostly failed experiments into progress, why the fixed budget is what makes the rule safe to apply blindly, and how git becomes the agent's memory. Then we will break it: a noisy measurement lets bad ideas through, and the interactive counts them.

Why the rule can be applied blindly

Keep if lower, discard if not, is only a safe rule when every run is comparable to the last. The fixed five-minute budget is what makes them comparable. A bigger model and a smaller model, a large batch and a small one, are judged on what they manage in the same window on the same card. There is no "better but slower" to argue about. The README lists this as the second of its three design choices: the fixed budget "makes experiments directly comparable regardless of what the agent changes (model size, batch size, architecture, etc)".

The rule has a second half that is easy to forget: the budget also makes a discard cheap. A bad idea costs five minutes and one line. The loop can afford to be wrong eleven times out of twelve.

Git as memory

The branch holds what worked; the ledger holds what was tried. Every experiment is committed before it runs. A keep leaves the commit in place and the branch "advances". A discard is a git reset to the commit before. By morning the branch history is the exact list of kept changes, in order, each with the diff that made it, and the ledger beside it lists every attempt including the ones that were reset away.

branch   baseline --> LR 0.04 --> (reset) --> drop warm-up --> (reset) --> (reset) --> batch tweak --> ...
ledger   keep         keep        discard     keep             discard     crash       keep
                                  GeLU                         width x2    OOM

This is why the ledger is not committed and why rewinding is discouraged. Rewinding past a keep throws away a tooth. program.md allows it "very very sparingly (if ever)". The branch is the agent's long-term memory of what works, and the ledger is its memory of what does not. Neither is in the agent's context window, which fills and forgets; both are files, which do not.

The arithmetic of a night

Small gains, mostly failures, add up because the failures cost nothing but time. Take program.md's hundred runs, suppose one idea in seven helps, and suppose a helpful idea is worth 0.003 of val_bpb:

runs           100
help           100 / 7 = about 14
gain each      0.003
night's gain   14 x 0.003 = 0.042                   from 0.9979 to about 0.956

The reported numbers are in this range. VentureBeat's account of one of Karpathy's nights: 126 experiments, val_bpb from 0.9979 to 0.9697, a gain of 0.028. His two-day run: about 700 experiments, about 20 kept, and the kept changes together cut the "time to GPT-2" on his leaderboard from 2.02 hours to 1.80, an 11% gain. Twenty keeps in seven hundred is a keep rate under 3%. The ratchet does not need a good night. It needs a long one.

Where the ratchet slips

A noisy measurement lets a bad idea measure well once, and the loop keeps it. Call it a false keep. The branch advances, the ledger records a gain, and the model is worse. Nothing in the nine steps can tell, because the nine steps trust the number. The pawl is worn.

Our site loop met this on its first night. Its program says: "Speed is noisy (plus or minus 8 points run to run). Every candidate gets two runs; compare the mean." And it raised the bar for a keep from "lower" to "mean score higher by more than 0.5". The voice cloner fixes the random seed in its generation parameters for the same reason. Both are the same repair: shrink the noise, or demand a gain larger than the noise.

Before you move anything in the model below, predict how many false keeps a hundred candidates produce when the noise is a third of a typical real gain.

Two runs per candidate halve the candidates and cut the noise by a factor of the square root of two, about 1.4. Four runs quarter the candidates and halve the noise. Whether that trade is worth it depends on how much a false keep costs you. In a training run it costs a little of the metric. In a website it can cost a page that worked.

Changes that do not keep to themselves

Kept changes are tested one at a time, but they live together. Change A was kept against the baseline; change B was kept against the baseline plus A. Nobody tested A without B, or B without A, or either against tomorrow's model. paddo.dev's write-up lists this under the method's limits as "No isolation between improvements". The reported evidence that it is often fine is Karpathy's own: VentureBeat says the roughly 20 improvements "transferred perfectly to larger models". The reported evidence that it is not always fine is Tobi Lütke's caveat about his best number: "probably somewhat overfit". Chapter 6 has both.

Try it yourself

Run the same experiment twice without changing anything, in whatever loop you have. Write down the two numbers. The gap between them is your noise, and any keep smaller than that gap was luck. If you cannot run the same experiment twice, you do not have a harness yet; you have a demo.

Common questions

Why not keep the best of several runs instead of the mean?

Because the best of several noisy runs is biased upward: it rewards the candidate that got lucky most. The mean is what the candidate will do on an ordinary run, which is what you are going to ship.

Could the agent keep a worse run on purpose, to explore?

program.md does not allow it and the ratchet depends on it not happening. Exploration happens in the ideas, not in the keeps. An idea that needs two bad steps to reach a good one has to be tried as one change.

What if the metric improves and the memory doubles?

That is the soft constraint. program.md says some increase is acceptable for a meaningful gain and a dramatic blow-up is not. The ledger records memory next to the metric so the morning reader can see what each keep cost.

Does a false keep get found later?

Only if something tests it again: a re-baseline after a harness change, a transfer to a larger model, or a human reading the diffs in the morning. The ratchet itself never looks back.

Carry this

• Keep if lower, discard if not, is safe because the fixed budget makes every run comparable and every discard cheap.
• The branch is the memory of what worked; the ledger is the memory of what was tried. Both are files, not context.
• A long night of mostly failures adds up: twenty keeps in seven hundred gave an 11% gain, as reported.
• Noise causes false keeps. Repeat runs and a minimum gain are the repair; both cost candidates.
• Kept changes are tested in sequence, not in isolation. Transfer to a bigger model is the check.

Check yourself

1. Why is the ledger kept out of git while the experiments are committed?

Answer

So the branch history is exactly the kept changes, which makes it the memory of what works, while the ledger records every attempt, discards included.

2. What is a false keep, and what causes it?

Answer

A change kept because it measured better once while really being worse. Measurement noise causes it; repeated runs and a minimum gain reduce it.

3. Running every candidate four times instead of once changes the noise by how much, and the number of candidates by how much?

Answer

The noise halves (the square root of four) and the candidates per night fall to a quarter.