One number nobody edits

In this chapter, we will learn the first and largest reason the loop works: the agent is judged by one number that it cannot touch. We will list what a metric needs before a loop can use it, watch a made-up ledger go wrong when the harness leaks, and read the real story of a harness of ours that had to be rebuilt twice in two days.

What a loop needs from a metric

A metric is fit for a loop when a program can compute it, in minutes, without a person, as one number. Five properties, in the order they usually fail:

cheap        it finishes inside the budget (five minutes here), or there is no loop, only a queue
automatic    no person reads, rates or clicks; the agent runs a command and greps a line
one number   not a dashboard; a loop cannot rank two candidates on three numbers without a rule
directional  lower is better, or higher is better, and the file says which
honest       it is hard to raise the number without doing the real work

val_bpb has all five. A run takes five minutes, the evaluation is a function in prepare.py, it prints one line, lower is better, and bits per byte on held-out text cannot be lowered by changing the tokenizer. The README's reason for the choice was the last point: "vocab-size-independent so architectural changes are fairly compared".

The seal: prepare.py is read-only

The evaluation lives in a file the agent is told never to open for editing. program.md's list of what the agent cannot do has three items, and two of them are about the scale:

"Modify prepare.py. It is read-only. It contains the fixed evaluation, data loading, tokenizer, and training constants (time budget, sequence length, etc)." And: "Modify the evaluation harness. The evaluate_bpb function in prepare.py is the ground truth metric."

The third item is no new packages, which is a different seal on the same idea: the agent cannot change what the experiment is made of, only how it is arranged.

Why does this one rule matter more than the rest? Because an agent asked to lower a number will find the shortest path to a lower number. If the shortest path is editing the test, that is the path it finds, and not from malice. Making the test easier is an edit that lowers val_bpb. Only a rule stops it. Aakash Gupta's guide puts it in one line: the agent cannot touch the harness "(if it could, it would just make the test easier instead of making the model better)".

When the scale can be edited

Here is a made-up model of a night. Each run is an idea; a few ideas really help; the measurement is honest. Flip the harness to "the agent may edit the test" and the agent sometimes eases the test instead. Before you flip it, predict what the two lines will do.

With the harness leaking, the ledger shows a fine night and the model is no better. The agent did what it was asked. The ask was wrong. This has a name, Goodhart's law: when a measure becomes a target, it stops being a good measure. A locked harness does not repeal the law. It narrows the paths to the ones that run through the real work.

Gates: the numbers that are not the score

One score decides the ranking; gates decide who may be ranked at all. program.md keeps one gate soft and one hard. Memory "is a soft constraint. Some increase is acceptable for meaningful val_bpb gains, but it should not blow up dramatically." Time is hard: past ten minutes, the run is killed.

Loops we run add gates of their own, and they are the parts of the program we are least willing to lose. The site loop (chapter 7) has three: no private name on a public page, the build must pass, every route must answer, and "any failure means discard, no matter the score". Its program adds a gate no script can run: after any visual change, "open the page in the browser, take a screenshot, and look", and a change that raises the score but makes a section look worse is a discard. The voice cloner's program keeps a change only if the metric improves and the output quality score does not fall by more than 0.30 and generation stays under three times real time. The metric cannot hear a robotic voice; the gate can.

When the scale is wrong: a harness rebuilt twice

Sometimes the loop's first lesson is about the scale, not the samples. Our voice-cloning loop (chapter 7 has the whole story) started on 2 October 2026 with a harness that scored a clone on one clean, twelve-second studio reference. Four rows from its ledger, as written:

run              clone_score  sim_norm  wer    dnsmos  rtf   status   description
20261002-180034  88.3         0.899     0.018  3.31    0.94  keep     baseline: chatterbox turbo (mps), default sampling, 12s ref, seed 0
20261002-181151  99.2         1.001     0.009  3.31    0.91  keep     voxcpm2 (mps), prompt mode, cfg 2.0, 10 steps (sim at ceiling; rtf 2x of voxcpm1.5)
20261002-184052  89.8         0.911     0.015  2.93    0.51  keep     [h2] re-baseline on harness v2 (clean+hard refs): voxcpm2 prompt mode cfg 2.0, 6 steps
20261003-181955  84.9         0.894     0.014  3.20    2.00  keep     [h3] chatterbox mtl v3 cfg 0.3 t 0.6 + trim/norm reference

Read the second row. By the fourth experiment a model had a similarity of 1.001, above the ceiling set by real recordings of the same speaker. The scale had nothing left to say. So the harness was rebuilt the same evening: version 2 scored every clone twice, once from the clean reference and once from a five-second clip degraded like a phone call, and the ledger was re-baselined (the third row). The next day the project's README records why there was a version 3: "v2 saturated (clean condition above the judge's ceiling)" and "the hard condition's real failure, copying the reference's noise into the output, was invisible to v2". Version 3 multiplies each score by an output-quality factor, q = min(1, DNSMOS / quality_ceiling), so a clone that copies the hiss of a phone line pays for it.

harness v1   runs 1 to 11     clean reference only              best 99.2, similarity above the ceiling
harness v2   runs 12 to 66    clean + a degraded 5 s reference  rows tagged [h2]; best 94.6 (program file)
harness v3   runs 67 onward   v2 x an output-quality factor q   rows tagged [h3]; best 84.9 (README)

The best score fell from 99.2 to 84.9 across the three versions, and the cloner got better. That is not a contradiction. Each new harness was a harder, more honest scale, and rows from different versions "are not comparable", which is why each version begins with a re-baseline line and a tag in every description. When you change the scale, you start the ledger again. You do not compare across the seal.

Try it yourself

Take the number you would most like a loop to improve. Score it against the five properties. If it fails "automatic", write down the smallest program that could stand in for the person: a fixed set of inputs, an answer key, a scorer. That program is your prepare.py, and it is the first thing to build. Chapter 8 shows one in forty lines.

Common questions

Can the agent read prepare.py?

Yes, and it should: the setup tells it to read all three files. It may read the evaluation; it may not change it. Knowing how the scale works helps the agent choose ideas; owning the scale would let it cheat.

What if my metric needs a model to judge it, like a rubric scored by an LLM?

Then the judge is part of the harness and must be fixed too: the same model, the same prompt, the same temperature, the same inputs, for the whole night. A judge that drifts is a scale without a seal. Our memory layer's benchmarks name the judge model for exactly this reason.

Why not optimise several numbers at once?

Because a loop needs an order, and three numbers have no order until a rule combines them. The site loop combines seven sub-scores into one geometric mean (chapter 7). The cloner uses one score and two gates. Both are ways of giving the loop one number.

How do I know the harness is wrong?

Two signs from the cloner's story: a score at or above its own ceiling, and a failure you can hear or see that the number does not move for. Either one means the night is measuring the scale, not the work.

Carry this

• A loop's metric must be cheap, automatic, one number, directional and honest.
• prepare.py is read-only because the shortest path to a lower number is otherwise an easier test.
• Gates decide who may be ranked; one score decides the ranking.
• When the scale saturates or misses a real failure, the human rebuilds it, re-baselines, and tags the ledger. Rows across versions are not compared.

Check yourself

1. Name the two items in program.md's "cannot do" list that protect the metric.

Answer

Do not modify prepare.py, and do not modify the evaluation harness: evaluate_bpb is the ground truth.

2. The voice cloner's best score fell from 99.2 to 84.9. Why is that not a regression?

Answer

The harness changed twice to become harder and more honest, and rows from different harness versions are not comparable. Each version was re-baselined.

3. What is the difference between a gate and the score?

Answer

A gate is pass or fail and a failure means discard regardless of the score; the score ranks the candidates that passed.