Part IV · Your turn
Chapter 8
Write your own loop
A complete research loop in two short Python files with no packages, run for real with its ledger, then the checklist and the pitfalls for pointing a loop at your own work.
On this page
In this chapter, we will build the smallest complete autoresearch loop: a sealed harness, one editable file, a ledger, and a program. It runs on any laptop with Python 3 in under a second a run. We will run the baseline and four experiments and show what they printed. Then the checklist for a real loop, the pitfalls the reports and our own ledgers agree on, and a set of questions to answer cold.
The harness: prepare.py
The harness owns the data, the metric and the budget, and prints the lines the loop greps. A hidden curve is sampled with noise. The agent's job is to learn it from 2,000 training points. The metric is the mean absolute error on 500 points the agent never sees. Lower is better. Fitting must finish in two seconds.
"""prepare.py: the fixed half of a toy research loop. READ-ONLY for the agent."""
import math, random, time
SEED = 7
N_TRAIN, N_VAL = 2000, 500
BUDGET_SECONDS = 2.0
def hidden(x):
return 0.6 * math.sin(2.1 * x) + 0.25 * x - 0.08 * x * x + 0.3
def dataset():
rng = random.Random(SEED)
def draw(n):
xs = [rng.uniform(-3, 3) for _ in range(n)]
return [(x, hidden(x) + rng.gauss(0, 0.15)) for x in xs]
return draw(N_TRAIN), draw(N_VAL)
def evaluate(fit):
"""fit(train) returns predict(x). Prints the summary block the loop greps for."""
train, val = dataset()
t0 = time.perf_counter()
predict = fit(train)
seconds = time.perf_counter() - t0
mae = sum(abs(predict(x) - y) for x, y in val) / len(val)
print(f"val_mae: {mae:.6f}")
print(f"fit_seconds: {seconds:.3f}")
print(f"within_budget: {'yes' if seconds <= BUDGET_SECONDS else 'NO'}")Forty lines, and all five properties from chapter 3: cheap (under a second), automatic, one number, lower is better, and honest, because the validation points are drawn inside the harness and never printed. The seed makes the scale the same every night.
The editable file: solve.py
The baseline predicts the mean of the training targets and nothing else. That is deliberately poor. The first run of a night is the baseline, as program.md says, and a poor baseline is still the right first line.
"""solve.py: the ONLY file the agent edits. fit(train) returns predict(x)."""
from prepare import evaluate
def fit(train):
mean = sum(y for _, y in train) / len(train)
return lambda x: mean
if __name__ == "__main__":
evaluate(fit)$ python3 solve.py val_mae: 0.408395 fit_seconds: 0.000 within_budget: yes
Four experiments, as run
Each experiment replaces fit, runs, and is kept or discarded by the ledger's rule. These were run on 10 October 2026; the output is what they printed. Before you read the ledger, predict which of the four loses.
Experiment 2, a straight line by least squares:
val_mae: 0.361450 fit_seconds: 0.000 within_budget: yes
Experiment 3, a polynomial of degree 5, fitted by solving the normal equations with Gaussian elimination, about twenty lines:
val_mae: 0.161001 fit_seconds: 0.003 within_budget: yes
Experiment 4, the average of the 25 nearest training points, with the points sorted once and found by binary search:
import bisect
from prepare import evaluate
K = 25
def fit(train):
pts = sorted(train)
xs = [x for x, _ in pts]
def predict(x):
i = bisect.bisect_left(xs, x)
lo, hi = max(0, i - K), min(len(pts), i + K)
near = sorted(pts[lo:hi], key=lambda p: abs(p[0] - x))[:K]
return sum(y for _, y in near) / len(near)
return predictval_mae: 0.129616 fit_seconds: 0.000 within_budget: yes
Experiment 5, the same with 2 neighbours instead of 25:
val_mae: 0.151605 fit_seconds: 0.000 within_budget: yes
And the ledger, written from those outputs:
experiment val_mae fit_seconds status description 1 0.408395 0.000 keep baseline: predict the mean 2 0.361450 0.000 keep straight line by least squares 3 0.161001 0.003 keep polynomial of degree 5 4 0.129616 0.000 keep average of the 25 nearest points 5 0.151605 0.000 discard average of the 2 nearest points [worse: 0.151605 vs 0.129616]
baseline to best 0.408395 to 0.129616 gain 0.278779, about 68% of the error gone keep rate 4 of 5, because the first four were the obvious ideas; the fifth was the first real test
Two neighbours lost to twenty-five because the data is noisy (the harness adds noise with a spread of 0.15) and two points average out less noise than twenty-five. The agent did not need to know that. The scale said 0.151605 against 0.129616, and the ledger says discard, with the reason in brackets as the cloner's program requires.
The program for the toy
## What you CAN do Modify solve.py. It is the only file you edit.
## What you CANNOT do Modify prepare.py. Install packages. Print the validation points.
## One experiment python3 solve.py > run.log 2>&1
grep "^val_mae:\|^fit_seconds:\|^within_budget:" run.log
## The ledger experiment val_mae fit_seconds status description (tab-separated)
## Keep or discard lower val_mae and within_budget yes -> keep; otherwise discard; equal and simpler -> keep
## NEVER STOP The human is asleep.
Point an agent at that file and the two Python files and go to bed. The next ideas are the ones a reader of prepare.py would have: more neighbours, a weighted average, a polynomial of degree 7, a sine term since the hidden function has one. Some will lose. The ledger will say which.
The checklist for a real loop
Build in this order, and do not start the loop until every line has a yes.
THE SCALE [ ] one number a script prints, with the direction (lower or higher) written down [ ] the harness is a file the agent is told never to edit, and the inputs it scores are not printed [ ] the baseline run twice: the gap between the two runs is written into the program as the noise THE SURFACE [ ] one file (or one folder) the agent edits, named in the program; everything else read-only [ ] no new dependencies, or a list of what is allowed THE BUDGET [ ] a fixed time per run, and a kill time (program.md: 5 and 10 minutes; the cloner: about 3 to 10 and 25) [ ] a cap for the night: hours, dollars, or experiments (the site: 180 minutes, $8, 6 experiments) THE LEDGER [ ] tab-separated, one row per run including crashes, a reason on every discard, a tag when the scale changes [ ] kept out of version control; the kept changes are the branch THE RULES [ ] keep: better metric and every gate passed; equal and simpler [ ] gates: the things that mean discard regardless of score (privacy, build, the result set unchanged) [ ] the simplicity rule, with an example at the size of your changes [ ] NEVER STOP, and where ideas come from when the list is empty [ ] nothing in the program requires asking a person THE MORNING [ ] a way to read the ledger and the diffs in ten minutes [ ] a way to send the result to visitors that the loop cannot run on its own
The pitfalls
Five failures that the reports and our own ledgers agree on.
the metric is gamed chapter 3; the harness leaks, or the metric rewards the wrong thing; seal it, re-run fresh overfit to the benchmark Lütke's own caveat; the check is inputs the loop never saw noise makes false keeps chapter 4; two runs, compare the mean, demand a minimum gain the agent runs dry program.md's answer: read the papers, re-read the files, combine near misses, go radical results tied to one box the README says so; the loop finds the best recipe for your machine, not for the world
And one that is ours. The scale is the hard part. The cloner rebuilt its harness twice in two days. The site added a gate that no script runs. The memory layer had to name its judge. Expect the first nights to be about the scale, and count that as the loop working.
Whiteboard it cold
If you can answer these without the book, you can run a loop.
1. Name the three files and who edits each. 2. State the nine steps of the loop. Which step is the only decision? 3. What are the five properties of a usable metric? 4. Why is the harness read-only? Give the one-sentence reason. 5. What is a false keep, what causes it, and what is the repair? 6. Quote the simplicity rule's three examples. 7. What does the never-stop paragraph say to do when ideas run out? 8. Give the keep rate of a strict metric on tuned code, and of a site being built, and say why they differ. 9. What does a ledger row need beyond the metric? Name three things. 10. What is the one command a loop that can deploy must not be able to run?
Try it yourself
Copy the two files, run the baseline, and write your own experiment 6 before you let an agent near it. Then give the
program to an agent with permissions off and the toy folder as its whole world, and read the ledger in the morning.
The folder with the files and the ledger is in the book's source under examples/toy.
Common questions
Is the toy too easy to teach anything?
It teaches the shape, which is the part people get wrong: the sealed scorer, the one file, the ledger with reasons, the baseline first. The first real loop you build will spend its first nights on the scale, exactly as the toy would if the noise were higher or the budget tighter.
What is the smallest real loop worth running?
One with a metric you already compute by hand. A test suite's run time, a page's size and speed, a prompt's accuracy on a fixed answer key, a query's p95. If you measure it already, the harness is half built.
How do I stop the loop spending money all night?
Put the cap in the harness or the scheduler, not in the agent's instructions. The site's loop is started by a scheduled job with a dollar budget, a minute budget and an experiment cap, and it cannot raise them.
What do I do on the first morning?
Read the discards first. Then the diffs of the keeps. Then change one line of the program, usually the noise rule or a gate, and run another night.
Carry this
- A complete loop is a sealed scorer, one editable file, a ledger and a program; the toy is two short files.
- Baseline 0.408395, best 0.129616, one discard with its reason: a real ledger from a real run.
- Build the scale first, run the baseline twice, write the noise into the program.
- Five pitfalls: a gamed metric, overfit, false keeps, running dry, results tied to one machine.
- The loop may stage; a person promotes.
Check yourself
1. In the toy, why are the validation points never printed?
Answer
Printing them would let the agent fit to the test. The harness draws them inside evaluate and reports only the mean error.
2. Why did 2 neighbours lose to 25?
Answer
The data carries noise with a spread of 0.15; averaging 25 points cancels more of it than averaging 2. The ledger shows 0.151605 against 0.129616.
3. What is the first thing to run on a new loop, and what is the second?
Answer
The baseline, as the first ledger line; then the baseline again, to measure the noise and write it into the program as the minimum gain.