The loop, step by step

In this chapter, we will walk through program.md the way the agent does: the setup it performs once, the seven steps it repeats all night, the ledger it writes, and what it does when a run crashes or runs long. By the end you will be able to play the night shift yourself, and the interactive will let you.

The setup, once

Before the first experiment, the agent and the human agree on a run tag, and the agent makes a branch for the night. program.md lists the steps:

1. agree a run tag from today's date, for example mar5; the branch autoresearch/mar5 must not already exist
2. git checkout -b autoresearch/mar5          from the current master
3. read README.md, prepare.py and train.py
4. check that ~/.cache/autoresearch/ holds the data shards and the tokenizer
   (if not, tell the human to run: uv run prepare.py)
5. create results.tsv with only its header row
6. confirm with the human, and go

Then the first run is always the same: "Your very first run should always be to establish the baseline, so you will run the training script as is." The baseline is the first line of the ledger. Every later line is compared with the best line so far.

The seven steps

Every experiment, all night, is the same list. program.md heads it "LOOP FOREVER". Here it is, lightly compressed; the quoted parts are the file's own words.

LOOP FOREVER
  1. look at the git state: which branch, which commit
  2. "Tune train.py with an experimental idea by directly hacking the code"
  3. git commit
  4. uv run train.py > run.log 2>&1        "redirect everything; do NOT use tee or let output flood your context"
  5. grep "^val_bpb:\|^peak_vram_mb:" run.log
  6. empty grep output means the run crashed: tail -n 50 run.log, read the trace, try a fix;
     "If you can't get things to work after more than a few attempts, give up"
  7. write one line to results.tsv
  8. "If val_bpb improved (lower), you 'advance' the branch, keeping the git commit"
  9. "If val_bpb is equal or worse, you git reset back to where you started"

Nine lines, and the only decision in them is step 8 against step 9. Notice step 4. The output of a training run is thousands of lines. The agent reads two of them. A loop that lets its own logs into its context fills up by midnight and starts forgetting the morning's lessons.

What the run prints

The script ends with a fixed summary block, and the loop greps it. program.md shows the block, with a note that the numbers will differ by platform:

val_bpb:          0.997900
training_seconds: 300.1
total_seconds:    325.9
peak_vram_mb:     45060.2
mfu_percent:      39.80
total_tokens_M:   499.6
num_steps:        953
num_params_M:     50.3
depth:            8

Two of these lines decide the night. val_bpb is the metric. peak_vram_mb is the soft constraint: "Some increase is acceptable for meaningful val_bpb gains, but it should not blow up dramatically." The rest is context for the next idea: how many steps fit in the budget, how big the model was, how busy the GPU was (mfu is model FLOPs utilisation, the share of the card's arithmetic that did useful work).

The ledger

results.tsv has five tab-separated columns and one line per experiment, crashes included. The file says why tabs: "commas break in descriptions". Here is the example from program.md:

commit   val_bpb   memory_gb  status   description
a1b2c3d  0.997900  44.0       keep     baseline
b2c3d4e  0.993200  44.2       keep     increase LR to 0.04
c3d4e5f  1.005000  44.0       discard  switch to GeLU activation
d4e5f6g  0.000000  0.0        crash    double model width (OOM)

commit       the short git hash, 7 characters
val_bpb      six decimals; 0.000000 for a crash
memory_gb    peak_vram_mb / 1024, one decimal; 0.0 for a crash
status       keep | discard | crash
description  short text; for a discard, why

Two details are easy to miss. The ledger is not committed: "do not commit the results.tsv file, leave it untracked by git". And discards are written down. A discard line is not a failure, it is the map of where not to go again, and by morning the discards outnumber the keeps many times over. Chapter 6 has the counts.

Crashes and long runs

A crash gets one honest attempt at a fix, then a line in the ledger. program.md: "If it's something dumb and easy to fix (e.g. a typo, a missing import), fix it and re-run. If the idea itself is fundamentally broken, just skip it, log 'crash' as the status in the tsv, and move on." A run that goes past ten minutes is treated the same way: "kill it and treat it as a failure (discard and revert)".

Rewinding further than the last commit is allowed and discouraged in the same breath: "If you feel like you're getting stuck in some way, you can rewind but you should probably do this very very sparingly (if ever)." The branch is meant to move one way.

Never stop

The one paragraph in capitals is about not asking. In full:

"NEVER STOP: Once the experiment loop has begun (after the initial setup), do NOT pause to ask the human if you should continue. Do NOT ask 'should I keep going?' or 'is this a good stopping point?'. The human might be asleep, or gone from a computer and expects you to continue working indefinitely until you are manually stopped. You are autonomous. If you run out of ideas, think harder: read papers referenced in the code, re-read the in-scope files for new angles, try combining previous near-misses, try more radical architectural changes. The loop runs until the human interrupts you, period."

Every coding agent is trained to check in. This paragraph exists because that habit, harmless at a desk, ends the night at 1 a.m. with a question nobody answers. The second half of the paragraph matters as much as the first: it tells the agent what to do with an empty idea list, which is the other way a night ends early.

Play the night shift

Six candidate runs are about to come in against a baseline of val_bpb 0.9979 at 44.0 GB. Before you press anything, decide which rule you would apply to a run that improves the metric by 0.0004 and adds twenty lines of code.

Try it yourself

Write the nine steps of the loop for something you work on, replacing train.py, the run command and the grep line with your own. Do not write the harness yet. If you cannot name the one line you would grep, you have found the first thing to build, and chapter 3 is about it.

Common questions

Why commit before the run and not after?

Because step 9 is a git reset. If the change is committed first, a discard is one command and nothing is lost. If it were committed only on success, a crash mid-run would leave an edited file and no record of it.

What counts as "equal" in step 9?

program.md keeps only on a strictly lower val_bpb; equal goes back. The simplicity rule in chapter 5 is the exception: an equal score with much simpler code is a keep.

Why is the ledger not committed?

The ledger is the night's record, not the code. Keeping it out of git means the branch history is exactly the kept changes, which is what makes the branch useful as memory in chapter 4.

How does the agent know what the baseline was?

It is the first line of results.tsv, and the current best is always the last keep line. The agent reads the ledger, not its own memory.

Carry this

• Setup once: a branch for the night, the files read, the ledger with a header, the baseline as the first run.
• Nine steps, repeated; the only decision is lower val_bpb or not.
• Grep two lines from the log; never let the log into the context.
• Crashes get one easy fix or a crash line; runs past ten minutes are killed; the ledger is never committed.
• Never stop, and when out of ideas, read, re-read, combine near misses, go radical.

Check yourself

1. What is the agent's first run of every night, and why?

Answer

The baseline: the script run as is. It is the first line of the ledger and the number every later run is compared with.

2. A run prints nothing when grepped for val_bpb. What happens next?

Answer

It crashed. The agent reads the last fifty lines of the log, fixes it if the fault is trivial and re-runs, or logs a crash line and moves on.

3. Why does the ledger use tabs and not commas?

Answer

Because the description column is free text and commas inside it would break a comma-separated file.