Why autoresearch works

In this note, we will learn what karpathy/autoresearch is, what the loop inside it does, what people got when they pointed it at their own problems, and the six rules that make it work. It is one page. Every number names its source.

What it is

Autoresearch is a small, real training setup plus a page of instructions that lets an AI coding agent run machine learning experiments on its own, all night. Andrej Karpathy published it in March 2026 under the MIT licence. His summary: "give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats. You wake up in the morning to a log of experiments and (hopefully) a better model."

The repository has three files that matter.

prepare.py   data, tokenizer, the evaluation     read-only. The agent must not touch it.
train.py     model, optimizer, training loop    the ONLY file the agent edits.
program.md   the instructions to the agent      the ONLY file the human edits.

That last line is the whole idea. In the README's words, "you're not touching any of the Python files like you normally would as a researcher. Instead, you are programming the program.md". The human programs the research organisation. The agent programs the model.

The loop

Every experiment is the same short list of steps, and the metric decides.

loop forever:
  1. change train.py with one idea, then git commit
  2. uv run train.py > run.log            training stops at exactly 5 minutes
  3. grep val_bpb from run.log             lower is better
  4. write one line to results.tsv         commit, metric, memory, keep | discard | crash, description
  5. improved?   keep the commit; the branch moves forward
     not?        git reset to where you started
  6. a run past 10 minutes is killed and counted as a failure

The log is tab-separated, so a description may contain commas. program.md shows what a night looks like:

commit   val_bpb   memory_gb  status   description
a1b2c3d  0.997900  44.0       keep     baseline
b2c3d4e  0.993200  44.2       keep     increase LR to 0.04
c3d4e5f  1.005000  44.0       discard  switch to GeLU activation
d4e5f6g  0.000000  0.0        crash    double model width (OOM)

The pace is stated as arithmetic, not as hope:

one experiment     about 5 min of training, plus startup and eval
per hour           about 12
one night's sleep  about 8 h x 12 = about 100 experiments      (program.md)

One instruction is in capitals: "NEVER STOP ... The human might be asleep ... The loop runs until the human interrupts you, period." If the agent runs out of ideas it is told to think harder: read the papers the code cites, re-read the files for new angles, combine earlier near misses, try something more radical.

What people got

The results that made the repository famous are small, measured, and were reported by the people who ran them.

Why it works

The loop succeeds because of what it forbids, not because of what the model knows. Six rules do the work.

1. One number, measured by code the agent cannot edit

The evaluation lives in prepare.py, and prepare.py is read-only. If the agent could change the test, the easiest "improvement" would be to make the test easier. Locking the scale is what keeps the ledger honest. The README's design note says it in six words: "One GPU, one file, one metric."

2. A fixed budget makes every run comparable

Training always stops at five minutes. A bigger model, a smaller batch, a new optimizer: each is judged on what it manages in the same window on the same machine. The agent never has to reason about "better but slower". Better means a lower val_bpb inside five minutes, and nothing else.

3. One file keeps every change reviewable

"The agent only touches train.py. This keeps the scope manageable and diffs reviewable." The script is about 630 lines, by the press reports, so it fits in the agent's context with room to think, and a person can read any kept diff in a minute the next morning.

4. Keep or discard, with git as the memory

A good idea advances the branch. A bad idea costs five minutes and leaves one line in the log. Rewinding further is allowed "very very sparingly (if ever)". The branch is a ratchet: it only moves forward, and every tooth was measured.

5. The human writes the program, not the code

The leverage moves from the experiment to the environment: what is allowed, what is forbidden, what counts, when to give up on a crash. Improving program.md improves every night after it. The README calls this the "research org code", and says the shipped version is a bare-bones baseline on purpose.

6. Simpler wins ties

From program.md: "A 0.001 val_bpb improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 val_bpb improvement from deleting code? Definitely keep. An improvement of ~0 but much simpler code? Keep." The Mac Mini night ended with a simpler model for the same reason.

How to copy it

You need three things, and a night.

• A metric a script can compute in minutes with no person in the loop. Not a feeling. A number.
• One file the agent may change, and a harness it may not. Everything else is read-only.
• A log with the discards left in. The failures are the map of where not to go again.

Then write the program: the setup, the loop, the budget, the keep rule, the never-stop rule and the simplicity rule. Where it fails is also known. A weak metric gets gamed on the first night, the agent does run out of ideas, kept changes are not isolated from each other, and the result is tuned to one machine. None of that stopped the people in the table. They wrote a better program.md and ran it again.

Common questions

Is this only for training language models?

No. The repository trains a small model because that was Karpathy's problem. Lütke ran the same loop on a templating engine written in Ruby, and the loops behind this shelf ran it on a database, a memory layer and a voice cloner. The only requirement is a number a script can measure.

Why five minutes, and not "until it converges"?

A fixed window makes every run comparable and keeps the night cheap. Twelve runs an hour means a bad idea costs five minutes. It also means the loop finds the best model for that window on that machine, which is the honest version of the question.

Does the agent get stuck repeating the same ideas?

It does, and program.md says what to do: read the papers the code cites, re-read the files for new angles, combine earlier near misses, try bigger changes. The log helps too. An idea that was discarded is one line away.

What is the catch?

The metric is all the agent sees, so a weak metric is optimised into nonsense. Lütke flagged his own best result as possibly overfit. Build the harness first, lock it, and spend your attention there.

Carry this

• One file the agent edits, one file the human edits, one number nobody edits.
• A fixed budget turns "is it better" into a comparison of two numbers.
• Keep or discard every run. The branch is a ratchet and the log is the map.
• Program the organisation, not the experiment. A better program.md improves every night after it.

Check yourself

1. Which file is the agent forbidden to edit, and why does that one rule matter most?

Answer

prepare.py, which holds the evaluation. If the agent could change the test, it would make the test easier instead of making the model better.

2. About how many experiments does the program expect in one night, and how is that figure arrived at?

Answer

About 100: roughly 12 an hour, because each run trains for 5 minutes, over the length of a night's sleep.

3. What does the simplicity rule say about a tiny improvement that adds twenty lines of hacky code?

Answer

Probably not worth keeping. A tiny improvement from deleting code is definitely kept, and an improvement of about zero with much simpler code is kept too.