The program, not the code

In this chapter, we will learn what program.md contains, section by section, and why the README calls it the file the human iterates on. We will read the simplicity rule, then look at what two of our own programs added after their first nights: gates, noise rules, helper scripts, a taste gate, and a ledger that explains itself. The interactive writes a skeleton program for a problem of your choosing.

Anatomy of program.md

The shipped program is a few hundred words in eight parts. In the order they appear:

1. what this is       "This is an experiment to have the LLM do its own research."
2. setup              the run tag, the branch, read the files, check the data, create the ledger, confirm
3. the task           "The goal is simple: get the lowest val_bpb."  what you CAN do (train.py only) and
                      CANNOT do (prepare.py, new packages, the evaluation)
4. the output         the summary block the script prints; grep two lines of it
5. the ledger         five tab-separated columns; crashes logged with zeros; not committed
6. the loop           LOOP FOREVER, nine steps; crash handling; the ten-minute kill
7. the simplicity rule
8. NEVER STOP         and the use case: about 100 experiments while the human sleeps

Count what is missing. There is no list of ideas to try. There is no description of the model. There is no praise and no motivation. The program is a procedure and a set of fences. The ideas come from the agent reading the code, the papers the code cites, and its own ledger.

The simplicity rule

Ties go to the simpler code, and a small gain can lose to the complexity it costs. This is the one place program.md asks for judgement, and it gives three examples so the judgement has edges:

"All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it." Then: "A 0.001 val_bpb improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 val_bpb improvement from deleting code? Definitely keep. An improvement of ~0 but much simpler code? Keep."

Without this rule the ratchet still turns, and after a hundred nights train.py is a thousand lines of special cases that each bought a thousandth. The VentureBeat account of a Mac Mini run ends on the same note: of the seven runs that worked, the lesson was that "the model got better by getting simpler".

What the human adds after the first night

The program grows by one rule per surprise. Two of our loops, both built from program.md, show the kind of rule that gets added. Everything below is quoted or counted from their program files.

The site loop

The program that improves madhudream.dev opens by naming its ancestry: it "is the program.md of github.com/karpathy/autoresearch adapted to a website". After its first experiments it had these rules that program.md does not:

gates          privacy_leaks must be 0, the build must pass, every route must answer;
               "any failure means discard, no matter the score"
noise          "Speed is noisy. Every candidate gets two runs; compare the mean."
a minimum      keep only if the mean score is higher by more than 0.5, or equal with a clear simplification
taste          after any visual change, open the page, screenshot it, look; a change that raises the score
               and makes a section look worse is a discard
memory         every 10 experiments, refresh a "What we learned" section, negative results included
maintenance    every 20 experiments, re-run the content pipelines and walk the site end to end

The taste gate is worth a second look. It is a rule the agent executes (take the screenshot, look at it) about a judgement no script makes. The program does not pretend the metric is complete. It names the gap and puts a procedure in it.

The voice cloner

The program that improves our voice clone calls itself "lifted from karpathy/autoresearch, adapted from LLM training to voice cloning". Its additions:

one editable file   experiment.py: "the only file you edit"; harness.py: "read-only"
helpers             scripts/log_result.py auto: appends the ledger row AND decides keep or discard by the rules,
                    then copies experiment.py to or from best_experiment.py
queues              scripts/run_batch.sh: "Queue 3-6 at a time; never run two experiments concurrently"
                    (the machine has crashed under parallel load before)
four keep rules     1. clone_score strictly improves   2. output quality may not drop by more than 0.30
                    3. generation stays under 3x real time   4. the simplicity rule, restated for this loop
its own timeout     "If a run exceeds 25 minutes, kill it, log crash (timeout) and revert."
a readme            "Update it every few experiments so the human can read the state of the research at any time."

Notice the helper that decides keep or discard. The rules were moved out of the agent's head and into a script, so that a tired context at 4 a.m. cannot misremember them. That is the direction every mature program takes: fewer instructions to follow, more rules the tools enforce.

The skeleton, for your problem

The shape does not change between a training run, a web page and a voice recipe; the names do. Pick one and read the result as a draft of your own program. Before you pick, decide which line you would have to build before the loop could start.

What not to put in the program

Three things do not belong in program.md. Permission to edit the harness, for the reasons of chapter 3. A long list of ideas, because the agent will run the list and stop thinking; a short backlog is fine, and the site's program keeps one, but the never-stop paragraph tells the agent where ideas come from when the list is empty. And anything the agent should ask about. The program is read by something that cannot ask, so every "check with the owner" is a night that ends at the first check.

Try it yourself

Take the skeleton above for the problem closest to yours and write the three lines the interactive leaves vague: the exact command that runs one experiment, the exact grep that reads its result, and the exact rule for keep. If any of the three needs a person, that is your next piece of work, not the agent's.

Common questions

Should the program list ideas to try?

A few, as a backlog, is useful on the first night. A long list turns the loop into a queue. program.md ships with none; the site's program keeps a short one and tells the agent to read the site as an investor would when it runs out.

How long should program.md be?

The shipped one is a few hundred words. The site's is much longer because it also describes the site. The test is not length; it is whether an agent that has never seen the project can run a correct night from the file alone.

Can the agent improve program.md itself?

It can propose. The README's division is that the human edits program.md, and the reason is the same as for the harness: the file holds the rules the agent is judged by. The site's program allows the agent to edit a companion file for a later session, and keeps the rules file to the owner.

What is the first rule to add after night one?

Usually a noise rule. Run the baseline twice before anything else and write the gap into the program as the minimum gain for a keep.

Carry this

• program.md is a procedure and a set of fences, in eight parts, with no ideas list and no motivation.
• The simplicity rule: a tiny gain with hacks is probably not worth it; an equal score with simpler code is a keep.
• The human adds one rule per surprise: gates, a noise rule, a minimum gain, a taste gate, a readme the agent keeps.
• Move rules out of the agent's head and into helper scripts; a script does not get tired at 4 a.m.
• Nothing in the program may require asking.

Check yourself

1. What does program.md say about a 0.001 improvement that comes from deleting code?

Answer

"Definitely keep."

2. Name two rules the site loop added that the original program does not have.

Answer

Any two of: hard gates (privacy, build, routes), two runs per candidate with the mean compared, a minimum gain of 0.5, the screenshot taste gate, a "what we learned" section every ten experiments.

3. Why did the voice cloner move the keep-or-discard decision into a script?

Answer

So the rules are enforced by a tool rather than remembered by the agent, which cannot misapply them late in a long night.