What people got

In this chapter, we will read the results that made the repository famous, with who reported each one and when. We will look at where the reports disagree, at the one person who flagged his own best number as suspect, and at what a keep rate does and does not tell you. The aim is not to doubt the results. It is to read them the way the loop itself would: one number, one source, one caveat.

The table

Every row below is a report, not a measurement of ours. Where a report cites the person's own post, the source column says so.

Where the reports disagree

The same first night is given three sizes. VentureBeat says 126 experiments. paddo.dev says 89 in the first session and 125 over two days across sessions. Aakash Gupta's embedded tweet from 8 March says 83 experiments and 15 kept, with validation loss "dropping from about 1.000 to about 0.977". These may be different nights, different sessions of one night, or one night counted three ways. The primary sources, Karpathy's own posts and the repository's discussion threads, are where to settle it, and this book has not. So: 83, 89 or 126, depending on who wrote it up.

The two-day figure is steadier. Fortune, VentureBeat and Gupta all give about 700 experiments and about 20 kept improvements, and Fortune and VentureBeat both give the 11% speed-up. Twenty keeps in seven hundred is what a strict metric on already hand-tuned code looks like.

Keep rates

A keep rate tells you what kind of loop it was, not how good. Here are the rates that can be computed from the reports, next to the rates from our own ledgers (chapter 7). Before you look, guess which is highest and why.

Karpathy's 3% is a strict metric on code a careful person had already tuned for years. Our site loop's 85% is a website being built out, where most experiments add a page or a section and the score mostly measures whether the new thing is sound. The voice cloner's 24% sits between: a strict metric, but a recipe with many untried knobs. A low rate on hand-tuned work is a compliment to the hands. A high rate on new work means the loop is doing the building, not the tuning. Neither is a grade.

How to read a claim from a loop

Four questions, in order.

1. Who measured it?        the person who ran it (primary) or a write-up of their post (secondary)
2. What was on the scale?  the exact metric, the exact budget, the exact hardware; "19% gain" of what?
3. Was the scale sealed?   could the loop touch the test; was the test the same all night; was it rerun fresh
4. Does it transfer?       to a bigger model, to new inputs, to tomorrow; the author's own caveat counts double

Run the table through the four. The two-day result passes the first three on the strength of several consistent reports and the fourth on the reported transfer to larger models. The Liquid result passes the first and fails the fourth by its author's own word. The Mac Mini result is a report of a report, but its lesson, simpler won, matches program.md's own rule and nobody's incentive.

Try it yourself

Find the primary post for any row in the table and compare its numbers with the row. If they differ, the row is wrong and the post is right; write down which. That is the whole method of this chapter, and it works on any benchmark claim, not only these.

Common questions

Why quote secondary sources at all?

Because they are what most readers will meet, and because three of them agreeing on 700 and 20 and 11% is itself evidence. The chapter names each one so you can go past it.

Is 11% a lot?

On code that an expert had tuned by hand for years, twenty small changes found in two days of machine time is the point. VentureBeat quotes a GitHub user asking whether the gains were noticeable and Karpathy replying that "these are real and substantial gains". Whether 11% matters to you depends on what you pay per hour of training.

What did the loop find that a person missed?

Fortune reports that the agent "caught oversights in attention scaling and regularization"; Gupta's guide says one find was "a bug in his attention implementation he'd missed entirely". Both are reports of Karpathy's account.

Why is the Mac Mini row worth a line?

Twenty-six of thirty-five runs failing is what a night looks like on small hardware, and the seven that worked still carried the lesson. A loop does not need most runs to work.

Carry this

• The two-day result, about 700 experiments, about 20 kept, 11% faster, is consistent across three reports.
• The first night is 83, 89 or 126 experiments depending on the report; the primary posts settle it, not this book.
• The best headline number came with its author's own overfit warning. Read the two together.
• A keep rate describes the kind of loop: strict metric on tuned code, 3%; a site being built, 85%.
• Four questions: who measured, what was on the scale, was it sealed, does it transfer.

Check yourself

1. Which result in the table carries its own author's caveat, and what is the caveat?

Answer

Lütke's Liquid run: 53% faster parse plus render, with "This is probably somewhat overfit."

2. Why is a 3% keep rate not a sign of a poor night?

Answer

It was a strict metric applied to code already tuned by hand for years; the kept changes still added up to an 11% gain. A low rate on tuned work is expected.

3. What single check answers "does it transfer"?

Answer

A fresh set of inputs the loop never saw, or a larger model the changes are applied to. The author's own caveat, where there is one, counts double.