The LLM lab · How an LLM Works: Novice to Ninja

In this chapter, we will learn by doing. Every interactive of the book is here on one page, in the order in which a request travels through the machine. Each one has a small task. Try to predict the result before you move anything. The answers to the tasks are at the end of the page, under "Common questions".

The idea

A language model guesses the next token, and a loop turns guesses into text. From Chapter 1. Task: press the button until the sentence is finished, and find the step where the model was most sure.

What is known is stored as the strength of connections. From Chapter 2. Task: how many times must you see something so that it still comes back, at least sometimes, after 20 years? Then set the years to 30 and try again.

Task: change two weights so that a horse counts as a car. Then ask yourself what the neuron has really learned.

One layer is one multiplication: Y = W · X. From Chapter 3. Task: switch all four features off, and then all four on. In which case does no answer win, and in which do two answers get the same score?

Task: reach about one billion weights in two different ways, once with many layers and once with a large width.

Text becomes numbers

A tokenizer cuts text into pieces from a fixed list. From Chapter 4. Task: type your own name. Is it one token?

Task: pick "A novel". By the rule of thumb, does it fit a window of 128,000 tokens?

An embedding gives each token a place, and tokens with similar use get nearby places. From Chapter 5. Task: find the token whose nearest neighbour surprises you most.

Task: pick the families one after the other, and find the one that sits furthest from all the others.

Inside the model

Attention lets a token take information from the tokens before it. From Chapter 6. Task: find a token and a head where more than 90% of the attention goes to one place.

Task: set the slider to 0 and move it up, one layer at a time. After which layer is France the highest guess for the first time?

Scores become chances, and one token is drawn. From Chapter 7. Task: raise all four scores by 1.0. What happens to the four chances?

Task: find the lowest temperature at which "Draw twenty" usually gives at least five different tokens. Press "Clear" and draw several times at each setting, because every draw is a matter of chance.

Training turns every weight a little, against the slope of the loss. From Chapter 8. Task: find the largest learning rate that still settles.

Task: move to the last checkpoint. What chance does the model give to the right letter, on text it never saw?

Using a model

A request is one JSON object. From Chapter 9. Task: build the request that makes an answer repeatable.

Every request carries the whole conversation. From Chapter 10. Task: with a system prompt of 500 tokens and 250 tokens per turn, after how many turns have you paid for one million input tokens?

Task: leave the three sliders where they are and switch from gpt-6-luna to gpt-6-astra. How many times larger is the bill for thirty days?

The model continues text in a likely way. Likely is not the same as true. From Chapter 11. Task: count on how many of the eight sentences the true answer is the model's first guess.

What the model lacks, the application puts into the prompt. From Chapter 12. Task: switch blocks on until all six questions can be answered. How many tokens does the prompt hold then, and which share of it is the user's question?

Size and speed

A model is a file of weights, and it must fit in memory to run. From Chapter 13. Task: compare GPT-3 with GPT-6 Astra. How many times larger is the second, if the reported figure is right?

Task: set 4 bits per weight, and find the largest model whose smallest home is still a laptop.

Task: pick "Qwen 1.5 B, our laptop" and leave the answer at 300 tokens. After how many seconds is the whole answer there, and who is faster, the model or the reader?

The whole journey

One request passes nine stations, and only four of them are the model. From Chapter 14. Task: walk the nine stations, then close the page and write them down from memory.

Try it yourself

The measured numbers of this book come from five short programs. They are in the folder books/llm/examples of the book's source, each next to a file with the output we got. Run the first one. It needs one library and no graphics card.

pip install tiktoken
python 01_tokens.py

Then change the sentence in the program to one of your own, and compare the result with the tokenizer on this page. The numbers must be the same.

Common questions

Are the numbers on this page real?

The measured ones are. Everything about GPT-2 (tokens, neighbours, attention, scores, the layer by layer guess) was read from the real model by gen_data.py. The training curve is a real run. The prices are published, the size of GPT-6 Astra is reported and not confirmed, and the connections of the brain are estimated, with a range of 100 to 1,000 trillion. The brain picture, the weights of the single neuron and of the small matrix, and the token sizes in the prompt packer are made up for illustration. The 20% working room is our rule of thumb.

Why GPT-2 and not a current model?

Because its weights are public, it is small enough to open on a laptop, and its design is the ancestor of the models in use today. The large models of today are not public. We can call them, and we cannot look inside them.

Does the page send what I type to a server?

No. Every interactive runs in your browser. The tokenizer downloads one file, the list of 50,000 merges, and then works on its own.

What are the answers to the tasks?

Here they are, in the order of the page. Where a task has many right answers, we give one. Three tasks have no answer that we could print: your own name, the neighbour that surprises you, and the nine stations from memory.

The loop: step 7, where "of" gets 99.0% after "is a member".
The connection: at 20 years, about 120 times, which gives a strength of 0.253. At 30 years no number is enough. Even 10,000 times leave 0.995 x 0.215 = 0.214, which is below the 0.25 that the picture asks for. The formulas are made up for illustration.
The neuron: for example, fur +2.0 and carries people +5.0. The sum for the horse is 2.0 + 5.0 - 3.0 = +4.0. The neuron has learned nothing about horses. It adds up what its inputs report.
The multiplication: with all features off the three scores are 0, and no answer wins. With all on, automobile and smartphone both get +1. The interactive reports a tie, because the table has no way to decide between them.
Counting weights: width 1,024 with 78 layers gives 1,035,019,264. Width 4,096 with 4 layers gives 1,015,574,528.
Words to tokens: 90,000 / 0.75 = 120,000 tokens, so it fits.
The map: the days of the week.
Attention: layer 5, head 12. Every token gives nearly all its attention to the token just before it.
The layers: layer 10, with 88%.
Softmax: nothing happens. Only the gaps between the scores count.
Temperature: about 0.7 to 0.8. Your own draws may say a little more or less.
One weight: 0.130. At 0.135 the loss grows.
The training run: 27%, from e ^ -1.316.
The request: temperature 0.
The conversation: 87 turns. 87 x 500 + 250 x 87 x 88 / 2 = 43,500 + 957,000 = 1,000,500 tokens.
The bill: $75,000 against $750, which is 100 times.
Fluent or true: 3 of 8.
The prompt: all six blocks are needed, which makes 3,445 tokens. The question is 20 of them, or 0.6%.
The ladder: 10 trillion / 175 billion = 57.1 times. The 10 trillion is reported and not confirmed.
The size: 26 billion weights. The file is 13 GB and needs 15.6 GB. At 27 billion it needs 16.2 GB, and a laptop with 16 GB is too small.
The wait: 16.9 seconds. The reader needs 56.3 seconds for the 225 words, so the model is faster.

Carry this

Text in, numbers through, text out. Text exists only at the two ends.
Every token of the answer is one trip through all the weights.
The model is fixed. What changes from request to request is the prompt.

Check yourself

1. Which interactives on this page show measured numbers from a real model, and which show a picture made for illustration? Name two of each.

Answer

Measured: the loop, the tokenizer, the neighbours, the map, attention, the layers, temperature, the training run, fluent or true, and the nine stations. Illustration: the strength of a connection in the brain, the single neuron, the multiplication by hand, and the sizes of the blocks in the prompt packer.

2. The conversation task needed 87 turns to reach one million input tokens. How many tokens does request 87 itself send?

Answer

request 87     500 + 87 x 250   =  22,250 tokens

One request is small. The million comes from sending the growing conversation 87 times.