Part V · Practice
Chapter 17
Whiteboard it cold, and what to read next
The whole book on two pages: the picture, the numbers, the formulas and the words. Then the plan for the books that follow, on the inside of the transformer and on serving with vLLM and SGLang.
On this page
In this chapter, we will learn to explain a language model without notes, at a whiteboard, in ten minutes. We need one picture, the numbers, the formulas and the words. At the end comes the plan for the books that follow this one.
The one picture
If you can draw this from memory and explain each box, you have understood the book.
"Maya lives in Paris, the capital of" text
|
[ tokenizer ] cut into pieces, replace by ids Chapter 4
|
6747 64 3160 287 6342 11 262 3139 286 9 numbers
|
[ embedding table ] one row of 768 numbers for each id Chapter 5
|
9 x 768 numbers
|
+---------------------+
| attention | tokens collect from tokens before Chapter 6
| feed-forward | each token is reworked: Y = W . X Chapter 3
+---------------------+ x 12 layers
|
9 x 768 numbers
|
[ read-out ] compare the last row with every token Chapter 7
|
50,257 scores
|
[ temperature, softmax ] scores become chances Chapter 7
|
_France 28.81 % _the 25.30 % _a 5.16 % ...
|
[ draw one ] by chance, according to the chances
|
_France ---------> add to the text, and go round again Chapter 1
The ten sentences
- A language model takes a text and returns a chance for every token that could come next.
- It writes by a loop: draw one token, add it to the text, and guess again.
- What it knows is stored in its weights, which are the strengths of connections, written as numbers.
- One layer is one multiplication, Y = W · X, followed by a bend.
- Text enters as tokens, and each token becomes a row of numbers, its embedding.
- In each layer, attention lets every token collect from the tokens before it.
- The weights were found by training: guess, measure the loss, move every weight a little.
- After training the weights are frozen. The model learns nothing while it is used and keeps nothing between requests.
- It produces what is likely. Nothing in it checks what is true.
- What it lacks must be put into the prompt: documents, examples, results of tools, memories.
The numbers
| What | Number | Kind | Chapter |
|---|---|---|---|
| Weights of GPT-2 small | 124,439,808 | measured | 3 |
| Its layers, heads in each layer, width | 12, 12, 768 | measured | 6 |
| Its vocabulary | 50,257 tokens | measured | 4 |
| Its context window | 1,024 tokens | measured | 10 |
| Its file | 548,105,171 bytes | measured | 13 |
| Its speed on our laptop, processor only | 123 tokens per second | measured | 13 |
| Multiplications for one token | 123,532,032 | measured | 13 |
| Chance it gives to "France" | 28.81% | measured | 1 |
| Words per token, English | 0.75 by rule, 0.87 measured | both | 4 |
| Loss of a blind guess among 50,257 tokens | 10.82 | derived | 8 |
| Our toy model | 30,829 weights, loss 3.367 to 1.316 | measured | 8 |
| Context window of the GPT-6 models | 1,050,000 tokens | published, September 2026 | 10 |
| Price of gpt-6-luna, per million tokens | $0.10 in, $0.50 out | published, September 2026 | 10 |
| Weights of GPT-3 | 175 billion | published | 3 |
| Weights of GPT-6 Astra | 10 trillion; another report says 4.8 trillion | reported, not confirmed | 13 |
| Neurons in a human brain | 86 billion; range 61 to 99 billion | estimated | 2 |
| Connections in a human brain | 200 trillion; range 100 to 1,000 trillion | estimated | 2 |
A weight is not a connection. Put the last three rows side by side for a sense of scale, and for nothing more.
The formulas
one neuron y = w1 x x1 + w2 x x2 + ... + bias Chapter 2 one layer Y = W . X then a bend Chapter 3 weights of a layer about 12 x width x width Chapter 3 tokens words / 0.75 Chapter 4 numbers entering tokens x width Chapter 5 looks of attention n x (n + 1) / 2 for n tokens, in each head Chapter 6 softmax chance = e ^ (score / temperature) / sum over all tokens Chapter 7 a gap of 1 in score makes a token 2.72 times as likely Chapter 7 loss - ln(chance given to the right token) Chapter 8 one training step new weight = old weight - learning rate x slope Chapter 8 cost of a request tokens in x price in + tokens out x price out, per million Chapter 10 a conversation n x system prompt + tokens per turn x n x (n + 1) / 2 Chapter 10 file size weights x bits per weight / 8 Chapter 13 memory to run file size x 1.2 (our rule of thumb) Chapter 13 time for an answer time to first token + tokens / tokens per second Chapter 13
The words
| Word | In one line |
|---|---|
| Activation function | the bend between two layers, such as ReLU: negative numbers become 0 |
| Agent | a program that lets a model ask for tools again and again until a task is done |
| API | the agreed form in which a program sends a request to a model and gets a response |
| Attention | the step in which a token collects a weighted mix of the tokens before it |
| Backpropagation | the method that finds the slope of every weight from one guess |
| Bias | a number added to a weighted sum; also: a leaning the model took over from its training text |
| Chat template | the pattern that joins a list of messages into one text |
| Context window | the largest number of tokens a model can take in at once, prompt and answer together |
| Cosine similarity | how nearly two vectors point the same way, from -1 to 1 |
| Embedding | the row of numbers that stands for a token: a place on the map |
| Feed-forward | the second half of a layer: each token is reworked alone |
| Few-shot | a prompt with a few solved examples in it |
| Fine-tuning | a second, shorter training that shapes how a model behaves |
| GPU | a graphics card: it does many small calculations at the same moment |
| Hallucination | a fluent answer that is false |
| Head | one questioner of attention; GPT-2 small has 12 in each layer |
| Inference | using a finished model; the weights are only read |
| Knowledge cutoff | the day at which the training text ends |
| KV cache | the store of work done on earlier tokens, so that it is not done again |
| Layer | one round of attention and feed-forward; GPT-2 small has 12 |
| Learning rate | the size of the step in training |
| Logit | the raw score of a token, before softmax |
| Loss | one number that says how wrong a guess was |
| Matrix | a table of numbers with rows and columns |
| Mixture of experts | a design in which each token uses only some of the weights of a layer |
| Open weights | the file of the model is published and can be downloaded |
| Overfitting | learning the training text by heart, which does not help with new text |
| Parameter | any learned number of the model: a weight or a bias |
| Pre-training | the first, long training: guess the next token on a very large amount of text |
| Prompt | the text that is given to the model |
| Quantization | storing each weight with fewer bits |
| RAG | retrieval-augmented generation: search first, then answer from what was found |
| Sampling | drawing a token by chance, according to the chances |
| Softmax | the formula that turns scores into chances that add up to 100% |
| Stateless | keeping nothing between two requests |
| Streaming | sending each token to the reader as soon as it is drawn |
| System prompt | the standing instructions, written by the developer |
| Temperature | the number by which scores are divided: low is careful, high is adventurous |
| Tensor | a block of numbers: the general word for vector and matrix |
| Token | a piece of text with a number of its own |
| Tokenizer | the program that cuts text into tokens and puts them together again |
| Tool | a function the model can ask for; the application runs it |
| Top-k, top-p | two ways of removing unlikely tokens before the draw |
| Transformer | the design of layers with attention, published in 2017 |
| Vector | a list of numbers |
| Vocabulary | the complete list of tokens a model knows; GPT-2 has 50,257 |
| VRAM | the memory of a graphics card |
| Weight | the strength of one connection, as a number |
What to read next
This book showed what a language model does. It left out how the inside is built in detail, and how a model is run for many users. Those are the subjects of the books that follow. Here is the honest state of the shelf. Two books can be read today: the memory book and Thinking in Vectors. Three are planned and not yet written: one on the inside of the transformer, one on serving with vLLM and SGLang, and one on system design.
You can read these today
| Book | What it adds to this one | Where |
|---|---|---|
| AI Agent Memory: Novice to Ninja | how an assistant remembers a user across conversations, although the model keeps nothing: 25 chapters | on this shelf |
| Thinking in Vectors | embeddings, similarity and search, from the first idea to a hundred million passages: 66 chapters | in the Reading Room |
Planned, and not yet written
None of these three books exists yet, and there is nothing to open or download. We name them so that you know where the road goes. They will appear on this shelf when they are ready. Until then, the list tells you which words to look for.
| Book | What it will open up |
|---|---|
| Inside the transformer | Attention with its real arithmetic: queries, keys and values. How modern models mark positions. Normalization. The KV cache. Mixture of experts. Models that reason before they answer. Models that read pictures and sound. Training at scale, and cheap fine-tuning with small added matrices. |
| Serving a model: vLLM, SGLang and the machines | How one graphics card answers many users at once by packing their requests together. How the memory of the card is shared out in pages. Reusing the work done on a shared beginning of a prompt. Quantization in practice. Answers that must follow a fixed form. Measuring time to first token and tokens per second under load. Choosing the machines, and what a token costs to produce. |
| System Design: Novice to Ninja | How large services are built, derived from the numbers that stopped fitting: caches, queues, databases, replicas. It includes the design of a service around a language model. |
Try it yourself
Set a timer to ten minutes. Take an empty sheet. Draw the one picture, write the ten sentences in your own words, and put five numbers next to the picture, each with its unit.
Then give the sheet to someone who has not read the book and ask them to explain it back to you. What they cannot explain, your drawing did not yet make clear.
Common questions
I understood the book. Can I build an application now?
You can build a first one. You know how to send a request, what it costs, where the model will fail and how to put the right text into the prompt. What this book did not cover is testing: how to measure whether your application gives good answers. Chapters 17 and 18 of the memory book show one way to do it.
Do I need the mathematics to go further?
For using models, no. For the book on the inside of the transformer you will want to be at ease with vectors and matrices. The first seven chapters of Thinking in Vectors teach vectors and similarity in the same plain way. Matrices we met in Chapter 3.
Will this book be out of date soon?
The prices, the names of the models and their sizes will be out of date within months, which is why each of them carries its date. The mechanism has been stable since 2017: tokens, embeddings, layers with attention, scores, a draw. We expect that part to last.
Where are the programs and the data of this book?
In the book's folder of the source: books/llm/examples holds the five programs, each next
to a file with the output we got. books/llm/gen_data.py produces the data that the
interactives show.
Carry this
- One picture: text, tokens, embeddings, layers, scores, chances, a draw, and around again.
- Ten sentences, fifteen formulas and one table of numbers hold the whole book.
- Every number has a kind: measured, derived, published, reported, estimated or made up for illustration. Say which, when you quote one.
- To read today: the memory book and Thinking in Vectors. Planned and not yet written: the inside of the transformer, serving with vLLM and SGLang, and system design.
Check yourself
1. Without looking: a model has a width of 1,024, and a prompt has 200 tokens. How many numbers enter the first layer, and how many looks does one attention head need?
Answer
numbers 200 x 1,024 = 204,800 looks 200 x 201 / 2 = 20,100
2. A request sends 2,000 tokens and receives 400, at $0.10 and $0.50 per million, the list prices of gpt-6-luna in September 2026. There are 50,000 such requests a day. What is the bill for 30 days?
Answer
input 2,000 x $0.10 / 1,000,000 = $0.0002 output 400 x $0.50 / 1,000,000 = $0.0002 request = $0.0004 one day $0.0004 x 50,000 = $20 30 days $20 x 30 = $600
3. Someone tells you that a new model has "a trillion parameters, more than the brain has neurons". What two things do you say?
Answer
First, ask how the number is known: published by the makers, or reported by someone else? Second, the right comparison for a weight is a connection and not a neuron. The brain has an estimated 86 billion neurons and an estimated 100 to 1,000 trillion connections. A trillion weights are a hundred times fewer than the low end of that range: 100 trillion / 1 trillion = 100. And a weight is a much simpler thing than a connection.