Part IV · Ninja
Chapter 14
One request, end to end
Follow one question from the keyboard to the answer, through nine stations, with the numbers of a real model. This is the whole book in one journey.
On this page
- The map of the journey
- Station 1: the request
- Station 2: text to tokens
- Station 3: tokens to vectors
- Station 4: twelve layers
- Station 5: scores
- Station 6: chances
- Station 7: one token is drawn
- Station 8: around again
- Station 9: tokens to text
- The journey in numbers
- What an application adds around the journey
- Try it yourself
- Common questions
- Carry this
- Check yourself
In this chapter, we will learn nothing new, and we will put everything together. We will follow one request from the moment it leaves the application to the moment the answer returns, through nine stations. At each station we will say what goes in, what comes out, and in which chapter we met it.
The map of the journey
1 the request JSON ordinary software 2 text to tokens 9 numbers ordinary software 3 tokens to vectors 9 x 768 numbers \ 4 twelve layers 9 x 768 numbers | the model: 5 scores 50,257 numbers | 124,439,808 weights 6 chances 50,257 chances / 7 one token is drawn 1 number ordinary software 8 around again back to station 3 ordinary software 9 tokens to text JSON ordinary software
Only stations 3 to 6 are the model. Everything else is ordinary software that any programmer could write. Press "Next station" below and watch what leaves each station, then read on for the detail of each.
We use GPT-2 small for the journey, because we can see into it, with the sentence of the book. Every
number in this chapter was measured on it by the program 03_gpt2_inside.py, unless we say
otherwise.
Station 1: the request
The application sends one JSON object to the server. It holds the name of the model, the text and the settings (Chapter 9).
{
"model": "gpt2",
"prompt": "Maya lives in Paris, the capital of",
"temperature": 0,
"max_tokens": 12
}GPT-2 is a plain completion model, so it takes a prompt and no list of messages. With a chat model there is one more step here: the chat template joins the messages into one text and adds the markers for each turn. In Chapter 9 our 19 words were 23 tokens, and the template added 13, which made 36.
Station 2: text to tokens
The tokenizer cuts the text into pieces from its list and replaces each by its id (Chapter 4).
(c) The prompt becomes token ids prompt "Maya lives in Paris, the capital of" pieces May | a | _lives | _in | _Paris | , | _the | _capital | _of ids 6747 64 3160 287 6342 11 262 3139 286
words in the prompt 7 tokens 9 "Maya" is two tokens, and the comma is one words per token 7 / 9 = 0.78
From here on, no text exists in the machine.
Station 3: tokens to vectors
Each id picks one row of the embedding table, and the row of its position is added (Chapter 5).
row 6342, the token _Paris [-0.219, +0.042, +0.121, -0.260, +0.028, -0.051, ... 762 more] 9 tokens x 768 numbers = 6,912 numbers enter the first layer
At this station the token "of" knows nothing about Paris. It carries the meaning of "of" alone, and the fact that it stands at place 9.
Station 4: twelve layers
In every layer, attention lets each token collect from the tokens before it, and the feed-forward step reworks each token alone (Chapter 6). Both are multiplications by weight matrices, Y = W · X (Chapter 3).
after the embedding _France 0.00 % top 3: _destro 65% _mathemat 26% _challeng 8% after layer 5 _France 0.00 % top 3: _the 88% _a 1% _his 1% after layer 9 _France 19.23 % top 3: _the 39% _France 19% _Europe 8% after layer 10 _France 87.64 % top 3: _France 88% _Europe 6% _the 2% after layer 12 _France 28.81 % top 3: _France 29% _the 25% _a 5%
These five lines are taken from the table of thirteen in Chapter 6. After each layer we asked what the model would guess if it had to guess now. In layer 9 one head looks from "of" back to "Paris" with 61% of its attention, and in the same layer France appears among the guesses. The shape never changes: 9 rows of 768 numbers go in, and 9 rows of 768 numbers come out.
weights in the twelve layers 12 x 7,087,872 = 85,054,464 token table = 38,597,376 position table = 786,432 final tidy-up = 1,536 all weights 85,054,464 + 38,597,376 + 786,432 + 1,536 = 124,439,808
Station 5: scores
The 768 numbers of the last token are compared with every row of the token table, which gives one score for every token (Chapter 7).
_France -69.61 _the -69.74 _a -71.33 _French -71.79 _Spain -72.31 ... and 50,252 more
Only the last of the 9 rows is used for the guess. The other 8 rows did their work inside the layers, as sources for attention.
Station 6: chances
The scores are divided by the temperature, and softmax turns them into chances that add up to 100% (Chapter 7).
_France 28.81 % _the 25.30 % _a 5.16 % _French 3.24 % _Spain 1.92 % the other 50,252 tokens together 100 - 64.43 = 35.57 %
Our request has temperature 0, and nothing can be divided by 0. By agreement, 0 is not a division at all: it tells station 7 to take the highest score. The chances above are those of the scores as they are, which is temperature 1.
Station 7: one token is drawn
One token is drawn by chance, according to the chances. With temperature 0 the highest is taken. Top-k and top-p can remove the tail before the draw (Chapter 7).
drawn _France id 4881 chance 28.81 %
Our request asked for temperature 0, so the token with the highest chance is taken on every trip, and you can repeat the journey and get our numbers. With a temperature above 0 the token would be drawn by chance, and every run would give a different text, as the three runs of Chapter 1 showed.
With streaming switched on, this token is sent to the application at once.
Station 8: around again
The new token is added to the text, and the longer text goes back to station 3 (Chapter 1).
trip 1 9 tokens in _France taken trip 2 10 tokens in , taken trip 3 11 tokens in _and taken ... trip 12 20 tokens in a new line is taken: max_tokens is reached, and the loop stops
The loop ends for one of two reasons. The model draws its end token, and the response says
finish_reason: stop. Or the limit of max_tokens is reached, as here, and the response says
finish_reason: length.
Station 9: tokens to text
The ids are turned back into pieces and joined. The server wraps the text in a JSON response and adds the token counts (Chapters 4 and 9).
{
"text": " France, and is a member of the French parliament.\n",
"finish_reason": "length",
"usage": { "prompt_tokens": 9, "completion_tokens": 12, "total_tokens": 21 }
}The bill is worked out from the last line (Chapter 10). And the answer is fluent and false, as we saw on the first pages of the book: Maya is a data engineer (Chapter 11).
The journey in numbers
tokens read 9 tokens written 12 trips through the model 12 weights used on each trip 124,439,808 multiplications, roughly 12 x 123,532,032 = 1,482,384,384 time on our laptop 12 tokens / 123 tokens per second = about 0.1 s
About one and a half billion multiplications for twelve tokens, in a tenth of a second, on a laptop without a graphics card. The count is rough: it takes each weight of a multiplication table as used once on each trip, which is what the KV cache makes possible. A model with 8 billion weights does about 65 times more work for each token, as we worked out in Chapter 13. The stations are the same.
What an application adds around the journey
An application does its work before and after the nine stations, and leaves the stations as they are.
the user types a question
|
v
+---------------------------------------------------------------+
| the application |
| looks up memories about the user (memory book) |
| searches documents for passages that fit (Chapter 12) |
| adds the system prompt and the tool descriptions |
| adds the conversation so far (Chapter 10) |
+-------------------------------+-------------------------------+
| one request
v
the nine stations of this chapter
|
v
+---------------------------------------------------------------+
| the application |
| if the model asked for a tool: runs it, and sends again |
| checks the answer, shows it, stores the conversation |
+---------------------------------------------------------------+
When an assistant "searches the web" or "remembers you", the box in the middle has not changed. The application around it has done more work before and after.
Try it yourself
Close the page and take a sheet of paper. Draw the nine stations from memory, with one line for each: what goes in, and what comes out. Then write next to each station whether it belongs to the model or to ordinary software.
Compare with the map at the top of this chapter. The stations you forgot are the chapters to read again.
Then explain the journey aloud to someone who has not read the book, in three minutes. Where you hesitate, you have found the place you have not yet understood.
Common questions
Where in the nine stations does the model "think"?
If anywhere, at station 4. That is where information moves between tokens and where the guess takes shape. A model that "reasons" before it answers does not have a further station. It runs the loop for longer and writes its reasoning as tokens before it writes the answer.
Is the journey the same for a model with a trillion weights?
The stations are the same. The numbers are larger: more layers, wider rows, a larger vocabulary. Large systems add machinery around the model, for example to pack the requests of many users together. None of that changes what happens to one request.
Which station takes the most time?
Station 4, followed by station 5. Cutting text into tokens and looking up rows is very little work beside them. In GPT-2 small the layers hold about two thirds of the multiplications, and the comparison with all 50,257 rows at station 5 holds the rest. Both are repeated for every token that is written.
matrices of the twelve layers 84,934,656 / 123,532,032 = 68.8 % token table, used at station 5 38,597,376 / 123,532,032 = 31.2 %
Does the model see my whole prompt on every trip?
In effect, yes: every new token may look back at every earlier token. In the machine, the work for the earlier tokens is kept in the KV cache and not repeated.
Why are softmax and the temperature counted as part of the model?
It is a matter of where one draws the line. The weights end at station 5, with the scores. Softmax is a fixed formula without weights, and the temperature is a setting of the request. We count station 6 with the model because the model was trained through this formula: its loss in Chapter 8 was computed from the chances.
Where do pictures enter the journey?
At station 3. A picture is cut into small squares, and a separate part of the model turns each square into a row of numbers of the same width as a token row. From station 4 on, the layers treat those rows as they treat text.
Carry this
- Nine stations: request, tokens, vectors, layers, scores, chances, draw, around again, text.
- Stations 3 to 6 are the model. The rest is ordinary software.
- Text exists at the two ends. In between there are only numbers.
- Our journey: 9 tokens read, 12 written, 12 trips, about 1.5 billion multiplications, a tenth of a second.
- Search, memory and tools are work the application does before and after. The nine stations stay the same.
Check yourself
1. A prompt has 50 tokens, and GPT-2 small writes an answer of 30 tokens. How many numbers enter the first layer on the first trip, and how many trips are there?
Answer
numbers 50 tokens x 768 = 38,400 trips 30 one for each token written
2. At which station is the temperature used, and at which max_tokens?
Answer
The temperature is used at station 6, where the scores are divided by it before softmax. A temperature of 0 is the exception: nothing is divided, and station 7 takes the highest score. max_tokens is used at station 8, where the loop checks whether it should go round again.
3. An assistant answers a question about this morning's news correctly. Which station made that possible?
Answer
None of the nine. The application searched for the news and put it into the prompt before station 1. What the weights hold ends at the knowledge cutoff.