Part I · Novice
Chapter 1
The machine that guesses the next word
A language model does one thing: it guesses what comes next. Watch a real model do it, one token at a time, and see why that small trick is enough to write an answer.
On this page
In this chapter, we will learn what a large language model is. We will not start with mathematics. We will start with the one thing the machine does, which is to guess the next word. Then we will watch a real model do it, one token after the other.
You have used one already
A language model is a program that is given some text and returns a guess for what comes next. The keyboard on your phone has a small one. Type "See you" and it offers "tomorrow", "soon" and "later". It has no idea who you are meeting. It has seen that those words often follow "See you".
An assistant such as ChatGPT or Claude is built on the same idea, made very much larger. The three letters LLM stand for large language model.
| Word | What it means here |
|---|---|
| Large | It has very many dials. The small model we open in this book has 124 million (we counted them). GPT-3, from 2020, has 175 billion (published by OpenAI). The makers of the newest models no longer publish the number. |
| Language | It works on text: questions, stories, programs, tables. |
| Model | A model is a small copy of something, made so that we can predict how the real thing behaves. A weather model predicts rain. A language model predicts text. |
One guess
The model does not return one word. It returns a chance for every word it knows.
We will use one sentence through the whole book. It is about Maya, who lives in Paris.
We give the model: Maya lives in Paris, the capital of We ask: what comes next?
We gave this sentence to a real model called GPT-2 small. OpenAI released it in 2019 and made its dials public, so anyone can download it and look inside. Here is what it answered.
token score (logit) probability
_France -69.61 28.81 %
_the -69.74 25.30 %
_a -71.33 5.16 %
_French -71.79 3.24 %
_Spain -72.31 1.92 %
_which -72.36 1.83 %
_Belgium -72.56 1.50 %
_Morocco -72.80 1.18 %
_Europe -72.91 1.06 %
_Switzerland -73.24 0.76 %
the ten together 70.77 %
tokens needed to reach 90 % 122 of 50,257
Let us read it slowly, because every later chapter returns to this table.
- The left column is a list of candidates. The mark
_stands for a space, so_Franceis the word "France" with a space in front. The model calls these pieces tokens. For now, read "token" as "word". Chapter 4 explains the difference. In the interactives the space is drawn as␣. - The middle column is a score, which programmers call a logit. A higher score means "fits better". Do not worry that the scores are negative. Only the gaps between them matter, and Chapter 7 shows why.
- The right column is the chance the model gives to each candidate. The table shows the ten best of 50,257 candidates. The chances of all 50,257 together are 100%.
- The last line says how flat the list is. If we add up the chances from the top, we need the 122 best candidates to reach 90%.
The model gave the right answer, France, the highest chance. But look how unsure it is.
chance of _France 28.81 % chance of anything else 100 - 28.81 = 71.19 % the model knows 50,257 tokens the ten best cover 70.77 % the other 50,247 share 100 - 70.77 = 29.23 %
This is the first thing to understand about a language model. It never knows. It
weighs. "The capital of the country" would also be good English, so _the gets 25%. A
language model is a machine for saying how likely each continuation is.
From one guess to a whole answer
To write a sentence, the model guesses one token, adds it to the text, and guesses again with the longer text. This loop is all there is. There is no second machine that plans the answer.
text so far the model one new token
+-------------------------------------+ +---------------+
| Maya lives in Paris, the capital of | --> | wall of dials | --> _France
+-------------------------------------+ +---------------+
^ |
| |
+--------------- add it to the end -------------------+
Before you press the button below, guess: what will the model write after "France"? Then press it, once for each token, and watch the chances. The count of tokens starts at 9, although the sentence has 7 words. The name "Maya" is cut into two tokens, and the comma is a token of its own.
These are the first ten steps, as our program printed them. The interactive goes two steps further.
Step 11 picks a full stop and step 12 picks a line break, drawn as ↵.
step 1 picks _France 28.8 % text so far: 'Maya lives in Paris, the capital of France' step 2 picks , 35.3 % text so far: 'Maya lives in Paris, the capital of France,' step 3 picks _and 32.2 % text so far: 'Maya lives in Paris, the capital of France, and' step 4 picks _is 16.1 % text so far: 'Maya lives in Paris, the capital of France, and is' step 5 picks _a 21.6 % text so far: 'Maya lives in Paris, the capital of France, and is a' step 6 picks _member 17.1 % text so far: 'Maya lives in Paris, the capital of France, and is a member' step 7 picks _of 99.0 % text so far: 'Maya lives in Paris, the capital of France, and is a member of' step 8 picks _the 73.9 % text so far: 'Maya lives in Paris, the capital of France, and is a member of the' step 9 picks _French 5.4 % text so far: 'Maya lives in Paris, the capital of France, and is a member of the French' step 10 picks _parliament 8.6 % text so far: 'Maya lives in Paris, the capital of France, and is a member of the French parliament'
Three things are worth noticing.
The model is sure only once. At step 7, after "is a member", the word "of" gets 99.0%. English leaves almost no other choice there. At step 9 the winner has only 5.4%, because a great many words could follow "the".
Each token costs one full trip. Ten tokens are ten trips through all the dials. A long answer takes longer than a short one for exactly this reason, and Chapter 13 counts the seconds.
The sentence is good English, and it is false. Maya is a data engineer. She is not a member of any parliament. The model did not lie, because it has no idea what is true about Maya. It continued the sentence in a likely way. Keep this example in mind. It explains most of what goes wrong with language models, and Chapter 11 comes back to it.
The same prompt, three more times
A prompt is the text we give to the model. In the run above we always took the candidate with the highest chance. Real assistants usually do something else: they pick by chance, according to the chances in the table. A candidate with 28.81% is picked about 29 times in 100.
the same prompt, three times, picking by chance at temperature 1.0
run 1: 'Maya lives in Paris, the capital of Central Europe. A committed activist and activist, she was elected as a'
run 2: 'Maya lives in Paris, the capital of which is part of her signature suburb. The 30-year-old'
run 3: "Maya lives in Paris, the capital of which borders Manila. He has a master's degree in History from Paris"
Three runs, three different texts, and none of them is sensible. This is a small model from 2019, and we chose it because its mistakes are easy to see. A modern model makes far better guesses. It works in the same way: one token, by chance, then the next. The word "temperature" in the output is the setting that controls how adventurous the picking is. We will turn it ourselves in Chapter 7.
Why is guessing the next word enough?
To guess the next word well, a machine must capture a great deal about the world that the text describes. This is the surprising part, so let us take three examples.
to continue this text ... ... the machine needs ------------------------------------------------ ------------------------------ "The capital of France is" a fact of geography "She put the ice in the sun, and an hour later how ice and heat behave it had" "def add(a, b): return" how a programming language works "Question: what is 12 x 12? Answer:" arithmetic, and the form of an answer
Nobody wrote rules for geography or ice into the machine. It was only ever asked to guess the next token, billions of times, on text written by people. To become good at that one task, the dials had to settle into positions that reflect facts, grammar, style and some reasoning. How that happens is the subject of Chapter 8.
A chat is the same trick with a costume on. The application writes a text that looks like a conversation, and it ends where the assistant's turn begins. The most likely continuation of such a text is a helpful answer. Chapter 9 shows that text letter by letter.
What the rest of the book opens up
Each later chapter opens one part of the loop. The map below shows which.
your question
|
v
text to numbers Chapter 4 (tokens) and Chapter 5 (embeddings)
|
v
the wall of dials Chapter 2 (weights), Chapter 3 (Y = W . X), Chapter 6 (attention and layers)
|
v
a chance for every Chapter 7 (scores, chances, temperature)
token, then one pick
|
v
numbers to text and around again, until the answer is complete
how the dials were set Chapter 8
how to call and pay for it Chapters 9 and 10
where it fails, and the fixes Chapters 11 and 12
size, speed, the whole trip Chapters 13 and 14
Try it yourself
You need nothing but a phone. Open a message, type "I am going to the", and then only tap the middle suggestion of the keyboard, twenty times. Read the result aloud.
You have just run the loop of this chapter by hand. The text will be grammatical and empty. Your phone's model is tiny, and it looks at only the last few words. Then ask yourself: what would the text be like if the model could look at the last 1,000 words, and had read a large part of the internet?
If you have a computer with Python, you can run the exact program behind this chapter. It is
03_gpt2_inside.py in the book's examples. It downloads GPT-2 small, a file of 548 MB, and
needs no graphics card.
pip install torch transformers
python 03_gpt2_inside.pyCommon questions
Is ChatGPT really just guessing the next word?
The model at its centre, yes. It produces a chance for every token, one token is picked, and the loop runs again. Around that centre there is ordinary software: it adds instructions, searches the web, runs tools and keeps the conversation. The later chapters show each of these. But no part of the system writes text in any other way.
If it only guesses, why are the answers so often right?
Because for a well trained model, the most likely continuation of a question is very often the correct answer. People wrote correct answers after such questions far more often than wrong ones. Where the training text was thin or mixed, the guess gets worse, and the model cannot feel the difference.
Does the model understand what it writes?
This book does not settle that question, and the people who build these models do not agree on it. What we can show is what the machine does: it turns text into numbers, multiplies them by its dials, and turns the result into chances. Whether that deserves the word "understand" is a matter of definition. The behaviour is the same either way.
Why did you use a model from 2019?
Because its dials are public, it fits on a laptop, and we can print every step. The dials of the largest assistants, such as ChatGPT and Claude, are not public, so nobody outside their makers can look inside them. Other modern models are public, and Chapter 9 runs one. They are all built on the same design as the one we open here, the transformer.
How does the model know when to stop?
One of the tokens it knows is a special one that means "the text ends here". When that token is picked, the loop stops. The application can also set a limit on the number of tokens, and the loop stops there even in the middle of a sentence.
What is a parameter?
It is another word for one dial. "Weight" is a third word for nearly the same thing. When you read that a model has 124 million parameters, it has 124 million dials. We will mostly say "weight", because that is the word the next chapter explains.
Carry this
- A language model takes text and returns a chance for every token it knows. GPT-2 small knows 50,257.
- Writing is a loop: guess one token, add it to the text, guess again. One token costs one trip through the whole model.
- The model weighs, it does not know. "France" had 28.81%, and 71.19% went elsewhere.
- A likely sentence is not always a true one.
Check yourself
1. An answer is 200 tokens long. How many trips through the model did it take to write it?
Answer
200. Every token is one trip. The prompt is read along the way, but each new token needs its own trip.
2. In the table, _France has 28.81% and _the
has 25.30%. If the model picks by chance 1,000 times, about how often will it pick one of these
two?
Answer
28.81 + 25.30 = 54.11 % 54.11 % of 1,000 = about 541 times
3. The model wrote that Maya is a member of the French parliament. Was that a bug in the program?
Answer
No. The program did exactly what it is built to do: it continued the text in a likely way. Nothing in the prompt said what Maya's job is, and the model has no way to look it up. The error is in expecting a guessing machine to know a private fact.