Why it makes things up

In this chapter, we will learn where a language model goes wrong, and why. We will ask a real model six kinds of question that it cannot answer reliably, and read what it says. Each mistake follows from something we learned in the earlier chapters, so we will be able to name its cause.

Fluent is not the same as true

A language model produces the most likely continuation of a text. Whether the continuation is true plays no part in the calculation.

Look back at what the machine does. It turns text into numbers, multiplies them by its weights, and ends with a chance for every token (Chapter 7). At no station on that way is there a step that checks a fact. There is no list of facts that could be checked against.

We can watch this in GPT-2 small. We gave it the beginnings of eight sentences that have one true ending. Each line shows the token we wanted, the chance the model gave it and its place in the model's list. The three best guesses follow, with their chances in percent. All are measured.

The capital of France is | wanted ' Paris' 3.2% rank 5 | top [(' the', 8.5), (' now', 4.8), (' a', 4.6)]
The capital of Australia is | wanted ' Canberra' 0.8% rank 17 | top [(' the', 9.0), (' a', 5.2), (' home', 4.2)]
Two plus two equals | wanted ' four' 5.4% rank 4 | top [(' one', 33.0), (' two', 10.8), (' three', 6.8)]
The opposite of hot is | wanted ' cold' 16.3% rank 1 | top [(' cold', 16.3), (' the', 8.2), (' a', 6.2)]
The largest planet in our solar system is | wanted ' Jupiter' 2.7% rank 5 | top [(' about', 9.4), (' the', 6.2), (' a', 5.8)]
The first person to walk on the moon was | wanted ' Neil' 2.9% rank 3 | top [(' a', 5.9), (' the', 3.6), (' Neil', 2.9)]
The Eiffel Tower is in the city of | wanted ' Paris' 7.0% rank 1 | top [(' Paris', 7.0), (' London', 5.8), (' New', 3.1)]
Once upon a | wanted ' time' 17.7% rank 1 | top [(' time', 17.7), (' certain', 8.5), (' moment', 3.1)]

sentences                                     8
the true ending is the model's first guess    3      cold, Paris (Eiffel Tower), time
the true ending is further down               5

Pick a sentence and look at the bars. Guess first: for "Two plus two equals", which number does the model think most likely?

"Two plus two equals one" is false. It is also a possible English sentence, and every bar in the interactive is a way in which an English sentence could go on. The model was trained to be good at exactly that.

Let us be fair to the model. A guess such as "the" is not false in itself, because "The capital of France is the city of Paris" is a true sentence. The experiment does not show eight errors. It shows that the order of the bars is an order of likelihood, in which truth has no say.

The word for a fluent answer that is false is hallucination. It is a misleading word, because it suggests that the model does something unusual when it happens. It does not. A true answer and an invented one are produced by the same steps. The model cannot feel the difference.

Six kinds of question to a larger model

Each of the six mistakes below has a cause that we can name. GPT-2 small is from 2019. We asked a newer and larger open model, Qwen2.5 with 1.5 billion weights, trained to be an assistant. It is still small by today's measure, so its mistakes are easy to provoke. Large models make the same kinds of mistake, less often.

1. A book that does not exist

     asked     "In two sentences, summarise the novel 'The Silent Orchard of Kepler' by Maya Lindqvist."
     answered  '"The Silent Orchard of Kepler" by Maya Lindqvist is a novel that explores themes of nature, loss, and the human connection to the environment. The story revolves around a young girl who discovers a hidden orchard in the forest, where she finds a magical tree that speaks to her. As she learns to care for the orchard and its inhabitants, she discovers the importance of preserving nature and the value of human connection. The novel emphasizes the beauty'

We invented the title and the author. The model wrote a summary with themes, a main character and a talking tree. Cause: the most likely continuation of "summarise the novel X" is a summary. "I have never heard of this book" is a much rarer continuation in the text the model learned from. The word "orchard" in the title was enough to supply the trees.

2. An event in the future

     asked     'Who won the football World Cup in 2038? Answer in one sentence.'
     answered  "I'm sorry, but the World Cup is an annual event, so it's impossible to predict the winner of the World Cup in 2038. The World Cup is held every four years, so the next World Cup will be in 2022."

The model rightly declines to name a winner. But it calls the World Cup an annual event, says in the next sentence that it is held every four years, and names 2022 as the next one. Cause: each sentence, taken alone, is a likely one. The model does read its earlier sentences while it writes, which makes agreement with them likely. No step makes it certain, and in a small model it fails often. The year 2022 is probably there because much of the text the model learned from was written before that tournament, when 2022 really was the next one.

3. Today's date

     asked     "What is today's date? Reply with the date only."
     answered  '2023-11-26'

We asked on 28 September 2026. Cause: the model has no clock. Its weights were frozen when training ended, and everything it knows is from text written before a certain day. That day is called the knowledge cutoff. For the model it is always about that day. The date it wrote is a likely date and no more. It need not be the exact day of the cutoff.

4. Arithmetic

     asked     'What is 4837 x 9216? Reply with the number only.'
     answered  '441004192'
     correct   44,577,792

the model's answer       441,004,192
the correct answer        44,577,792
too large by a factor    441,004,192 / 44,577,792  =  about 9.9

The answer begins with 44, as the correct one does, and it has one digit too many. Cause: the model does not calculate. It writes the digits of a number that looks like the result of such a multiplication. We saw in Chapter 4 that a tokenizer cuts a long number into pieces. The model guesses one piece after another, from left to right. A person who multiplies on paper starts at the right and carries.

5. Letters inside a word

     asked     'How many times does the letter r appear in the word strawberry? Reply with the number only.'
     answered  '3'
     correct   3
     asked     'How many times does the letter e appear in the word nevertheless? Reply with the number only.'
     answered  '3'
     correct   4
     asked     'How many times does the letter e appear in the word bookkeeper? Reply with the number only.'
     answered  '4'
     correct   3

One right out of three (the lines marked "correct" were counted by our program). We print the right one too, because it teaches something: a correct answer does not prove that the model can do the task. Cause: the model never sees letters. It sees the numbers of tokens, and "strawberry" may be a single one. The question about strawberry is famous and has been written about often, so the right answer may be the familiar one and no more.

6. A private fact

     asked     "What is the name of Maya's cat? Reply with the name only."
     answered  'Whisker'

Maya's cat is called Miso. Cause: the model has never heard of Maya. We asked for a name only, so it gave one, the kind of name a cat would have. This is the most important case for anyone who builds an application. What is not in the weights and not in the prompt does not exist for the model. It will fill the gap with something likely.

The causes, side by side

Every repair in the last column has the same form: give the model what it lacks. That is the subject of the next chapter, where we repair three of these six answers: the private fact, the date and the sum.

What the model inherits from its text

A model repeats the patterns of the text it was trained on, including the patterns we would rather it did not have. If doctors are mostly "he" and nurses mostly "she" in the training text, the model will lean the same way when it writes a story. If most of the text is English and about rich countries, the model knows those best. This is called bias. It is a different thing from the bias of Chapter 2, the number added to a sum; the two only share a word. The makers work against it in fine-tuning, the second phase of training that we met in Chapter 8, and they have not removed it.

What a language model is good at

A language model is strongest when the material it needs is in the prompt. This chapter may leave the wrong impression, so let us set it right. The same mechanism that invents a book does a great deal very well.

The left column has one thing in common. In each case the material is in front of the model, or the subject was covered thousands of times in its training text. The model is at its best as a worker on text that you hand to it. It is at its worst as an oracle that is asked for facts from memory.

python 05_wrong_and_fixed.py