Part II · Apprentice
Chapter 7
From scores to a word
The model ends with 50,257 scores. Turn them into chances, turn the temperature, cut off the tail, and draw. Here is why the same question can get different answers.
On this page
In this chapter, we will learn how the model gets from its last 768 numbers to one word. We will turn scores into chances with a formula called softmax, and see what the temperature does to real scores. Then we will learn the two settings, top-k and top-p, that keep a model from writing nonsense.
One score for every token
A logit is the raw score the model gives to one token. After the last layer, the 768 numbers of the final token are compared with every row of the token table, by one more multiplication. The result is 50,257 scores. For our sentence, "Maya lives in Paris, the capital of", these are the five highest (measured on GPT-2 small).
final numbers of the last token 768 rows of the token table 50,257 multiplications 50,257 x 768 = 38,597,376 for one single guess
token score (logit) probability
_France -69.61 28.81 %
_the -69.74 25.30 %
_a -71.33 5.16 %
_French -71.79 3.24 %
_Spain -72.31 1.92 %
The scores look strange. They are all negative, and they are close together. That does not matter, because a score has no meaning alone. Only the gaps between scores count.
gap between _France and _the -69.61 - (-69.74) = 0.13 nearly equal gap between _France and _a -69.61 - (-71.33) = 1.72 a clear lead
Softmax: from scores to chances
Softmax is the formula that turns a list of scores into chances that are all positive and add up to 100%. It has three steps.
- Raise the number e, which is about 2.718, to the power of each score. A calculator has a key for it.
- Add up all the results.
- Divide each result by that sum.
Let us do it by hand with four tokens. We keep the real gaps, rounded to one decimal, and move the scores to friendlier values. Moving all scores by the same amount changes nothing, as we will see.
token score e ^ score divided by the sum chance
_France 3.0 20.09 20.09 / 44.15 = 45.5 %
_the 2.9 18.17 18.17 / 44.15 = 41.2 %
_a 1.3 3.67 3.67 / 44.15 = 8.3 %
_French 0.8 2.23 2.23 / 44.15 = 5.0 %
------- -------
sum 44.15 100.0 %
(the four rounded values add up to 44.16; the sum of the unrounded values is 44.15)
Why the power of e, and not something simpler? Because it has two properties we need. It turns every score, even a negative one, into a positive number. And it turns a gap between two scores into a ratio between two chances.
a gap of 1.0 e ^ 1.0 = 2.72 the token is 2.72 times as likely a gap of 2.0 e ^ 2.0 = 7.39 a gap of 5.0 e ^ 5.0 = 148 _France over _a, real gap 1.72 e ^ 1.72 = 5.58 check with the real chances 28.81 / 5.16 = 5.58
Move the four sliders below. They start at the scores of our table. First guess: if you raise all four scores by 1.0, what happens to the chances?
With four tokens, France gets 45.5%. In the real model it has 28.81%, because the real softmax runs over all 50,257 tokens and the thousands of unlikely ones take their share together.
the ten together 70.77 %
tokens needed to reach 90 % 122 of 50,257
tokens that share the first 90 % 122 tokens that share the last 10 % 50,257 - 122 = 50,135
This long row of unlikely tokens is called the tail. It will matter in a moment.
Drawing a token
Sampling means drawing one token by chance, where each token is as likely to be drawn as its chance says. This is the spin of the wheel. The other choice is to take the token with the highest chance every time. That is called greedy picking.
| Way of picking | What you get | Good for |
|---|---|---|
| Greedy: always the highest | the same answer every time; it tends to be flat and to repeat itself | pulling facts out of a text, sorting into classes, tests |
| Sampling: draw by chance | a different answer each time; livelier, and sometimes wrong | writing, ideas, conversation |
Temperature
Temperature is a number by which every score is divided before softmax. It makes the wheel more even or less even.
chance of a token = e ^ (score / temperature) divided by the sum over all tokens temperature 1.0 the scores are used as they are temperature 0.5 every gap is doubled: the leaders take more temperature 2.0 every gap is halved: the tail takes more temperature 0 not a division at all: by agreement it means greedy picking
We applied five temperatures to the real scores.
(h) Temperature divides every score before the probabilities are computed
token T=0.2 T=0.7 T=1.0 T=1.5 T=2.0
_France 65.69 % 46.67 % 28.81 % 6.95 % 1.71 %
_the 34.29 % 38.76 % 25.30 % 6.37 % 1.60 %
_a 0.01 % 4.00 % 5.16 % 2.21 % 0.72 %
_French 0.00 % 2.06 % 3.24 % 1.62 % 0.57 %
_Spain 0.00 % 0.98 % 1.92 % 1.14 % 0.44 %
_which 0.00 % 0.91 % 1.83 % 1.11 % 0.43 %
Read the row of _France from left to right. At 0.2 it has 65.69%. At 2.0 it has 1.71%.
Now compare the columns for 1.0 and 2.0. Every token of the table has fallen. Where did the chance go? It
went into the tail.
at temperature 2.0, the six tokens above 1.71 + 1.60 + 0.72 + 0.57 + 0.44 + 0.43 = 5.47 % everything else 100 - 5.47 = 94.53 %
At temperature 2.0, about 95 draws in 100 land on a token that is not among the six best. The text falls apart. Turn the temperature yourself, from 0.10 to 2.00, and draw twenty tokens at each setting. The interactive shows the ten best tokens and one bar for all the others. Guess first: at which temperature do "all other tokens" together get more than half of the chance?
In the interactive, "all other tokens" pass one half between the temperatures 1.15 and 1.20. A small rise above 1.0 is enough to hand most draws to the tail.
Cutting off the tail: top-k and top-p
Top-k keeps only the k tokens with the highest chances. Top-p keeps the smallest group of tokens whose chances add up to p. Everything else is removed from the wheel, and the remaining fields are widened so that they fill it again.
at temperature 1.0
top-k = 5 keep _France, _the, _a, _French, _Spain
their chances 28.81 + 25.30 + 5.16 + 3.24 + 1.92 = 64.43 %
widened again _France 28.81 / 64.43 = 44.7 %
top-p = 0.5 add up from the top until 50 % is reached
_France 28.81 %
+ _the 54.11 % 50 % is passed: stop
2 tokens stay _France 28.81 / 54.11 = 53.2 %
The temperature is applied first, and the cut is made on the chances that result. The two cuts differ in one useful way. Top-k always keeps the same number of tokens. Top-p keeps few tokens when the model is sure and many when it is unsure.
after "is a member", the model gave _of 99.0 % top-p = 0.9 keeps 1 token the model is sure: no room for play after "Paris, the capital of" top-p = 0.9 keeps 122 tokens the model is unsure: many may come
Set the temperature in the interactive to 2.0, where the text fell apart. Then switch top-k to 40. The tail is gone, and the draws are sensible words again.
Which settings to use
The right setting depends on whether the task has one right answer or many good ones.
| Task | Temperature | Why |
|---|---|---|
| Pull a date or a name out of a text | 0 | there is one right answer, and it should be the same tomorrow |
| Sort into classes, write program code | 0 to 0.3 | small slips are costly |
| Answer questions, summarise | 0.3 to 0.7 | natural wording, little risk |
| Write a story, find names and ideas | 0.8 to 1.2 | variety is the point |
| Almost none | above 1.5 | rarely useful, as our bakery showed |
These ranges are common practice and not a law. The right value depends on the model, so try a few. Providers advise changing temperature or top-p and leaving the other alone, because both do a similar job.
Try it yourself
Do one softmax by hand, with a calculator. Three tokens have the scores 2.0, 1.0 and 0.0.
e ^ 2.0 = 7.39
e ^ 1.0 = 2.72
e ^ 0.0 = 1.00
sum = 11.11
chances 7.39 / 11.11 = 66.5 %
2.72 / 11.11 = 24.5 %
1.00 / 11.11 = 9.0 %
Now do it again at temperature 0.5. Divide each score by 0.5 first, which gives 4.0, 2.0 and 0.0. You should get about 86.7%, 11.7% and 1.6%. The leader has grown and the last has nearly vanished.
Common questions
Why does the same question give me different answers?
Because the token is drawn by chance. If the first draw differs, the text is different from there on, and every later guess is made for a different text. Two answers can part at the second word and end in different places.
Can I get the same answer every time?
Set the temperature to 0. Many services also accept a seed, a number that fixes the throws of the dice, so that the same request draws the same tokens. Even then, providers promise only "mostly the same", because the machines that run the model change over time.
Is a high chance a sign that the model is right?
It is a sign that the model has seen such continuations often. For well known facts that goes together with being right. For rare or private facts it does not. The chance measures how usual a continuation is. It does not measure whether it is true.
Why are the scores negative?
They come out of a multiplication and can land anywhere. Softmax only uses the gaps. If we added 70 to every score of our table, the chances would be the same to the last digit.
Do reasoning models also draw by chance?
Yes. A model that "thinks" before it answers writes its thinking as tokens, in the same loop, and then writes the answer. Some providers fix the temperature for such models and ignore or refuse the value you send. Read the documentation of the model you use.
What is the difference between a chance and a probability?
None in this book. "Probability" is the word of the mathematicians, written as a number from 0 to 1. We say "chance" and write it in percent. 0.2881 and 28.81% are the same thing.
Carry this
- The model ends with one score, a logit, for each of its 50,257 tokens. Only the gaps between scores count.
- Softmax: raise e to each score, add up, divide. A gap of 1.0 makes a token 2.72 times as likely.
- Temperature divides the scores. Low: the leaders take nearly all. High: the tail takes over. 0 means always the highest.
- Top-k keeps a fixed number of tokens. Top-p keeps as many as are needed to reach a share.
- Temperature sets the variety of the answers. It does not make them true.
Check yourself
1. Two tokens have the scores 4.0 and 1.0, and there are no others. What are their chances?
Answer
e ^ 4.0 = 54.60 e ^ 1.0 = 2.72 sum = 57.32 chances 54.60 / 57.32 = 95.3 % 2.72 / 57.32 = 4.7 %
2. At temperature 1.0 our model gives _France 28.81%. With
top-k set to 2, what is its chance?
Answer
the two that stay 28.81 + 25.30 = 54.11 % _France 28.81 / 54.11 = 53.2 %
3. A colleague sets the temperature to 0 "so that the model stops making things up". Will it work?
Answer
No. The answers will be the same each time, which helps with testing. But the model will still give its most likely continuation, and where that is false it will be false every time. What helps against made-up answers is giving the model the facts, which is the subject of Chapter 12.