How the weights were learned · How an LLM Works: Novice to Ninja

In this chapter, we will learn where the weights come from. We will train a single weight by hand, step by step. Then we will train a small language model from random numbers on a laptop and read what it writes at each stage. At the end we will know what training, loss, learning rate and fine-tuning mean.

The loss: how wrong was the guess?

The loss is one number that says how wrong the model was. Zero is perfect, and larger is worse.

For a language model the loss is computed from one thing only: the chance the model gave to the token that really came next. This way of measuring is called the cross-entropy loss. Suppose the training text says "Maya lives in Paris, the capital of France". The model read up to "of" and gave _France a chance of 28.81%.

The formula uses ln, the natural logarithm, which undoes the power of e of Chapter 7: ln(0.2881) is the power to which e must be raised to give 0.2881. For a chance below 100% that power is negative, and the minus sign makes the loss positive.

loss  =  - ln(chance given to the right token)          ln is a key on a calculator

chance  100 %       loss  = - ln(1.00)    =  0.00       sure and right: perfect
chance   28.81 %    loss  = - ln(0.2881)  =  1.24       our model on our sentence
chance    1 %       loss  = - ln(0.01)    =  4.61       nearly missed it
chance    0.002 %   loss  = - ln(0.00002) = 10.82       a blind guess among 50,257 tokens

A model that knows nothing gives every token the same chance, 1 in 50,257. Its loss is ln(50,257) = 10.82. Training begins there and works downwards.

Notice what is not needed: a person who marks answers as right or wrong. The text is its own answer key. Every position in every text is a question, "what comes next?", with the answer printed one token further on. The loss of a piece of text is the average over all its positions. This is why language models can be trained on amounts of text that no team of people could ever label.

Training one weight by hand

Gradient descent is the method of moving a weight a small step in the direction that lowers the loss. Let us do it with the smallest model there is: one weight, no bias.

the model            guess = w · x
the hidden rule      y = 3 · x        (the model does not know this)
the examples         x = 1, 2, 3, 4      y = 3, 6, 9, 12
the start            w = 0

With w = 0 the model guesses 0 for everything. For this small example we measure the loss as the average of the squared errors.

errors        0 - 3,  0 - 6,  0 - 9,  0 - 12      =  -3, -6, -9, -12
squared       9,  36,  81,  144
loss          (9 + 36 + 81 + 144) / 4             =  67.5

Now the question of hot and cold: should w go up or down, and how strongly does the loss react? The answer is a number called the slope, or gradient. It says how much the loss changes for each small turn of the weight. A negative slope means "turn the weight up and the loss will fall". We can measure it by turning the weight a very little and computing the loss again.

loss at w = 0          67.5
loss at w = 0.01       (2.99² + 5.98² + 8.97² + 11.96²) / 4              =  67.0508
change of the loss     67.0508 - 67.5                                =  -0.4492
slope                  -0.4492 / 0.01                                =  about -45

the exact rule for this loss: the average of 2 x error x input
                       (2 x -3 x 1) + (2 x -6 x 2) + (2 x -9 x 3) + (2 x -12 x 4)  =  -180
                       -180 / 4                                      =  -45

We do not jump the whole way. We multiply the slope by a small number, the learning rate, and move the weight by that amount, against the slope.

new weight  =  old weight  -  learning rate x slope

step 1      0       - 0.05 x (-45)       =  2.25
step 2      2.25    - 0.05 x (-11.25)    =  2.8125
step 3      2.8125  - 0.05 x (-2.8125)   =  2.9531

    step   w        loss (mean squared error)   slope
       0   0.0000   67.5000                     -45.0000
       1   2.2500   4.2188                      -11.2500
       2   2.8125   0.2637                      -2.8125
       3   2.9531   0.0165                      -0.7031
       4   2.9883   0.0010                      -0.1758
       6   2.9993   0.0000                      -0.0110
       8   3.0000   0.0000                      -0.0007
      12   3.0000   0.0000                      -0.0000

After eight steps the weight is 3.0000, to four decimals. The model has found the hidden rule. It was only told, after each guess, which way was warmer.

The learning rate decides the size of each step. Move the slider, which runs from 0.005 to 0.140, and press "One step" or "Ten steps". Guess before you try: what happens if the learning rate is twice as large, 0.100? And at 0.140?

With small steps the weight creeps. With large steps it jumps over the target and swings from side to side. With steps that are too large, each jump lands further away than the one before, and the loss grows without end.

in our example the slope is always 15 x (w - 3)       at w = 0:  15 x (-3)  =  -45

distance from 3 after a step  =  distance before x (1 - 15 x learning rate)

learning rate 0.050     1 - 0.75  =   0.25      a quarter of the distance is left
learning rate 0.100     1 - 1.50  =  -0.50      the other side of 3, half as far away
learning rate 0.140     1 - 2.10  =  -1.10      the other side, further away: it never settles
the border              2 / 15    =   0.1333    above it, every step lands further away

And with billions of weights?

Backpropagation is the method that works out the slope of every weight in the model from a single guess. Turning each weight a little and guessing again, as we did above, would need one guess for every weight. Backpropagation gets all the slopes in one go. It starts at the loss and works backwards through the layers, and it gives each weight its share of the blame. After one guess, every weight knows whether to go up or down, and by how much.

1. take a piece of text           "Maya lives in Paris, the capital of"   and the answer  "_France"
2. let the model guess            chance given to _France: say 28.81 %
3. compute the loss               - ln(0.2881) = 1.24
4. backpropagation                a slope for every weight
5. move every weight a little     new = old - learning rate x slope
6. take the next piece of text    and repeat, many millions of times

A real training run

We built a small language model and trained it from random weights. To keep it small, its tokens are single letters. It reads 8 letters and guesses the ninth. Its training text is the memory book on this shelf.

(c) A tiny language model, trained from random weights
    training text    306,963 characters from the memory book, lower case
    vocabulary       29 tokens (one token = one character): ' ,.abcdefghijklmnopqrstuvwxyz'
    the model        reads 8 characters, guesses the next one
      E   shape (29, 16)       464 weights
      W1  shape (128, 192)  24,576 weights
      b1  shape (192,)         192 weights
      W2  shape (192, 29)    5,568 weights
      b2  shape (29,)           29 weights
      total            30,829 weights
    a blind guess    1 of 29 tokens: loss = ln(29) = 3.367

E is an embedding table with 29 rows of 16 numbers. W1 and W2 are two weight matrices with a bend between them, and b1 and b2 are their biases. The model has no attention. It lays the rows of its 8 letters side by side and works on all of them at once.

numbers that go into W1      8 letters x 16 numbers      =  128
smaller than GPT-2 small     124,439,808 / 30,829        =  about 4,000 times

    step     loss on text it trains on   loss on text it never saw   a sample, starting from 'the '
         0   3.367                       3.367                       'the lybygjuim smgtfkaicesflzysmfbzmaptox,mk,pvoe'
       100   2.554                       2.583                       'the nv wefsdi seboke laer nytlif wh rors nh rre '
      1000   2.066                       2.086                       'the mpay mome shas mamber, wtin. wh thes miclore'
      5000   1.511                       1.575                       'the locred. of one leach, its meast trit lides f'
     15000   1.277                       1.395                       'the it yourself . folling its on the store answe'
     30000   1.171                       1.316                       'the explainn and the jobohy. nobey, rows in time'

    training took    16 seconds, 30,000 steps of 128 examples = 3,840,000 guesses corrected

Step 0. Random weights, random letters. The loss is 3.367, exactly the loss of a blind guess among 29 tokens.
Step 100. It has found that spaces come every few letters and that some letters are more common than others.
Step 1,000. The groups of letters have the length of words. Full stops and commas appear.
Step 5,000. Real words arrive: "of", "one", "its".
Step 30,000. "the", "and", "rows", "in", "time". It even begins "explain". Most of the rest looks like English and is not.

Move through the training below. Before you do, guess the loss after 2,000 steps.

A loss can be turned back into a chance, because e ^ -loss undoes the ln. The loss is an average over many letters, so the result is the typical chance of the right letter, on text the model never saw.

loss at the start    3.367      chance of the right letter    e ^ -3.367  =   3.4 %
loss at the end      1.316      chance of the right letter    e ^ -1.316  =  26.8 %

the model became     26.8 / 3.4  =  about 8 times better at guessing the next letter

One more test shows what the model learned. After the eight letters "the mode" we asked for its chances.

    after 'the mode' the trained model expects
      'l'   99.0 %
      'r'    0.3 %
      ' '    0.2 %
      'c'    0.1 %
      's'    0.1 %

The memory book uses the word "model" very often. The small model is 99% sure, and it is right.

From our toy to a real model

Pre-training is the first, long phase, in which a model learns to guess the next token on a very large collection of text. The P in GPT stands for pre-trained. The method is the one we used. The scale is not.

	Our toy	A large model
Weights	30,829	billions, and by reports that are not confirmed, trillions
Training text	306,963 characters, one book	a large part of the public internet, books and program code
Machine	one laptop	thousands of graphics cards in a data centre
Time	16 seconds	weeks to months
Method	guess the next token, compute the loss, move every weight a little

The right column gives rough sizes, because the makers of the large models do not publish exact figures. Our program and the large ones use a refined form of the step, which also looks at the slopes of the steps before. The idea stays the same.

Fine-tuning is a second, much shorter phase of training on chosen examples, to shape how the model behaves. A model that has only been pre-trained continues any text. Give it a question and it may continue with another question. To make an assistant, the makers train it further on examples of good conversations: a question, then a helpful answer. Later phases teach it to follow instructions, to refuse harmful requests and to say when it does not know.

random weights
     |   pre-training: guess the next token on a very large amount of text
     v
a model that continues any text
     |   fine-tuning: examples of questions with good answers
     v
an assistant
     |   training ends: the weights are frozen and the file is copied to the servers
     v
you use it. Nothing you type changes the weights.

Inference is the word for using a finished model. During inference the weights are only read. The model does not learn from your conversation, and it knows nothing that happened after its training text was collected. That date is called the knowledge cutoff, and Chapter 11 shows what follows from it.

Try it yourself

Train the model of this chapter on your own laptop. It needs one library and about sixteen seconds.

pip install numpy
python 02_neuron_and_training.py

Then change one thing at a time in the program and run it again.

Set HID from 192 to 32. The model has fewer weights. How much worse is the final loss?
Set CTX from 8 to 3. The model sees only three letters. What happens to the samples?
Replace the training text by a text of your own, of at least 100,000 characters. The program reads it into the variable raw. Whose style do the samples take on?

Common questions

Does the model learn from my chats?

Not while you chat. The weights are frozen during use. A provider may use conversations later as training text for a future model. Its terms of use say whether it does, and many offer a switch to refuse.

Why does training need so much text?

Each step moves each weight only a little, and there are very many weights. Our toy needed 3,840,000 corrected guesses for 30,829 weights and was still far from English. A model with billions of weights, which should also know facts about the world, needs many thousands of times more.

Why not take large steps and finish sooner?

Because large steps jump over the target, as the interactive shows. In practice the learning rate is raised gently at the start and lowered towards the end.

Is the training text stored inside the model?

Not as text. The model stores weights, and the weights hold the patterns of the text. But patterns can be strong: a passage that appeared very often in the training text can sometimes be written out by the model word for word.

Can I fine-tune a model myself?

Yes. Open models can be fine-tuned on one graphics card, and several providers offer fine-tuning of their models as a service. For most tasks it is not the first thing to try. A better prompt, or giving the model the right documents, is faster and cheaper, as Chapter 12 shows.

Carry this

Loss = -ln(chance given to the right token). A blind guess among 50,257 tokens has a loss of 10.82.
One step of training: guess, compute the loss, find the slope of every weight, move every weight a little against its slope.
The learning rate is the size of the step. Too small is slow, and too large never settles.
Our toy went from a loss of 3.367 to 1.316 in 30,000 steps and 16 seconds.
Pre-training teaches the model to continue text. Fine-tuning makes it an assistant. After that the weights are frozen.

Check yourself

1. A model gives the right next token a chance of 50%. What is the loss?

Answer

loss  =  - ln(0.50)  =  0.693

2. A weight is 2.0, its slope is +0.8 and the learning rate is 0.1. What is the weight after the step?

Answer

new weight  =  2.0 - 0.1 x 0.8  =  1.92

The slope is positive, so the weight goes down.

3. During training, the loss on the training text keeps falling, and the loss on the text set aside starts to rise. What is happening, and what would you do?

Answer

The model is overfitting: it is learning its training text by heart. Stop the training at the point where the loss on the text set aside was lowest, or give the model more training text.