Part I · Novice
Chapter 3
Y = W · X: the one equation
A whole layer of neurons is one multiplication. Do it by hand with twelve weights, learn the words matrix and tensor, and count the 124,439,808 weights of a real model.
On this page
In this chapter, we will learn the one equation that a language model repeats from its first layer to its last: Y = W · X. We will work it out by hand with twelve weights and learn what a matrix and a tensor are. We will see why one small bend between the layers is needed, and count the weights of a real model down to the last one.
One neuron, written short
The weighted sum of a neuron is written y = w1·x1 + w2·x2 + w3·x3 + ..., with one term for each input.
In Chapter 2 we built a neuron with four inputs. The letters are only names for what we already did. The dot means "times". We write it wherever the letter x is a name, so that the two cannot be confused.
x1, x2, x3, x4 the inputs wheels, engine, fur, carries people w1, w2, w3, w4 the weights +2.0, +3.0, -4.0, +1.5 y = w1 · x1 + w2 · x2 + w3 · x3 + w4 · x4 = 2.0 x 1 + 3.0 x 1 + -4.0 x 0 + 1.5 x 1 = 6.5 (and the bias of -3.0 brings it to 3.5)
The inputs are called features. A feature is one measurable property of the thing we are looking at. The x values change with every thing we show to the neuron. The w values stay the same. They are what the neuron has learned.
Many neurons at once: the matrix
A matrix is a table of numbers with rows and columns. When several neurons look at the same inputs, we write the weights of each neuron as one row. The rows together are the matrix W.
Let us take three neurons. Each answers one question about a thing: is it an automobile, is it an animal, is it a smartphone? They all look at the same four features. The weights are made up for illustration.
wheels engine fur screen
W = row 1 [ +2 +3 -3 -1 ] automobile
row 2 [ -2 -3 +4 -2 ] animal
row 3 [ -1 -1 -2 +5 ] smartphone
X = [ 1 1 0 0 ] a thing with wheels and an engine
To multiply W by X, we take each row in turn and do the weighted sum of a single neuron. We get one number for each row.
y1 automobile = (+2 x 1) + (+3 x 1) + (-3 x 0) + (-1 x 0) = +5 y2 animal = (-2 x 1) + (-3 x 1) + (+4 x 0) + (-2 x 0) = -5 y3 smartphone = (-1 x 1) + (-1 x 1) + (-2 x 0) + (+5 x 0) = -2 Y = [ +5, -5, -2 ] the highest score wins: automobile multiplications 3 rows x 4 features = 12
This is the whole meaning of Y = W · X. X is what we see, as a list of numbers. W is what the model knows, as a table of numbers. Y is one score for each possible answer.
Before you use the interactive, guess the three scores for a car with a big screen on its dashboard: wheels 1, engine 1, fur 0, screen 1. Which answer wins, and by how much?
Try the toy dog on wheels. It has wheels and fur. The fur outweighs the wheels, and "animal" wins with a modest +2. Nothing in the table says what to do with a thing the model has never met. It adds up what it has.
Vector, matrix, tensor
A tensor is a block of numbers with any number of directions. The word sounds difficult. It is only the general name for things we have already met.
| Name | Shape | Example in this chapter | How many numbers |
|---|---|---|---|
| Scalar | one number | the bias, -3.0 | 1 |
| Vector | a list | X = [1, 1, 0, 0] | 4 |
| Matrix | a table: rows and columns | W, 3 rows by 4 columns | 12 |
| Tensor | any of these, or a block with more directions | a stack of 12 tables like W | its sides multiplied: 12 x 3 x 4 = 144 |
The file of a language model is a collection of named tensors. When we opened GPT-2 small, we found two
tables at the entrance and then the tensors of its first layer. Programmers count from 0, so the first
layer is named h.0.
transformer.wte.weight (50257, 768) 38,597,376
transformer.wpe.weight (1024, 768) 786,432
transformer.h.0.ln_1.weight (768,) 768
transformer.h.0.ln_1.bias (768,) 768
transformer.h.0.attn.c_attn.weight (768, 2304) 1,769,472
transformer.h.0.attn.c_attn.bias (2304,) 2,304
transformer.h.0.attn.c_proj.weight (768, 768) 589,824
transformer.h.0.attn.c_proj.bias (768,) 768
transformer.h.0.ln_2.weight (768,) 768
transformer.h.0.ln_2.bias (768,) 768
transformer.h.0.mlp.c_fc.weight (768, 3072) 2,359,296
transformer.h.0.mlp.c_fc.bias (3072,) 3,072
transformer.h.0.mlp.c_proj.weight (3072, 768) 2,359,296
transformer.h.0.mlp.c_proj.bias (768,) 768
The names are those of the programmers and we do not need them yet. Look at the shapes. A shape with two numbers, such as (768, 3072), is a matrix: a W. Its size is the two numbers multiplied.
our toy W 3 x 4 = 12 weights one real W 768 x 3,072 = 2,359,296 weights
A shape with one number, such as (3072,), is a vector: the biases that belong to the matrix above it. The model is the same sum we did by hand, with larger tables.
Why one bend is needed
Two multiplications in a row can always be replaced by one, so layers that only multiply would be pointless. The model therefore bends the numbers a little between one layer and the next.
Let us see the problem with the smallest possible example. Take a layer that multiplies by 3, followed by a layer that multiplies by 2.
layer 1 y = 3 · x layer 2 z = 2 · y = 2 · 3 · x = 6 · x one layer with weight 6 does the same
The same holds for tables: two matrices applied one after the other can be multiplied into a single matrix. However many such layers we stack, the result is one multiplication. The cure is to put a simple rule between the layers. The best known one is called ReLU: a negative number becomes 0, and a positive number stays as it is.
ReLU(+5) = 5 ReLU(-5) = 0 ReLU(-2) = 0 Y = [ +5, -5, -2 ] after ReLU = [ 5, 0, 0 ] "automobile" passes on, the two others are silent
The bend is called an activation function. With it, the second layer can no longer be folded into the first, and a stack of layers can describe curves, steps and exceptions. The squashing between 0 and 1 in Chapter 2 was such a bend too. GPT-2 uses a smoother relative of ReLU, called GELU. The idea is the same.
X --> [ W1 . X ] --> bend --> [ W2 . ] --> bend --> [ W3 . ] --> ... --> Y
layer 1 layer 2 layer 3
Counting the weights of a real model
The number of weights follows mostly from two choices: how wide the model is and how many layers it has. The width is how many numbers describe one token while it travels through the model. In GPT-2 small the width is 768 and there are 12 layers.
We can count the weights from the shapes printed above. In the names, attn is attention,
mlp is the feed-forward step and ln is a tidy-up step. One more tidy-up step
follows the last layer.
token table 50,257 tokens x 768 = 38,597,376 position table 1,024 positions x 768 = 786,432 one layer attention 768 x 2,304 + 2,304 + 768 x 768 + 768 = 2,362,368 feed-forward 768 x 3,072 + 3,072 + 3,072 x 768 + 768 = 4,722,432 two tidy-up steps 4 x 768 = 3,072 one layer in all = 7,087,872 twelve layers 12 x 7,087,872 = 85,054,464 final tidy-up 2 x 768 = 1,536 total 38,597,376 + 786,432 + 85,054,464 + 1,536 = 124,439,808
Our program counted the weights inside the real file and printed the same number.
token embeddings (wte) 38,597,376 31.0 % position embeddings (wpe) 786,432 0.6 % layer norms 38,400 0.0 % attention, 12 layers 28,348,416 22.8 % feed-forward, 12 layers 56,669,184 45.5 % total 124,439,808
Later chapters explain these words: the token table in Chapter 5, attention and feed-forward in Chapter 6. "Layer norms" are the tidy-up steps. They keep the numbers from growing too large or too small on the way.
Now change the width and the number of layers. Guess first: if you double the width, does the number of weights double?
How the numbers grew
From 2018 to 2020 the published number of weights grew about 1,500 times, and press reports suggest it has grown much further since. For the early models the makers published the numbers. For the recent ones they do not.
| Model | Year | Weights | How we know |
|---|---|---|---|
| GPT-1 | 2018 | 117 million | published by OpenAI |
| GPT-2 small | 2019 | 124,439,808 | we counted them |
| GPT-2 XL | 2019 | 1,557,611,200 | derived with our formula; OpenAI says 1.5 billion |
| GPT-3 | 2020 | 175 billion | published by OpenAI |
| GPT-6 Astra | 2026 | about 10 trillion | reported in the press; not confirmed by OpenAI |
| Human brain, connections | about 200 trillion | an estimate, with a range of 100 to 1,000 trillion; see Chapter 2 |
GPT-3 / GPT-1 175,000,000,000 / 117,000,000 = about 1,500 times in two years GPT-6 Astra / GPT-3 10,000,000,000,000 / 175,000,000,000 = about 57 times in six years brain / GPT-6 Astra 200,000,000,000,000 / 10,000,000,000,000 = 20 times
Notice what grew. The features X did not become trillions. A sentence still enters as a handful of tokens, each described by hundreds or thousands of numbers. What grew is W, the tables of weights: more layers, and wider ones. Chapter 13 shows what such sizes mean for memory and speed.
Try it yourself
Take a sheet of paper. Write a matrix W with two rows and three columns, for two neurons that answer "is it a fruit?" and "is it a vehicle?". The three features are: grows on a tree, has wheels, is sweet. Choose six weights that seem sensible to you.
Then write X for an apple, a truck and a toy car made of chocolate. Work out Y for each. How many multiplications did you do in all?
one thing 2 rows x 3 features = 6 multiplications three things 3 x 6 = 18 multiplications
Common questions
Is a language model really only this one equation?
Almost. Its layers are made of multiplications by weight matrices, additions of biases, the bend, and the tidy-up steps. One more ingredient, attention, decides which tokens are mixed together. It uses weight matrices too, and it also multiplies the numbers of one token by those of another, which is not linear. Chapter 6 shows how.
What is the difference between a parameter, a weight and a feature?
A weight is one number of a matrix W. A bias is one number that is added. "Parameter" is the common name for both: every number that was learned in training. A feature is something else. It is one of the inputs X, and it changes with every text. The model stores its parameters. It does not store features.
Do the features inside a real model have names such as "has wheels"?
No. We chose named features so that the sums could be followed. Inside a real model a token is described by 768 numbers that nobody named. Researchers have found that some directions in those numbers stand for things one can describe, but most cannot be put into a word.
Why are the multiplications done on graphics cards?
A matrix multiplication is a very large number of small multiplications that do not depend on each other. A graphics card was built to do many small calculations at the same moment, for the millions of points of a picture. That is the right tool for this job too. Chapter 13 returns to it.
Why 768? It seems an odd number.
It is a choice of the designers, and no law of nature. It is 12 times 64, which divides evenly among the 12 attention heads of each layer that Chapter 6 explains. Larger models use larger widths: the largest GPT-2 uses 1,600.
Do I need to be good at mathematics to go on?
You need multiplication, addition and, in a few places, the number e. If you followed the twelve multiplications of the three neurons, you have all the mathematics this book uses.
Carry this
- Y = W · X. X is what comes in, W is what the model learned, Y is one score for each answer.
- A vector is a list, a matrix is a table, a tensor is the general word. A model file is a collection of tensors.
- Between the layers the numbers are bent, for example by setting negative numbers to 0. Without the bend, many layers would be worth one.
- GPT-2 small: width 768, 12 layers, 124,439,808 weights. One layer is about 12 × width × width.
- Weights grew from 117 million in 2018 to a reported, and not confirmed, 10 trillion in 2026. The inputs did not grow in the same way. The tables did.
Check yourself
1. With the matrix W of this chapter, work out Y for a smartphone: wheels 0, engine 0, fur 0, screen 1.
Answer
y1 automobile = (+2 x 0) + (+3 x 0) + (-3 x 0) + (-1 x 1) = -1 y2 animal = (-2 x 0) + (-3 x 0) + (+4 x 0) + (-2 x 1) = -2 y3 smartphone = (-1 x 0) + (-1 x 0) + (-2 x 0) + (+5 x 1) = +5 smartphone wins
2. A matrix has the shape (1024, 4096). How many weights does it hold, and how many bytes is that at 4 bytes per weight?
Answer
1,024 x 4,096 = 4,194,304 weights 4,194,304 x 4 B = 16,777,216 bytes = about 16.8 MB
3. A model has a width of 768. You may either double its layers from 12 to 24, or double its width to 1,536 and keep 12 layers. Which gives more weights in the layers?
Answer
24 layers, width 768 24 x (12 x 768 x 768 + 13 x 768) = 170,108,928 12 layers, width 1,536 12 x (12 x 1,536 x 1,536 + 13 x 1,536) = 339,978,240
Doubling the width gives about twice as many as doubling the layers, because each table grows in two directions.