Part II · Apprentice
Chapter 5
Embeddings: numbers become meaning
A token id is only a house number. The embedding table gives each token a place on a map, where Monday lives next to Tuesday. Look at a real table and its neighbourhoods.
On this page
In this chapter, we will learn how a token number becomes something the model can calculate with. We will meet the embedding table, the first and largest table of weights in the model. We will look up real rows, measure which tokens are neighbours, and draw a map of 63 of them.
The problem with ids
A token id says which token it is, and nothing about what the token means.
_France id 4881 _Paris id 6342 _banana id 25996
If we put the ids straight into Y = W · X, the model would treat 6342 as "a bit more than 4881". Paris would be a slightly larger France. That is meaningless. The ids are positions in a list, and the list has no order that matters.
So the model does not calculate with the id. It uses the id to look something up.
The embedding table
The embedding table has one row for every token of the vocabulary, and every row is a list of numbers that the model learned. Such a list is called an embedding or a vector. The length of the list is the width of the model.
(d) Each id picks one row of the embedding table table shape (50257, 768) = 50,257 rows x 768 numbers row 6342 _Paris [-0.219, +0.042, +0.121, -0.260, +0.028, -0.051, ... 762 more] row 3139 _capital [+0.057, -0.007, +0.091, +0.064, -0.029, +0.058, ... 762 more]
This is the real table of GPT-2 small. Token 6342 picks row 6342. No calculation is needed for that. It is like opening a book at a page number.
tokens The _capital _of _France _is
ids 464 3139 286 4881 318
| | | | |
v v v v v look up one row for each id
rows [768 nrs][768 nrs][768 nrs][768 nrs][768 nrs]
5 tokens x 768 numbers = 3,840 numbers go into the first layer
rows 50,257 numbers in each row 768 weights in the table 50,257 x 768 = 38,597,376 share of the whole model 38,597,376 / 124,439,808 = 31.0 %
Nearly a third of GPT-2 small is this one table. The numbers in it are weights like all the others. At the start of training they were random. They got their values in the same way as every other weight, which Chapter 8 explains.
What do the 768 numbers mean?
No single number of an embedding has a meaning we can name. What carries meaning is the direction of the whole list, compared with the direction of other lists.
A position on a paper map needs 2 numbers. A position in a room needs 3. A token gets 768, because words differ in many more ways than two or three: thing or action, living or not, large or small, polite or rude, and hundreds of ways that have no name. Nobody can picture 768 directions. We do not have to. One rule is the same in 2 directions and in 768: tokens that are used in similar ways get rows that point in similar directions.
To compare two rows we use the cosine similarity. Draw each row as an arrow from the centre of the map. The cosine is a number that says how nearly the two arrows point the same way. The length of the arrows plays no part in it. The program at the end of this chapter works it out with one multiplication.
same direction cosine 1.0 used in the same way similar direction cosine 0.6 related at right angles cosine 0.0 nothing in common opposite cosine -1.0 opposite
The four lines above show the scale. We measured two real pairs on the table of GPT-2 small.
cosine(_Paris, _France) = 0.634 cosine(_Paris, _banana) = 0.211
Paris and France 0.634 Paris and banana 0.211 ratio 0.634 / 0.211 = 3.0 the cosine with France is three times as high
Paris and banana have nothing to do with each other, and still their cosine is 0.211 and not 0. In this table all rows lean a little in one common direction, so even unrelated tokens reach about 0.2. Read 0.2 as "nothing in common" here, and judge every other value against it.
Who lives next door?
The nearest neighbours of a token are the rows with the highest cosine to its own row. For six tokens we asked: of all the other 50,256 rows, which point most nearly the same way?
(e) Rows that lie close together (cosine similarity in the embedding table) _Paris Paris 0.83 _France 0.63 _French 0.58 _London 0.58 France 0.57 _Copenhagen 0.57 _France France 0.83 _French 0.71 _Germany 0.70 _Spain 0.68 _Italy 0.68 French 0.66 _cat _Cat 0.71 _cats 0.69 Cat 0.64 cat 0.60 _dog 0.55 _Cats 0.54 _Monday _Tuesday 0.89 _Thursday 0.85 _Wednesday 0.84 _Friday 0.82 _Sunday 0.81 _Saturday 0.80 _happy happy 0.70 _Happy 0.69 _happier 0.67 _unhappy 0.66 Happy 0.66 _pleased 0.63 _doctor _doctors 0.78 _physician 0.77 doctor 0.71 _Doctor 0.68 _physicians 0.66 Doctor 0.65
Read the line for _Monday. Its six nearest neighbours are the six other days of the week.
Nobody gave the model a calendar. Read the line for _cat. In
Chapter 4 we saw that "cat", " cat" and "Cat" are different tokens with different
ids. Here they have found each other again: the model learned that they are nearly the same thing.
Pick one of the twenty tokens below and read its eight nearest neighbours. Before you look, guess the nearest one.
A map of 63 tokens
A flat map shows many rows at once, at the price of throwing most of their directions away. We took 63 tokens from eight families: countries, cities, animals, days, numbers, food, feelings and jobs. There are eight of each, and seven days. We cannot draw 768 directions, so we flattened the rows onto the two directions in which these 63 differ most.
Pick a family and watch where its dots sit. Guess before you look: which family will sit furthest away from all the others?
measured on the 768 numbers, not on the flat map average cosine, two tokens of the same family 0.53 average cosine, two tokens of different families 0.26 ratio 0.53 / 0.26 = about 2
The days of the week sit in one tight group, far from everything. Countries and cities form a second group. Animals, food and jobs overlap on the flat map. They are not mixed up in the model. They are apart in some of the other 766 directions, which a page cannot show. The map keeps only 17.9% of the spread (measured), and it says so under the picture.
Where is the token in the sentence?
A position embedding is a second row of 768 numbers that says at which place in the text a token stands.
The token table gives "dog" the same row wherever it stands. But "the dog bit the man" and "the man bit the dog" are different events. So GPT-2 has a second, smaller table with one row for each position: row 1 for the first token, row 2 for the second, up to row 1,024. The row of the position is added, number by number, to the row of the token. These rows are weights too, and they were learned in training together with the token rows. That is why the sum still works: the model learned to tell from it both which token it is and where it stands.
token row of _Paris [-0.219, +0.042, +0.121, ...] + position row of place 5 [ ... ... ... ...] = what enters the first layer: "the token _Paris, standing at place 5" position table 1,024 rows x 768 = 786,432 weights
The size of this table is the reason why GPT-2 cannot read more than 1,024 tokens at once. It has no row for position 1,025. Newer models mark the position in other ways and can read far longer texts, which is the subject of Chapter 10.
The table is only the start
The row from the table is the meaning of a token alone, before it has seen its sentence. The word "bank" gets the same row in "the river bank" and in "my bank account". Which bank is meant can only be decided by looking at the neighbours. That is the work of the layers. Each layer changes the 768 numbers of every token a little, using the tokens around it. After the last layer, the numbers of "bank" in the two sentences are quite different.
id --> row from the table the token alone this chapter
--> after layer 1 a little of its sentence Chapter 6
--> ...
--> after layer 12 the token in its sentence, Chapter 7
ready for the guess
Try it yourself
With Python, you can ask the real table about any token. This prints the nearest neighbours of a word of your choice.
import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
tok = GPT2TokenizerFast.from_pretrained("openai-community/gpt2")
table = GPT2LMHeadModel.from_pretrained("openai-community/gpt2").transformer.wte.weight.detach()
unit = table / table.norm(dim=1, keepdim=True) # make every arrow the same length
word = " Monday" # mind the space in front
row = tok.encode(word)[0]
scores, ids = (unit @ unit[row]).topk(7) # Y = W . X, once more
for s, i in zip(scores[1:], ids[1:]):
print(f"{tok.decode([int(i)])!r:14} {float(s):.2f}")' Tuesday' 0.89 ' Thursday' 0.85 ' Wednesday' 0.84 ' Friday' 0.82 ' Sunday' 0.81 ' Saturday' 0.80
These are the values of the _Monday line above. Look at the line with the comment "once
more". Once every arrow has the same length, the cosine of two rows is their product: multiply them number
by number and add up. So finding the neighbours of a token is the multiplication of Chapter 3: the whole
table, times one row. Try " king", " Tokyo" and " seven".
Common questions
Who decided that Monday and Tuesday belong together?
Nobody. The rows started as random numbers. In training, the model had to guess the next token in a very large amount of text. "Monday" and "Tuesday" are followed by the same kinds of words, such as "morning", "evening" and "at noon". Rows that lead to the same guesses were pushed in the same direction, step by step.
Can I read a meaning out of one of the 768 numbers?
Usually not. There is no number for "is an animal" and another for "is large". Such properties are spread across many numbers at once, as directions that do not line up with any single number. Finding out what the directions stand for is a field of research of its own, called interpretability.
Is the famous "king - man + woman = queen" true?
It was found in 2013 in smaller embedding tables that were built for that purpose, and it holds there roughly, not exactly. In the table of GPT-2 we measured a cosine of 0.657 between "king" and "queen", so the two are close neighbours. We did not test the arithmetic, and we make no claim about it here.
Why are the cosines so far from 1?
A cosine of 1 would mean two tokens are used in exactly the same way, and then one of them would be unnecessary. Even "Monday" and "Tuesday" reach only 0.89. At the other end, unrelated tokens sit near 0.2 in this table. What matters is the order: which tokens are nearer than others.
Does a bigger model have longer rows?
Yes. The width grows with the model. GPT-2 small uses 768 numbers, the largest GPT-2 uses 1,600, and the Qwen model we call in Chapter 9 uses 1,536, as its published design says. Longer rows can hold finer differences, and they make every table in the model larger.
Is the embedding table used again at the end?
In GPT-2, yes. To turn the final 768 numbers into a score for each token, the model multiplies them with every row of the same table. A token whose row points the same way as the final numbers gets a high score. The table is stored once and counted once in the 124,439,808 weights. Not every model shares its table in this way. Chapter 7 starts from there.
Carry this
- A token id is a house number. The embedding is the place on the map.
- The embedding table of GPT-2 small has 50,257 rows of 768 numbers: 38,597,376 weights, 31% of the model.
- Tokens that are used in similar ways have rows that point in similar directions. Cosine similarity measures it: 0.634 for Paris and France, 0.211 for Paris and banana.
- Near means "appears in the same places", which can even be the opposite meaning.
- The row is the token alone. The layers turn it into the token in its sentence.
Check yourself
1. A model has a vocabulary of 200,000 tokens and a width of 4,096. How many weights does its embedding table hold?
Answer
200,000 rows x 4,096 numbers = 819,200,000 weights
2. "happy" and "unhappy" have a cosine of 0.66. Does the model think they mean the same?
Answer
No. The table says they appear in the same kinds of sentence, which is true. The difference between them lies in other directions, and the layers use it. Nearness in the table is about use, and similar meaning is only one reason for similar use.
3. A sentence of 9 tokens enters GPT-2 small. How many numbers go into the first layer?
Answer
9 tokens x 768 numbers = 6,912 numbers