Part II · Apprentice
Chapter 4
Tokens: text becomes numbers
A computer can only multiply numbers, so the first step is to turn text into them. Cut your own sentences with a real tokenizer, and find out how many words a token is worth.
On this page
In this chapter, we will learn how a sentence becomes a list of numbers. We will meet the tokenizer, the small program that stands in front of every language model. We will cut real sentences into tokens, see why "strawberry" can be one token or three, and measure how many words a token is worth.
Why text must become numbers
A model can only multiply and add, and one cannot multiply a letter. The equation Y = W · X of Chapter 3 needs X to be numbers. So before the model sees anything, the text is replaced by numbers, and after the model has answered, numbers are replaced by text.
"The capital of France is" --> tokenizer --> [464, 3139, 286, 4881, 318] --> model
|
" Paris" <-- tokenizer <-- [6342] <----+
A token is a piece of text that has a number of its own. The tokenizer is the program that cuts text into tokens. The complete list of tokens that a model knows is called its vocabulary.
Three ways to cut a text
A token is usually smaller than a word and larger than a letter. To see why, let us look at the two simpler choices first.
| Cut the text into | Size of the list | The problem |
|---|---|---|
| letters | very small, about 100 for English with its digits and signs | a sentence becomes very long, and a single letter carries almost no meaning |
| whole words | endless | new words, names, typing mistakes and other languages are not on the list |
| pieces of words | fixed; 50,257 and 200,019 in the two lists of this chapter | none of the above: common words are one piece, rare words are built from several |
Nearly all current language models use the third way. The list is made once, before training, by a simple procedure. The best known one, used by the GPT models, is called byte pair encoding.
It starts with single bytes. A byte is the unit in which a computer stores text, and it has 256 possible values. An English letter is one byte. A letter of many other scripts takes two or three, and an emoji takes four or more. Then the procedure looks for the pair of pieces that occurs together most often in a large amount of text, and glues that pair into a new piece. It repeats this tens of thousands of times. GPT-2's list was built with exactly 50,000 such merges.
single bytes to start with 256 merges 50,000 one special token, "end of text" 1 vocabulary of GPT-2 256 + 50,000 + 1 = 50,257
A real tokenizer at work
The tokenizer replaces each piece of text by its number in the list. We ran the
tokenizer of GPT-2 on a short sentence. In the output, the mark _ stands for a
space, and r50k_base is the name our library gives to GPT-2's list.
(b) The running sentence, GPT-2 vocabulary (r50k_base). A leading space is shown as _ text "The capital of France is" tokens 5 pieces The | _capital | _of | _France | _is ids 464 3139 286 4881 318 text " Paris" tokens 1 pieces _Paris ids 6342
Five words became five tokens. That is a coincidence of this sentence, as we will see at once. Notice
the spaces. The space belongs to the word that follows it. The token is
_France, with the space in front, and its number is 4881. "France" without a space is a
different token with a different number.
The number of a token is called its id. It is only a position in the list. Token 4881 is not "larger" than token 464 in any sense that matters, in the same way that house number 12 is no better than house number 3.
Common words are one token, rare words are several
(d) Common words are one token. Rare words are split into pieces. 'cat' 1 token(s) cat ' cat' 1 token(s) _cat 'Cat' 1 token(s) Cat 'vegetarian' 3 token(s) ve | get | arian ' vegetarian' 1 token(s) _vegetarian 'tokenization' 2 token(s) token | ization 'unbelievably' 4 token(s) un | bel | iev | ably 'Maya' 2 token(s) May | a 'strawberry' 3 token(s) st | raw | berry ' strawberry' 1 token(s) _strawberry
There are three lessons in these ten lines.
The same word can be different tokens. "cat", " cat" and "Cat" are three tokens with three ids. To the model they are three different things at first, and it must learn in training that they are related.
A space changes the cut. " strawberry" with a space is one token, because that is how the word appears in the middle of a sentence. "strawberry" without a space is rare. It appears mostly at the very start of a text or straight after a quotation mark, so it is built from the three pieces "st", "raw" and "berry". "Maya" is cut in the same way, into two. Together with the comma, a token of its own, that is why the sentence of Chapter 1 has 7 words and 9 tokens.
The pieces do not follow meaning. "raw" inside "strawberry" has nothing to do with raw food. The pieces were chosen because they are frequent, and for no other reason. This is why language models have trouble with questions about letters, such as "how many r are in strawberry". The model never sees letters. It sees the number of a piece. Chapter 11 tests this on a real model.
Now try it. Type your own name, a long word, a number, a line of another language. Before each one,
guess the number of tokens. The interactive draws a space as ␣. It uses GPT-2's list, which
cuts most other scripts into single bytes and shows them as codes. The Telugu greeting of the next table
takes 24 tokens there (measured).
Numbers, code and other languages
(e) Numbers, code and other languages (o200k_base)
'1234567890' 10 characters 4 token(s)
'3.14159' 7 characters 4 token(s)
"print('hello')" 14 characters 4 token(s)
'https://interactive.carebun.com' 31 characters 7 token(s)
'नमस्ते' 6 characters 4 token(s)
'నమస్కారం' 8 characters 5 token(s)
'こんにちは' 5 characters 1 token(s)
'😀' 1 characters 1 token(s)
This part of the experiment used a newer vocabulary, called o200k_base, with 200,019 tokens.
OpenAI made it for its later chat models. Even there, a greeting in Hindi or Telugu takes four or five
tokens for one word. The Japanese greeting is so frequent that it has a piece of its own. English is cheap
because the lists were built mostly from English text. A long number is cut into groups of
digits, which is one reason why arithmetic is hard for a language model.
How many words is a token worth?
As a rule of thumb, one token is about three quarters of a word of English, or about four characters. OpenAI publishes this rule for planning. It is a rule of thumb and no more, so we measured it.
(f) How many words is a token worth? One paragraph of plain English. r50k_base 80 words 449 characters 92 tokens words per token 0.87 characters per token 4.88 o200k_base 80 words 449 characters 92 tokens words per token 0.87 characters per token 4.88
our paragraph 80 words / 92 tokens = 0.87 words per token the rule of thumb 0.75 words per token 1,000 words by the rule 1,000 / 0.75 = 1,333 tokens 1,000 words as we measured 1,000 / 0.87 = 1,149 tokens
Our paragraph was made of short, common words, so it needed fewer tokens than the rule says. A text with names, numbers, code or rare words needs more. The rule of 0.75 is on the careful side for plain English, and that is the right side to be on when you plan a budget.
Why do we care about the count? Because everything about a language model is measured in tokens. That covers how much text it can read at once, how long an answer may be, how fast it writes, and what it costs. Chapter 10 does that arithmetic.
The most text a model can read at once is called its context window, which Chapter 10 explains. Pick a text below or move the two sliders, and watch how many tokens the text needs and which of three windows it fits.
And back again
Turning numbers back into text is a lookup in the same list, in the other direction.
(g) Decoding is a table lookup in the other direction ids [464, 3139, 286, 4881, 318, 6342, 13] text 'The capital of France is Paris.'
Seven numbers, seven pieces, glued together without anything in between. The spaces are already inside the pieces. Token 13 is the full stop.
The tokenizer is ordinary software. It contains no weights and has learned nothing about meaning. It cuts and it looks up. All the knowledge is in what comes next: the table that gives every token number a place on a map, which is the subject of Chapter 5.
Try it yourself
The program behind this chapter is the shortest in the book. It needs one library and runs in a second.
pip install tiktoken
python 01_tokens.pyOr write your own in a few lines.
import tiktoken
enc = tiktoken.get_encoding("r50k_base")
ids = enc.encode("The capital of France is Paris.")
print(ids)
print([enc.decode([i]) for i in ids])[464, 3139, 286, 4881, 318, 6342, 13] ['The', ' capital', ' of', ' France', ' is', ' Paris', '.']
Then take a page of your own writing, count its words, and count its tokens. What is your own number of words per token?
Common questions
Why not simply use whole words?
Because the list would never be complete. People invent words, misspell them, and write in thousands of languages. With pieces of words, any text at all can be written: in the worst case, byte by byte. With whole words, every unknown word would have to become a token that means "unknown", and its meaning would be lost.
Is a token the same as a syllable?
No. Syllables follow how a word is spoken. Tokens follow how often a run of letters appears in text. "unbelievably" has five syllables and is cut into the four tokens "un", "bel", "iev" and "ably".
Does the model see my capital letters and my typing mistakes?
Yes. "Cat" and "cat" are different tokens. A misspelt word is cut into different pieces from the correct one. Large models have seen enough misspelt text to cope well, but the mistake is visible to them and it usually costs extra tokens.
Do pictures and sound become tokens too?
In models that handle them, yes. A picture is cut into small squares and sound into short slices of time. Each square or slice is turned into a list of numbers that travels through the model next to the text tokens. The principle of this chapter stays: everything becomes numbers before the model sees it.
Who decides the size of the vocabulary?
The designers of the model. A larger vocabulary makes texts shorter in tokens, which saves work. It also makes the tables at the two ends of the model larger. In GPT-2 small, the token table alone holds 38,597,376 of the 124,439,808 weights, which is 31%.
What is the "end of text" token?
A special token that is not part of any normal text. In training it marked where one document ended and the next began. When the model picks it while writing, the application stops the loop. Chat models have a few more special tokens, which mark where each speaker's turn begins and ends.
Carry this
- A tokenizer cuts text into pieces from a fixed list and replaces each piece by its id. GPT-2's list has 50,257 entries.
- Common words are one token. Rare words, names, numbers and most other languages take several.
- The space belongs to the following word:
_Franceis token 4881. - One token is about 0.75 words of English. We measured 0.87 on plain text. Plan with 0.75.
- Every model has its own tokenizer. Ids and counts cannot be carried from one model to another.
Check yourself
1. A report has 6,000 words of English. About how many tokens is it, by the rule of thumb?
Answer
6,000 words / 0.75 words per token = 8,000 tokens
2. Why is " strawberry" one token and "strawberry" three?
Answer
Because the list was built from frequency. In running text the word nearly always follows a space, so the piece with the space in front was frequent enough to get its own entry. The form without a space is rare, and it is put together from the pieces "st", "raw" and "berry".
3. You tokenize a text with the tokenizer of one model and send the ids to another model. What happens?
Answer
The second model looks up each id in its own list, where the same number stands for a different piece of text. It reads a meaningless sequence and answers with nonsense. No error is reported as long as every id exists in the second list. Only an id beyond the end of that list stops the program.