How big is a model?

In this chapter, we will learn what the size of a model means in practice. We will account for every byte of a real model file. We will climb the ladder of sizes from our toy to the largest models of 2026 and compare them with the human brain. Then we will work out how much memory a model needs, why it runs on graphics cards, and how fast it writes.

A model is a file

A trained model is a file that holds its weights, one number after the other. We opened the file of GPT-2 small and accounted for every byte.

(a) The model is a file of numbers
  weights           124,439,808
  bytes per weight  4 (a float32)
  weights x 4       497,759,232 bytes
  also in the file  12 fixed tables of 1,024 x 1,024 (not learned)   50,331,648 bytes
                    a header that names every tensor                14,291 bytes
  sum               497,759,232 + 50,331,648 + 14,291 = 548,105,171 bytes
  file on disk      model.safetensors   548,105,171 bytes

A float32 is a common way for a computer to store a number with decimals. It takes 32 bits. A bit is the smallest unit of storage, a single 0 or 1, and 8 bits make a byte, so a float32 is 4 bytes. The twelve fixed tables are a detail of this particular file: they tell each token that it may not look to its right, and they were stored along with the weights. The header is a list of contents.

weights        497,759,232 / 548,105,171   =  90.8 % of the file
fixed tables    50,331,648 / 548,105,171   =   9.2 %
header              14,291 / 548,105,171   =   0.003 %

There is no program code in the file and no text. The "knowledge" of the model is 124,439,808 numbers, and you could print them on paper. At 5,000 numbers to a page it would be a book of about 25,000 pages.

The ladder of sizes

The size of a model is its number of weights. Chapter 3 showed how it grew. Here is the whole ladder, with the human brain at the top. Each bar says how we know the number. Pick two rungs to compare, and read how many times larger one is than the other.

from GPT-2 small to GPT-6 Astra      10,000,000,000,000 / 124,439,808         =  about 80,000 times
from GPT-6 Astra to the brain       200,000,000,000,000 / 10,000,000,000,000  =  20 times

the same, with the ends of the ranges
smallest gap    100 trillion / 10 trillion     =  10 times
largest gap   1,000 trillion / 4.8 trillion    =  about 208 times

The 10 trillion is reported and not confirmed, and the 200 trillion is an estimate. If both are right, the largest model of 2026 is eighty thousand times GPT-2 small, and the brain is twenty times larger still. That sounds as if the machines had nearly arrived. Four cautions:

• The top two numbers are soft. OpenAI has not published the size of GPT-6 Astra. Ten trillion is the figure most often repeated, and another report says 4.8 trillion. The estimates for the brain differ by a factor of ten.
• A weight is not a connection. A weight is one number that never changes after training. A connection in the brain is a living structure that changes every day. One connection may be worth many weights, and nobody knows how many.
• Size is room, and says little about what is in it. A model of 2026 with a given number of weights is far more capable than a model of 2020 with the same number, because training improved.

The fourth caution needs a new term. Some large models are no longer one solid block. They are built as a mixture of experts. Inside each layer the feed-forward step exists in many copies with different weights, called experts. A small switch chooses a few of them for each token. Such a model holds far more weights than it uses on one trip. Several open models are built in this way. Whether GPT-6 Astra is one of them has not been published, so we cannot say how many of its reported weights work on each token.

Bits per weight

Quantization means storing each weight with fewer bits. The file shrinks in proportion, and the weights become a little less exact.

It is surprising that 16 values can be enough for a weight. It works because no single weight matters much. The answer of the model is the sum of millions of small contributions, and small errors in them largely cancel.

Will it fit?

To run at a useful speed, a model must sit in memory completely, because nearly every weight is needed for every token. In a mixture of experts each token uses only some of the weights, but the next token may need others, so all of them must be at hand. Memory here means the working memory of the machine, called RAM, and not its disk. The memory of a graphics card is called VRAM.

a model with 8 billion weights, at 16 bits
file               8,000,000,000 x 2 bytes     =  16,000,000,000 bytes  =  16 GB
working room       about 20 % on top           =  3.2 GB
memory needed      16 GB x 1.2                 =  19.2 GB

a laptop with 16 GB of memory                     does not fit
a graphics card with 24 GB                        fits

the same model at 4 bits
file               8,000,000,000 x 0.5 bytes   =  4 GB
memory needed      4 GB x 1.2                  =  4.8 GB        fits a laptop, and many phones

We count 1 GB as 1,000,000,000 bytes. The working room is for the numbers the model holds while it reads: 768 or more numbers for every token of the prompt, in every layer. Twenty percent is our rule of thumb and not a measurement. With a very long prompt it is more.

Choose a size and a number of bits. Guess first: how many data-centre cards of 80 GB would a model of 10 trillion weights need at 16 bits? That is the size reported, and not confirmed, for GPT-6 Astra.

10 trillion weights at 16 bits
file               10,000,000,000,000 x 2 bytes   =  20,000 GB  =  20 TB
memory needed      20,000 GB x 1.2                =  24,000 GB
cards of 80 GB     24,000 / 80                    =  300 cards

This is why the largest models exist only in data centres, and why you reach them through the API of Chapter 9.

Why graphics cards?

A graphics card, or GPU, does a very large number of small calculations at the same moment. It was built for pictures, where millions of points must be worked out for every frame of a game.

The work of a model has the same nature. Look at one weight matrix of GPT-2, a table of 768 by 3,072 weights. It turns the 768 numbers of a token into 3,072 weighted sums, and no sum needs the result of another. They can all be done at once.

one matrix of GPT-2 small       768 x 3,072                =    2,359,296 multiplications
all the matrices, one token                                =  123,532,032 multiplications

(m) Speed on this laptop (CPU only, no graphics card)
  one pass over the 9-token prompt       14.2 ms
  100 new tokens                          0.81 s  = 123 tokens per second
  weights inside multiplication tables    123,532,032
  each is used about once per new token, so one token costs roughly that many multiplications

multiplications for one token          123,532,032
tokens per second, measured                    123
multiplications per second     123,532,032 x 123     =  about 15 billion

Our laptop did about 15 billion multiplications a second with its ordinary processor, which is called the CPU. That was enough for a model of 124 million weights. Now take a model with 8 billion weights.

multiplications for one token    8,000,000,000 / 123,532,032   =  about 65 times as many
tokens per second, same speed    123 / 65                      =  about 2

This is an estimate from our one measurement, and it assumes that the processor keeps its pace. Two tokens a second are 2 x 0.75 = 1.5 words a second, slower than anyone reads. A graphics card is built to do this kind of work many times faster.

How fast does it write?

Two numbers describe the speed of a model: the time to the first token, and the tokens per second after that.

Both rows are our own measurements. For GPT-2 the time to the first token is one pass over its prompt of 9 tokens.

an answer of 300 tokens at 18 tokens per second
writing         300 / 18              =  16.7 s
in all          0.217 s + 16.7 s      =  16.9 s

you read        300 x 0.75 words      =  225 words
at 4 words per second   225 / 4       =  56.3 s

Four words a second is a common estimate for silent reading. The time to the first token is mostly the reading of the prompt. With a provider, the way over the internet comes on top. The tokens per second are the trips of the loop. Set the length of the answer and the speed below, and watch which is faster, the model or the reader.

At 18 tokens a second the model writes 18 x 0.75 = 13.5 words a second, faster than a person reads, so with streaming nobody waits. For a program that needs the whole answer before it can go on, 16.9 seconds is a long time. This is the reason why speed matters most in agents, where one task takes many requests.

Open and closed, small and large

A model has open weights when its maker publishes the file, and it is closed when the file stays with the maker.

Providers sell their models in sizes, as we saw on the price list of Chapter 10: a small, fast and cheap one, a middle one, and a large one that costs a hundred times as much as the small. A good habit is to start with the smallest model that does the task well, and to move up only where the answers are not good enough.

memory of my machine                        ______ GB
room for the model      memory / 1.2      =  ______ GB
weights at 4 bits       room x 2          =  ______ billion     (0.5 bytes per weight)
weights at 16 bits      room / 2          =  ______ billion     (2 bytes per weight)

16 bits     70,000,000,000 x 2 bytes     =  140 GB
 4 bits     70,000,000,000 x 0.5 bytes   =   35 GB

memory     140 GB x 1.2   =  168 GB
cards      168 / 80       =  2.1, so 3 cards

writing     600 / 40        =  15 s
in all      0.5 s + 15 s    =  15.5 s