Part III · Journeyman
Chapter 10
The context window and the bill
Everything about a model is counted in tokens: how much it can read, how much it may write, and what you pay. Do the arithmetic once, and see why a long chat costs far more than it seems.
On this page
In this chapter, we will learn the two limits that every user of a language model meets: how much text fits into one request, and what a request costs. We will do the arithmetic with published prices, and we will find out why a conversation of forty turns costs 3.6 times as much as one of twenty.
The context window
The context window is the largest number of tokens a model can take in at one time, counting the prompt and the answer together.
Everything must fit: the system prompt, the whole conversation so far, any documents, the new question, and the answer that is still to be written.
+--------------------------- the context window ----------------------------+
| system | earlier turns of the | documents, | the new | room for |
| prompt | conversation | memories | question | the answer|
+----------+----------------------------+------------+----------+-----------+
max_tokens
Why is there a limit at all? We met two reasons. GPT-2 has a position table with 1,024 rows, so it has no way to mark a token at place 1,025 (Chapter 5). And the work of attention grows with the square of the length, so a very long text is very costly to read (Chapter 6).
| Model | Year | Context window | About how much English |
|---|---|---|---|
| GPT-2 | 2019 | 1,024 tokens | 768 words: about two pages |
| GPT-6 Luna, Sol and Astra | 2026 | 1,050,000 tokens | 787,500 words: about nine novels |
GPT-2 1,024 tokens x 0.75 words per token = 768 words GPT-6 1,050,000 tokens x 0.75 words per token = 787,500 words a novel of 90,000 words 787,500 / 90,000 = about 9 novels growth in seven years 1,050,000 / 1,024 = about 1,025 times
The two window sizes are published by OpenAI. Those of the GPT-6 models are from its model pages, as we read them on 28 September 2026. The same pages give a separate limit for the answer: 128,000 tokens at most. The 0.75 words per token is the rule of thumb of Chapter 4. The novel of 90,000 words is a round number of ours. So the words and the novels are estimates.
The bill
A provider charges for tokens read and for tokens written, at two different prices, quoted per million tokens. Tokens read are called input. Tokens written are called output.
| Model | Input, per million tokens | Output, per million tokens |
|---|---|---|
| gpt-6-luna, the small one | $0.10 | $0.50 |
| gpt-6-sol, the middle one | $2.00 | $10.00 |
| gpt-6-astra, the large one | $10.00 | $50.00 |
These are OpenAI's list prices as published on 28 September 2026. Prices change often, so take them as an example and look up the current ones.
Let us price one request. Maya asks her cooking assistant a question. The application sends 1,000 tokens: a system prompt, some earlier turns and the question. The answer has 300 tokens. We made up these two sizes, and the 100,000 requests a day below, for illustration.
with gpt-6-luna input 1,000 tokens x $0.10 / 1,000,000 = $0.0001 output 300 tokens x $0.50 / 1,000,000 = $0.00015 one request = $0.00025 100,000 requests a day $0.00025 x 100,000 = $25 a day thirty days $25 x 30 = $750
the same request with gpt-6-astra input 1,000 tokens x $10 / 1,000,000 = $0.01 output 300 tokens x $50 / 1,000,000 = $0.015 one request = $0.025 100,000 requests a day $0.025 x 100,000 = $2,500 a day thirty days $2,500 x 30 = $75,000
One request costs a fraction of a cent. That is why a single question feels free. The bill appears when the request is multiplied by a hundred thousand requests a day. Pick a model and move the three sliders. Guess first: if the answer is three times as long, does the cost triple?
the answer three times as long, with gpt-6-luna input 1,000 tokens x $0.10 / 1,000,000 = $0.0001 as before output 900 tokens x $0.50 / 1,000,000 = $0.00045 one request = $0.00055 against the first request $0.00055 / $0.00025 = 2.2 times
The cost does not triple, because the input stayed the same.
Why writing costs more than reading
Every token written is a full trip through the model. Tokens that are read can be worked on together, in one pass.
When the model reads a prompt of 1,000 tokens, it can work on all 1,000 at the same moment, because they are all there. When it writes 300 tokens, it must make 300 trips, one after the other, because each token depends on all the tokens before it. On the list above, a token written costs five times a token read: $0.50 / $0.10 = 5.
our request: share of the tokens that are output 300 / 1,300 = 23 %
share of the bill that is output $0.00015 / $0.00025 = 60 %
Less than a quarter of the tokens make up more than half of the bill. If you want to save money, ask for shorter answers first.
A conversation costs more than its turns
Because the model is stateless, every request carries the whole conversation so far, and all of it is paid for again. We saw this in Chapter 9: the model knew the name of the cat only when the earlier messages were sent again.
Let us count, with sizes made up for illustration. The system prompt has 500 tokens. Each turn, a question with its answer, adds 250 tokens.
request 1 500 + 1 x 250 = 750 tokens
request 2 500 + 2 x 250 = 1,000 tokens
request 3 500 + 3 x 250 = 1,250 tokens
...
request 20 500 + 20 x 250 = 5,500 tokens
paid for, all 20 requests together
system prompt 20 x 500 = 10,000
turns 250 x (1 + 2 + 3 + ... + 20)
250 x 210 = 52,500
total = 62,500 tokens
The conversation itself is only 20 x 250 = 5,000 tokens long. We paid for 62,500. Now let it run twice as long.
40 turns
system prompt 40 x 500 = 20,000
turns 250 x (1 + 2 + ... + 40)
250 x 820 = 205,000
total = 225,000 tokens
twice the turns 40 / 20 = 2
the tokens paid for 225,000 / 62,500 = 3.6 times
To add the numbers from 1 to n quickly, multiply n by n + 1 and halve the result. For 20 that is 20 x 21 / 2 = 210. Because n is multiplied by itself, the sum grows with the square of the turns.
Move the number of turns below and watch the two lines: what one request sends, and what has been paid for in all. Guess first: with these numbers, at which turn is a window of 128,000 tokens full?
the window is full when 500 + n x 250 = 128,000
n = (128,000 - 500) / 250 = 510 turns
The slider for the turns ends at 400, so the window of 128,000 tokens is never full in the interactive. Make the window smaller, or the turns longer, to see it overflow.
In this simple count we charged every token at the input price, including the 250 new ones of each turn. In truth the answer inside each turn is first charged once as output, at the higher price, so the real bill is a little above ours.
What applications do about it
An application keeps the bill down by sending less text, or by sending text that the provider has already worked on. There are five common methods.
| Method | What it does | What it costs |
|---|---|---|
| Cut | send only the last few turns | everything older is forgotten |
| Summarise | replace old turns by a short summary | details are lost with each rewriting |
| Remember | store facts outside the model, and send only the few that fit the question | a memory system must be built |
| Retrieve | search documents, and send only the passages that fit | a search system must be built |
| Cache | let the provider reuse its work on a prompt that begins the same way | no extra work, but the beginning of the prompt must stay the same, token for token |
On OpenAI's price list, cached input costs a tenth of normal input: $0.10 / 10 = $0.01 per million for gpt-6-luna. A long system prompt that is the same in every request is therefore much cheaper than it looks. The earlier turns of a conversation also begin the same way each time, so caching softens the growth we counted above. It does not remove it, and the window fills as fast as before.
The third row is a subject of its own. The memory book on this shelf, AI Agent Memory: Novice to Ninja, is about nothing else. The fourth row is called retrieval, and we meet it in Chapter 12.
Try it yourself
Work out the monthly bill of an application you use or would like to build. You need four numbers.
requests per day ______ tokens sent with each request ______ (words / 0.75, plus the system prompt) tokens written in each answer ______ price per million, input and output ______ and ______ one request = tokens sent x input price / 1,000,000 + tokens written x output price / 1,000,000 one month = one request x requests per day x 30
Then do it a second time with a model that costs twenty times as much. Is the better answer worth the difference for your task? For many tasks, such as sorting messages into classes, the small model is enough.
Common questions
I pay a fixed price per month for a chat assistant. Where are the tokens?
They are counted all the same, behind the scenes. A subscription is a flat rate with limits: after a certain amount of use the service slows down or asks you to wait. The provider has done the arithmetic of this chapter for an average user.
What happens when my text does not fit into the window?
The server refuses the request with an error. Some applications avoid the error by silently dropping the oldest turns. That is why a long chat can suddenly "forget" what was said at the beginning.
Do the tokens of the system prompt cost money on every request?
Yes. They are read each time. A system prompt of 2,000 tokens, sent 100,000 times a day, is 200 million tokens a day. At $0.10 per million that is $20 a day, and with caching about a tenth of it.
Do "thinking" tokens cost money?
Yes. A model that reasons before it answers writes its reasoning as tokens. They are charged as output even when the application does not show them. An answer of 100 visible tokens can stand on top of several thousand tokens of reasoning.
Will prices keep falling?
They have fallen steeply for a given level of quality. In the same years, applications began to send much more text with each request, and the most capable models stayed expensive. It is safer to plan with the arithmetic than with a hope.
Is it cheaper to run an open model on my own machine?
It depends on how busy the machine is. A graphics card costs the same per hour whether it answers one request or a thousand. With steady, heavy use your own machine can be cheaper. With light use the provider is, because you pay only for what you send. Chapter 13 looks at the machines.
Carry this
- The context window holds the prompt and the answer together. GPT-2: 1,024 tokens. The GPT-6 models of 2026: 1,050,000.
- Price = tokens read x input price + tokens written x output price, per million. Output costs five times input on the 2026 list.
- One request at the small model: $0.00025. A hundred thousand a day for thirty days: $750.
- Every request resends the conversation. In our example, twice the turns cost 3.6 times the tokens.
- The cures are to cut, summarise, remember, retrieve and cache.
Check yourself
1. A request sends 4,000 tokens and gets an answer of 500 tokens, with gpt-6-sol at $2 and $10 per million. What does it cost?
Answer
input 4,000 x $2 / 1,000,000 = $0.008 output 500 x $10 / 1,000,000 = $0.005 request = $0.013
2. A conversation has 10 turns of 300 tokens each, and no system prompt. How many input tokens are paid for in all, if every request resends everything?
Answer
300 x (1 + 2 + ... + 10) = 300 x (10 x 11 / 2) = 300 x 55 = 16,500 tokens
3. You must lower the bill of an application, and you may halve only one thing. The requests send 800 tokens and the answers have 600. Which do you shorten?
Answer
The answers. With output at five times the price of input, the 600 tokens written cost as much as 3,000 tokens read, against only 800 that are read. Halving the answers saves far more than halving the prompt would.
in units of the input price input 800 output 600 x 5 = 3,000 total 3,800 answers halved input 800 output 300 x 5 = 1,500 total 2,300 prompt halved input 400 output 600 x 5 = 3,000 total 3,400