The paper on the desk: state

In this chapter, we will learn how to prepare the state, which is the text a decision model reads before it answers. We will see the three shapes a state can take, count one call in tokens, find the size limit, and then bury a tenant's message under 150 old ones to measure what that costs.

What the state is

The state is the content we ask the model to judge, sent in the state field of the request. In Chapter 2 a request had three fields: model, state and questions.

The documentation defines it like this: "State is the content you ask a System One model to evaluate. It could be a support message, a passage of text, or the current state of your application." Two rules follow.

• One state per request. The documentation says: "All questions see the same state and are evaluated independently."
• Nothing else is sent. The request has no other field for background, and the documentation warns: "Do not rely on knowledge stored in model weights when current information can come from your own knowledge base." If the clerk should know that flat 14B has a boiler contract, that fact must be on the desk.

Three shapes, text only

A state is a string, an object or an array. A string is one piece of text. An object is a set of named fields, written in JSON with curly brackets. An array is a list, written with square brackets.

Most of our experiments use the simplest shape. This function from experiments/desk.py builds the state for one row of the dataset:

def state_of(row):
    return "Tenant message:\n" + row["text"]

For real work the documentation recommends the second shape: "Use an object for most requests so each part of the state has a descriptive name and its relationships remain clear." Here is an object state for the desk. It is an invented example, not one we measured.

{
  "new_request": "My bedroom radiator is stone cold and it is freezing outside. Flat 14B. Let yourselves in if I am out.",
  "flat": {"number": "14B", "open_jobs": 1},
  "heating_contract": "Boiler and radiators are covered. Portable heaters are not."
}

Named fields give us something to point at. To aim a question at one part of the state, the documentation says to "name it in the instructions with a dot-and-index path to its key, including the backticks". A backtick is the ` character. The documentation's example is "Does `ticket.messages[0].text` request a refund?". For our object, a question could ask "Is the problem in `new_request` covered by `heating_contract`?".

Tokens: how the paper is measured

A token is a small piece of text, such as a short word, a part of a long word or a punctuation mark, and it is the unit in which a model's input is counted and billed. Before a model reads text, software called a tokenizer cuts the text into tokens.

The documentation names no tokenizer for Jev and no tool to count tokens in advance. So we do not count them ourselves. We read the count that every response reports in usage.input_tokens. The state and the questions are both input, so both are counted.

In experiment 0002, a call with one tenant message and the desk's three questions used 694 input tokens on average. The published price is $0.042 per million input tokens, and the answers are free.

tokens in one call (measured, mean of 215 calls)      694
price per token (published)                           $0.042 / 1,000,000
cost of one call      694 x 0.042 / 1,000,000     =   $0.000029
cost of 1,000 calls   0.000029 x 1,000            =   $0.029

How is that total divided between the state and the questions? A response reports one number, so experiment 0027 sent seven versions of one request, built so that two rows differ in one part only. The message is the one from Chapter 2: 173 characters about water dripping through a ceiling light. A bare Noul is one yes/no statement with no criteria.

the message is 173 characters, 33 words

state              questions                         input tokens
one word ('x')     one bare Noul, no criteria             272
the message        one bare Noul, no criteria             313
one word ('x')     trade (Choice, 6 options)              419
the message        trade                                  460
the message        trade + severity                       562
the message        trade + severity + may_enter           625
the message twice  trade + severity + may_enter           667

the call of Chapter 2, part by part
  the smallest call: a one-word state and one bare Noul           272
  the message                                 313 - 272     =      41
  the trade Choice in place of the bare Noul  419 - 272     =     147
  the severity Score                          562 - 460     =     102
  the may_enter Noul                          625 - 562     =      63
  total               272 + 41 + 147 + 102 + 63             =     625

characters per token          173 / 41        =  about 4.2
the message's share           41 / 625        =  6.6%

The 272 tokens are spent before the message and the real questions arrive. They cover the one-letter state, a four-word statement, and whatever the service adds to every request. TypeSafe does not itemise that part. The lesson is plain all the same: on a short message, the questions and the fixed part weigh far more than the state.

The 625 belongs to one short message. The average call in experiment 0002 was 694 tokens, with the same three questions, so the difference is the messages.

longer messages, same questions      694 - 625    =  69 more tokens
the average message of our dataset   41 + 69      =  about 110 tokens
at 4.2 characters per token          110 x 4.2    =  about 460 characters

So two sizes are in play, and we will say which one we mean. The Chapter 2 message is 41 tokens. An average message of our dataset is about 110, nearly three times as long. Both are small next to the questions, and that changes as soon as the state grows.

The context limit

The context limit is the largest number of tokens one request may contain. It decides how much paper fits on the desk.

Two published numbers exist, and we give both. TypeSafe's documentation says: "64k tokens per request; 32k tokens for state plus the longest question", where k means thousand. The first budget covers the state plus all questions. The second covers the state plus the single longest question. OpenRouter, the service through which our experiments call Jev, lists a context length of 32,000 tokens and describes it as "the state you send plus the questions". That may be TypeSafe's second budget in looser words, but neither page says so.

Our desk calls through OpenRouter, so the safe reading is the stricter one: the state and all the questions together stay under 32,000 tokens. How many old messages from our dataset is that?

budget (published)                                       32,000 tokens
one message and three questions (measured, 0008)            712 tokens
left for more state                 32,000 - 712      =  31,288 tokens
one old message (derived, 0008)     (17,137 - 712) / 150  =  about 110 tokens
old messages that fit               31,288 / 110      =  about 284

The 110 comes from experiment 0008, later in this chapter, where 150 old messages of average length raised a call from 712 to 17,137 tokens. It matches the average message we derived above. Messages as short as the one in Chapter 2 would fit in larger numbers.

The largest calls we sent averaged 17,137 tokens, a little over half of that budget. We have not tested what happens near the limit or above it.

What belongs on the desk

A good state holds everything the questions need and nothing they do not. The documentation gives three pieces of advice, and we adopt them.

The last piece comes with a reason. The documentation says: "Accuracy falls as the state grows with content unrelated to the decision. Unrelated detail acts as a distractor." It calls the effect context rot: answers getting worse as unneeded material piles up.

One more kind of content belongs on the desk: facts the back office has already worked out. The clerk does not do sums. If a question depends on a total or on a number of days, code should compute it and write it into the state as a named field. In experiment 0009 we asked whether a repair quote was over an approval limit, on 100 invented quotes. The model was right 92% of the time when it had to add up the raw line items, and 100% of the time when the code had put the total in the state (the 100 quotes have 92 different totals, so that row is 92 different requests). Chapter 10 is about this split.

A cluttered desk, measured

Experiment 0008 adds unrelated old messages to the state and records tokens, latency, cost and accuracy. Latency is the waiting time for one answer. As in Chapter 2, p50 is the middle call and p95 is the time that 95 calls in 100 beat.

We took 60 of our labelled requests (from the half of the dataset called the test split). Each state was an object. The field new_request held the message to judge, and the field older_requests_from_other_flats held 0, 5, 20, 60 or 150 other messages. Every question ended with "Judge only `new_request`." This is what the program printed. Tokens and cost are means over the 60 calls.

old messages added   input tokens   p50 ms   p95 ms   $ per call   trade   severity   may_enter
         0                 712      278      342   $0.000030   100.0%    96.7%   100.0%
         5                1270      249      341   $0.000053    98.3%    96.7%   100.0%
        20                2929      260      331   $0.000123    98.3%    96.7%   100.0%
        60                7351      270      377   $0.000309    98.3%   100.0%   100.0%
       150               17137      333      416   $0.000720    98.3%    98.3%   100.0%

• Tokens grew in a straight line, about 110 for every message added.
• Latency hardly moved. The median with 5 old messages was lower than with none, so differences of 30 ms are noise between runs. Only at 150 messages is there a clear rise.
• Accuracy moved by a request or two. With any old messages present, 59 of the 60 trades were right (98.3%) against 60 with none, and no more were lost as the pile grew thirty times larger. Severity gained one or two requests, which is as likely to be chance as the loss was.
• Cost followed the tokens.

tokens     17,137 / 712              =  24.1 times
cost       0.000720 / 0.000030       =  24 times
latency    333 - 278 = 55 ms,   55 / 278   =  20% slower

per 1,000 calls     0.000030 x 1,000  =  $0.030      bare message
                    0.000720 x 1,000  =  $0.720      with 150 old messages

The clutter cost 24 times the money and a fifth more time, and bought nothing.

Before you touch the slider below, predict one number: with 40 old messages in the state, what do 1,000 calls cost? Then move the slider. Positions between the five measured points are read from a straight line between their neighbours, and the interactive says so when it does that.

python3 experiments/0008-state-size/main.py

tokens      712 + 40 x 110              =  5,112
one call    5,112 x 0.042 / 1,000,000   =  $0.000215
1,000 calls 0.000215 x 1,000            =  $0.215