Many questions, one call · Decision Models with Jev: Novice to Ninja

In this chapter, we will learn how to ask many questions about one state in a single call. We will measure what extra questions cost in time and in tokens, check that they do not disturb each other, meet the speculative question, and learn the one situation in which a second call is needed.

Independent and in parallel

A batch is a set of questions sent together in one call, about the same state. The desk has batched from its first call: trade, severity and permission travel in one request.

The documentation makes two promises about a batch: "Every question is evaluated in parallel and in isolation against the same state in one go."

In parallel means at the same time, not one after another. The documentation concludes: "Adding questions barely changes the response time."
Independent, or "in isolation", means that no answer influences another: "One question's answer is not hidden context for another. You can add or remove questions without changing the others' results."

Both are published claims, so we tested them.

What we measured

Experiment 0007 sends the same 12 tenant messages with 1, 3, 10, 30 and 100 questions per call and records latency and tokens. Latency is the waiting time for one answer, and p50 is the middle call: half were faster. The token column is the mean of the 12 calls.

The questions come from a pool of ten: the desk's three, plus seven small ones such as "The tenant is angry." and "The message gives a flat number." A call with 30 questions repeats the pool three times, and a call with 100 repeats it ten times.

questions per call   p50 ms   p95 ms   input tokens   tokens per extra question   $ per call
        1              222      285          539                               $0.000023
        3              233      297          704                82             $0.000030
       10              237      284          893                39             $0.000038
       30              275      302         1919                48             $0.000081
      100              305      360         5510                50             $0.000231

Independence: the trade question asked alone, and asked as question 1 of 30. Largest change in any option's probability:
  per request: 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00    max 0.00

The same question asked ten times in ONE call (q0, q10, q20 are all the trade question in the 30-question call):
   r049 [('electrical', 1), ('electrical', 1), ('electrical', 1)]
   r253 [('plumbing', 1), ('plumbing', 1), ('plumbing', 1)]
   r091 [('heating', 1), ('heating', 1), ('heating', 1)]
   r183 [('pests', 1), ('pests', 1), ('pests', 1)]

First, the speed. Going from 1 question to 100 added 83 ms.

latency    305 - 222           =  83 ms more for 99 more questions
           83 / 99             =  0.84 ms per extra question
           83 / 222            =  37% slower for 100 times the questions

tokens     5,510 - 539         =  4,971 more tokens
           4,971 / 99          =  about 50 tokens per extra question

Second, the independence. We asked the trade question alone, and then as the first of 30 questions. For each of the 12 messages we compared the six probabilities of the two answers. The largest difference was 0.00. The last block shows the trade question at three places inside one 30-question call, and the three copies agree. (Its heading says ten times. A 30-question call holds three copies.)

Those 12 messages were clear ones, with every probability near 0 or 1. Experiment 0037 repeated the check where the clerk is unsure: 30 contested requests, on which the writer and a second reader disagreed about severity. It asked the severity Score five times alone and five times as one of 30 questions.

mean spread of the score across 5 calls, asked alone:        0.054
mean spread of the score across 5 calls, asked among 30:     0.043
mean gap between the alone average and the batched average:  0.011
...
requests where the nearest level of the two averages differs: 0 of 30

An unsure answer moves a little between identical calls, so that movement is the yardstick. The spread is the highest of the five scores minus the lowest. Asked alone, it was 0.054 on average. The mean gap between asking alone and asking in a batch was 0.011, smaller than that, and no request changed its nearest level. Independence held here too, within the noise. Experiment 0035 saw the same on one unsure trade answer over 20 calls each way: confidence 0.66 to 0.75 alone, 0.69 to 0.78 with two other questions, mean 0.72 both times.

One call against N calls

The alternative to a batch is N calls with one question each, and by our numbers it costs more tokens and, run in a row, more waiting. Each of the N calls carries the tenant message again, plus whatever the service adds to every request. And if each call waits for the one before, the waiting adds up.

Take ten questions. One call with ten questions was measured: 893 tokens and 237 ms. A call with one question was also measured: 539 tokens and 222 ms. Ten such calls in a row are arithmetic.

                    one call, 10 questions      10 calls, 1 question each
tokens              893  (measured)             10 x 539     =  5,390
latency (p50)       237 ms  (measured)          10 x 222 ms  =  2,220 ms

latency ratio       2,220 / 237     =  9.4 times
token ratio         5,390 / 893     =  6.0 times

per day at 1,000 messages (an invented volume), at $0.042 per million tokens
one call each       893 x 1,000 x 0.042 / 1,000,000     =  $0.038
ten calls each      5,390 x 1,000 x 0.042 / 1,000,000   =  $0.226

The money is small either way: the difference is 19 cents a day. The time is the real difference. A program waiting for the desk gets its answers in a quarter of a second or in more than two seconds.

The one-question call asked the trade question, a Choice with six described options. A call with one short yes/no question is smaller: in experiment 0027, on one short message, a bare Noul took 313 tokens against 460 for the trade question. The smallest call we sent used 272 tokens, and ten calls of that size use 2,720, three times the 893. And ten calls sent at the same moment would finish sooner than 2,220 ms, but they would still pay for the state ten times and count as ten requests against the published limit of 80 requests per second.

TypeSafe publishes its own comparison, on 13 questions about a long article: 0.27 s for one call and 2.71 s for 13 calls in a row. One of its pages calls this "11.5x cheaper and 9.6x faster" and another "12.2x cheaper and 10.0x faster".

Predict first. With 30 questions, how many times longer do 30 calls in a row take than one call? The interactive uses the five measured points, draws a straight line between them for other numbers of questions, and says so when it does.

Speculative questions

A speculative question is a question we ask before we know whether its answer will be needed. The documentation calls the pattern "speculative fan-out": "Ask every question your code might need, including ones whose answer only matters for some inputs, and let the code decide which answers to use."

If a message is about pests, Rosa wants to know which animal. The slow design asks for the trade first, waits, and then sends a second call. The speculative design asks both at once.

The documentation phrases such questions conditionally, for example "If this is a shipping problem, which kind is it?". For the desk it could read as follows. This question is invented, and we have not measured it.

"pest_kind": {
  "type": "choice",
  "instructions": "If this is a pest problem, which animal is it?",
  "criteria": {
    "insects": "Ants, wasps, cockroaches, bed bugs or other insects.",
    "rodents": "Mice or rats.",
    "birds": "Pigeons or other birds.",
    "other": "Another animal, not stated, or not a pest problem."
  }
}

The back office reads the answer only when it needs it.

if answers["trade"]["choice"] == "pests":
    animal = answers["pest_kind"]["choice"]
# For every other trade the answer also came back. The code never reads it.

What does the speculation cost? At the pool's average, about 50 tokens. A described Choice like this one takes more: the trade Choice took 147 more than a bare Noul.

one speculative question       about 50 tokens
cost per call                  50 x 0.042 / 1,000,000        =  $0.0000021
cost per 1,000 calls           0.0000021 x 1,000             =  $0.0021

the alternative: a second call in a row, about 222 ms more on every pest message

At the average, a fifth of a cent per 1,000 messages buys back a fifth of a second. The documentation says the same without numbers: "Asking a question you might not need is close to free."

Note the "other" option. A speculative question is also asked about messages it does not fit, and the answer is still one of the options. The unneeded answer needs somewhere harmless to land.

When a second request is really needed

A dependent decision is one whose request cannot be written until an earlier answer is known. The documentation is strict about what counts: "The dependency is real only when your code cannot build the second request until it has the first answer: it needs the answer to fetch more data for the state, to decide what the state is made of, or to pick the next question's options."

It gives three examples from its cookbooks, which are its pages of worked examples.

The documentation's example	What the code does between the two requests	What is new in the second
Skill suggestion	"ranks 182 skills in one request, then fetches the full text of the top three and judges them again"	more data in the state
Structure recovery	"merges lines into blocks from those answers, then classifies the blocks, which did not exist until the first request had answered"	a different state
Hierarchical classification	"uses each Choice answer to decide which options the next request offers"	different options

An invented case for the desk: once the trade is known, the back office looks up which contractors of that trade are on call today, puts their notes in the state, and asks the clerk to pick one. The first request cannot hold that list, because the code does not yet know which list to fetch. Chapter 11 is about lists of candidates like this one.

The follow-up about the animal was not a real dependency. It could be asked against the original message, so it belongs in the first call. The documentation's rule: "Two requests are the exception, not the rule. If the second request's questions could have been asked against the original state, ask them in the first request and let the code ignore the ones it doesn't need."

What a batch cannot do, and how big it can be

Inside one call, the documentation says, no answer becomes input for another question, so a batch cannot reason in steps. Suppose Rosa wrote these two questions, which are invented:

q1   "How severe is the problem described?"
q2   "Given that severity, should the tenant be told to leave the flat?"

By that rule the model answering q2 does not see the answer to q1. It sees the state and the words of q2, and it reads them literally, as Chapter 8 showed. There are two repairs. Write q2 so that it stands alone: "The message describes a situation in which the tenant should leave the flat now." Or drop q2 and let the code decide from the severity level. Either way the chain moves into the back office, the subject of Chapter 10.

How many questions fit in one call? The documentation states no maximum count. The bound is the token budget from Chapter 7: 64k tokens for the state plus all questions, and 32k for the state plus the longest question, while OpenRouter lists 32,000 tokens.

questions of the size in our pool, about 50 tokens each, after a first call of 539 tokens

under 64,000 tokens      (64,000 - 539) / 50    =  about 1,269 questions
under 32,000 tokens      (32,000 - 539) / 50    =  about 629 questions

These are estimates from a straight line, and the largest call we sent had 100 questions. Questions also differ in size. In experiment 0011 a call with one Choice of 255 options used 6,860 tokens, more than our whole call of 100 small questions. Count tokens, not questions.

Try it yourself

Take the decision you sketched in Chapter 7. Write down every question your code might ever ask about one state, including the ones that matter only for some inputs. Mark each one: does it need an earlier answer before it can be written at all? Only those go into a second call.

With the book's repository and an OpenRouter key you can repeat the measurement. This experiment turns the cache off for its timing runs, so every run makes real calls. Ours cost under one cent.

python3 experiments/0007-batch/main.py

Is the row for 100 questions still within about 100 ms of the row for one?

Common questions

Does the order of the questions matter?

No page of the documentation says that it does, and the independence it promises implies that it does not. In our run the trade question gave the same answer at positions 0, 10 and 20 of one call, on the four messages we printed. The order of the options inside one Choice is a different matter, covered in Chapter 13.

Do more questions make each answer worse?

The documentation says no: "adding more questions does not create context-rot". Our checks agree as far as they go: on 12 clear messages the trade probabilities were the same alone and among 30 questions, and on 30 unsure ones the severity score moved less between alone and batched (0.011) than across five identical calls (0.054). A larger state is a different thing, and Chapter 7 measured it separately.

Is this the same as a batch API with overnight processing?

No. Some services offer a separate, slower and cheaper way to submit many requests at once. The TypeSafe documentation describes no such thing. In this book a batch means several questions in one ordinary call.

What if I need the same question asked about 50 different messages?

That is 50 states, and a request has one state. Send 50 requests. Batching puts many questions on one state.

Carry this

Questions in one call are answered independently and in parallel. That is published, and our small tests agreed: largest change 0.00 on 12 clear messages, and on 30 unsure ones a gap of 0.011, inside the 0.054 wobble between identical calls.
Measured p50: 1 question 222 ms, 100 questions 305 ms, about 50 tokens per extra question.
Ten questions as ten calls in a row: 9.4 times the wait and 6.0 times the tokens.
Ask every question the code might need and let the code ignore the rest. A second request is for a decision that cannot be written until the first answer is known.
Published: no answer feeds another inside a call. The limit is the token budget, not a number of questions.

Check yourself

1. Rosa needs 30 questions answered about each message. Using the measured rows, compare one call of 30 questions with 30 calls in a row of one question each.

Answer

one call        275 ms              1,919 tokens      (measured)
30 calls        30 x 222 = 6,660 ms         30 x 539 = 16,170 tokens

latency ratio   6,660 / 275      =  24.2 times
token ratio     16,170 / 1,919   =  8.4 times

2. Which of these needs a second request? (a) "If the tenant mentions a pet, which animal?" (b) "Which of today's three available plumbers suits this job?", where the list comes from a rota that the code reads once it knows the trade.

Answer

Only (b). Question (a) can be asked against the original message, so it is a speculative question for the first call. Question (b) cannot be built until the code knows the trade and has fetched the rota, which is the documentation's test for a real dependency.

3. A colleague puts "Is the answer to the previous question trustworthy?" in the same call as the question it refers to. What will happen?

Answer

By the documentation's rule, the model answering it sees no other question and no other answer, so "the previous question" points at nothing. To judge how far an answer can be trusted, the code should read the probabilities and the confidence that came back with it, as in Chapter 6.