Speed, cost and scale

In this chapter, we will learn to do the arithmetic of speed and cost for any volume. We will read latency the way engineers do, with percentiles. We will work out the real bill of Rosa's desk, find that it is almost nothing, and then scale the same sums to a hundred million requests a month, next to the language models we measured.

Percentiles: p50 and p95

A percentile is the value below which a given share of the measurements fall. Latency is the time from sending a request to holding the answer. Sort all the latencies from fastest to slowest. The p50, also called the median, is the one in the middle: half the calls were faster. The p95 is the one that 95 of every 100 calls beat.

Experiment 0002 made 215 calls, each with the three desk questions. Its last lines read:

latency per call (three questions): p50 251 ms   p95 386 ms   max 1342 ms
input tokens per call: 694   cost per call: $0.000029   per 1,000 requests: $0.029

calls slower than the p50    50% of 215   =  about 107 calls took more than 251 ms
calls slower than the p95     5% of 215   =  10.75, so about 11 calls took more than 386 ms
the slowest call             1,342 ms     =  5.3 times the p50   (1,342 / 251)

We quote p50 and p95 together. The p50 tells Rosa what a tenant usually sees. The p95 tells her what to plan for.

What we measured

Every Jev latency in this book was measured from one laptop, on one day, through OpenRouter. OpenRouter is a service that passes our call on to TypeSafe, so each number includes that extra step and our own network. Read the numbers as "what one ordinary user saw", and not as the speed of the model itself. The gpt models were called on Azure OpenAI, Microsoft's service for OpenAI's models, and the other two language models through OpenRouter.

The pattern is that the work grows a great deal and the time grows a little. In the batch experiment of Chapter 9, a hundred times the questions cost 83 ms more at the p50 (305 against 222). With only 12 calls per row those percentiles are rough: the p95 of a row is its second-slowest call.

Now the vendor's side. These are published claims, not our findings. TypeSafe writes that "Most queries complete in about 100 ms" on one page, "150ms" on another, and "70ms-500ms" in its launch post. It also says where its own numbers come from: "our published evals are generally run from our laptops on the West Coast (this is where our service is currently based)".

Our p50 values, 216 to 333 ms, sit inside the widest of those ranges and above the shortest claim. We went through one more service and we do not know our distance from their machines, so this is no contradiction. One outside test reported a p50 of 264 to 276 ms, close to ours. Measure from where your own servers stand.

The price of one call

Jev's published price is $0.042 per million input tokens, and output is free. A token is a small piece of text, as in Chapter 7. Input tokens are everything we send: the state and the questions. So the cost of a call is one multiplication.

tokens in one desk call      694                                    (measured mean, 0002)
price                        $0.042 per 1,000,000 tokens            (published)
cost of one call             694 x 0.042 / 1,000,000   =  $0.0000291
cost of 1,000 calls          $0.0000291 x 1,000        =  $0.029

The derived $0.0000291 agrees with the $0.000029 that the response itself reported.

Where do the tokens go? The documentation does not say how they are counted, so experiment 0027 measured it. It sent the first call of Chapter 2 in seven sizes and read the count that every response reports. A bare Noul is a yes/no question with no criteria.

the message itself:            313 - 272 = 41 tokens  (about 4.2 characters per token)
the trade Choice over a bare Noul: 419 - 272 = 147 tokens
adding the severity Score:     562 - 460 = 102 tokens
adding the may_enter Noul:     625 - 562 = 63 tokens
a second copy of the message:  667 - 625 = 42 tokens
fixed overhead of the smallest call (one-word state, one bare Noul): 272 tokens

smallest call: one word of state, one bare yes/no question    272
swap that question for the trade Choice              + 147  =  419
the message, 173 characters                          +  41  =  460
the severity Score                                   + 102  =  562
the may_enter Noul                                   +  63  =  625     the call of Chapter 2
everything except the message     625 - 41    =  584
mean message in experiment 0002   694 - 584   =  110 tokens     (derived)

Three things follow. The smallest call we sent is 272 tokens, which is $0.011 per 1,000 calls. The three questions weigh far more than the message: 147 + 102 + 63 is 312 tokens against 41. And a question is not a fixed size: a short extra question cost about 50 tokens in experiment 0007, while the desk's own, with their options and described levels, cost 63 to 147.

A larger call is a second measured point. The catalogue design of Chapter 11 asks nine questions in one call and carries its candidates in the state (experiment 0030).

catalogue call    1,496 x 0.042 / 1,000,000   =  $0.0000628     $0.063 per 1,000   (printed: $0.063)
against the desk  1,496 / 694                 =  2.2 times the tokens and the cost
time              p50 234 ms against 251 ms    =  no slower

The bill at every size

At the size of one building, the cost of the model is too small to matter. Harbor Court has 120 flats. We do not have Rosa's real numbers, so we assume two requests per flat per month. Replace it with your own count.

requests per month     120 flats x 2            =  240          (assumed)
Jev, per month         240 x $0.0000291         =  $0.007       under one cent
Jev, per year          $0.007 x 12              =  $0.084
gpt-5.1, per month     240 x $0.000756          =  $0.18        ($0.756 per 1,000, measured in 0003)
gpt-5.1, per year      240 x $0.000756 x 12     =  $2.18
model time, per month  240 x 251 ms             =  60 seconds at the p50

Eight cents a year against two dollars a year. If Rosa picks the clerk for her desk, cost is not the honest reason. Two other things are.

• Speed. In the whole-desk test of Chapter 11, the clerk with the back office answered in 250 ms at the p50. The letter writer gpt-5.1 took 1,281 ms, and gpt-5-mini took 6,426 ms.
• Control. The clerk's answers are typed and come with probabilities. The back office answered 87.9% of requests automatically, got 98.9% of those right, and handed 13 unsure ones to a person. The gpt-5.1 letters were right 88.8% of the time, and nothing told the desk which ones to check.

Below, the sliders start at Harbor Court. Before you move them, predict: at what monthly volume does the gap between Jev and gpt-5.1 first reach $1,000 a month?

The gap is about $0.73 per 1,000 requests, so it reaches $1,000 near 1.4 million requests a month.

Scaling a cost is one multiplication: the cost of one request times the number of requests. The table uses the measured cost per 1,000 requests of each engine on the same 215 requests (experiment 0003), and of the two whole-desk designs on 107 requests (experiment 0017). The gpt costs are measured tokens times list prices; OpenRouter reported the cost of the other two.

The interactive carries more decimals than this table, so it shows $2,915 where the table shows $2,900.

Two notes on the last column. gpt-5.1 wrote these requests and labelled them, so its 96.7% has a home advantage. And the data is synthetic and cleaner than real mail.

TypeSafe's home page claims "193.6x Faster, 444.6x Cheaper", with the footnote "based on workflows for System One tasks", and its launch post adds: "we expect that these are on the higher end of real world gains." Ours:

against gpt-5.1                 speed  1,064 / 251   =   4.2 times     cost  0.756 / 0.029   =  26 times
against gpt-5-mini              speed  2,490 / 251   =   9.9 times     cost  0.458 / 0.029   =  16 times
against gemini-2.5-flash-lite   speed    518 / 251   =   2.1 times     cost  0.046 / 0.029   =  1.6 times
whole desk, gpt-5.1 letters     speed  1,281 / 250   =   5.1 times     cost  0.953 / 0.029   =  33 times

Our ratios are far below the claim. That is not a contradiction either. We compared one short call with one short call. The vendor's footnote says its figure is for workflows, and it expects real gains to be lower. The lesson is to compute the ratio for your own task. Note the third line: a cheap language model came within about a factor of two on both speed and price.

Rate limits, and where the time goes

A rate limit is the most traffic a provider accepts from one customer per second. The documentation lists 80 requests per second and 100K tokens per second for Jev. A request over either limit is refused with the error code 429, "Too Many Requests", and the documentation tells us to wait a little longer after each refusal before trying again. It also warns that the limits "can change without notice".

seconds in a 30-day month     30 x 24 x 3,600            =  2,592,000
100 million requests a month  100,000,000 / 2,592,000    =  38.58 requests per second, on average
tokens per second             38.58 x 694                =  26,775
published limits              80 requests per second, 100,000 tokens per second

On average, a hundred million desk requests a month fit under both limits. Traffic is rarely even. If the busiest hour carries three times the mean, 116 requests per second is over the limit, and you need a queue or a higher limit from the vendor.

Where does the time go inside one request? The whole-desk experiment timed the code that runs after the model answers: the lookup, the gate, the template.

  latency p50 250 ms, p95 386 ms; the code after the model call took 7 microseconds at most

the model call, p50 250 ms   =  250,000 microseconds
the back office     7 microseconds at most
share of the wait   7 / 250,000   =  0.003%

Experiment 0039 put a timer on the sums of Chapter 10 as well: a total, a count and a date difference, for 100 cases each.

2. the three kinds of app-side arithmetic for 300 cases, timed: 95 microseconds for all 300 (best of 200 runs), about 315 nanoseconds per case

one case             95 / 300          =  0.32 microseconds

The call is the wait. A faster back office gains almost nothing. Fewer calls gain the most, which is why Chapter 9 put all the questions into one.

When the letter writer is still right

A language model is the right tool when the result must be text, or when the judgment needs several steps of reasoning. The first is a published limit: Jev does not generate text or explanations, and it accepts text only, so images and audio must be turned into text first. The second is also the vendor's own advice: tasks "requiring extended reasoning, such as complex mathematics or chess-like planning, may be better suited to large reasoning models."

There is a third case, and it needs care. On all 215 requests gpt-5.1 scored 96.7% where Jev scored 92.1%, with a home advantage we cannot size. A common design, the cascade, sends only the unsure requests to the stronger model. Experiment 0024 ran it on the 107 test requests, with the gate of Chapter 12 picking the unsure ones.

                                     right   $ / 1,000   mean ms
jev alone                            96.3%       0.029       284
gpt-5.1 alone                        95.3%       0.757      1087
cascade: jev, then gpt-5.1 if unsure   96.3%       0.120       420

escalated to gpt-5.1: 13 of 107 (12.1%).  On those, Jev alone was right on 10, gpt-5.1 on 10.

cost of the pair    0.120 / 0.029   =  4.1 times Jev alone
gain in accuracy    96.3 - 96.3     =  0 points

The cascade bought nothing here. On the 13 hard requests gpt-5.1 was right on 10, exactly as Jev was. On this half of the data gpt-5.1 alone was not even ahead: 95.3% against 96.3% is one request in 107. A cascade pays only when the second model is better on the hard cases. Measure that first.

cd experiments
python3 0002-accuracy/main.py

cost of one call    2,000 x 0.042 / 1,000,000   =  $0.000084
per month           5,000 x $0.000084           =  $0.42

5% of 80   =  4 calls