The same answer twice

In this chapter, we will learn what happens when the same request is sent to the clerk again. We will measure how far an answer moves over thirty identical calls, count how often a repeat changes what the desk does on 100 requests, and learn to pin a model version. Then we will treat our questions like code and give them a test that runs on every change.

Repeatability and determinism

Repeatability is how closely the answers agree when the same request is sent again. Determinism is the strict form of it: the same input always gives exactly the same output, down to the last digit. Ordinary code is deterministic.

Language models usually offer two settings for this. Temperature controls how much chance is used when the model picks its next word. A seed is a number that fixes the chance, so that a run can be replayed. Jev has neither. A documentation write-up notes that "reasoning models and TypeSafe run without a temperature setting", and no seed field is documented.

TypeSafe does not claim determinism. Its website asks "Is Jev deterministic?" and answers: "Determinism means returning the same result for an identical input. This is less valuable than consistency. We define consistency as making similar decisions when the meaning stays similar, even if the wording changes. Jev is designed for consistency." That is not a yes.

TypeSafe has published its own numbers on this. All of the following is published, not measured by us.

• Fourteen Noul questions, each asked 15 times: the mean standard deviation of the probability was 0.0102. Standard deviation is the typical distance of a value from the average. One question's answers ran from 0.43 to 0.53, "crossing a 0.5 decision threshold". The write-up adds a caveat: each call also carried a fresh id in the state, so the test cannot separate that from pure repetition.
• Eight Choice questions, each asked 15 times: the most common label came back 90.8% of the time, and the winner flipped on 2 of the 8 questions. With a rule that only accepts a top probability of at least 0.60, agreement rose to 99.2%, and 74.2% of answers passed the rule.
• A third write-up found most answers identical over 5 repeats, while two Nouls "carry a little run-to-run sampling noise".

The documentation does not explain where the noise comes from.

What we measured

We sent five requests 30 times each, with nothing cached, and kept the smallest and the largest value of every answer. That is 150 calls with the three desk questions (experiment 0005). The requests are synthetic, written and labelled by gpt-5.1. The first three come from the agreed set. The last two, r003 and r004, come from the contested set, where the two labels disagreed about severity.

request   choice flips   trade p(top) min..max   severity score min..max   may_enter noul min..max
  r049         0           1.00 .. 1.00            0.01 .. 0.04              0.02 .. 0.02
  r253         0           1.00 .. 1.00            1.71 .. 1.79              0.98 .. 0.98
  r091         0           1.00 .. 1.00            1.00 .. 1.00              0.98 .. 0.98
  r003         0           1.00 .. 1.00            0.17 .. 0.24              0.93 .. 0.94
  r004         0           1.00 .. 1.00            0.59 .. 0.77              0.95 .. 0.97

30 calls per request, nothing cached. 'choice flips' = number of different winners minus one.

The width of each range is the largest value minus the smallest.

severity score    r049   0.04 - 0.01   =  0.03
                  r253   1.79 - 1.71   =  0.08
                  r091   1.00 - 1.00   =  0.00
                  r003   0.24 - 0.17   =  0.07
                  r004   0.77 - 0.59   =  0.18     the widest
may_enter Noul    r004   0.97 - 0.95   =  0.02     the widest
trade, top p      all    1.00 - 1.00   =  0.00

No winner changed: in 150 calls the chosen trade was the same every time, at a printed probability of 1.00. The answers are still not identical: a score moved by up to 0.18 on a scale of 0 to 3. And the doubtful request moved most. r004 sits between level 0 and level 1 and has the widest range, while r091 sits exactly on level 1 and did not move.

The limits: five requests, one day, one laptop, every call through OpenRouter. Five requests show that a wobble happens. They cannot say how often it changes what the desk does.

Why it matters at a threshold

A threshold turns a small movement into a different action whenever the answer sits near the line. Far from the line, a movement of 0.18 changes nothing. On the line, it changes everything. So the useful question is not "does the number move?" but "how often does the action change?"

Experiment 0032 asked that. It took the first 100 test requests and sent each one 10 times, with nothing cached. Then it counted the requests on which a repeat changed something the desk does: the trade, the nearest severity level, permission at 0.5, or the route. The route is the choice of the gate from Chapter 12, which answers automatically or hands an unsure request to a person.

100 requests, 10 uncached calls each (1,000 calls)

requests where a repeat changed the trade                  0
requests where a repeat changed the nearest severity level 4
requests where a repeat changed permission at 0.5          0
requests where a repeat changed the route (auto or person) 5

spread of the severity score across 10 calls: median 0.01, 90th percentile 0.08, largest 0.17; exactly 0.00 on 38 requests
spread of the permission Noul across 10 calls: median 0.00, 90th percentile 0.01, largest 0.23
mean spread where mean confidence is 0.9 or more (70 requests): 0.008;  where it is below 0.6 (9 requests): 0.111

The spread of a request is its largest value minus its smallest over the 10 calls. The median is the middle request, and nine requests in ten were at or below the 90th percentile.

sure answers (70 requests)    mean spread 0.008
unsure answers (9 requests)   mean spread 0.111        0.111 / 0.008  =  14 times wider

• Most answers hardly move. The median spread of the score is 0.01, and on 38 requests it is exactly 0.00. The trade never changed, and permission never crossed 0.5, although one Noul moved by 0.23. A trade can change where a message is torn between two trades: in experiment 0044, six identical calls changed it on 0 of 291 requests and on 2 of 60 such messages.
• The answers that wobble are the unsure ones. Where the confidence is 0.9 or more the mean spread is 0.008. Where it is below 0.6 it is 0.111.
• At a line, the wobble becomes an action. The desk rounds the score to the nearest level, so its lines are at 0.5, 1.5 and 2.5. On 4 requests in 100 the same message was rounded to two different levels, and a level decides the promise the tenant reads. On 5 the gate sent the same message to a person on some calls and answered it automatically on others.

The second finding softens the third. A score that sits on a rounding line is usually an unsure one, and the gate exists to stop unsure answers. The confidence formula of Chapter 5 shows it for a request torn between two neighbouring levels.

probabilities         level 1: 0.5     level 2: 0.5
score                 1 x 0.5 + 2 x 0.5     =  1.5     exactly on the line
spread around level 1 0.5 x 1               =  0.5
even spread, 4 levels (1.5 + 0.5 + 0.5 + 1.5) / 4  =  1.0
confidence            1 - 0.5 / 1.0         =  0.5     below the gate at 0.6, so a person reads it

That is arithmetic, not a count: the program did not print whether the four requests whose level changed were also the ones sent to a person.

Where we draw a line decides whether a wobble can be seen. With the desk's lines, none of the five ranges of experiment 0005 crossed. Now invent a rule, "a score of 0.70 or more gets a visit this week". Request r004 ran from 0.59 to 0.77, so it would get a visit on some calls and not on others.

Below, each bar is one request's measured range from experiment 0005. The line starts at 0.50. Before you move it, predict: between which two values must the line stand for r253 to be routed two ways?

The answer is anywhere above 1.71 and up to 1.79. Three habits keep the wobble out of your system.

• A band, not a line. Put a band that goes to a person around every line. A wobble at the edge of the band then only changes whether Rosa looks at the request, as it did for 5 requests in 100, and no longer changes what the tenant is promised.
• Decide once. Store the answer with the request. If the tenant reloads the page, the back office reads the stored answer and does not ask again.
• Never branch on the second decimal. A rule that separates 0.69 from 0.71 is a rule about noise.

Questions do not disturb each other

Independence means that an answer does not change when other questions share its call. The documentation says: "Every answer is independent. One question's answer is not hidden context for another." This matters for testing. If it holds, adding a fourth question to the desk cannot quietly change the first three.

Chapter 9 measured it. We asked the trade question alone, and again as the first of 30 questions, on 12 requests.

Independence: the trade question asked alone, and asked as question 1 of 30. Largest change in any option's probability:
  per request: 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00    max 0.00

No probability changed in the second decimal. But those were 12 clear requests, the easy case. So experiment 0037 took 30 contested requests and asked the severity question five times alone and five times as one of 30 questions, with nothing cached.

mean spread of the score across 5 calls, asked alone:        0.054
mean gap between the alone average and the batched average:  0.011
requests where the nearest level of the two averages differs: 0 of 30

Asked alone, the unsure scores moved by 0.054 on average between identical calls. Sharing the call moved their averages by 0.011 on average, less than that wobble, and no level changed. Experiment 0035 saw the same on one unsure trade answer: a mean confidence of 0.72 alone and 0.72 in the three-question call. So independence held where the clerk is unsure too, within the noise. These small tests agree with the documentation and do not prove it for every case.

Versions and aliases

An alias is a model name that points at whichever version is current. The documentation lists jev-latest as an alias for jev-1.13.0 today. It warns: "An alias moves when a new release ships, so the answers behind it can change without a change on your side."

For a desk with tuned thresholds this is the larger risk. A wobble moves a few requests in a hundred. A new version can move all of them at once. The documentation's advice: "If you have tuned confidence thresholds against a specific version, pin that version's ID instead of the alias". To pin is to write the exact version, jev-1.13.0, in the model field.

The response also tells you which version answered, in its own model field. Our experiments went through OpenRouter and asked for typesafe/jev-1.13. The first line of experiment 0002 shows what came back.

requests: 215   model: typesafe/jev-1.13-20260917

Log that field. The documentation has no page that says what changed between versions, so the name is all you will have when answers shift.

Test questions like code

A golden set is a fixed collection of inputs together with the answers you have agreed are right. A regression test is a test you run again after every change, to check that what worked yesterday still works. A question is a small piece of program too: change one word in a level description and the behaviour changes.

Split the golden set in two. Tune wording and thresholds on the dev half. Report on the test half, which you never tune on. Our desk has 108 dev and 107 test requests.

Here is the shape of such a test. The floors are invented numbers; yours come from your first run on the dev half.

# A sketch. GOLDEN is your labelled dev half; ask() wraps the API call; read() turns answers into labels.
PINNED = "jev-1.13.0"
FLOOR = {"trade": 0.97, "severity": 0.85, "may_enter": 0.97}   # invented floors

def test_questions_still_work():
    right = {name: 0 for name in FLOOR}
    for row in GOLDEN:
        reply = ask(row["text"], QUESTIONS, model=PINNED)
        assert reply["model"] == PINNED          # the version that answered
        got = read(reply["answers"])             # labels, not raw probabilities
        for name in FLOOR:
            right[name] += got[name] == row[name]
    for name in FLOOR:
        assert right[name] / len(GOLDEN) >= FLOOR[name], name

Two details matter. The test compares labels and shares, not exact probabilities, because we have just seen probabilities move. And it runs every question on every change: a new wording, a new option, a new threshold, a new model version.

cd experiments
python3 0005-repeatability/main.py
python3 0032-repeat-at-scale/main.py

0005     5 requests x 30     =  150 calls       150 x $0.000029    =  $0.0044
0032     100 requests x 10   =  1,000 calls     1,000 x $0.000029  =  $0.029

width      0.24 - 0.17   =  0.07
line 0.5   0.24 is below 0.5, so the whole range is below the line
result     level 0 on all 30 calls