Where it holds, where it breaks

In this chapter, we will learn where a decision model can be trusted and where it cannot. TypeSafe publishes a list of nine weak spots of Jev. We will read that list and put our own measurements next to each line. Where our test showed the weakness, we give the number. Where it did not, we say how small, clean or easy the test was.

The jagged edge

A model's jagged edge is the uneven line between the tasks it does well and the similar-looking tasks it does badly. The line is jagged because it does not follow our sense of easy and hard. A clerk can judge how severe a leak is in a quarter of a second and still miscount a short list of flats.

The documentation has a page with this name, "jaggedness", for jev-1.13, last reviewed on 2 October 2026. It lists nine failure modes, each with a fix. In the table, the first two columns are published and the third is measured.

Before you open the rows below, predict. The table says "not shown" three times, and its option-order row says "mildly". For each of those four, what about our test could have made it too easy?

Our tests showed five weak spots: literal reading, arithmetic, dates, hostile text and, mildly, option order. Three tests came back clean (indirection, large state, contradictory criteria). Those three and the option-order test were small, clean or easy, and the sections below say how.

Sums, counts and dates

Arithmetic is any answer that a short program can compute exactly, and the documentation says to keep it out of the model. It says "Jev is not a calculator", and that the model "does not count reliably".

Chapter 10 printed this test in full (experiment 0009). The clerk decides whether a repair quote is over the 250 limit, first from the raw line items, then from a total that the code has added up. A second test counts a flat's earlier requests in a raw log. Each has 100 generated cases. From the raw data the clerk was right on 92.0% of the sums and 95.0% of the counts. When the code computed the number first, it was right on 100.0%, on 92 and 86 different requests (0043).

Experiment 0038 found where the weakness sits. All eight wrong sums called a quote over 250 when its true total was 202 to 243. Of the 15 quotes within 60 of the limit, 8 were wrong. Of the 85 further away, none was. Only 2 of those 15 were above the limit (0043), so "all eight said over" is not yet evidence of a lean.

The documentation says the model "reads dates as text, not as ordered quantities". Our first test, in 0009, scored 100.0% from raw dates. It was easy: one date format, one fixed "today", and gaps of 1 to 30 days, so only 30 different dates (0043).

Experiment 0041 is the harder test: each of 100 requests was opened 10 to 21 days before its own "today".

(a) 'The request was opened more than 14 days before today.'  100 distinct pairs of dates, written out in full
    true gap in days:       10    12    13    14    15    16    18    21
    cases:                  10    10    10    30    10    10    10    10
    correct:                10    10    10    23    10    10    10    10
    total correct: 93 of 100
    the app sends days_open: 100 of 100

All 7 misses are requests opened exactly 14 days ago, which is not more than 14. When the code sent the number of days, all 100 were right.

Experiment 0040 went to that line: 20 pairs of dates at each gap from 12 to 16 days (99 of the 100 distinct, every call fresh), three date formats, and four wordings of the statement.

the request was opened ...                                                   12   13   14   15   16   (correct of 20; true gap in days)
more than 14 days before today                                               20   19   15   20   20
more than 14 days before today. Exactly 14 days is NOT more than 14 days     18   19   10   11   10
at least 15 days before today                                                20   20   17   20   20
14 or more days before today                                                 15    6   20   19   19

For whole days, "more than 14" and "at least 15" are the same condition, and they scored alike: 94 and 97 of 100, with the misses at 13 and 14 days. "14 or more" moves the line down a day, and its misses moved with it: 19 of the 21 fell at 12 and 13 days, for 79 of 100. Stating the boundary made things worse: 68 of 100, with about half wrong at 14, 15 and 16 days.

The documentation's own fix splits the work: the model extracts the month and the day, each as a Choice, and the code compares. Experiment 0041 tried it on 90 different numeric dates such as "08/10/26".

A day above 12 cannot be a month, and all 56 such dates were read correctly. A day of 1 to 12 can, and only 7 of those 34 were right. Only 6 of the 27 misreadings were a plain swap of month and day. One line in the state, "This building writes dates as day/month/year.", with each question told to use it, made all 34 right.

Words read at face value

Literal reading means the model answers the words on the page, and not the intention behind them. The documentation says the model "answers the question you wrote, not the one you meant. Scoping words, negations, and implied conditions are read at face value." A scoping word is a word that sets how far a condition reaches, such as "all" against "at least one".

The four date wordings above belong here: 68 to 97 of 100 for one rule. The documentation's advice is: "Put boundary cases in the criteria." Ours went into the instruction, so that advice is untested here.

The other case is a missing description. In Chapter 8 the severity question lost its four described levels and got bare labels in their place. Exact answers fell from 88.0% to 29.6% of 108 dev requests: 95 right against 32.

Small scoping words gave no trouble. Experiment 0022 described 60 invented flats with four radiators each, 49 of the descriptions different (0043), and asked whether at least one radiator is cold, all are, only the bedroom one is, or none is. Every answer was right. Every radiator had a plain sentence of its own, so the test was too easy to tell us anything.

Indirection is an answer that needs more than one look-up through the state. The documentation says the model "may struggle with tasks that require additional levels of indirection". Experiment 0023 asked one statement, "The request comes from a flat on the fourth floor.", three ways: with the floor in the state, with the flat number, and with a tenant's name plus a list of eight tenants and their flats (the last two with a note that a flat number starts with its floor).

All three scored 100.0% on 80 cases. Only the third way had 80 different requests: the first two repeat 8 floors and 17 flat numbers (0043). Two look-ups through eight tidy rows is a short chain.

Contradictory criteria are criteria that disagree with their own instruction. The documentation's example: "a Noul where true maps to no and false maps to yes will perform worse." Experiment 0031 did exactly that to the permission question, on the 108 dev requests.

Still 100.0% right: the instruction won. The only trace is the mean probability on real permissions, which slipped from 0.97 to 0.92. On a question nearer the line, such a slip could change answers, and we have not tested one.

A crowded desk

Irrelevant state is everything on the paper that the question does not need. The documentation says: "Accuracy falls as the state grows with content unrelated to the decision."

In Chapter 7 we buried each new request under 150 old messages from other flats, on 60 test requests (experiment 0008). The mean call grew from 712 to 17,137 tokens. Trade went from 100.0% to 98.3% and severity from 96.7% to 98.3%: one request lost, one gained. The test was kind: the old messages sat in their own named field, and each question said to judge only the new request.

Three limits sit outside that list.

Many options. A Choice accepts up to 255 options. We added invented teams to the six trades (experiment 0011, 80 test requests). With 6 and 12 options the clerk was right on 100.0%, with 36 on 93.8%, with 255 on 92.5%.

No option that fits. A Choice must return one of its options. In experiment 0025 all 40 hand-written messages that belong to no trade, such as a stuck lift, were forced into one, 24 of them at a confidence of 0.9 or more. The documentation advises an other option. Here it caught 38 of 40 and cost one real request in 80.

Other languages. The documentation says other languages than English "currently have lower accuracy". On 60 test requests that gpt-5.1 translated into five languages (experiment 0013; the questions stayed in English), trade and permission held at 100.0%. Severity was 96.7% in English and 90.0% to 95.0% in the others, with Hindi the lowest. The mean severity confidence stayed between 0.88 and 0.90, so it gave no warning.

The order of the options

Order bias is a lean toward an option because of where it stands in the list, and not because of what it says. The documentation admits it: "the order of a Choice's options can affect the answer, and jev-1.13 leans toward the option that comes first."

Experiment 0042 asked the trade question six times per request, each time with the six trades in a different order and every call fresh: on all 291 requests, then on 60 messages that gpt-5.1 wrote to sit between two trades.

   messages whose chosen trade changed with the order: 4 of 291 (1.4%)
   the winner was the option listed first in 288 of 1746 calls (16.5%; six options, so about 16.7% if order did not matter)

A flip alone does not prove an order effect, because an unsure answer can also change between identical calls (Chapter 15). So experiment 0044 ran the control: the same messages in one fixed order. The spread is the winner's largest probability minus its smallest; nine messages in ten sit at or below the 90th percentile.

60 close-call messages: the same order, six fresh calls each
   messages whose chosen trade changed between identical calls: 2 of 60
   spread of the winner's probability across the six calls: median 0.03, 90th percentile 0.10, largest 0.16
   for comparison, six DIFFERENT orders in experiment 0042: 3 of 60; spread median 0.07, 90th percentile 0.18, largest 0.34

On the 291 requests no trade changed between identical calls, against 4 across the orders, and the largest spread was 0.13 against 0.28. So reordering moves the probabilities somewhat more than repeating does: a modest, real order effect. It changed the winner only on messages torn between two trades, such as hot water that is plumbing or heating. The first-listed option won no more than one call in six (15.0% on the close calls), so the documented lean toward the first position did not show at six options.

Long lists we did not shuffle. One outside test reported that 13% of choices changed when options were reordered.

What the clerk never does

Jev does not write text and does not explain its answers. The documentation says the model "is not trained to generate text", and that System One models do not "generate explanations of their reasoning". This ninth row we did not test: there is no text answer to ask for. So the reply to a tenant comes from a template that the back office fills, as in Chapter 11.

cd experiments
python3 0041-dates-distinct/main.py

on the line   23 / 30          =  76.7%
elsewhere     70 / 70          =  100.0%
overall       (23 + 70) / 100  =  93.0%