How sure is sure

In this chapter, we will learn what the numbers next to the stamps are worth. We will separate three words that are easy to mix up: probability, confidence and calibration. Then we will check them against our own data. One band of answers looks badly wrong, and we will track down why.

Three words

Probability is the number on one possible answer, confidence is a summary of all the probabilities of one answer, and calibration is a property of many answers taken together.

Confidence holds no new information. The documentation calls it "a statistic computed from the probability distribution the answer already gives you" (a distribution is the full set of probabilities of one answer). In experiment 0014 the published formulas reproduced the returned confidence of 215 Choices to within 0.006 and of 215 Scores to within 0.000.

Calibration is a claim about groups

A model is calibrated when, among all the answers it gave a probability of about p, a share of about p turn out to be true. TypeSafe's documentation spells it out: outcomes given 0.2 should occur about 20% of the time, and outcomes given 0.8 about 80% of the time. It adds: "These rates describe groups of predictions, not a guarantee about any single answer."

TypeSafe says that System One models, of which Jev is the first, are "trained for calibrated decisions". That is the vendor's claim. As far as our research notes could find, TypeSafe has published no calibration measurement for Jev. So we measure it ourselves, on our own task, and we claim nothing beyond that task.

To measure calibration we need to know what was true, so we must be exact about our data. The messages are synthetic. A language model, gpt-5.1, wrote 336 of them from labels we chose first, and a second gpt-5.1 call labelled each one without seeing our label. The two agreed on 260 messages, and we kept 215 of them, at most 60 per severity level: the agreed set. On 76 they disagreed: the contested set, and 66 of those disagreements were about severity. For a contested message, "truth" is only the writer's intention, and a second reader has already said otherwise.

Does confidence follow correctness?

A useful confidence is lower on the answers that are more likely to be wrong. Experiment 0004 tests this twice with the severity Score of Chapter 5. First, the unclear messages. On the 215 agreed requests the mean severity confidence was 0.87. On the 66 contested ones it was 0.67. A line at 0.8 separates them partly:

confidence below 0.8     contested   63.6% of 66    =  42 messages
                         agreed      22.8% of 215   =  49 messages

So the clerk hesitated on about two thirds of the messages that two readers could not agree on, and on fewer than a quarter of the others. Second, the agreed set alone, in buckets of confidence:

agreed set: exact-level accuracy by confidence bucket
confidence      n    correct
 0.00-0.50     13     38.5%
 0.50-0.70     27     77.8%
 0.70-0.90     28     96.4%
 0.90-0.97     30     96.7%
 0.97-1.00    117    100.0%

wrong answers     13 - 13 x 38.5%  =  13 -  5  =  8
                  27 - 27 x 77.8%  =  27 - 21  =  6
                  28 - 28 x 96.4%  =  28 - 27  =  1
                  30 - 30 x 96.7%  =  30 - 29  =  1
total             8 + 6 + 1 + 1    =  16 of 215         215 - 16 = 199 right = 92.6%

Every bucket is more often right than the one before it, and 14 of the 16 wrong answers sit in the two lowest. But the first bucket holds 13 answers, so its 38.5% is a rough figure.

The trade Choice of Chapter 3 gives a rougher picture. Experiment 0026 takes all 291 messages, agreed and contested, with the writer's trade as the truth on the contested ones, and sorts them by trade confidence:

requests: 291   trade correct: 282   wrong: 9

trade confidence      n    wrong   correct
  0.00 - 0.60        6      1     83.3%
  0.60 - 0.80        9      5     44.4%
  0.80 - 0.95       12      2     83.3%
  0.95 - 1.00      264      1     99.6%

Eight of the nine wrong answers sit below 0.95. But these bands do not rise in order, and the two lowest are tiny: one answer in a band of 6 moves its share by 1 / 6 = 16.7 points. Low confidence is a useful flag and not a fine ruler. It also marks the answers that move when we ask again. In experiment 0032, 100 requests were sent 10 times each. The severity score moved by 0.008 on average where confidence was 0.9 or more, and by 0.111 where it was below 0.6.

Does the probability mean what it says?

A calibration table sorts the answers into bands of stated probability and counts how many in each band were true. Experiment 0010 does this for the danger Noul of Chapter 4: "The situation described is a danger to people or to the building." It uses 281 messages, the 215 agreed ones and the 66 contested on severity. A message counts as dangerous when its label is level 3.

CALIBRATION, all requests: of the requests Jev gave a probability in this band, how many were dangerous?
band          n    said (mean p)   were dangerous
0.0 - 0.1     95       0.06             0.0%
0.1 - 0.3     70       0.16             0.0%
0.3 - 0.5     26       0.40             3.8%
0.5 - 0.7     30       0.60            20.0%
0.7 - 0.9     23       0.81            73.9%
0.9 - 1.0     37       0.94            94.6%

The two ends are close. Below 0.3 the clerk said "probably not" 165 times and none was dangerous. Above 0.9 it said 0.94 on average and 94.6% were. The middle looks broken. In the band 0.5 to 0.7 the clerk said "more likely than not" and, by our labels, was right one time in five. We had three suspects: a dial that reads too high, noisy labels on the contested messages, and bands too small to trust. Then we found a fourth.

The suspect we had not questioned

It was the answer key. Look at what the clerk was asked. The Noul's true description reads: "Fire, smoke, sparks, gas, flooding, risk of electric shock, or anything that could injure someone or damage the building soon." Our truth was the severity label, written for the ruler of Chapter 5. A message labelled level 2 can describe something a careful reader would also call a danger under that sentence. We had marked the answers to one question with the key of a neighbouring question.

Experiment 0029 writes a matching key. Three language models (gpt-5.1, gpt-5-mini and claude-haiku-4.5) each read the Noul's own statement and criteria, and answer yes or no for every message. The new truth is the majority of the three. Then the same Jev answers go into the same bands.

requests: 281.  the three readers agreed unanimously on 232 (82.6%)
dangerous by majority of readers: 85   dangerous by the old label (severity 3): 59   the two truths differ on 42 requests
Brier score of Jev: against the old label 0.080, against the readers' majority 0.073
...
band          n    said (mean p)   dangerous, old label   dangerous, readers' majority
0.0 - 0.1     95       0.06              0.0%                    0.0%
0.1 - 0.3     70       0.16              0.0%                    5.7%
0.3 - 0.5     26       0.40              3.8%                   19.2%
0.5 - 0.7     30       0.60             20.0%                   73.3%
0.7 - 0.9     23       0.81             73.9%                   73.9%
0.9 - 1.0     37       0.94             94.6%                  100.0%

band 0.5 to 0.7          30 answers, said 0.60 on average
if calibrated            0.60 x 30    =  18 dangerous
old label                20.0% of 30  =   6 dangerous        12 too few
readers' majority        73.3% of 30  =  22 dangerous         4 too many

The band that looked broken is now 4 messages above the stated 0.60, not 12 below it. On the 49 messages where the readers split, the same experiment printed a mean Jev answer of 0.46: the clerk hesitated where readers could not settle. Most of the failure was ours. A probability is calibrated against the question that was asked, not against a neighbouring label.

Three cautions keep this honest. The readers are models, not people, and they may share blind spots with each other and with Jev. The bands are small: one message in a band of 30 moves its share by 1 / 30 = 3.3 points. And the new column is not a straight line: the band 0.3 to 0.5 said 0.40 and got 19.2%. Over the whole table the clerk still leans towards danger:

expected if every band were calibrated
95 x 0.06 + 70 x 0.16 + 26 x 0.40 + 30 x 0.60 + 23 x 0.81 + 37 x 0.94  =  98.7
dangerous     59 by the old label, 85 by the readers' majority

One number for a whole table: the Brier score

The Brier score is the average of (probability - truth) squared, where truth is 1 if the thing happened and 0 if it did not. It is 0 for a perfect forecaster. Lower is better. Three invented answers show how it works:

said 0.9, was dangerous        (0.9 - 1) squared  =  0.01
said 0.2, was not dangerous    (0.2 - 0) squared  =  0.04
said 0.6, was not dangerous    (0.6 - 0) squared  =  0.36

Brier score    (0.01 + 0.04 + 0.36) / 3   =  0.41 / 3  =  0.137

Squaring punishes a confident mistake. Two lazy forecasters give a yardstick:

always says 0.5     every error is 0.5 squared                    =  0.250
always says 0       wrong only on the 59 dangerous:  59 / 281     =  0.210     (old label)

Experiment 0018 asks the same danger question, about the same 281 messages, in four ways, and scores them against the old label. Jev returns the number. For "token odds" we ask gpt-5.1 for one word, yes or no, and read the probability the model itself gave to each word. For "says a number" we ask gpt-5.1 to write a probability between 0 and 1. The fourth way is the same request to a cheaper model, gpt-5.6-luna. Chapter 18 explains token odds properly.

way                             Brier  right at 0.5  distinct values  in 0.1-0.9  p50 ms  $ / 1,000
jev                             0.080         88.3%               79       53.7%     235      0.019
gpt-5.1, token odds             0.103         89.3%               18        3.2%    1012      0.349
gpt-5.1, says a number          0.088         88.6%               53       59.1%     996      0.445
gpt-5.6-luna, says a number     0.122         83.6%               38       22.4%    1428      0.098

"Right at 0.5" counts 0.5 or more as a yes. "Distinct values" is how many different numbers a way produced. All four beat the lazy 0.210. By this label Jev has the lowest score, though 0.080 against 0.088 is a small gap on noisy labels. The token odds are the instructive row. They were right at 0.5 most often, and they were almost never unsure: only 3.2% of their answers fell between 0.1 and 0.9. A forecaster who always says "certain" pays in full for each mistake.

That ranking does not survive a change of truth. Experiment 0038 scored the same four sets of answers against the readers' majority as well:

way                             severity label  readers majority
jev                                      0.080             0.073
gpt-5.1, token odds                      0.103             0.068
gpt-5.1, says a number                   0.088             0.063
gpt-5.6-luna, says a number              0.122             0.066

Against the readers all four score between 0.063 and 0.073, and Jev is last, not first. One caution: gpt-5.1 was one of the three readers, so its two rows are marked partly by itself. It also wrote the messages. Even so, we cannot say that Jev's probabilities are better calibrated than a language model's: which way looks best depends on the answer key. The last two columns of the first table are firmer. Jev's number arrived four to six times sooner at the median, at a fifth of the cost or less.

Before you switch between the four ways below, predict: whose dots will sit furthest from the dashed line, and in which band? Then go back to Jev, change the truth from the old label to the three readers' majority, and watch the dot of the 0.5 to 0.7 band.

python3 experiments/0010-thresholds/main.py
python3 experiments/0029-matched-truth/main.py
python3 experiments/0018-diy-probabilities/main.py

rule: the highest threshold that misses nothing dangerous on dev -> 0.41
TEST half at 0.41:  caught 33, false alarms 23, missed 0, correctly left alone 85   precision 58.9%  recall 100.0%

(0.9 - 0) squared  =  0.81
(0.1 - 0) squared  =  0.01
(0.8 - 1) squared  =  0.04
(0.5 - 1) squared  =  0.25
Brier score   (0.81 + 0.01 + 0.04 + 0.25) / 4  =  1.11 / 4  =  0.278

if calibrated     0.81 x 23    =  18.6 dangerous
by either truth   73.9% of 23  =  17 dangerous
gap               18.6 - 17    =  1.6 messages