Choice: the tray stamp · Decision Models with Jev: Novice to Ninja

In this chapter, we will learn the first of the three stamps. We will write a Choice, read the probabilities it returns, and compute its confidence by hand for two real answers. Then we will look at the trades the clerk got wrong, at a paper that fits no tray, and at what we measured when we changed the wording, the length of the list and the order of the options.

What a Choice is

A Choice is a question whose answer is exactly one option from a list that you supply. This is Rosa's trade question, the standard one of our experiments.

"trade": {
  "type": "choice",
  "instructions": "Which trade should handle this request?",
  "criteria": {
    "plumbing": "Leaks, drains, taps, toilets, water pressure.",
    "electrical": "Power, lights, sockets, fuse box.",
    "heating": "Radiators, boiler, hot water, air conditioning.",
    "appliance": "Oven, fridge, washing machine, dishwasher.",
    "pests": "Insects, mice, rats, birds.",
    "locks": "Doors, locks, keys, windows, intercom."
  }
}

Each entry of criteria is one tray. The key is the option: the name that comes back in the answer. The value is its description: the card on the tray.

The documentation adds three facts. The option names and the descriptions are both sent to the model. A description may be null, JSON's word for no value, and then the option "is interpreted by its name alone". And one Choice accepts up to 255 options.

Probabilities that sum to 1

The answer to a Choice is a probability for every option, and the probabilities sum to 1. A probability is a number from 0 to 1. The field choice is simply the option with the largest one. Here are two real answers: the dripping ceiling of Chapter 2, and a sparking hob that we meet again below.

Option	Dripping ceiling (experiment 0001)	Sparking hob (experiment 0002, request r159)
plumbing	0.99	0
electrical	0.01	0.71
heating	0	0
appliance	0	0.29
pests	0	0
locks	0	0
`choice`	plumbing	electrical

dripping ceiling   0.99 + 0.01 + 0 + 0 + 0 + 0   =  1.00
sparking hob       0 + 0.71 + 0 + 0.29 + 0 + 0   =  1.00

In the first answer one tray took almost everything. In the second the clerk was torn between two. The winner alone would hide that.

Confidence: how far above an even split

The confidence of a Choice is one number from 0 to 1 that says how far the top probability stands above an even split. An even split means every option gets the same share, 1/n for n options. TypeSafe publishes the formula:

confidence = (p_max - 1/n) / (1 - 1/n)

p_max   the probability of the winning option
n       the number of options

With six options an even split is 1/6, about 0.1667. We work it out for both answers.

dripping ceiling    p_max = 0.99
                    (0.99 - 0.1667) / (1 - 0.1667)  =  0.8233 / 0.8333  =  0.988
                    the response said 0.99

sparking hob        p_max = 0.71
                    (0.71 - 0.1667) / (1 - 0.1667)  =  0.5433 / 0.8333  =  0.652
                    the recorded answers of experiment 0002 hold 0.65

The small gaps are what rounding to two decimals would cause (Chapter 2). Experiment 0014 repeated this check on all 215 trade answers, and the largest gap was 0.006.

Only the top probability enters the formula, together with the number of options: the same top probability means less when there are fewer options to beat. And confidence describes the shape of the answer. It is not a promise that the answer is right.

The box below starts on the sparking hob: electrical 71, appliance 29. Predict before you touch it. If you pull appliance down to 0, what will the confidence be? And if you set all six sliders to the same value? The second button loads the documentation's quick start example, which has three options and probabilities of 0.85, 0.15 and 0.

From stamp to code

A Choice maps to a match statement: one branch of code for each option. The comparison comes from TypeSafe's chief executive, in a forum comment. TypeSafe says the answer is constrained to the options we supply, so the branches can be written in advance.

# a sketch, not the desk's real code
answer = answers["trade"]
if answer["confidence"] < 0.6:
    hand_to_rosa()                      # the clerk was torn
else:
    match answer["choice"]:
        case "plumbing":   book("plumber")
        case "electrical": book("electrician")
        case "heating" | "appliance" | "pests" | "locks":
            book(answer["choice"])
        case "other":      hand_to_rosa()   # the seventh tray, added below

The 0.6 is the line that the desk of Chapter 21 uses. Why the last branch? A Choice has no floor: constrained to our options, a paper that fits no tray is still put in one. Experiment 0025 sent 40 short messages, written by us, that fit none of the six trades: a stuck lift, a parking bay, a rent query, a thank-you. It asked with the six trades, and again with a seventh option, other.

SIX TRADES ONLY: the clerk must pick one anyway
  mean confidence 0.80   confidence below 0.6 on 10 of 40   at or above 0.9 on 24
WITH A SEVENTH OPTION, other
  picked other: 38 of 40   mean confidence 0.96
...
80 real test requests: correct with six options 80, with seven 79; sent to other by mistake: 1

The parking bay went to locks at 0.98. Confidence alone did not flag most of these off-topic papers. The line at 0.6 would have stopped 10 of the 40 and booked a tradesperson for the other 30.

The documentation's advice is the fix: "Add an other or none of the above option when the list might not cover every input". Our seventh card read "Anything that is not a repair for one of the trades above: lifts, parking, noise, rent, post, questions, thanks." It took 38 of the 40, and one of 80 repair requests. We wrote that card knowing the 40 messages, so this was a friendly test.

The mistakes: the sparking hob and eight more

A wrong Choice is an answer with the right shape and the wrong option. Experiment 0002 sent 215 labelled tenant messages. The messages are synthetic: gpt-5.1 wrote them from our labels, and a second, blind gpt-5.1 call agreed with every label we kept.

trade      (choice)  correct            99.5 %
...
trade mistakes:
  r159 label appliance  Jev electrical p=0.71 (label got 0.29)  Hi, this is flat 3B

correct    (215 - 1) / 215  =  214 / 215  =  99.5%

In request r159 "the electric hob suddenly started sparking from the back left ring, with a burning plastic smell". Our label says appliance. The clerk said electrical.

Our label came second with 0.29, and the returned confidence was 0.65. Is low confidence a usable alarm, then? One mistake cannot say. Experiment 0026 asked the trade question alone on 291 requests: the 215, plus 76 contested ones, where the blind second label disagreed with ours. On those we count our own label as the truth, so some of the "wrong" answers are arguments. In r020, hot water has stopped: our label says plumbing, the clerk said heating, and the heating card lists "hot water".

requests: 291   trade correct: 282   wrong: 9
...
trade confidence      n    wrong   correct
  0.00 - 0.60        6      1     83.3%
  0.60 - 0.80        9      5     44.4%
  0.80 - 0.95       12      2     83.3%
  0.95 - 1.00      264      1     99.6%

at 0.95 or more   264 / 291  =  90.7% of the requests       263 / 264  =  99.6% right
line at 0.6       hands over 6            catches 1 of 9      9 - 1  =  8 mistakes go through
line at 0.8       hands over 6 + 9 = 15   catches 1 + 5 = 6   9 - 6  =  3 mistakes go through

Most mistakes sit below 0.95. The hob's confidence was 0.71 in this call and 0.65 in the three-question call of experiment 0002. Experiment 0035 made 20 new calls each way: the confidence ranged from 0.66 to 0.75 alone and from 0.69 to 0.78 with the two other questions, mean 0.72 both ways. So the gap is the size of the movement between calls on an unsure answer, and we see no effect of asking three questions in one call (Chapter 15).

So a line at 0.6 lets the hob through. Three mistakes stay above 0.8, one of them at 0.99. Low confidence points at doubtful answers without catching all of them. The lower bands are small, so their percentages are rough. Chapter 12 shows how to choose a line on one half of the data and test it on the other.

Wording, long lists and option order

A Choice has three things you can change: the words, the number of options and their order.

Wording. Experiment 0006 changed one thing at a time on 108 of the 215 requests.

CHOICE (trade)                                         correct   mean confidence
  baseline: names + one-line descriptions               99.1%      0.98
  names only, descriptions null                         97.2%      0.94
...
  opaque names A-F, descriptions kept                  100.0%      0.97
  opaque names A-F, no descriptions (nonsense)          10.2%      0.45

baseline 99.1% of 108  =  107 right       names only 97.2% of 108  =  105 right
letters, no cards 10.2% of 108  =  11 right     blind guessing 108 / 6  =  18 right

The clerk needs meaning in at least one place. Clear names without cards cost two requests. Letters A to F with cards cost none. Letters without cards did no better than guessing, and still came back with a mean confidence of 0.45.

Long lists. Experiment 0011 kept the six trades and added invented departments, first 30 named ones such as roofing and painting, then numbered supplier contracts, up to the limit of 255. Each list was shuffled once, so the order differs between rows too. It used 80 requests.

options   correct   mean p(top)   p50 ms   p95 ms   input tokens   $ per call
     6    100.0%      0.98         226      343          529     $0.000022
...
    36     93.8%      0.96         231      322         1055     $0.000044
...
   255     92.5%      0.96         258      339         6860     $0.000288

right at 255 options     92.5% of 80  =  74     (6 wrong)
tokens per extra option  (6,860 - 529) / (255 - 6)  =  6,331 / 249  =  25.4

The waiting time barely moved, the bill grew with the list, and accuracy fell from 80 right to 74. Experiment 0033 looked at which options won on the misses.

36 options: 5 misses of 80 (answers read from cache: True)   winners: {'heating': 2, 'painting': 3}
...
255 options: 6 misses of 80 (answers read from cache: True)   winners: {'heating': 2, 'painting': 4}

No miss went to a numbered contract. All went to "painting" or "heating". The painting misses include a front door with "a big scratch and chipped paint", a fair job for a painter, and the heating misses are hot-water taps again. So part of the drop is our label. Painting joined the list at 36 options, where the first five misses appear, and 219 further options added one more.

Order. The documentation warns that "the order of a Choice's options can affect the answer, and jev-1.13 leans toward the option that comes first". Its remedy is to "reorder the options to double check that the answer stays consistent".

From the lab: six orders, then one order six times

Experiment 0042 sent the trade question with its options in six different orders, every call fresh: (a) on the 291 requests above, and (b) on 60 messages that gpt-5.1 wrote to sit between two trades.

(a) agreed + contested requests: 291 messages x 6 orders = 1746 fresh calls
   messages whose chosen trade changed with the order: 4 of 291 (1.4%)
   the winner was the option listed first in 288 of 1746 calls (16.5%; six options, so about 16.7% if order did not matter)
...
(b) close-call messages: 60 messages x 6 orders = 360 fresh calls
   messages whose chosen trade changed with the order: 3 of 60 (5.0%)
   the winner was the option listed first in 54 of 360 calls (15.0%; six options, so about 16.7% if order did not matter)
   spread of the first-order winner's probability across the six orders: median 0.07, 90th percentile 0.18, largest 0.34
   when one of the two plausible trades won, it was the one listed earlier of the two in 203 of 360 calls (56.4%)

Answers also move between identical calls, so experiment 0044 is the control: the same messages, one order, six fresh calls each.

291 agreed + contested requests: the same order, six fresh calls each
   messages whose chosen trade changed between identical calls: 0 of 291
...
60 close-call messages: the same order, six fresh calls each
   messages whose chosen trade changed between identical calls: 2 of 60
   spread of the winner's probability across the six calls: median 0.03, 90th percentile 0.10, largest 0.16

First, we see no lean to the first position at six options: the first-listed option won 16.5% and 15.0% of the calls, and no lean at all would give 1 / 6 = 16.7%. Second, reordering does move the probabilities, modestly. On a close call the winner's highest and lowest probability lay 0.07 apart at the median and 0.34 at most across six orders, against 0.03 and 0.16 across six identical calls. Third, the winner changed only on messages torn between two trades: on 4 of 291 requests and 3 of 60 close calls across orders, against 0 and 2 across identical calls. The 56.4% is above half, but 60 messages in six fixed orders are too few independent cases to call it a lean. Longer lists were not tested. Experiments 0015 and 0021 are void: they replayed saved replies, and our store ignored option order.

Try it yourself

Take the request file from Chapter 2 and change the state to "The lift has been stuck between floors since this morning, flat 9A." Run it with the six trades. In experiment 0025 this message went to electrical with a confidence of 0.89.

Then add a seventh option, "other", with the card quoted above. Before you run it, write down the probability you expect for other. Compute the confidence yourself with n = 7, and compare it with the confidence field.

Common questions

Should I act on confidence or on the top probability?

Both come from the same probabilities. The documentation calls its confidence "one reasonable way to summarize a distribution, not the only one", and suggests trying the top probability itself, or the ratio of the top to the runner-up. It notes that 0.5 is weak among two options and strong among ten. Confidence corrects for that.

Can a Choice return two answers?

No. choice is always one option, though the probabilities show the runner-up. If several labels can be true at once, TypeSafe's guide for coding agents suggests one Noul per label (Chapter 4).

Do I really need descriptions?

On our desk, clear names alone lost two requests in 108. Descriptions cost tokens. Write them, and read them as the clerk will: our heating card says "hot water", and hot-water messages went to heating.

Does a confidence of 1.0 mean the answer is right?

No. The documentation says of a Score at confidence 1.0 that this "describes the model's answer, not a guarantee that the answer is correct". The same holds for a Choice: one of our nine mistakes came at 0.99.

Carry this

A Choice returns one option, a probability for every option (they sum to 1) and a confidence.
Confidence = (p_max - 1/n) / (1 - 1/n). Sparking hob: (0.71 - 0.1667) / 0.8333 = 0.652.
Trade was right on 214 of 215 synthetic requests, and on 282 of 291 with the contested ones. A line at 0.6 catches 1 of the 9 mistakes, a line at 0.8 catches 6.
Meaning must sit in the names or in the descriptions. With neither, 11 of 108 were right.
Confidence alone did not flag most off-topic papers: 24 of 40 landed in a trade at 0.9 or more. An other option took 38 of the 40.

Check yourself

1. A Choice with four options returns a top probability of 0.50. What is its confidence? What would the same 0.50 give with two options?

Answer

four options   (0.50 - 0.25) / (1 - 0.25)   =  0.25 / 0.75   =  0.33
two options    (0.50 - 0.50) / (1 - 0.50)   =  0 / 0.50      =  0

Half the probability is a weak lead among four options and no lead at all among two.

2. Two answers to a three-option Choice are (0.6, 0.4, 0) and (0.6, 0.2, 0.2). Which has the higher confidence?

Answer

Neither. Only the top probability enters the formula, so both have (0.6 - 0.3333) / 0.6667 = 0.4. To tell them apart, read the probabilities: in the first, one rival is close behind.

3. Rosa's list has six trades and no other. A tenant writes that the bicycle store is full. What comes back, and what should change?

Answer

One of the six trades comes back, and nothing forces its confidence to be low: in experiment 0025, 24 of 40 such messages came back at 0.9 or more. Add an other option with a clear description, and let the code send that branch to a person.