Part V · Practice
Chapter 22
The decision lab
Seventeen of the book's interactives on one page, each with a small task, and one new simulator in which you set the gate for twenty recorded requests.
On this page
In this chapter, we will learn by playing. Seventeen interactives from the earlier chapters are collected here, each with a small task: set something, predict, then look. At the end is a new one, where you run the back office for twenty recorded requests.
One rule for the whole page: guess before you look. No interactive here calls a model. Three interactives stay at home, tied to their text: the sorting game of Chapter 1, the weak spots of Chapter 13 and the translator of Chapter 17.
The three stamps
A stamp is one typed question: Choice picks one option, Noul gives the probability of yes, and Score places the paper on a scale.
Recorded requests. Pick a message you have not read yet. Predict the trade, the level and whether the permission dial is near 0 or near 1, then show the recorded answers. Home: Chapter 2.
Choice confidence. It opens on the one wrong trade of experiment 0002, the sparking hob: electrical 71, appliance 29. Now spread the 29 over the five other trades, as 6, 6, 6, 6 and 5, and predict whether the confidence moves. Home: Chapter 3.
The Noul line. Switch to the test half and set the threshold to 0.41. Predict how many dangerous requests are missed, then look at the false alarms. Home: Chapter 4.
The ruler. The sliders start at 27 on level 1 and 73 on level 2. Move the 27 from level 1 to level 3, and predict the new score and whether the confidence changes. Home: Chapter 5.
How sure, and how it was asked
A probability is only useful if it means what it says, and it depends on how the question was worded.
Calibration bands. Start with Jev and read the band from 0.5 to 0.7. Predict what that row shows when you change the truth from the old label to the three readers' majority. Then switch to "gpt-5.1, token odds" and predict the share of answers between 0.1 and 0.9. Home: Chapter 6.
The wording lab. It opens on "Score: severity" with the baseline wording. Predict the accuracy with bare labels before you click "Bare labels". Then switch to "Choice: trade" and look for the one wording that breaks it. Home: Chapter 8.
The same call, thirty times. Stay on the severity score and slide the threshold into the widest bar. Predict what your code would do with that request from one call to the next. On the 100 requests of experiment 0032, each sent 10 times, the desk's route changed on 5. Home: Chapter 15.
Token odds. Raise the raw score of "no" until it equals the raw score of "yes". Predict the token odds of "yes" first. The sliders are illustrative. Home: Chapter 18.
What a call costs
The cost of a call is its input tokens times the price, and its time changes little with the number of questions.
A bigger state. Drag the old messages to 150. Predict which grows more, the time or the cost per 1,000 calls. Home: Chapter 7.
One call or many. Set the questions per message to 30. Predict how the time of one call with 30 questions compares with 30 calls of one question each. Home: Chapter 9.
Scale. Move the requests per month from 240 to about one million and predict the monthly bill for Jev. Then switch the engine to gpt-5.1. Home: Chapter 16.
A voice turn. Switch the decision step from Jev to gpt-5-nano. Predict whether the whole turn still fits inside the target of 800 milliseconds. Only the decision step is measured. Home: Chapter 19.
The back office
The back office is the ordinary code around the model: facts, rules, thresholds, templates and actions.
Who does what. Sort the twelve tasks into code or one of the three stamps. Before each pick, ask whether code could work out the answer exactly. Home: Chapter 10.
One request through the desk. Pick "Warm fridge" and step forward to the stamps. Predict whether the gate will let it through. Home: Chapter 11.
The cheapest line. Set both costs to $50, so that a missed danger is no dearer than a false alarm. Predict which way the cheapest line moves. Home: Chapter 12.
The injection lab. It opens on the fake staff note with the plain question. Predict the share for Jev, then change the defence to the fenced instruction and predict again. Home: Chapter 14.
The pattern picker. Read each of the seven needs and name the pattern aloud before you click it. For the cascade, experiment 0024 measured the pair: sending the unsure 13 of 107 requests to gpt-5.1 left accuracy at 96.3% and raised the cost from $0.029 to $0.120 per 1,000. Home: Chapter 20.
Play the back office
The desk simulator lets you set the gate for twenty recorded requests and see which replies would have gone out wrong. The requests are the twenty of Chapter 2: one short message for each pair of trade and level where the dataset had one. They are synthetic, like the whole dataset. The stamps are the answers Jev gave in experiment 0002. The model is not called. Only the gate runs, in your browser. It makes the three checks on the stamps from Chapter 21, and leaves out the check on the flat number.
You set three things.
- The severity confidence line. A severity answer with confidence below the line goes to Rosa. It starts at 0.60.
- The permission band. A permission dial strictly inside the band goes to Rosa. It starts at 0.20 to 0.80.
- Level 3 always to a person. When this is on, every request stamped as a danger also goes to Rosa.
The trade line stays at 0.60. A row is right when the trade, the rounded level and the permission all match our labels. At the start, 18 of the 20 are answered automatically and 2 of those 18 are wrong.
automatic 18 / 20 = 90.0% automatic and right (18 - 2) / 18 = 88.9%
Now predict. If you raise the severity line from 0.60 to 0.80, how many requests will Rosa read, and how many wrong replies will still go out? Then move the slider.
Try it yourself
In the desk simulator, find the setting that sends no wrong reply and gives Rosa the fewest requests to read. Then turn on "level 3 always to a person" and count what that rule costs her. Last, put your numbers into the program of Chapter 21 and run its eight requests again.
Common questions
Do these interactives call the model?
No. None of them calls any server. They replay answers that our experiments recorded on 7 October 2026, or do arithmetic on them.
How do I know which numbers are real?
Each interactive marks its numbers as measured, derived or illustrative. Most measured ones come from
data-lab.js, a file that a script writes from the experiment outputs. The rest are copied into the
interactive's code from an experiment's printed output.
Why are there only twenty requests in the simulator?
Because twenty fit in one table that you can read. They are the first message of at most 330 characters for each pair of trade and level. Six trades and four levels make 24 pairs, and 20 of them had such a message.
Can I try my own message?
Not on this page, because nothing here calls the model. Add your message to the OURS list in
the program of Chapter 21 and run it.
Carry this
- Guess before you look. No model is called on this page.
- A gate is a trade: the severity line at 0.80 stopped both wrong replies and gave Rosa 4 right ones to read.
- Twenty rows can teach how a gate behaves. They cannot choose its thresholds.
Check yourself
1. In the desk simulator you raise the severity line to 0.80 and leave the rest alone. The tiles show 13 automatic replies, none of them wrong, and say that 3 of the requests handed to Rosa had wrong stamps. How many does Rosa read, and how many of those were right all along?
Answer
Rosa reads 20 - 13 = 7 of which wrong stamps 3 right all along 7 - 3 = 4 automatic share 13 / 20 = 65.0%
Stopping the 2 wrong replies took Rosa from 2 reads to 7, and 4 of the 7 were not needed.
2. In the ruler, 27 moves from level 1 to level 3 while 73 stays on level 2. What happens to the score and to the confidence?
Answer
score before 1 x 0.27 + 2 x 0.73 = 1.73 score after 2 x 0.73 + 3 x 0.27 = 2.27 confidence, both 1 - 0.27 x 1 / 1.0 = 0.73
The score moves by about half a level. The confidence does not move, because the 0.27 is still one level away from the most likely level.
3. Why is "level 3 always to a person" a rule for the back office and not a question for the clerk?
Answer
Because it is a policy, not a judgment. The clerk judges how severe the problem is. Whether a person must see every danger is Rosa's decision, and it belongs in code that she can read.