Writing questions that work

In this chapter, we will learn how to write a question that a decision model answers well. We will take the documentation's rules one at a time, try each on Rosa's desk, and then look at an experiment that changed the wording of each stamp in small ways.

One judgment per question

An atomic question asks for one judgment that a knowledgeable person could make in a few seconds. Atomic means that it cannot usefully be split further. The documentation calls this rule "probably the most important concept in this guide".

It gives a pair of examples. "Does this message convey urgency?" is a good question. "Analyze this message and determine the best course of action" is not, because it "needs slow reasoning, and it is a signal to break the task into small questions and compose the answers in code".

Suppose Rosa's first idea was one big Choice: "What should the desk do with this message?", with options such as "send a plumber now". That single question hides three judgments. The desk already splits them.

The combining happens in code, where it can be read, tested and changed.

level = int(answers["severity"]["score"] + 0.5)      # nearest whole level
promise = ["at the next monthly inspection", "within 3 working days",
           "within 4 hours", "within 1 hour"][level]

The documentation names the benefit: "Broad questions hide several judgments behind one answer. Atomic questions expose those judgments so you can inspect, tune, and combine them in code." If Harbor Court decides that a lost toilet deserves one hour and not four, Rosa edits a list. No question changes.

The model answers the question you wrote

Literal reading means that the model takes every word of a question at face value. The documentation says that Jev "answers the question you wrote, not the one you meant. Scoping words, negations, and implied conditions are read at face value."

A scoping word is a word that narrows a condition. The desk's Noul ends with the words "while they are out". Now imagine this tenant. The message is invented, and we have not measured it.

Tenant:  Come any evening after six, I will be at home to let you in.

Has this tenant given permission to enter the flat? Yes. Permission to enter "while they are out"? No. Had Rosa written the shorter statement and meant the longer one, the model would have answered the shorter one correctly, and the desk would have been wrong.

The documentation's remedy: "state the exact condition in the instructions. Be specific. Put boundary cases in the criteria."

Our first test of literal reading was too easy. Experiment 0022 described 60 invented flats with four radiators each, and asked four statements built on the scoping words at least one, all, only and none. Every answer was right. Experiment 0043 counted 49 different descriptions among the 60 flats, so that is 4 x 49 = 196 different requests. Each radiator had its own plain sentence.

A second test did find errors, where the condition compares numbers. Experiment 0040 put two dates in the state, today's and the day a request was opened, and asked one Noul with no criteria. It tried four wordings of a 14-day rule on requests opened 12 to 16 days ago. Each column holds 20 pairs of dates in three written formats (99 of the 100 pairs differ), and every call was fresh. A cell counts right answers out of 20. In the last row code sends the number of days: five values, each sent 20 times.

the request was opened ...                                                   12   13   14   15   16   (correct of 20; true gap in days)
more than 14 days before today                                               20   19   15   20   20
more than 14 days before today. Exactly 14 days is NOT more than 14 days     18   19   10   11   10
at least 15 days before today                                                20   20   17   20   20
14 or more days before today                                                 15    6   20   19   19
the app sends days_open; `days_open` is more than 14                         20   20   20   20   20

more than 14 days                20 + 19 + 15 + 20 + 20   =   94 of 100
  with the boundary spelled out  18 + 19 + 10 + 11 + 10   =   68 of 100
at least 15 days                 20 + 20 + 17 + 20 + 20   =   97 of 100
14 or more days                  15 +  6 + 20 + 19 + 19   =   79 of 100
the app sends days_open          5 x 20                   =  100 of 100

For whole days, "more than 14" and "at least 15" are the same rule, and they scored 94 and 97. Eight of their nine misses sit in the 14-day column, on the line itself. "14 or more" draws the line a day lower, and its misses moved with it: 6 right of 20 at 13 days. The documentation says that Jev "reads dates as text, not as ordered quantities". None of the four wordings was safe next to the line.

Spelling out the boundary made it worse. The sentence added in the second row took it from 94 to 68, with about half of the answers wrong at 14, 15 and 16 days. We put it in the instruction, not in the criteria where the documentation says to put boundary cases, and we have not tested the criteria. The dependable fix is the last row, and Chapter 10 builds it: code subtracts the dates.

Two more rules belong here.

• Instructions and criteria must agree. The criteria are the descriptions of the possible answers. The documentation says to "treat the criteria as an extension of the instruction", and warns that "a Noul where true maps to no and false maps to yes will perform worse". The desk's Noul is aligned: the statement is about permission, and its true text reads "The message clearly says staff may enter without the tenant present."
• Write directly. The documentation says that instructions "carrying double negatives or complex indirection are answered less reliably".

We tested the first rule on the desk's Noul, and the warning did not reproduce. Experiment 0031 asked the permission statement about 108 labelled requests in four versions: the desk's criteria, no criteria, the two criteria swapped so that true describes no permission, and criteria about the tenant's politeness. Every version put every request on the right side of 0.5. Swapped criteria only pulled the mean answer for the 61 tenants who gave permission from 0.97 to 0.92. One easy Noul does not make contradictory criteria harmless.

Describe situations, not degrees

A level of a Score should describe a situation that a message can be matched against, not a degree such as "low" or "high". This is the documentation's first rule for levels: "Describe situations, not degrees."

Experiment 0006 uses 108 labelled requests, the half of our dataset called the dev split (explained below). It asks one question per call and changes one thing at a time. "Exact" is the share of requests where the nearest whole level equals the labelled level. The mean absolute error is the average distance between the returned score and the labelled level, in levels.

SCORE (severity)                                       exact     mean abs. error
  baseline: four described levels                       88.0%      0.15
  bare labels: none / low / medium / high               29.6%      0.73
  numbers only: 0 / 1 / 2 / 3                           32.4%      0.67
  levels given in reverse order (scored back)           88.0%      0.15

The second row replaces the four descriptions with the words none, low, medium and high, and changes nothing else.

baseline       88.0% of 108   =  95 requests at the labelled level
bare labels    29.6% of 108   =  32 requests
difference     95 - 32        =  63 requests lost by removing the descriptions
in points      88.0 - 29.6    =  58.4

• Bare labels. "Medium" gives the model nothing to compare a cold radiator with. A described level names the lost service, and a message either matches it or does not.
• Numbers only. The digits 0 to 3 do hardly better: 35 requests against 32. The documentation says: "The model doesn't see a level's number or its neighbours", so "numbers in the descriptions or the instructions don't help".
• Reversed order. We listed the descriptions from danger down to cosmetic and turned the score back round in code. The result is the baseline again. That fits the documentation's statement that "every level is evaluated separately", though it does not prove how the model works inside.

Where the wording did not matter

A robust question is one whose answers stay the same when its wording changes without changing its meaning. The other two stamps were robust in this experiment, with one instructive exception.

CHOICE (trade)                                         correct   mean confidence
  baseline: names + one-line descriptions               99.1%      0.98
  names only, descriptions null                         97.2%      0.94
  ...
  vague instruction ('Category?')                       99.1%      0.99
  opaque names A-F, descriptions kept                  100.0%      0.97
  opaque names A-F, no descriptions (nonsense)          10.2%      0.45

one request        100 / 108       =  0.93 points
99.1% of 108                       =  107 right
97.2% of 108                       =  105 right     two more wrong without descriptions
10.2% of 108                       =  11 right
guessing one of six   100 / 6      =  16.7%, about 18 right

Removing the descriptions cost two: names such as "plumbing" are ordinary words that carry meaning. The one-word instruction "Category?" left the score where it was: the six described options still say what is being asked. Renaming the options A to F gained one request, which is not a finding.

The last row is the exception. Options named A to F with no descriptions leave nothing on the card that says what A is. The model was right 11 times in 108, no better than guessing. Mean confidence fell from 0.98 to 0.45. The confidence number noticed that the question was empty.

The dots hide the row for options in reverse order: a fault in our test program compared the baseline with itself there. Experiment 0042 redid it with fresh calls.

(c) dev half, 108 requests: correct with the options in the usual order 107, in reverse order 106; the two orders chose differently on 1

It also tried six orders on all 291 written requests. The trade changed on 4, and on 3 of 60 messages written to sit between two trades. With one order called six times (experiment 0044) it changed on 0 and on 2, and on the 60 the winner's probability moved by a median of 0.03, against 0.07 across orders (largest 0.16 against 0.34). So reordering has a modest real effect on the probabilities, and it changed the winner only on messages torn between two trades. The first-listed option won 16.5% and 15.0% of calls, against 16.7% if order did not matter, so we saw no lean to the first position at six options. Longer lists were not tested.

Now the Noul. The last column is the mean distance of the answers from 0 or 1. A small number means decisive answers.

NOUL (may_enter)                                       correct   mean distance from 0 or 1
  baseline: statement + true/false criteria            100.0%      0.028
  statement only, no criteria                          100.0%      0.036
  phrased as a question                                100.0%      0.033
  negated statement (answers flipped back)             100.0%      0.054

All four wordings got all 108 requests right. A single negation ("has NOT given permission"), sent without criteria, was handled correctly once our code flipped the answers back, with answers half as far again from 0 and 1 as the statement without criteria.

Before you use the interactive, predict one thing. The Choice tab has six wordings. Which one costs two requests, and which one almost all of them? Then pick each wording and read the question that was sent.

Test a wording before you trust it

A wording test runs the old and the new wording on the same labelled examples and compares the results. The documentation gives the reason: "Two wordings of the same scale can behave differently on your data."

Our 215 agreed requests are divided into two halves. The dev split has 108 requests, and we may look at it as often as we like. The test split has 107 requests. It is kept for the final number. If we tuned a wording on the rows we report, we would be reporting how well we fitted those rows.

Change one thing. Run both wordings on dev. If the new one is not better by more than a request or two, keep the old one. If it is, run it once on test and report that number.

A checklist for every question:

1. It asks for one judgment.
2. The whole question is in instructions, not in the id.
3. The exact condition is written out, scoping words included. No question compares numbers or dates.
4. The criteria agree with the instructions: yes means yes.
5. Each Score level describes a situation and makes sense when read alone.
6. Each Choice option has a description that separates it from its neighbours.

python3 experiments/0006-wording/main.py

numbers only    32.4% of 108   =  35 requests
baseline        88.0% of 108   =  95 requests
fewer           95 - 35        =  60 requests