Noul: the yes/no dial

In this chapter, we will learn the second of the clerk's three stamps: the Noul. It answers a yes/no question with one number between 0 and 1. We will write one for Rosa's desk, see why a number is more useful than a plain yes or no, and turn it into an if statement. Then we will move a line over 281 real answers and count what each position costs.

A statement and a number

A Noul is a yes/no question, and its answer is one number from 0 to 1: the probability that the answer is yes. A probability is a way of writing "how likely" as a number. 0 means never, 1 means always, and 0.5 means a coin toss.

In Chapter 3 the clerk picked one tray out of six. Here there is one statement, and the only question is whether it is true. This is the Noul that Rosa's desk asks about every message, exactly as our experiments send it:

"may_enter": {
  "type": "noul",
  "instructions": "The tenant has given permission to enter the flat while they are out.",
  "criteria": {
    "true": "The message clearly says staff may enter without the tenant present.",
    "false": "The message does not give that permission, or asks staff to wait."
  }
}

In Chapter 2 the tenant of flat 4B wrote that water was dripping through the bathroom ceiling light, and ended with "I am at work until 6, you can let yourself in." This is what experiment 0001 printed for our Noul. The answer has two fields and nothing else:

    "may_enter": {
      "type": "noul",
      "noul": 0.98
    }

The documentation tells us how to read the number: a value near 1 is a strong yes, a value near 0 is a strong no, and a value near 0.5 means the model gives yes and no similar probability.

Is the dial reading the sentence we think it reads? Experiment 0028 sends that message once as it is and four more times with one thing changed. The columns are the trade and its probability, the severity score and its confidence, and our Noul.

edit                                trade           p  severity  conf  may_enter
the original                        plumbing     0.99      2.99  0.99       0.98
permission sentence removed         plumbing     0.99      2.99  0.99       0.03
permission reversed                 plumbing     0.98      2.98  0.99       0.02
a slow drip, not through the light  plumbing     1.00      0.93  0.92       0.98
the light, not the water            electrical   1.00      1.47  0.53       0.98

Take the permission sentence out, and the dial falls from 0.98 to 0.03. Replace it with "please wait until I am home", and it reads 0.02. The other two stamps stay within 0.01 of where they were. In the last two rows we change the problem and keep the sentence: the other stamps move, and the dial stays at 0.98. We wrote these five messages ourselves, so this is a demonstration, not a measurement.

The documentation does not explain the odd name. A forum comment by TypeSafe's chief executive says that "noul" is short for Bernoulli. In statistics, a Bernoulli trial is anything with exactly two outcomes, such as one toss of a coin.

Why a number, and not a plain yes

A number lets our code decide where "yes" begins, and that point should depend on what a mistake costs. A model that answered only true or false would have made that choice for us, silently, at a point we cannot see.

The permission question can go wrong in two ways. If the desk sees permission where there is none, a plumber lets himself into a home uninvited. If it sees none where there was permission, somebody makes one extra phone call. The two mistakes do not cost the same.

So the clerk reports the dial, and the back office holds the line. The line is called a threshold: a fixed number that the answer is compared with.

The first two columns are the documentation's advice. The third is ours. The documentation adds that values in the middle can go to a person instead of to either path.

The if statement

The same forum comment says that a Noul maps to an if statement. Here is that if for the desk. The lines 0.8 and 0.2 come from an example in the documentation. They are a starting point, not a recommendation.

YES, NO = 0.8, 0.2                       # our lines, kept in one place

p = answers["may_enter"]["noul"]         # 0.98 for the dripping ceiling
if p >= YES:
    book_visit(tenant_may_be_out=True)
elif p <= NO:
    ask_tenant_for_a_time()
else:
    hand_to_rosa()                       # the clerk was unsure: a person decides

Writing the statement

The instruction of a Noul is a question or a statement that is either true or false of the paper on the desk. The documentation gives four pieces of advice, and we tested the ones we could.

• A statement works as well as a question. "The tenant has given permission" and "Has the tenant given permission?" are both allowed. The documentation says to try both on our own data.
• The criteria are optional. criteria holds a true and a false description of what a yes and a no mean. The documentation says the instruction alone is enough for most Nouls, and to keep whichever version gives better answers. It warns that criteria which contradict the instruction "will perform worse".
• Make a high value mean yes. "Is the message free of personal data?" turns the meaning around, and the documentation warns that "code that reads it later will get it backwards". It also says that instructions with double negatives are answered less reliably.
• One condition per Noul. "The tenant is angry and the problem is urgent" asks two things, and then the number means less. Ask two Nouls and combine them in code.

"Try it yourself" below tests the first three on our own data. A fifth piece of advice we measured ourselves: do not let the dial do arithmetic on dates. Experiment 0040 put today's date and a request's opening date in the state, with 100 fresh calls per wording: 20 at each gap from 12 to 16 days. "More than 14 days before today" was right on 94, and on 15 of the 20 at exactly 14 days. "14 or more days" was right on 79, and on 6 of 20 at 13 days. "At least 15 days" was right on 97. Adding "Exactly 14 days is NOT more than 14 days" to the instruction did worse: 68. Experiment 0041 asked the first wording about 100 other pairs of dates: 93 right, all 7 misses at exactly 14 days. The documentation gives the same advice: jev-1.13 "reads dates as text, not as ordered quantities". When code counted the days and sent the number, all 100 of those were right (Chapter 8 and Chapter 10).

How sure, without a confidence field

A Noul answer has no confidence field, because the one number already says how sure the clerk is. In Chapter 3 a Choice returned six probabilities and a confidence that summarised them. A Noul has only two outcomes, yes and no. If yes has 0.98, no has 0.02, and there is nothing left to summarise.

Sometimes we still want a "how sure" number, for example to send every unsure answer to Rosa with one rule. The documentation gives a formula: the distance from 0.5, doubled. The bars mean "drop the minus sign".

confidence = | 2 x p - 1 |

p = 0.98      | 2 x 0.98 - 1 |  =  | 0.96 |   =  0.96     a sure yes
p = 0.69      | 2 x 0.69 - 1 |  =  | 0.38 |   =  0.38     a hesitant yes
p = 0.50      | 2 x 0.50 - 1 |  =  | 0.00 |   =  0.00     a coin toss
p = 0.02      | 2 x 0.02 - 1 |  =  | -0.96 |  =  0.96     a sure no

The 0.98 and the 0.02 are two rows of the table above. The 0.69 is from the recorded answers of experiment 0002. It belongs to an angry message about the building's front door lock that says "You may LET YOURSELVES IN WHILE I'M OUT". Our label for that message is yes. The clerk leaned yes without much conviction. Notice that 0.98 and 0.02 are equally sure. Confidence says how far the dial is from the middle, not which way it points.

A line that costs something

Every position of the threshold trades one kind of mistake for the other. The permission question is too easy to show this. At a line of 0.5 the clerk agreed with the label on all 215 requests of our agreed set, the messages on which a second reader confirmed our labels (experiment 0002). So we take a harder Noul.

Experiment 0010 asks one statement about each message: "The situation described is a danger to people or to the building." It uses 281 messages: the 215 agreed ones, plus 66 on which the writer and a second reader disagreed about severity. A message counts as truly dangerous when its label is severity level 3. Those labels are noisy. A language model, gpt-5.1, wrote all the messages from labels we chose first, and on the 66 contested ones "the truth" is only the writer's intention. Chapter 6 tries a second truth.

Four words describe what can happen to one message. It is caught when it is dangerous and flagged. It is a false alarm when it is flagged and not dangerous. It is missed when it is dangerous and not flagged. Otherwise it is left alone. The table header adds the usual short names: TP (true positive), FP (false positive) and FN (false negative). Here is the half of the data we use for choosing a line, called the dev half: 140 messages, 26 of them dangerous.

DEV half
threshold   caught (TP)   false alarms (FP)   missed (FN)   precision   recall
   0.05         26               98               0         21.0%    100.0%
   0.10         26               67               0         28.0%    100.0%
   0.20         26               38               0         40.6%    100.0%
   0.30         26               27               0         49.1%    100.0%
   0.50         25               15               1         62.5%     96.2%
   0.70         23                5               3         82.1%     88.5%
   0.80         20                3               6         87.0%     76.9%
   0.90         16                2              10         88.9%     61.5%
   0.95          8                0              18        100.0%     30.8%

The last columns bring two new words. Precision is the share of flagged messages that were truly dangerous. Recall is the share of dangerous messages that were flagged. Let us redo the row for 0.90 by hand.

flagged      16 + 2          =  18
precision    16 / 18         =  88.9%
recall       16 / (16 + 10)  =  16 / 26  =  61.5%

A high line looks precise and lets 10 dangerous messages through. A low line misses nothing and sends the emergency crew to 98 flats that did not need it. No row is "the right one" until somebody says what a missed gas leak costs and what a wasted call-out costs. Chapter 12 does that: it picks a line on this half and checks it on the other half.

The interactive below holds all 281 answers, and its buttons show either half alone. It starts at a line of 0.5, where the test half's row, printed by the same experiment, is 33 caught, 17 false alarms and 0 missed:

caught          25 + 33     =   58
false alarms    15 + 17     =   32
missed           1 +  0     =    1
precision       58 / (58 + 32)  =  58 / 90  =  64.4%

Before you move the slider, predict: how low must the line go before nothing dangerous is missed, and how many false alarms will that cost?

python3 experiments/0006-wording/main.py

NOUL (may_enter)                                       correct   mean distance from 0 or 1
  baseline: statement + true/false criteria            100.0%      0.028
  statement only, no criteria                          100.0%      0.036
  phrased as a question                                100.0%      0.033
  negated statement (answers flipped back)             100.0%      0.054

dev requests: 108; 61 give permission

version                                                  correct at 0.5  mean p when permission given  when not given
aligned criteria (the desk's question)                           100.0%                          0.97            0.02
no criteria                                                      100.0%                          0.96            0.04
criteria swapped: true describes NO permission                   100.0%                          0.92            0.03
criteria about something else (is the tenant polite?)            100.0%                          0.95            0.04

flagged      23 + 5         =  28
precision    23 / 28        =  82.1%
recall       23 / (23 + 3)  =  23 / 26  =  88.5%

| 2 x 0.15 - 1 |  =  | -0.70 |  =  0.70
| 2 x 0.85 - 1 |  =  |  0.70 |  =  0.70