Part I · Novice
Chapter 5
Score: the ruler
The third stamp places a paper on a scale that we describe in words. We work a score and its confidence out by hand, and measure what happens when the levels are described badly.
On this page
In this chapter, we will learn the third stamp: the Score. It answers "how much" by placing the paper on a scale whose steps we describe in words. We will read a real answer, work the score and its confidence out by hand, and see a measurement in which the same model fell from 88.0% to 29.6% because the steps were named and not described. At the end we will know which of the three stamps to reach for.
Levels on a ruler
A Score is a question whose answer is a position on a scale, and the scale is an ordered list of described levels. A level is one step of the scale, written as a sentence. This is the Score that Rosa's desk asks about every message, exactly as our experiments send it:
"severity": {
"type": "score",
"instructions": "How severe is the problem described?",
"criteria": [
"Cosmetic. Nothing stops working.",
"Inconvenient. Something works badly but the flat is usable.",
"An essential service is lost: no water, no heat, no power, no working toilet or no secure door.",
"Danger to people or to the building: fire, gas, flooding, electric shock."
]
}Three rules come from the documentation. The list runs from the low end to the high end. A level's number is its place in the list, and counting starts at 0, so "Cosmetic" is level 0 and "Danger" is level 3. And a Score has at least two levels and at most ten.
We never write the numbers ourselves. The order of the list is the numbering.
The score is a weighted mean
The score is each level number multiplied by its probability, added up. This kind of sum is called a probability-weighted mean: an average in which likely levels count for more.
Take another message from our dataset. A tenant in flat 14B writes that there is no cold water from any tap, while the hot water still runs. This is from the recorded answers of experiment 0002, with the long legend lines shortened. Experiment 0014 prints the same numbers.
"severity": {
"type": "score",
"score": 1.73,
"legend": { "0": "Cosmetic. ...", "1": "Inconvenient. ...", "2": "An essential service ...", "3": "Danger ..." },
"probabilities": { "0": 0, "1": 0.27, "2": 0.73, "3": 0 },
"confidence": 0.73
}probabilities gives one probability for each level, and together they add up to 1.
legend repeats our descriptions next to their numbers, so that the answer can be read on its
own. The clerk put 0.73 on "an essential service is lost" and 0.27 on "inconvenient". Now the score:
score = 0 x 0 + 1 x 0.27 + 2 x 0.73 + 3 x 0
= 0 + 0.27 + 1.46 + 0
= 1.73
The paper sits about three quarters of the way from level 1 to level 2. When the code needs one whole level, we take the nearest. Our experiments do it by adding 0.5 and cutting off the decimals.
nearest level 1.73 + 0.5 = 2.23 cut the decimals: level 2
Level 2 is also the label of this message. The same check on all 215 answers is experiment 0014. It found that the returned score and our own sum never differed by more than 0.02. A gap of that size is what rounding would cause: experiment 0040 counted the 3,010 numbers in those recorded answers, and all but 2 have at most two decimals.
Score confidence
Score confidence is one number from 0 to 1 that says how tightly the probability is gathered around the most likely level. It is 1 when everything is on one level. TypeSafe publishes the formula, so we can work it out ourselves. It has three steps.
1. spread for each level: probability x distance from the most likely level, added up
2. even spread the same kind of distance for a clerk with no idea at all (equal probability
on every level, measured from the middle of the scale)
3. confidence 1 - spread / even spread and never below 0
For four levels the middle of the scale is 1.5, so the even spread is always the same number:
even spread = ( |0 - 1.5| + |1 - 1.5| + |2 - 1.5| + |3 - 1.5| ) / 4
= ( 1.5 + 0.5 + 0.5 + 1.5 ) / 4 = 4 / 4 = 1.00
Now the cold-water message. Its most likely level is 2.
spread = 0 x 2 + 0.27 x 1 + 0.73 x 0 + 0 x 1 = 0.27 confidence = 1 - 0.27 / 1.00 = 0.73 returned: 0.73
And the mouse message. Its most likely level is 3, and a lot of probability sits two levels away.
spread = 0 x 3 + 0.32 x 2 + 0.02 x 1 + 0.66 x 0 = 0.66 confidence = 1 - 0.66 / 1.00 = 0.34 returned: 0.34
Experiment 0014 repeated this for all 215 answers. The largest gap between the formula and the returned confidence was 0.000.
Why distance matters
The documentation gives two three-level examples, (0, 0.5, 0.5) and (0.5, 0, 0.5). In both, the top level has probability 0.5. For three levels the even spread is (1 + 0 + 1) / 3 = 0.67.
(0, 0.5, 0.5) torn between neighbours spread 0.5 x 1 = 0.5
confidence 1 - 0.5 / 0.67 = 0.25
(0.5, 0, 0.5) torn between the two ends spread 0.5 x 2 = 1.0
confidence 1 - 1.0 / 0.67 = -0.5, floored at 0
A clerk who cannot choose between "inconvenient" and "essential" is a little unsure. A clerk who cannot choose between "cosmetic" and "danger" does not know what the paper says. A ruler has an order, and this formula uses it. The Choice formula of Chapter 3 looks only at the top probability and would call both cases the same.
Before you touch the sliders below, predict: if we move 20 points of weight from level 2 to level 0, will the confidence fall by more or by less than if we move them to level 3?
How good was the ruler?
On our 215 agreed requests, the nearest level matched the label 92.6% of the time, and it was within one level 99.5% of the time. This is experiment 0002. Rows are the labels, columns are the nearest level of Jev's score.
severity: label (rows) against Jev's nearest level (columns) label 0 1 2 3 0 56 3 1 0 1 0 51 6 0 2 0 3 46 2 3 0 0 1 46
exact 56 + 51 + 46 + 46 = 199 199 / 215 = 92.6% two levels off 1 (a level 0 read as level 2) within one 215 - 1 = 214 214 / 215 = 99.5%
Almost every miss is a neighbour, which is what we hope for from a ruler. Two cautions belong here. The messages are synthetic: a language model, gpt-5.1, wrote them from labels we chose, and we kept only messages on which a second, blind reading by the same model agreed. And severity was the label the two readings disagreed on most. Of the 76 messages we set aside, 66 were set aside because of severity. Chapter 6 looks at those.
Describe situations, not degrees
A level description must say what a paper at that level looks like, because the description is all the clerk has to go on. The documentation puts it as "Describe situations, not degrees". It says each level is judged on its own against the state (the paper on the desk), and that the model does not see a level's number or its neighbours. So "worse than the previous level" means nothing, and neither does "medium".
We tested it. Experiment 0006 changes only the four descriptions. It runs on 108 of the 215 requests, the half we allow ourselves to experiment on, called the dev half. The last column is the mean distance between the score and the label, in levels.
SCORE (severity) exact mean abs. error baseline: four described levels 88.0% 0.15 bare labels: none / low / medium / high 29.6% 0.73 numbers only: 0 / 1 / 2 / 3 32.4% 0.67 levels given in reverse order (scored back) 88.0% 0.15
described levels 88.0% of 108 = 95 right bare labels 29.6% of 108 = 32 right lost by naming and not describing 88.0 - 29.6 = 58.4 points
The question, the messages and the model were identical. Only our sentences changed. The last row is the calm one: with the same four descriptions listed from danger down to cosmetic, and the score turned back, accuracy was again 88.0%. On this test the clerk read the descriptions, not their position.
The documentation adds three more pieces of advice, which we repeat without having tested them:
- Use as many levels as you can describe distinctly, up to 10. Three is fine.
- Keep one Score to one dimension. A level that says "punctual and smart and experienced" measures three things.
- If the top of the scale has a rare extreme that needs different handling, give it its own level. Our level 3 is that level: it is the only one that sends somebody within the hour.
Try it yourself: pick the stamp, then sort
Now that we know all three stamps, here is how the documentation tells them apart.
| Stamp | Fits when | Desk question | In code it becomes |
|---|---|---|---|
| Choice | the answer is one of a known set of options with no order between them | which trade? | a match |
| Noul | a clean yes/no question where the probability itself is the useful signal | may we enter? | an if |
| Score | the answer falls on a spectrum and we can describe each point | how severe? | sorting, or a threshold |
When two stamps seem to fit, the documentation says to prefer the one whose answer the code can act on directly. It also warns against stretching a Noul into a ruler: 0.5 on "the problem is severe" would mean "I cannot tell", not "medium".
The Score's own use is sorting. Rosa's morning queue is one line of code, and the promise made to the tenant is a lookup that the back office owns. Write the two for your own desk, then decide what should happen to an answer whose confidence is 0.34.
queue = sorted(requests, key=lambda r: r["severity"]["score"], reverse=True)
PROMISE = {3: "within 1 hour", 2: "within 4 hours",
1: "within 3 working days", 0: "at the next monthly inspection"}
level = int(answer["score"] + 0.5) # 1.73 becomes 2
print(PROMISE[level]) # within 4 hoursCommon questions
Does a score of 1.73 mean the problem is 73% of the way to "essential service lost"?
No. It means the clerk split its belief 0.27 to 0.73 between two descriptions. The documentation asks us not to read an exact magnitude between two levels from a score, and says Jev's score levels are "weak in numerical calibration". Use the score to rank, to round, or to compare with a line.
Why not ask a Choice with four options named after the levels?
A Choice has no order. It could not tell us that a miss by one level is milder than a miss by three, its confidence would ignore distance, and there would be no number to sort by. If the answers have no in-between at all, the documentation says to use a Choice or several Nouls.
Can I compare two Scores with different numbers of levels?
Yes. The documentation's advice is to divide each score by its top level number. Our severity score of 1.73 on a scale that tops out at 3 becomes 1.73 / 3 = 0.58.
Will the same message always get the same score?
No. In experiment 0032 we sent 100 requests 10 times each. Across the 10 calls the severity score moved by 0.01 at the median and by 0.17 at most, and the nearest level changed on 4 of the 100 requests. The unsure answers moved most: the mean movement was 0.008 where confidence was 0.9 or more and 0.111 where it was below 0.6. A score near 1.5 can round either way, so read the confidence before acting on the level.
What does a low confidence on a Score usually mean?
The documentation names three causes: the levels overlap for this paper, the question measures more than one thing, or the paper does not say enough. The mouse message looks like the first kind, though nothing in the answer says so: our descriptions do not say whether mice are a nuisance or a danger to health.
Should I put example messages inside the level descriptions?
The documentation allows a level to be an object with examples and reports a case where a fitting example raised confidence from 0.35 to 0.96. It also warns that higher confidence does not show the answer is right. We did not test examples. Test them on labelled messages before keeping them.
Carry this
- A Score is an ordered list of 2 to 10 described levels. The first is level 0.
- Score = sum of level number x probability. For (0, 0.27, 0.73, 0) that is 1.73, and the nearest level is 2.
- Confidence = 1 - spread / even spread, never below 0. Probability far from the top level costs more than probability next to it.
- Measured: 92.6% exact and 99.5% within one level on 215 synthetic requests.
- Measured: described levels 88.0%, bare labels 29.6%. Describe situations, not degrees.
Check yourself
1. An invented answer on our four-level ruler has probabilities (0.10, 0.60, 0.30, 0). Work out the score, the nearest level and the confidence.
Answer
score 0 x 0.10 + 1 x 0.60 + 2 x 0.30 + 3 x 0 = 0.60 + 0.60 = 1.20 nearest 1.20 + 0.5 = 1.70, cut the decimals = level 1 most likely level 1 spread 0.10 x 1 + 0.60 x 0 + 0.30 x 1 + 0 x 2 = 0.40 confidence 1 - 0.40 / 1.00 = 0.60
2. What is the even spread for a ruler with five levels?
Answer
middle of the scale (5 - 1) / 2 = 2 even spread (2 + 1 + 0 + 1 + 2) / 5 = 6 / 5 = 1.2
A longer ruler has a larger even spread, so the same stray probability one level away costs less confidence.
3. A colleague's Score has the levels "low", "medium", "high" and "very high". What would you expect, and what would you change?
Answer
We measured a mistake of this kind (the labels none, low, medium, high): bare labels gave 29.6% against 88.0% for described levels. The documentation says each level is judged on its own, and "medium" describes nothing that can be matched against a message. Rewrite every level as a situation, such as "an essential service is lost: no water, no heat", and test the new wording on labelled messages.