A field guide to patterns

In this chapter, we will learn to recognise the shapes that systems built on a decision model keep taking. There are nine in this guide: four that TypeSafe's documentation names and five from its cookbooks. For each we give the need, the shape, the desk's version and the evidence. Then we combine them into Rosa's whole desk and list four shapes to avoid.

What a pattern is

A pattern is a named arrangement of questions and code that answers one recurring need. None of them is a product or a setting. Each is a few lines of ordinary code around a call that returns typed answers.

Evidence in this chapter comes in two kinds, and we mark each one. Measured means one of our experiments tested the shape on the desk's synthetic requests. Published, not tested by us means TypeSafe describes it and we have not checked it.

Cover the middle column and predict the pattern for each need. Then check yourself with the interactive. It holds six of these nine rows as seven needs, with a sketch for each: it splits the row about text into replies, under this book's name "candidates and templates", and extraction.

Four patterns the documentation names

TypeSafe's patterns page lists four: speculative fan-out, confidence-gated routing, composite scoring and intent routing.

Speculative fan-out

The need: many judgments about one paper. The documentation says to "Send many questions in a single call, including speculative ones, and let your code decide what's relevant". A speculative question is one whose answer only matters for some inputs.

state  -->  ONE call:  trade?  severity?  may_enter?  danger?  if a leak, is water near electrics?
                              |
        code reads the answers it needs and ignores the rest

Measured on 12 requests, experiment 0007, from Chapter 9:

one call, 100 questions       5,510 tokens     $0.000231     305 ms at the median
100 calls, 1 question each    100 x 539  =  53,900 tokens     100 x $0.000023  =  $0.0023
tokens saved                  53,900 / 5,510   =  9.8 times fewer

The trade answer did not move when 29 other questions were added: the largest change in any probability was 0.00. Experiment 0037 ran the same check on the severity score of 30 contested requests: the mean gap between asking alone and among 30 questions was 0.011, inside the mean spread of 0.054 between identical calls. A second data point comes from the phone line of Chapter 19. Experiment 0034 put five decisions about one moment of a call into one request, on 24 hand-written moments: p50 216 ms and 758 tokens, where a single turn-detection question took 231 ms in experiment 0016.

Confidence-gated routing

The need: a wrong action costs more than a person's minute. The documentation puts it in one line: "The answer tells you what; confidence tells you whether to act."

answer + how sure  -->  sure enough for THIS action?  --yes-->  act
                                                       --no--->  a person

Measured, experiment 0017: with a gate at confidence 0.6, 94 of 107 requests were answered automatically, and 93 of those 94 were fully right. Of the 13 sent to a person, Jev's level was wrong on 3. Measured, experiment 0010: a line at 0.41 chosen on one half of the data caught 33 of 33 dangers on the other half, with 23 false alarms. Chapter 12 has both.

Composite scoring

The need: one number to sort a queue by. Ask for several narrow answers, scale each to the range 0 to 1, and weight them in code. The desk could rank the day's jobs this way. The weights and the two flats below are invented.

priority  =  0.6 x severity / 3  +  0.25 x p(vulnerable tenant)  +  0.15 x p(has happened before)

flat 4B   severity 1.8, 0.10, 0.20     0.6 x 0.6 + 0.25 x 0.10 + 0.15 x 0.20   =  0.360 + 0.025 + 0.030   =  0.415
flat 9A   severity 1.8, 0.90, 0.80     0.6 x 0.6 + 0.25 x 0.90 + 0.15 x 0.80   =  0.360 + 0.225 + 0.120   =  0.705

Two jobs of equal severity now have an order, and changing a priority means changing a weight in code. Published by TypeSafe, not tested by us: we have not compared such a queue with a person's ordering.

Intent routing

The need: send each request to the right handler. One Choice, one code path per option, and an "other" option for what does not fit. The documentation lists three kinds of handler: "deterministic logic, a specialist LLM, or a human". The desk's trade question is this pattern. Measured, experiment 0002: it was right on 99.5% of 215 requests. Measured, experiment 0025: the "other" option matters. With six trades only, 40 messages that belong to no trade were all forced into one, 24 of them at a confidence of 0.9 or more. With "other" added, 38 of 40 went there.

Five more from the cookbooks

TypeSafe's cookbooks show five further shapes: the cascade, reranking, self-consistency, select instead of generate, and probabilities as features. We measured the first and the fourth. The other three are published by TypeSafe and not tested by us.

Cascade

A cascade is a chain of steps ordered from cheap to expensive, where a case moves on only when the earlier step was unsure. The cookbook's own chain puts a cheap model first, Jev in the middle and a reasoning model last. Ours starts with the clerk.

every request  -->  the clerk  --sure-->  done
                              --unsure-->  the letter writer, or a person

The desk already has the first half: the gate. Experiment 0024 added the second half on the 107 test requests. The 13 requests that the gate stopped went to gpt-5.1, and a request counts as right when all three answers match the labels.

                                     right   $ / 1,000   mean ms
jev alone                            96.3%       0.029       284
gpt-5.1 alone                        95.3%       0.757      1087
cascade: jev, then gpt-5.1 if unsure   96.3%       0.120       420

escalated to gpt-5.1: 13 of 107 (12.1%).  On those, Jev alone was right on 10, gpt-5.1 on 10.
...
cost arithmetic per 1,000 requests: 0.029 (jev on all) + 0.121 x 0.757 (gpt-5.1 on the escalated share) = 0.121

cost of the pair      0.120 / 0.029   =  4.1 times Jev alone       0.757 / 0.120  =  6.3 times less than gpt-5.1 alone
gain in accuracy      96.3 - 96.3     =  0 points
on the 13 hard ones   Jev right on 10, gpt-5.1 right on 10

No gain here. The pair was exactly as right as Jev alone, at four times the cost and a mean of 420 ms against 284. The table's $0.120 adds up the real cost of the 13 calls, and the arithmetic line uses the mean price, which is why it ends on 0.121. A cascade pays only when the second model is better on the hard cases. Here it was not, even though gpt-5.1 wrote this data. A person as the second step is a different matter: that is the desk of Chapter 21.

Rerank by Noul probability

To rerank is to take a list that a cheap search has already found and put it in a better order. Ask one Noul per candidate, in one call, and sort by the number. For the desk: "This past job describes the same fault as the new request", asked of twenty past jobs. Top-1 accuracy is the share of queries whose first result is the right one. TypeSafe's cookbook reports that reranking raised it "from 5% to 18%" over a keyword search on 40 legal queries. That is their number on their data.

Self-consistency

Self-consistency means asking the same thing several times and acting only when the answers agree. TypeSafe's cookbooks repeat a question and compare the answers. For the desk: the danger statement asked three times, and a person if the answers disagree. Rewording the statement each time is our own variant, not the cookbook's. Chapter 15 measured how far identical repeats move. We have not tested agreement as a gate, with repeats or with rewordings.

Select instead of generate

When a value or a sentence is needed, let code or a language model propose candidates, and let the clerk pick one. For extraction: a pattern match finds every flat number in the message, and a Choice picks the tenant's own. For replies: the candidates and templates of Chapter 11.

text  -->  candidates ["14B", "4B", "not stated"]  -->  Choice picks one  -->  code uses it

Measured for replies, experiment 0017: replies assembled from fixed phrases were fully right on 93 of 94. gpt-5.1 writing the whole reply was right on 95 of 107, which is 88.8%. To be fair to the letter writers, the same pattern check found no promise outside the policy in their replies, or in the 94 assembled ones (experiment 0045).

Measured again, experiment 0030, with a real catalogue: five candidates, nine questions in one call, and no gate. The whole reply was right on 96 of 107 requests, which is 89.7%, at a p50 of 234 ms and $0.063 per 1,000. The errors sat with the model's stamps: the promise was wrong on 6 requests, safety advice was added wrongly 5 times and missed once, and the master-key line was missed once. Some replies had two errors, so 11 replies were wrong. The two candidates that code decided had no errors. The facts in that test were generated, and its answer key is computed from our labels. The safety advice and the master-key line were each kept when a Noul said it was relevant (0.5 or more) and a Score said it was useful (1.0 or more). Experiment 0039 re-read those answers, and the Score added nothing: the Noul alone gave the same 96 of 107, and the Score alone gave 91. The extraction form is published and not tested by us.

Probabilities as features

A feature is one input number of a small statistical model that you train on your own history. The documentation suggests it for "learned composition": the probabilities from many narrow questions go in, and the model learns the weights that composite scoring made us choose by hand. It needs labelled outcomes, such as which jobs later turned out to be urgent.

Combining patterns: the whole desk

A real system is several patterns in a row, with code at every joint. Rosa's desk uses four of the nine, and could add two more.

message
   |
ONE call                 trade, severity, may_enter  (+ speculative extras)         fan-out, intent routing
   |
gate                     any answer unsure?  --yes-->  a person                     confidence-gated routing
   | no                                                 (or a slower model)         cascade: no gain in 0024
look up and fill         promise by level, access line, safety line                 select instead of generate
   |
queue                    priority = weights x answers                               composite scoring, not tested

The top three rows are the desk of Chapter 21, measured in experiment 0017 with its three standard questions. The model appears once, in the first row. Every other row is code.

Four shapes to avoid

An anti-pattern is a shape that looks reasonable and is known to cause trouble. These four come from the documentation, and we have measurements for three of them.

• One broad question. "What should we do with this request?" hides several judgments. The documentation: "Broad questions hide several judgments behind one answer." Split it and combine the answers in code.
• Arithmetic in the model. The documentation: "Jev is not a calculator." Measured, experiment 0009: asked whether a quote's line items passed a limit, Jev was right on 92% of 100 cases from raw items and on 100% when the code sent the total. Dates fail the same way near a line. Asked on 100 varied pairs of dates whether a request was opened "more than 14 days before today", Jev was right on 94, and on 15 of the 20 opened exactly 14 days before. Reworded as "at least 15 days before today", it was right on 97 (experiment 0040). When the code sent the number of days, it was right on 100 of 100 distinct cases (experiment 0041).
• A second call that was not needed. The documentation: "Two requests are the exception, not the rule." If the second question could have been asked against the first state, ask both at once.
• One threshold for actions of different risk. The documentation: "A confidence threshold is not one number." In our experiments the gate for a form reply was confidence 0.6 (0017), and the line on the danger statement that missed nothing was 0.41 (0010). A missed danger costs far more than a slow reply, so the two lines differ.

The price of a needless second call, from experiment 0007:

two calls, one after the other     222 + 222   =  444 ms      (p50, 1 question each, experiment 0007)
one call with three questions      233 ms
wasted per request                 444 - 233   =  211 ms

python3 experiments/0017-pipeline/main.py

severity scaled     2.4 / 3   =  0.8
priority            0.6 x 0.8 + 0.25 x 0.20 + 0.15 x 0.60   =  0.480 + 0.050 + 0.090   =  0.620

second step     0.20 x $0.757   =  $0.151
the pair        0.029 + 0.151   =  $0.180 per 1,000