Candidates and templates · Decision Models with Jev: Novice to Ninja

In this chapter, we will learn how a system can answer in full sentences when its model cannot write a word. We will read a small catalogue of things Rosa's desk may say, let the clerk point at the ones that fit, and let code assemble the reply. The catalogue was run on 107 requests, so we will also see which of its decisions went wrong, and who made them.

Select, do not generate

To select is to pick from a list that people wrote in advance, and to generate is to compose new text. A language model generates. A decision model can only select, so the design must turn "what shall we say?" into "which of these shall we say?"

This limit is also the point. A sentence that was written and approved in advance cannot contain a new promise.

A candidate and its parts

A candidate is one thing the system could say or do, written in advance by people. A list of candidates is a catalogue. In the catalogue we ran, each candidate has five parts.

An id: a short name for our code, such as send-trade.
An eligibility rule: a condition on the facts that code checks. If it is false, the candidate is not considered. Facts are the values the code knows or computes (Chapter 10).
A claim: one plain sentence saying what the candidate is for. The clerk judges it against the message.
One or more allowed actions, each with a description the clerk can match.
A template: the wording of the reply with slots, which are named gaps such as {promise} that code fills in.

Here is the catalogue of experiment 0030.

id	Eligible when (checked by code)	Claim	Allowed actions
`send-trade`	always	A tradesperson will attend.	`ATTEND_1H`, `ATTEND_4H`, `ATTEND_3D`, `NEXT_INSPECTION`
`safety-advice`	always	Tell the tenant what to do right now to stay safe.	`STOP_TAP`, `SWITCH_OFF`, `LEAVE_AND_VENTILATE`, `KEEP_CLEAR`
`enter-with-master-key`	always	Staff will let themselves in while the tenant is out.	`USE_MASTER_KEY`
`repeat-visit-apology`	`open_requests_from_flat >= 2`	Apologise that this flat has several requests open.	`APOLOGISE`
`out-of-hours-note`	`out_of_hours` is true	Explain that the office is closed and when it reopens.	`NOTE`

Two of the five, as they stand in experiments/0030-candidates/main.py. The dots mark the candidates we left out. In Python, lambda f: starts a small function of the facts f.

CATALOGUE = {
  "send-trade": {"eligible": lambda f: True, "claim": "A tradesperson will attend.",
      "actions": {"ATTEND_1H": "Danger to people or to the building: fire, gas, flooding, electric shock.",
                  "ATTEND_4H": "An essential service is lost: no water, no heat, no power, no working toilet or no secure door.",
                  "ATTEND_3D": "Inconvenient. Something works badly but the flat is usable.",
                  "NEXT_INSPECTION": "Cosmetic. Nothing stops working."},
      "template": "A member of our {trade} team will attend {promise}."},
  ...
  "repeat-visit-apology": {"eligible": lambda f: f["open_requests_from_flat"] >= 2, "claim": "Apologise that this flat has several requests open.",
      "actions": {"APOLOGISE": "Acknowledge the earlier open requests."},
      "template": "We are sorry that you have {open_requests_from_flat} other requests still open; we will review them on the same visit."},
  ...
}

The four actions of send-trade carry the four descriptions of the severity scale, and each stands for one response time: within 1 hour, within 4 hours, within 3 working days, at the next monthly inspection. Here an action only selects words. In a working desk it would also book the van.

What the app sends

The request carries the message, the facts and the candidates as the state, and a few small questions about each candidate. Our dataset holds only messages, so the experiment makes up two facts for each request with a random generator: how many requests the flat already has open, and whether the message arrived outside office hours.

Code works first. It checks the five eligibility rules, then sets aside what it can decide without a reader. The apology depends on one fact alone, so it never goes to the clerk. The out-of-hours note waits for a rule that we will meet below. Three candidates remain for the paper on the desk.

state = {"tenant_message": r["text"], "facts": f,
         "candidates": {cid: {"claim": c["claim"], "allowed_actions": c["actions"]} for cid, c in ask.items()}}
q = {"trade": {"type": "choice", "instructions": "Which trade should handle the problem in `tenant_message`?", "criteria": {t: None for t in TRADES}}}
for cid, c in ask.items():
    q[cid + "__relevant"] = {"type": "noul", "instructions": f"Candidate `{cid}` belongs in the reply to `tenant_message`.",
                             "criteria": {"true": "What the tenant describes makes this candidate's claim apply.", "false": "The claim does not apply to what the tenant describes."}}
    q[cid + "__useful"] = {"type": "score", "instructions": f"How useful is candidate `{cid}` to this tenant right now?",
                           "criteria": ["Not useful.", "Routine background.", "Addresses a concrete need in the message.", "Addresses a pressing need in the message."]}
    if len(c["actions"]) > 1:
        q[cid + "__action"] = {"type": "choice", "instructions": f"Which allowed action of candidate `{cid}` fits what the tenant describes?", "criteria": c["actions"]}

Here ask holds the three remaining candidates, and f the facts. The first question is the trade, a Choice with names only (None means no description). Then each candidate gets a Noul (does it belong in the reply?) and a Score (how useful is it?). A candidate with several allowed actions gets one more Choice.

trade                                                    1
relevant + useful, for 3 candidates       3 x 2     =    6
which action, for send-trade and safety-advice      =    2
one call                                  1 + 6 + 2 =    9 questions

The nine questions cost little: a p50 of 234 ms and a mean of 1,496 input tokens per call. The desk's usual three questions took a mean of 694 tokens in experiment 0002 and a p50 of 250 ms in experiment 0017.

tokens     1,496 / 694                      =  2.2 times the three-question call
cost       1,496 x 0.042 / 1,000,000        =  $0.000063, so $0.063 per 1,000 requests
latency    234 ms against 250 ms               no slower in these two runs

Code assembles the reply

Assembly is the step where code keeps the candidates the clerk judged to fit, adds the ones that code decides, and fills each template. The experiment fixed four rules before it ran.

A candidate on the paper is kept when its Noul is at least 0.5 and its Score is at least 1.0. Its action is the answer of its Choice.
The apology is added whenever its eligibility rule passed. No stamp is involved.
A reply always promises a visit. If send-trade was not kept, code adds it anyway.
The out-of-hours note is added when the fact says the office is closed and the chosen promise is not "within 1 hour". This rule joins a fact and a stamp.

catalogue: 5 candidates, written by people
      |   [CODE]   check eligibility on the facts, set aside what code decides
      v
3 candidates on the paper
      |   [MODEL]  one call, 9 questions
      v
9 typed answers
      |   [CODE]   apply the four rules, fill the slots, join the sentences
      v
the reply

The program printed three assembled replies, and these are the first two. The line in square brackets shows the labels and the generated facts of each request.

three assembled replies:
  [plumbing, level 2, may_enter True, facts {'open_requests_from_flat': 2, 'out_of_hours': False}]
    A member of our plumbing team will attend within 4 hours. We will let ourselves in with the master key, as you agreed. We are sorry that you have 2 other requests still open; we will review them on the same visit.
  [pests, level 2, may_enter True, facts {'open_requests_from_flat': 2, 'out_of_hours': True}]
    A member of our pests team will attend within 3 working days. We will let ourselves in with the master key, as you agreed. We are sorry that you have 2 other requests still open; we will review them on the same visit. Our office is closed now and reopens at 8 am.

Look at where each piece of the second reply came from. Every sentence is a template that people wrote. The words "pests" and "within 3 working days" were picked by the clerk. The "2" is a fact. The last sentence was added by rule 4. But the label says level 2, and level 2 should be promised "within 4 hours". The clerk picked the wrong action, and a fixed phrase was the wrong fixed phrase.

The measured result

Experiment 0030 ran this catalogue on the 107 test requests with no gate, so every reply was sent and every reply was graded. A gate is a rule that hands unsure cases to a person, and Chapter 12 builds one. A reply counts as right when it holds exactly the candidates that the labels and the facts call for, and the right promise.

test requests: 107   candidates in the catalogue: 5   questions per call: 9 to 9
right set of candidates in the reply          100 of 107  (93.5%)
right response-time promise                   101 of 107  (94.4%)
whole reply right (set and promise)           96 of 107  (89.7%)
latency p50 234 ms, p95 325 ms   input tokens 1496   cost per 1,000 requests $0.063

candidate                 decided by       should+did   missed   added wrongly
send-trade                model (action)       107        0           0
safety-advice             model                 26        1           5
enter-with-master-key     model                 48        1           0
repeat-visit-apology      code (a fact)         41        0           0
out-of-hours-note         code (fact + stamp)    28        0           0

wrong replies                107 - 96        =  11
  wrong set of candidates    107 - 100       =   7     safety advice 5 + 1, master key 1
  wrong promise              107 - 101       =   6
  wrong in both ways         7 + 6 - 11      =   2

Every error belongs to the model. The two candidates that code decided were right on all 107 requests. The safety advice was the weak one: added 5 times where the labels did not call for it, and missed once. The master key was never offered without permission, which would be the costly mistake, and was left out once.

The experiment has limits. The facts were generated, not taken from real records. The answer key is computed by rule from the labels and those facts, so the code-decided candidates could hardly fail: the key applies the same rule to the same facts. The key also shares any mistake in the labels. Advice "added wrongly" means added where the label is below level 3, which is not always bad advice. The thresholds 0.5 and 1.0 were fixed before the run.

Did the Score earn its place beside the Noul? Experiment 0039 took the same stored answers and graded safety advice and the master key again under other keep rules.

   rule                                         whole reply right  added wrongly  missed
   Noul >= 0.5 and Score >= 1.0 (as run)                 96 of 107              5       2
   Noul >= 0.5 alone                                     96 of 107              5       2
   Score >= 1.0 alone                                    91 of 107             12       0
   Score >= 2.0 alone                                    77 of 107              3      25
   Noul >= 0.8 alone                                     87 of 107              2      16

The Noul alone gives the same 96 of 107. The Score alone is worse at either line, and a stricter Noul line of 0.8 misses 16. In this catalogue the Score was dead weight for the keep decision: it added questions to every call and changed no reply. That is one catalogue on 107 synthetic requests.

Experiment 0017 ran a simpler design on the same 107 requests: the three stamps of Part I, a table of response times, two access lines and a gate. The four response times are four candidates and the Score selects one. Its replies were graded on less, the response time and the access line only, so the two rows below did not sit the same exam.

                               right, sent   wrong, sent   handed to Rosa
0017  three stamps, a gate          93             1              13
0030  catalogue, no gate            96            11               0
gpt-5.1 writes the reply (0017)     95            12               0

0017 automatic share     94 / 107   =  87.9%       right among those   93 / 94  =  98.9%
cost per 1,000           0030 $0.063     0017 $0.029     gpt-5.1 $0.953

The catalogue said more in each reply and sent 11 wrong ones, with no gate to stop any. The gated desk sent 1 wrong reply and gave Rosa 13 to read. A language model writing the replies sent 12 wrong ones with nothing to mark them. Each candidate that the model decides is one more place to be wrong, which argues for a gate on a richer catalogue. We did not run one.

The interactive steps one request through the simpler desk of experiment 0017. Before you start, predict: of the seven stages, in how many does the model do the work, and what share of the total time does that stage take?

What we give up

A system that selects can say only what is in its catalogue. We give up variety, since every tenant with a level 2 plumbing fault reads the same sentence. We give up warmth, since a template cannot respond to a tenant's own words. And we give up anything nobody thought of in advance, which is what the hand-over to Rosa is for.

Sometimes a real letter is needed. The published advice is to pair the two models. OpenRouter's guide to Jev says: "you can use Jev to make the decision, then use a chat model to explain it". The letter writer is called for the words only, and the decision stays with the stamps.

Try it yourself

Run the experiment. The calls are cached, so it is free.

cd experiments
python3 0030-candidates/main.py

Then write three candidates for a desk you know: the eligibility rule as a condition on facts only, the claim as one sentence, and the template with every number as a slot. Mark each candidate: decided by the model, by code alone, or by code from a fact plus a stamp.

Common questions

Why check eligibility in code and relevance with the model? Are they not the same?

Eligibility is about facts: does this flat have two open requests? Code answers that exactly. Relevance is about the message: does it describe something that calls for safety advice? Only a reader can answer that.

Can one answer depend on another, such as the action on the relevance?

Not inside one call. The documentation says questions in a call are answered independently. The experiment asks for the action of the safety advice every time, and code ignores the answer when the candidate is not kept.

How large can a catalogue grow?

A Choice takes up to 255 options according to the documentation, and every question costs tokens: three candidates on the paper made a call of 1,496 tokens. Eligibility rules keep the request small, because code removes candidates before the model sees any.

Is the reply right more often than a language model's?

Not clearly. On our 107 synthetic requests gpt-5.1 sent 95 right replies, the gated desk 93 and the catalogue 96 on a different key. The difference is in the wrong ones: the gated desk sent 1 and handed 13 to a person, and the other two sent 12 and 11 unmarked. gpt-5.1 also wrote the requests, which helps it.

Carry this

Select, do not generate: the model points at things that people wrote in advance.
A candidate has an id, an eligibility rule (code), a claim and allowed actions (judged by the model) and a template with slots (filled by code).
Split the catalogue three ways: decided by the model, by code alone, or by code from a fact plus a stamp. Give the model only the first kind.
Measured on 107 synthetic requests with no gate: five candidates, nine questions, p50 234 ms, $0.063 per 1,000; whole reply right on 89.7%. Code-decided candidates made 0 errors. Every error was the model's. The Noul alone kept the same candidates as the Noul and the Score together.
The simpler gated desk of experiment 0017 answered 87.9% automatically, 98.9% of those right.

Check yourself

1. A catalogue has 40 candidates. For one request 6 pass their eligibility rules. Code decides 2 of the 6 alone. Of the 4 that go to the clerk, 3 have several allowed actions. With one trade question, how many questions are in the call?

Answer

trade                                 1
relevant + useful     4 x 2     =     8
which action                    =     3
one call              1 + 8 + 3 =    12 questions

2. Using experiments 0030 and 0017, how many of every 1,000 requests would each desk send out with a wrong reply?

Answer

catalogue, no gate     11 / 107  =  10.3%     about 103 in 1,000
three stamps, a gate    1 / 107  =   0.9%     about   9 in 1,000, plus 121 handed to a person (13 / 107)

This scales results on 107 synthetic requests, graded on different keys. Real mail will differ.

3. The out-of-hours template reads "Our office is closed now and reopens at 8 am." What is wrong with it?

Answer

The opening time is typed into the wording. If the hours change, someone must find every template that repeats them. Make it a slot filled from a fact, as the apology does with {open_requests_from_flat}, so that the number lives in one place.