Build the desk

In this chapter, we will learn to build the whole maintenance desk as one short program and to read it line by line. The desk is 64 lines of Python. One of those lines calls the model. We will run it on eight requests, two of which we wrote to trip it, and we will finish with a list of its flaws.

The whole desk on one page

A desk is a short program in which ordinary code does five jobs and a decision model does one. Our file is experiments/0019-build-the-desk/main.py. It is the desk of experiment 0017 from Chapter 11 with two additions: a fact step before the call and an audit record after it.

tenant message
     |
     v
1  facts    code    flat number, open requests for that flat       before the call
2  stamp    MODEL   trade, severity, may_enter                     one request, three questions
3  level    code    round the severity score to a whole level
4  gate     code    is there any reason to hand this to Rosa?
5  write    code    fill a form letter, or a note to Rosa
6  audit    code    one line for the day book

lines in the file                               103
the desk (lines 1 to 64)                         64
the runner that prints                          103 - 64             =  39

of the 64
opening note 4, imports 5, blank lines 6        4 + 5 + 6            =  15
the stamp function                              3      (line 30 is the call)
back office: rule book 10, facts 5,
             gate 8, write 9, desk 14           10 + 5 + 8 + 9 + 14  =  46
check                                           15 + 3 + 46          =  64

Line 30 is the only line that calls the model. Forty-six lines of back office stand around it.

The file starts with its imports. jev is the function in experiments/lib.py that sends one request to the decision model. QUESTIONS holds the three questions of the desk: trade, severity and may_enter. They live in experiments/desk.py, which the experiments share, so the standard questions keep one wording.

import json, re, sys, time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from lib import jev, load
from desk import QUESTIONS

The rule book and the facts

The rule book is the part of the desk that is plain data: promises, phrases and thresholds that a person can read and change without touching the model.

# ---- the rule book: all of it is ordinary data that Rosa can read and change ----
PROMISE = {3: "within 1 hour", 2: "within 4 hours", 1: "within 3 working days", 0: "at the next monthly inspection"}
ACCESS = {True: "We will let ourselves in with the master key, as you agreed.", False: "We will call you first to agree a time."}
SAFETY = {"electrical": "Do not touch the switch or socket.", "plumbing": "If you can reach the stop tap, turn it off.",
          "heating": "If you smell gas, open the windows and leave the flat.", "appliance": "Switch the appliance off at the wall.",
          "pests": "Keep food covered.", "locks": "Stay where you can be reached by phone."}
MIN_CONF = 0.6                      # trade or severity confidence below this: a person reads it
NO, YES = 0.2, 0.8                  # a permission dial strictly between these: a person reads it
FLOORS, LETTERS = 30, "ABCD"        # our Harbor Court: 30 floors x 4 flats per floor = 120 flats
OPEN = {"14B": ["pests"]}           # open requests by flat. A database table in real life.

The first three tables are copied from experiment 0017. PROMISE turns a severity level into a response time. ACCESS holds the two sentences about entering the flat. SAFETY holds one sentence per trade, used only at level 3.

The next two lines are the thresholds. MIN_CONF is 0.6, the value experiment 0017 fixed before looking at its test requests. NO and YES are 0.2 and 0.8, the example band on the Noul page of TypeSafe's documentation.

The last two lines are our own. We gave Harbor Court 30 floors with four flats on each, so 12B is a flat and 12E and 31A are not. OPEN is the list of open requests, with one invented entry.

floors x flats per floor     30 x 4  =  120 flats

A fact is something code can know for certain without asking a model. The fact step runs before the call.

def facts(text):
    """BEFORE the call: what code can know for certain. No model is asked."""
    m = re.search(r"\bflat\s+(\d{1,2})\s?([" + LETTERS + r"])\b", text, re.I)
    flat = f"{int(m[1])}{m[2].upper()}" if m and 1 <= int(m[1]) <= FLOORS else None
    return {"flat": flat, "open": OPEN.get(flat, [])}

The line that starts with m = holds a regular expression, a small pattern language for text. It looks for the word "flat", then a number of one or two digits, then a letter from A to D. If the number is a real floor, we have a flat, and we look up its open requests.

We do not send these facts to the model, because none of the three questions needs them. The gate and the reply use them. This is the rule of Chapter 10: what code can work out, code works out.

One line for the model

The stamp step is the single line that sends the message and the three questions to the decision model and gets three typed answers back.

def stamp(text):
    """The ONLY line in this file that calls a model: one request, three questions."""
    return jev("Tenant message:\n" + text, QUESTIONS)

The first argument is the state, the paper on the desk: the words "Tenant message:" and the message itself. The second is the three questions. The model answers them independently in one request, as we saw in Chapter 9. Back come a trade and a severity score, each with probabilities and a confidence, and a number between 0 and 1 for permission.

The model never sees the promise table, the thresholds or the phrases of the reply. It cannot quote a response time, because it was never told one.

Every other line would run with the network cable pulled out. The first call of this book, in Chapter 2, asked these three questions about one short message: 625 input tokens, 311 milliseconds, $0.000026. Over the 215 requests of experiment 0002 the call took 251 milliseconds at the median, used 694 input tokens on average and cost $0.000029. The timings are from one laptop on one day, through OpenRouter.

The gate, the letter and the day book

The gate is the code that decides, from the facts and the confidences, whether the desk may answer by itself.

def gate(f, a):
    """AFTER the call: every reason to hand this request to Rosa. An empty list means the desk may answer."""
    why = []
    if f["flat"] is None: why.append("no valid flat number found")
    if a["trade"]["confidence"] < MIN_CONF: why.append("unsure of the trade")
    if a["severity"]["confidence"] < MIN_CONF: why.append("unsure of the severity")
    if NO < a["may_enter"]["noul"] < YES: why.append("unsure about permission to enter")
    return why

The three checks on the answers are those of experiment 0017. A Choice and a Score come with a confidence, which we compare with MIN_CONF. A Noul has none, so we ask whether the dial stopped in the middle band. The check on the flat is new.

The gate returns a list of reasons, not a yes or a no. An empty list means "go ahead". Anything else opens the note to Rosa.

The reply is put together from fixed phrases, so the desk cannot promise anything that is not in the rule book.

def write(f, a, level, why):
    """Fill a form letter from fixed phrases, or a hand-off note. Nothing is generated."""
    trade, enter = a["trade"]["choice"], a["may_enter"]["noul"] >= YES
    if why:
        return f"TO ROSA ({'; '.join(why)}). The stamps suggest: {trade}, level {level}, {'may enter' if enter else 'call first'}."
    text = f"Thank you. A member of our {trade} team will attend {PROMISE[level]}. {ACCESS[enter]}"
    if level == 3: text += f" {SAFETY[trade]}"
    if trade in f["open"]: text += f" We have added this to your open {trade} request."
    return text

The model picked the trade and the level. Code picks the sentences. At level 3 it adds a safety sentence, and one more if the flat has an open request for the same trade.

def desk(text):
    """One request, start to finish. Returns everything, including the one-line audit record."""
    t0 = time.perf_counter(); f = facts(text); t1 = time.perf_counter()
    r = stamp(text); a = r["answers"]
    t2 = time.perf_counter()
    level = int(a["severity"]["score"] + 0.5)               # the nearest whole level
    why = gate(f, a)
    out = {"facts": f, "answers": a, "level": level, "route": "rosa" if why else "auto", "text": write(f, a, level, why)}
    out["audit"] = {"flat": f["flat"], "trade": a["trade"]["choice"], "trade_conf": a["trade"]["confidence"],
                    "score": a["severity"]["score"], "sev_conf": a["severity"]["confidence"],
                    "may_enter": a["may_enter"]["noul"], "route": out["route"], "why": why,
                    "model_ms": round(r["latency_s"] * 1000), "cached": r["cached"]}
    out["code_us"] = (t1 - t0 + time.perf_counter() - t2) * 1e6
    return out

Line 56 rounds the score to the nearest whole level, because the promise table has four rows and not a slider. A score of 1.73 becomes level 2.

An audit record is one line per request that holds what the desk knew, what the clerk stamped and what the desk did. When Rosa finds a wrong reply next month, this line tells her whether the clerk was sure and wrong, or the gate was too loose. It is also the data from which Chapter 12 chooses better thresholds.

Eight requests

We ran the desk on six requests from the test half of the dataset and on two that we wrote ourselves. The dataset is synthetic and cleaner than real mail: gpt-5.1 wrote the messages from labels we chose, and a second gpt-5.1 call labelled them blind.

OURS = [("two problems in one message",
         "Hello, flat 9C here. Two things. The kitchen tap has been dripping for a week, and since this morning the socket "
         "by the cooker crackles and gives a small spark when I plug the kettle in. I am out until six, please let yourselves in."),
        ("not a maintenance request",
         "Hi Rosa, flat 21A. Just to say thank you for sorting the boiler so quickly last week, the engineer was lovely. "
         "Also, do you know what time the bins are collected on bank holidays?")]
data = {r["id"]: r for r in load("requests")}
jobs = [(i, data[i]["text"], data[i]) for i in ("r071", "r310", "r253", "r118", "r067", "r117")]
jobs += [(name, text, None) for name, text in OURS]

The remaining lines of the file only print. Here is the first request.

[1] r071   dataset labels: electrical, level 3, may enter True
  message    hi this is flat 12b theres a buzzing noise and small sparks coming from the light switch in the hallway when we turn i...
  facts      flat 12B, open requests for this flat: none
  trade      electrical  confidence 1.00   top two: electrical 1.00, locks 0.00
  severity   score 3.00 -> level 3  confidence 1.00   p0..p3: 0.00 0.00 0.00 1.00
  may_enter  0.98
  route      auto
  text       Thank you. A member of our electrical team will attend within 1 hour. We will let ourselves in with the master key, as you agreed. Do not touch the switch or socket.
...
  audit      {"flat": "12B", "trade": "electrical", "trade_conf": 1, "score": 3, "sev_conf": 1, "may_enter": 0.98, "route": "auto", "why": [], "model_ms": 380, "cached": true}

The clerk is sure of all three stamps. The second request shows the open request at work.

[2] r310   dataset labels: pests, level 0, may enter False
...
  facts      flat 14B, open requests for this flat: ['pests']
...
  text       Thank you. A member of our pests team will attend at the next monthly inspection. We will call you first to agree a time. We have added this to your open pests request.

Now the two that went to Rosa.

[4] r118   dataset labels: heating, level 3, may enter False
  message    Hi, this is from flat 14B. The radiator in the bedroom is making loud banging noises and shaking the wall whenever the...
...
  severity   score 2.39 -> level 2  confidence 0.40   p0..p3: 0.00 0.30 0.00 0.70
...
  route      rosa
  text       TO ROSA (unsure of the severity). The stamps suggest: heating, level 2, call first.

[6] r117   dataset labels: heating, level 3, may enter False
  message    hi my flat is 3B and something is really wrong with the heating system in the bedroom radiator its making loud banging...
  facts      flat None, open requests for this flat: none
  trade      heating  confidence 0.87   top two: heating 0.89, electrical 0.11
  severity   score 3.00 -> level 3  confidence 1.00   p0..p3: 0.00 0.00 0.00 1.00
...
  route      rosa
  text       TO ROSA (no valid flat number found). The stamps suggest: heating, level 3, call first.

In request 4 the clerk put 0.30 on level 1 and 0.70 on level 3. The confidence is 0.40, below our line, and the rounded level would have been 2 when the label says 3. The gate caught a wrong stamp. In request 6 every stamp was sure and right. It went to Rosa only because the tenant wrote "my flat is 3B", and our pattern wants the number straight after the word "flat".

Request 5 is the one that got away.

[5] r067   dataset labels: electrical, level 2, may enter False
  message    Flat 3B here, and I am COMPLETELY FED UP with this building. Half my sockets went DEAD last night, including the one t...
...
  severity   score 2.69 -> level 3  confidence 0.69   p0..p3: 0.00 0.00 0.31 0.69
...
  route      auto
  text       Thank you. A member of our electrical team will attend within 1 hour. We will call you first to agree a time. Do not touch the switch or socket.

The label says level 2. The clerk said level 3 with confidence 0.69, above our line, so the desk promised "within 1 hour". This is the request that experiment 0017 counted as its one wrong reply among 94 automatic ones.

Last, the two messages we wrote.

[7] two problems in one message   written by us, no label
...
  trade      electrical  confidence 0.95   top two: electrical 0.96, plumbing 0.04
  severity   score 3.00 -> level 3  confidence 1.00   p0..p3: 0.00 0.00 0.00 1.00
...
  route      auto
  text       Thank you. A member of our electrical team will attend within 1 hour. We will let ourselves in with the master key, as you agreed. Do not touch the switch or socket.

[8] not a maintenance request   written by us, no label
...
  trade      heating  confidence 0.94   top two: heating 0.95, plumbing 0.02
  severity   score 0.07 -> level 0  confidence 0.93   p0..p3: 0.95 0.03 0.02 0.00
...
  route      auto
  text       Thank you. A member of our heating team will attend at the next monthly inspection. We will call you first to agree a time.

Request 7 holds two problems. The desk answered the sparking socket and said nothing about the dripping tap. The trade confidence was 0.95, so the gate saw nothing odd. Request 8 is a thank-you note with a question about bins. The desk promised a heating visit, with confidence 0.94 and 0.93.

summary: 8 requests, 6 answered automatically, 2 handed to Rosa.
  the six dataset requests: automatic and matching the labels 3, automatic and not matching 1, handed to Rosa with a wrong stamp 1.

Every answer here came from our call cache, the stored replies of earlier runs, so each model_ms is the time of the first call: 181 to 380 milliseconds. The code took 2 to 43 microseconds, timed by the perf_counter lines. Eight requests are a demonstration. The measurement is experiment 0017: 107 requests, 94 answered automatically, 93 of those right.

Honest flaws

A flaw is honest when we can point to the line of code or the line of output that shows it. Here are eleven.

1. Two problems in one message. Request 7 lost the dripping tap. A Choice returns one tray, and the 0.04 on plumbing was the only trace. A Noul such as "The message reports more than one separate problem" might catch it. We have not tested that.
2. A message that is not a request. Request 8 got a visit nobody asked for. The six trays have no "none of these". TypeSafe's documentation advises a seventh option, other. Experiment 0025 tried it: other took 38 of 40 off-topic messages, and 1 of 80 real requests by mistake. This file does not have it.
3. A wrong reply went out. Request 5 was sent at confidence 0.69. A line at 0.7 would have stopped it. A line at 0.74 would also stop request 3, which was right.
4. Thresholds chosen once. 0.6 was fixed before the test requests were looked at, and 0.2 and 0.8 come from a documentation example. In experiment 0026 a trade line at 0.6 caught 1 of 9 wrong trades. A line at 0.8 caught 6.
5. Synthetic data. A language model wrote the messages and checked the labels we chose. Real mail is messier. Of the 76 messages the two labelling passes disagreed on, 66 disagreed on severity, so the label that matters most is also the least certain.
6. One safety sentence per trade. SAFETY["heating"] is about gas. Request 6, read in full, reports a burning smell "like plastic or wiring" from a radiator. With a flat number it would have been told to open the windows if it smelled gas. In experiment 0030 the clerk picked the advice from four actions, and for five heating requests it picked three different ones. We did not grade those picks.
7. A brittle fact. The pattern missed the flat in 34 of 215 messages. Request 6 was a level 3 danger that waited for Rosa because of word order.
8. No fence around the tenant's text. In experiment 0012 a fake staff note typed inside a message forced level 3 on 100% of 60 mild requests. Chapter 14 has the defence. This file does not use it.
9. English only. In experiment 0013 severity was right on 96.7% of English messages and on 90.0% to 95.0% in five other languages.
10. No monitoring. The audit lines are printed and forgotten. Nothing warns us if the automatic share drops. In experiment 0032, repeating the same call moved the route of 5 requests in 100. Chapter 15 has the regression test this desk lacks.
11. No plan for failure, and no action. If the model call fails after the retries in lib.py, the program stops. A real desk would send the paper to Rosa. And nothing here books the van.

cd experiments
python3 0019-build-the-desk/main.py

eight fresh calls     8 x $0.000029  =  $0.000232     (mean cost per call from experiment 0002)

most likely level            3
spread                       0.30 x |1 - 3|                 =  0.60
even spread, four levels     (1.5 + 0.5 + 0.5 + 1.5) / 4    =  1.00
confidence                   1 - 0.60 / 1.00                =  0.40
gate                         0.40 < 0.60                    ->  hand to Rosa