The back office: do the work in code

In this chapter, we will learn how to divide a job between the clerk and the back office. We will state the division as one rule, give names to the two things that code produces (facts and signals), and measure what happens when the clerk is asked to do sums anyway. At the end we will explain, word by word, one sentence that sums up this design.

One rule for who does what

Code does anything that can be computed, looked up or counted, and the clerk does only what needs a judgment on text.

The rule comes with a test that we can apply to any task. Could a short program get it exactly right, every time, from data the application already holds? If yes, it is work for code. If the answer depends on what a person meant by their words, it is work for a stamp.

"Is the quote over 250?" passes the test: a program can add and compare. "Does the tenant sound frightened?" fails it.

Before reading on, predict. The list below has twelve tasks from Rosa's desk. How many of them do you think belong to code, and how many to the clerk? Write your two numbers down, then sort the tasks.

Facts and signals

A fact is a value that the code knows or computes, and a signal is a yes, a no or a category that the code derives from facts with a rule. These are the two things the back office writes on the paper before the clerk sees it.

A fact is a plain value: a total, a number of days, a count, a flat number. Here is one fact being computed. The quote is an invented example in the shape that our experiment uses.

labour                                  80
tap cartridge      2 x 62        =     124
waste trap         2 x 48        =      96
sealant            1 x 14        =      14
quote_total        80 + 124 + 96 + 14  =  314

A signal is one step further. The code applies a rule to a fact, and the result is a yes or a no.

rule      over_approval_limit  =  quote_total > approval_limit
today     314 > 250            =  true

Now the paper on the desk can carry both. This paper is invented, with the field names of experiment 0009.

{
  "message": "The kitchen tap has dripped for two weeks and the cupboard under it is wet.",
  "facts":   { "quote_total": 314, "days_open": 16, "requests_from_this_flat": 3 },
  "signals": { "over_approval_limit": true, "open_longer_than_14_days": true, "repeat_requester": true }
}

The message is the tenant's. Everything else was produced by ordinary code.

The same question, asked three ways

Experiment 0009 asks three desk questions that are arithmetic, each in three ways: with raw data, with the number computed by the app, and with the answer computed by the app. The app is short for the application: the program we write around the model, which is the back office. Each way was tried on 100 generated cases, with the true answer computed in code. In ways B and C, cases with the same number make identical requests. Experiment 0043 counted the different ones: in way B, 92 for the quotes and 86 for the counts.

The three questions are these. Is this quote over the 250 limit that needs the owner's approval? Has this request been open for more than 14 days? Is this at least the third request from this flat? All three are asked as a Noul (Chapter 4).

Here is what the program printed for the quotes.

quote over the 250 approval limit   (100 cases, 81 true, 19 false; always answering the majority scores 81%)
   how it was asked            correct   mean distance from 0 or 1
   A raw line items            92.0%        0.121
   B app sends the total      100.0%        0.010
   C app sends the answer     100.0%        0.132

Reading the table

Raw sums were wrong 8 times in 100. In the block about counting, the raw log was wrong 5 times in 100. Whenever the app did the arithmetic, every answer was right. The fair comparison for the 92% is the lazy answer: 81 of the 100 quotes were over the limit, so always saying "yes" scores 81%.

Experiment 0038 looked at the 8 wrong sums.

quotes within 60 of the line: 15, of which wrong: 8; quotes further away: 85, of which wrong: 0

All 8 were quotes under the limit that the clerk called over it, with true totals from 202 to 243. Only 2 of the 15 quotes near the line were over it (experiment 0043), so eight errors one way are not yet evidence of a lean towards "over". What we saw is this: 85 right of 85 far from the line, and 8 wrong of 15 within 60 of it. A rule about money matters exactly there.

A harder test for dates

The date block of experiment 0009 scored 100% in all three ways. It was easy: one date format, one "today" and gaps of 1 to 30 days, so its 100 cases are only 30 different requests (0043 again). Experiment 0041 is harder and has no repeats: 100 different pairs of dates, written out in full, with gaps of 10 to 21 days. Thirty pairs sit exactly on the 14-day line. The truth is computed in code.

(a) 'The request was opened more than 14 days before today.'  100 distinct pairs of dates, written out in full
    true gap in days:       10    12    13    14    15    16    18    21
    cases:                  10    10    10    30    10    10    10    10
    correct:                10    10    10    23    10    10    10    10
    total correct: 93 of 100
    the app sends days_open: 100 of 100

Now the documented weakness shows, in one place. Away from the line the clerk was right on 70 of 70. On the line it was right on 23 of 30. Given days_open as a number, it was right on 100 of 100. Rewording did not rescue the raw question: Chapter 8 tried four wordings and none was safe at the line. And (today - opened).days > 14 needs no clerk at all.

The documentation's own remedy for dates is this: "Extraction is a judgment, so give it to the model. Arithmetic is not, so keep it in code." To extract is to pull a value out of text. The clerk picks the month with one Choice of twelve options and the day with another of 31, and code compares. Part (b) of experiment 0041 tried the reading step on 90 different numeric dates, one for every day from 8 October 2026 to 5 January 2027. A date is right when the month and the day are both right.

(b) extracting day and month from 90 distinct numeric dates such as 08/10/26
                                         day is 1 to 12 (could be a month)   day is 13 to 31         all
    no hint about the format                                       7 of 34          56 of 56    63 of 90
    state says 'day/month/year'                                   34 of 34          56 of 56    90 of 90
    code: text.split(chr(47))                                     34 of 34          56 of 56    90 of 90

A day of 13 or more can be read only one way, and the clerk read all 56 correctly. A day of 12 or less could be a month, and there the clerk was right on 7 of 34. Nothing in the request said that Harbor Court writes the day first. That is a fact the application knows, not a judgment. The hint was one field on the paper, "This building writes dates as day/month/year.", and a pointer to it in each question. With it, all 34 were right. The third row is code that splits the text at the slashes, which needs no clerk when the format is known. The program also printed how the 27 wrong readings went: 6 swapped month and day, 5 took the month from the day position and kept the day, and 16 were something else, such as 08/10/26 read as August 26.

So a date the office already holds is subtracted in code. A date that exists only in a tenant's words is extracted by the clerk, with the format stated. The comparison is always code.

Accuracy is only one of three arguments

Even at 100% against 100%, as with the easy dates, two arguments remain. One is cost and speed. Code uses no tokens, and experiment 0039 timed the sums, subtractions and counts for 300 cases like ours: 95 microseconds in all, the best of 200 runs.

one model call (p50)                      251 ms  =  251,000 microseconds
all 300 computations in code               95 microseconds
ratio                 251,000 / 95     =  about 2,600

The other is auditability: being able to show afterwards how an answer was reached. A total computed in code can be re-run. A dial that stopped at 0.93 comes with no explanation.

What the documentation says

TypeSafe's guide to building with Jev is a workflow of eight steps, and the first is "Use code when you can". Under it the documentation says: "Keep deterministic work in code. It is reliable and cheap." Deterministic means that the same input always gives the same output.

The documentation also lists weak spots that code covers. On arithmetic: "Jev is not a calculator." On counting: the model "does not count reliably". On dates: it "reads dates as text, not as ordered quantities." Chapter 13 goes through the whole list.

The whole division, drawn

The model sits in one step of the pipeline, and code owns every step around it. A pipeline is a fixed sequence of steps that each request passes through.

the tenant's message  +  the desk's own records
        |
        v
[CODE]   compute facts and signals      totals, days, counts, lookups       microseconds, no tokens
        |
        v     the paper on the desk (state)  +  the questions
[MODEL]  stamp: Choice, Noul, Score     typed answers with probabilities    p50 251 ms, 694 tokens
        |
        v
[CODE]   apply thresholds and rules     act, ask Rosa, or do nothing
        |
        v
[CODE]   write from templates           every number comes from a fact
        |
        v
[CODE]   act                            book the van, send the reply, keep the record

The two measured numbers are from experiment 0002: one call with the desk's three questions. A threshold is a line drawn on a probability, and a template is a form letter with gaps. Chapter 12 and Chapter 11 build them.

The design in one sentence

The whole design fits in one sentence of engineer's shorthand: "provide Choice, Noul and Score inferences and do most of the work on the app compute side." The sentence is our own summary, not a vendor's. We take it apart from left to right.

• "provide". The model gives answers and our program receives them. The word is also a limit: the model does not act and does not write.
• "Choice, Noul and Score". The three stamps of Part I. The tray stamp answers "which one of these?". The yes/no dial answers "how likely is this statement to be true?". The ruler answers "where on this described scale?". There is no fourth kind of answer.
• "inferences". An inference is one answer that a trained model produces for one input. It is a judgment, not a calculation, and it comes with probabilities that say how sure the model was. The plural matters: one call returns many small answers (Chapter 9).
• "and do most of the work". Work means tasks here, not waiting time. In the drawing above the model has one box of five. That box took nearly all the time we measured, yet every other task belongs to code: computing facts, applying rules, drawing lines, filling in words, acting.
• "on the app compute side". The app is our own program. "Compute" is ordinary processing: arithmetic, look-ups and comparisons, with no model and no tokens. Every call has two sides, the model's service and our program. Ours is the back office.

In plain words: ask the model only for small typed judgments, and let your own program do everything that can be computed, looked up, decided by a rule or written from a form letter. The book calls those judgments stamps, and engineers call them inferences. This chapter measured the reason. When the clerk was given the office's work, it made mistakes. Its sums and date comparisons went wrong only near a line. The office made none.

Here is the sentence applied to Rosa's desk.

cd experiments
python3 0009-app-side-compute/main.py

21 September to 30 September     =   9 days
30 September to 7 October        =   7 days
days_open                9 + 7   =  16
open_longer_than_14_days 16 > 14 =  true