An interactive textbook
Decision Models with Jev
Novice to Ninja
Choice, Noul and Score from the first call to a production design: how a decision model stamps typed answers with probabilities, and how your own code does everything else. Every claim is measured on one running example, a building's maintenance desk.
Start readingOpen the decision lab
24 chapters · about 5.1 hours · 38 interactives · no background needed
How to read this book
- In order. Each chapter uses only what the earlier ones taught. Part I needs no background at all.
- With your hands. Before you move a slider, guess what will happen. The guess is where the learning is.
- With suspicion. Every number is marked as measured by us, published by a vendor, or derived. The measured ones come with the program that printed them.
- With your own problem in mind. The maintenance desk is a template. By Part II you should be sketching your own three stamps.
Part I · Novice
What a decision model is, the first call, and the three stamps: Choice, Noul and Score.
01The clerk with three stampsSoftware does not want a letter. It wants an answer it can act on. Meet the clerk who gives exactly three kinds of answer, and the office that does everything else.11 min · interactive02Your first callOne tenant message, three questions, one answer. We read the request and the response of a real call field by field, run it with curl, and measure where its 625 tokens went.12 min · interactive03Choice: the tray stampOne option from a list you wrote, a probability for every option, and one number for how sure. We work the confidence formula by hand, look hard at the trades the clerk got wrong, and measure what happens to a paper that fits no tray.12 min · interactive04Noul: the yes/no dialThe second stamp answers yes or no with a number from 0 to 1. We learn why a number beats a plain yes, how to write the statement, and how our own code turns the number into an if.11 min · interactive05Score: the rulerThe third stamp places a paper on a scale that we describe in words. We work a score and its confidence out by hand, and measure what happens when the levels are described badly.11 min · interactive06How sure is sureProbability, confidence and calibration are three different things. We define each, then check the clerk's numbers against our own labels, and find out why one band looked badly wrong.12 min · interactive
Part II · Apprentice
The state, good questions, many questions in one call, and the back office: doing the work in code.
07The paper on the desk: stateThe state is everything the clerk may read. We shape it, count it in tokens, find its limit, and measure what 150 unneeded messages do to speed, accuracy and the bill.11 min · interactive08Writing questions that workOne judgment per question, written exactly, with levels that describe situations. We change one thing at a time on 108 requests and see which changes matter: one of them takes severity from 88.0% to 29.6%.11 min · interactive09Many questions, one callA hundred questions in one call took 305 ms where one took 222 ms. We measure that, check that the questions do not influence each other, and learn when a second call is truly needed.11 min · interactive10The back office: do the work in codeThe clerk stamps and the office works. One rule decides who does what, two experiments show why, and one sentence about Choice, Noul and Score is taken apart word by word.12 min · interactive11Candidates and templatesWrite down in advance everything the desk may say or do, let the clerk point, and let code fill in the words. We run a catalogue of five candidates on 107 requests and read exactly where it went wrong.13 min · interactive12Thresholds and routingA probability does nothing until code draws a line on it. We price the two kinds of mistake, choose the line on one half of the data, judge it on the other, send the unsure cases to Rosa, and test whether a bigger model would read them better.13 min · interactive
Part III · Journeyman
Where it holds and where it breaks, untrusted text, repeatability, speed, cost and scale.
13Where it holds, where it breaksThe vendor lists nine weak spots of its own model. We put a measurement next to eight of them, show which five our tests reproduced, one of them only mildly, and say plainly why the three clean results do not clear the model.14 min · interactive14Untrusted text and the trust boundaryA tenant types a fake staff note into a harmless message, and every engine we tried sends the van. We measure three attacks and two defences, and find that who wrote a sentence is a fact only your code can know.12 min · interactive15The same answer twiceWe send five requests thirty times each, then a hundred requests ten times each. No trade changes in those calls, but the severity level changes on 4 requests in 100 and the route on 5, and the answers that wobble are the unsure ones. Then we pin a version and test questions the way we test code.14 min · interactive16Speed, cost and scaleHarbor Court's model bill comes to less than one cent a month. We do the latency and cost arithmetic for one building and for a hundred million requests, and say honestly when cost is the reason and when it is not.14 min · interactive
Part IV · Ninja
OpenAI's Decisions API, what might be inside, decisions in a voice agent, and a field guide to patterns.
17OpenAI's Decisions APIA second vendor now sells the same three stamps under different field names. We read its published guide, map it to Jev field by field, and write our code so the engine can be swapped. We could not call it, and we say so.12 min · interactive18What might be insideHow a language model turns text into odds, why a sentence is slow and a stamp is not, what we measured when we built the yes/no dial ourselves, and where knowledge about Jev's inside stops and guessing begins.13 min · interactive19Decisions in a voice agentRosa's desk gets a phone line. A voice agent has a fraction of a second to decide whether the caller has finished, whether to stop talking, and what to do next. We place nine such decisions, test seven of them on text we wrote ourselves, and budget the time.14 min · interactive20A field guide to patternsNine named shapes that decision systems keep taking, each with the need it answers, a sketch, the desk's version and the evidence: ours where we measured, the vendor's where we did not. Then four shapes to avoid.13 min · interactive
Part V · Practice
Build the whole desk, play in the lab, forty questions, and the whiteboard.
21Build the deskThe whole maintenance desk in 64 lines of Python: facts, one model call, a gate, form letters and an audit line. We run it on eight requests, two of them written to trip it, and list what it gets wrong.14 min22The decision labSeventeen of the book's interactives on one page, each with a small task, and one new simulator in which you set the gate for twenty recorded requests.20 min · interactive23Questions readers askForty short questions about decision models, each answered in a few sentences with the chapter to read and the number we measured.15 min24Whiteboard it coldThe whole book on one page: the three stamps, the request and the answer, the pocket numbers, the weak spots, and a thirty-minute script for explaining it at a whiteboard.12 min