Decisions in a voice agent · Decision Models with Jev: Novice to Ninja

In this chapter, we will learn where a decision model fits in an agent that talks on the phone. Rosa gives the maintenance desk a phone line. We will draw the loop such an agent runs, place nine split-second decisions in it, test seven of them on hand-written text, and work out a time budget for one turn.

The loop of a voice agent

A voice agent is a program that holds a spoken conversation by repeating four steps: listen, decide, think and speak. Each pass through the steps is one turn.

Two of the steps are machines we have not met. Speech recognition turns sound into written words. Speech synthesis turns written words into sound. Between them sit the clerk and the letter writer.

LISTEN   caller speaks  -->  speech recognition  -->  the words so far
                                                        |
DECIDE   the clerk stamps:  finished?  what do they want?  hand to a person?
                                                        |
THINK    the letter writer composes a sentence, only when one is needed
                                                        |
SPEAK    speech synthesis  -->  the agent talks, and keeps listening
                                                        |
         caller makes a sound  -->  DECIDE again:  stop talking, or carry on?

On paper, Rosa's desk ran the decide step once per message. On the phone it runs many times per call, and the caller is waiting each time.

Nine split-second decisions

Three words in the list need explaining: turn detection, barge-in and backchannel. Turn detection is deciding whether the caller has finished speaking or has only paused. A barge-in is the caller speaking while the agent is still talking, in order to interrupt. A backchannel is a small sound such as "mm-hm" that means "I am listening, go on". The agent must stop for a barge-in and keep talking through a backchannel.

The list of nine below comes from a post by Amit Shekhar. It is a list of ideas, not of results. Our copy of the post cuts off the last item after "Determine the next state or", so the end of row 9 is our reading. The stamp and the desk's version in each row are our own design. A tool, in row 7, is a function the agent's code can run, such as booking a visit.

#	Decision	Stamp	The desk's version	Tested by us
1	Turn detection	Noul	"The caller has finished what they wanted to say in `caller_so_far`."	0016: 40 of 40
2	Barge-in detection	Noul	"The agent should stop talking because of what the caller said in `caller_said`."	0016: 40 of 40
3	Intent classification	Choice	Report a problem, ask about status, change a booking, complain, or other.	0034: 23 of 24
4	Agent routing	Choice	The trade question of Chapter 3.	Not tested on calls
5	Human escalation	Noul	"A human should take over this call now."	0034: 23 of 24
6	Confirmation detection	Noul	"The assistant must ask the caller to confirm before acting on `caller_said`."	0034: 24 of 24
7	Tool selection	Choice	Book a visit, look up a request, cancel a visit, or no tool.	0034: 21 of 24
8	Response verification	Noul	"`agent_draft_reply` is safe and appropriate to speak to the caller."	0034: 24 of 24
9	Conversation state	Choice	The next step: ask for the flat number, ask about access, read back the booking, or end the call.	Not tested

We tested seven of the nine, all on text that we wrote ourselves. Row 4 reuses the trade question, which experiment 0002 measured on written messages and not on calls. Row 9 is a design.

What we measured

Two experiments tested seven of the decisions on short pieces of text that we wrote by hand, with no audio. Experiment 0016 took turn detection and barge-in, on 80 snippets. For turn detection: 20 finished remarks such as "It's flat 12C." and 20 unfinished ones such as "My radiator is leaking and". For barge-in: the agent is in the middle of a sentence about booking a plumber, and the caller says one of 20 interruptions ("wait, tomorrow doesn't work") or one of 20 backchannels ("mm-hm"). Each snippet went to Jev alone and uncached, which means no stored reply was reused, so the times are real round trips from a laptop.

TURN DETECTION: has the caller finished?
  correct at 0.5: 40 of 40     latency p50 231 ms, p95 277 ms, max 347 ms (uncached)
  closest calls: "It started yesterday." 0.82; "Around six in the evening." 0.86; "No." 0.87; "It's flat 12C." 0.89

BARGE-IN: should the agent stop talking?
  correct at 0.5: 40 of 40     latency p50 239 ms, p95 358 ms, max 393 ms (uncached)
  closest calls: "is that the earliest" 0.80; "I'm not home tomorrow" 0.82; "actually it's the kitchen" 0.85; "fine" 0.15

gpt-5-nano on turn detection: correct 39 of 40   latency p50 2165 ms, p95 3132 ms
gemini-2.5-flash-lite on turn detection: correct 40 of 40   latency p50 499 ms, p95 1131 ms

The p50 is the typical call: half were faster. The p95 is a slow call: 19 in 20 were faster. Jev was right on all 80, and its least sure answer was 0.80, still far from the line at 0.5. The same turn-detection task went to two language models, gpt-5-nano on Azure OpenAI and gemini-2.5-flash-lite through OpenRouter, which had to write a JSON reply.

gpt-5-nano               2,165 / 231   =  9.4 times slower than Jev, and 39 of 40
gemini-2.5-flash-lite      499 / 231   =  2.2 times slower than Jev, and 40 of 40

Experiment 0034 took five more decisions and put all five questions into one call. It used 24 moments from invented calls. Each moment holds what the caller just said and the reply the agent is about to speak. We wrote the moments, and we wrote the answer key.

24 hand-written call moments, five questions in one call

  intent classification (Choice, 5 options)    23 of 24
  human escalation (Noul)                      23 of 24
  confirmation needed (Noul)                   24 of 24
  tool selection (Choice, 4 options)           21 of 24
  response verification (Noul)                 24 of 24

one call for all five: p50 216 ms, p95 287 ms, max 1142 ms; input tokens 758; $0.032 per 1,000 moments

every miss:
  tool      key cancel_visit    got none            (0.56)  "I need to cancel tomorrow's visit, I won't be home."
  escalate  key False           got True            (0.79)  "There are sparks coming out of the socket behind the TV."
  tool      key book_visit      got none            (0.87)  "Mice again, in the cupboard under the sink."
  tool      key cancel_visit    got none            (0.77)  "Please cancel everything, I'm moving out on Friday."
  intent    key change_booking  got other           (0.77)  "Give the spare key for my flat to my brother when he comes."

Five misses in 120 answers, and some of them are ours.

Tool selection, 21 of 24. In two of the three misses the caller asks to cancel, and the agent's draft asks the caller to confirm ("Shall I go ahead?" in one, "Do you want me to do that?" in the other). The clerk chose "none", which we had described as "No tool: answer, confirm or hand over." Our key says "cancel_visit". With a confirmation still pending, "none" is arguably the better answer, and the fault is in our key. The third miss is real: the mice needed a visit, and the clerk chose no tool.
Escalation, 23 of 24. The clerk wanted a person for the sparking socket, at 0.79. Our key said the agent could handle it by sending an electrician. Our own criterion said "Someone may be in danger", so this false alarm is defensible, and it is the cheap kind of error.
Intent, 23 of 24. A key for a brother went to "other", where our key had stretched "change a booking" to cover it.
Verification, 24 of 24. Four drafts were unsafe: a guaranteed arrival in ten minutes, a promise of money, blame, and a neighbour's phone number. The clerk gave each a probability of being safe between 0.03 and 0.08.

tool selection as keyed                        21 of 24   =  87.5%
if the two pending confirmations count as right  23 of 24   =  95.8%
one moment                                     1 of 24    =  4.2 points

A latency budget for one turn

A latency budget is a list of the steps between two events, with the time each may take, and a total that must stay under a target. Latency is the wait between asking and getting. Here the two events are the caller's last word and the agent's first sound.

Only one line of our budget is measured: the decision step. The other three, and the target of 800 ms, are illustrative numbers.

speech recognition, last words        100 ms    illustrative
decision step (turn detection)        231 ms    measured p50, experiment 0016
letter writer, first words            300 ms    illustrative
speech synthesis, first sound         120 ms    illustrative
whole turn                            100 + 231 + 300 + 120   =  751 ms
slack under a target of 800 ms        800 - 751               =   49 ms

on a slow call (Jev's p95, 277 ms)    100 + 277 + 300 + 120   =  797 ms     3 ms of slack
five questions in one call (216)      100 + 216 + 300 + 120   =  736 ms     64 ms of slack   (0034)
the same on a slow call (287)         100 + 287 + 300 + 120   =  807 ms     7 ms over
gemini-2.5-flash-lite decides (499)   100 + 499 + 300 + 120   =  1,019 ms   219 ms over
gpt-5-nano decides (2,165)            100 + 2,165 + 300 + 120 =  2,685 ms   1,885 ms over

With these numbers the typical turn fits when the clerk decides, and does not fit with either of the two language models we timed. The slow-call lines matter most: a budget that holds for the typical call and not for the slow one fails at least one turn in twenty, and with five questions the slow call is already 7 ms over. Before you move a slider, predict: if speech recognition took 200 ms and not 100, which step would you shorten to get back under 800? The buttons switch between the clerk with one question, the clerk with five, and the two language models we timed.

Several decisions in one call

Decisions that are needed at the same moment and read the same words belong in one request. Chapter 9 found that extra questions cost tokens and almost no time. The two voice experiments agree.

one question, turn detection (0016)    p50 231 ms
five questions in one call (0034)      p50 216 ms     758 tokens     $0.032 per 1,000 moments
cost of that call                      758 x 0.042 / 1,000,000   =  $0.0000318
five calls, one after another          5 x 231   =  1,155 ms     (derived; we did not run this)

At the median, five answers came back no slower than one. We read the 15 ms between the two rows as noise: the desk's own three-question call had a p50 of 233, 251 and 278 ms in three of our experiments. The nine decisions fall at three moments.

At every pause. Turn detection, and with it intent, routing, escalation, confirmation, tool and next state as speculative questions. If the caller has not finished, the code throws those six answers away. If the caller has finished, they are already there, and no second round trip is needed.
While the agent speaks. Barge-in, alone, on every sound the caller makes.
After the letter writer has drafted. Response verification. It depends on the draft, so it is a second request. This is the dependent decision of Chapter 9. In experiment 0034 the draft was already written into each moment, which is why verification could share the call there.

Thresholds, and who keeps what

Each of the nine decisions gets its own line, set by what a wrong yes and a wrong no cost. This is Chapter 12 applied to a phone call.

Decision	A wrong yes costs	A wrong no costs	So the line
Barge-in	The agent stops for an "mm-hm" and picks up again. Cheap.	The agent talks over an objection.	sits low: stop easily
Turn detection	The agent cuts the caller off.	A silence, then the agent answers late.	sits near the middle; wait a little when unsure
Confirmation needed	The agent asks once more. Cheap.	A visit is cancelled, or a key handed over, that nobody confirmed.	sits low: ask easily
Human escalation	A person spends a few minutes.	A danger is handled by a machine.	sits low, as in Chapter 12

Our snippets and moments cannot place these lines. On barge-in, any line between 0.16 and 0.79 gives 40 of 40, which tells us the test was too easy to choose one. Lines come from labelled calls of your own.

The split of work is the one the whole book argues for. The clerk stamps: finished, interrupting, which trade, which tool, which step. The letter writer chooses the words to say, and only where a fixed phrase will not do. Code keeps the state machine, the fixed list of steps a call can be in and the moves allowed between them, and it owns the tools. The clerk may pick "book a visit". Only code books it, after checking that the step allows it.

Try it yourself

Run both experiments. They make 184 uncached calls between them and cost about a cent.

python3 experiments/0016-voice/main.py
python3 experiments/0034-voice-seven/main.py

Then make it harder. Add five remarks to FINISHED that end on a trailing word, such as "I think that's it, so", and five to BACKCHANNEL that are not quite one, such as "okay but".

Common questions

Do real voice systems decide turn-taking from the words alone?

We tested words alone. A real system also has the audio, and our test says nothing about how much that adds. Treat the Noul as one signal that your code can combine with a pause timer.

Is 800 ms the right target?

It is our illustrative number. Pick yours by listening: build the loop, add a delay you control, and find where callers talk over the agent.

Would Jev be faster inside a data centre?

We do not know. Our 231 ms was measured from a laptop over the public internet, through OpenRouter. TypeSafe says "Most queries complete in about 100 ms". Measure from where your agent will run.

Does OpenAI's Decisions API cover this too?

OpenAI's guide describes a voice pattern in which its speech model keeps talking and listening while the application calls Decisions to pick an action from a fixed list. That is row 7 of our table. We did not run it, as Chapter 17 explains.

Carry this

A voice agent loops: listen, decide, think, speak. The clerk owns "decide".
Measured on 80 hand-written text snippets: turn detection 40 of 40, barge-in 40 of 40, p50 231 and 239 ms. No audio, no noise.
Five more decisions in one call on 24 hand-written moments: 23, 23, 24, 21 and 24 of 24, at a p50 of 216 ms. Two of the three tool misses are arguably our key.
Budget: 100 + 231 + 300 + 120 = 751 ms. Only the decision step is measured. Budget for the p95, not the p50.
One line per decision. Stopping for an "mm-hm" is cheap; missing a danger is not.

Check yourself

1. Your steps take 150 ms, 231 ms, 400 ms and 100 ms. The target is 900 ms. What is the slack on a typical turn, and on a slow turn where the decision takes 358 ms?

Answer

typical     150 + 231 + 400 + 100   =  881 ms       900 - 881   =  19 ms of slack
slow        150 + 358 + 400 + 100   =  1,008 ms     1,008 - 900 =  108 ms over

The budget holds on the typical turn and breaks on the slow one.

2. In experiment 0034 the caller says "I need to cancel tomorrow's visit", the agent's draft asks "Shall I go ahead?", and the clerk picks the tool "none" where our key said "cancel_visit". Who is right, and what should the code do either way?

Answer

Arguably the clerk: nothing should be cancelled before the caller confirms, and "none" was described as covering "confirm". The lesson is about the key and about the code. Write the key for the moment the question is asked. And let code own the tool: it cancels only when the step of the call allows it, whatever the clerk picked.

3. Why is response verification a second request, when the other decisions share one?

Answer

It judges the letter writer's draft, and the draft does not exist until the first decisions have been made and the letter writer has run. Our code cannot build the request earlier. The documentation names that as the case where a second request is needed.