Part V · Production
Chapter 21
Observability and evaluation
Seven timestamps per turn, ten metrics, a test set with the names removed, a simulated caller who is annoyed, and a human who listens to a sample every day.
On this page
In this chapter, we will learn how to see what a voice agent is doing and how to tell whether a change made it better. We cannot improve what we cannot see, and for a voice agent, seeing means visibility at the level of a single turn. We will set up the per-turn trace, list the metrics worth tracking, and build the two kinds of test that run before a prompt or a model is changed.
The per-turn trace
A per-turn trace is a log of the timestamp of every stage in one turn, so that we can find exactly where the latency went in any slow turn. For every turn, we log seven moments.
user speech end --> turn detected --> STT final --> LLM first token --> LLM done --> TTS first byte --> audio played
Each gap between two neighbours is one stage of the budget. A slow turn is a slow gap, and the trace points at it. One rule matters more than the rest: voice-to-voice latency must be measured on the client, at the user's ear, because that is what the user actually feels. A server that reports a fast turn while the caller is still waiting is measuring the wrong thing.
The metrics to track
A metric is a number we track across many calls, so that a change in it tells us something changed. These are the ten that matter for a voice agent.
| Metric | What it tells us |
|---|---|
| Voice-to-voice latency at p50, p95, p99 | How the conversation feels, for the typical turn and for the slow ones. |
| Time to first token, time to first audio byte | Whether the LLM and the TTS are holding their share of the budget. |
| Turn detection errors: early cut-offs and late responses | Whether we interrupt people or keep them waiting. |
| Interruption count and false interruption count | Whether barge-in works, and whether coughs and "okay" stop the agent. |
| STT Word Error Rate on a sample of calls | Measured against human transcription, so it is a true error rate. |
| Task completion rate | Did the user get what they called for? |
| Human transfer rate | How often the agent could not finish the job. |
| Average call duration | Whether calls are getting longer or shorter. |
| Tool call failure rate | Whether the business systems behind the tools are healthy. |
| Cost per minute | Whether the design still makes business sense. |
Recordings and transcripts are stored too, with consent, for debugging, quality review and training data. The consent part is not optional, and Chapter 22 says what else has to be true of those recordings.
Offline evaluation
Offline evaluation means testing a new prompt or a new model against a fixed set of calls before any real caller meets it. There are two ways to build that test, and we use both.
The first is a test set. We collect real call transcripts and audio, remove the personal details, run the new version on them, and compare the result with the old version. Same inputs, two outputs, one diff.
The second is a simulated caller. This is an LLM that plays the role of a user with a specific goal, for example "You want to return a damaged item and you are annoyed", and talks to our agent. Because the caller is a program, we can run thousands of test conversations automatically, and we can give the caller any mood or goal we want to test.
Simulated caller: I want to return this thing, it arrived broken, and honestly I am fed up. Agent: I am sorry to hear that. Let me pull up the order, one moment.
Human review
Human review means sampling a small percentage of calls every day and having a person rate them. This catches things no metric catches, like a tone that sounds rude. The latency can be perfect and the task completed, and the caller can still have been spoken to badly. Only a person hears that.
Try it yourself
Take one slow turn you imagine from the order example in Chapter 11 and write down the seven timestamps for it, with a made-up 900 ms gap between LLM first token and LLM done. Which metric from the table would have moved first? Which chapter has the fix?
Common questions
Why measure latency at the ear and not at the server?
Because the user feels the whole path, including the network back to them and the client's jitter buffer. The server only sees its own part.
Why is Word Error Rate measured on a sample?
Because it is measured against human transcription, and a person has to transcribe each sampled call. That is expensive, so we do it on a sample rather than on every call.
If we have a simulated caller, why keep a test set of real calls?
A simulated caller is still a model, and it talks the way a model thinks a user talks. Real transcripts hold the mumbling, the odd phrasing and the surprises that no simulation invents.
Carry this
- Per-turn trace: seven timestamps, from user speech end to audio played. A slow turn is a slow gap.
- Measure voice-to-voice latency at the user's ear, at p50, p95 and p99.
- Ten metrics, from latency and turn detection errors to task completion and cost per minute.
- Before any change: a test set of real calls with personal details removed, and a simulated caller with a goal and a mood.
- A human rates a small daily sample. Rudeness has no metric.
Check yourself
1. Name the seven timestamps in a per-turn trace, in order.
Answer
User speech end, turn detected, STT final, LLM first token, LLM done, TTS first byte, audio played.
2. What is a simulated caller, and why do we want one?
Answer
An LLM playing a user with a specific goal, such as returning a damaged item while annoyed. It lets us run thousands of test conversations automatically before a change goes live.
3. Every metric is green. What can still be wrong, and how would we know?
Answer
The agent's tone can sound rude. Only human review of a daily sample of calls catches that.