An interactive textbook
Design a Real-Time Voice AI Agent
Novice to Ninja
A system that listens, thinks, acts and talks back within a fraction of a second. The five components, the two ways to build it, the 800 millisecond budget, interruptions, tools, telephony, scale, the edge cases, and how to present it all in an interview.
Start readingOpen the latency budget
24 chapters · about 2.3 hours · 16 interactives · no background needed
How to read this book
- In order. Each chapter is five or six minutes and uses only what the earlier ones taught. The whole book is about two hours.
- With your hands. Sixteen chapters have an interactive. Guess first, then move the slider or press the button.
- With the numbers from the source. Every millisecond, kilobit and dollar in this book comes from the blog post it follows. Nothing was measured here, and the interactives say so.
- Recall before you read on. Every chapter ends with questions and folded answers, and the later chapters ask you to recall the earlier ones.
Part I · The problem
What a voice agent is, why real-time voice is hard, what we must build, and how big it is.
01What is a Voice AI Agent?Voice plus AI plus Agent, five real examples, and the one number that drives every design decision in this book: 800 milliseconds.5 min02Why real-time voice is hardSeven problems that a text chatbot never had: no Enter key, a tiny time budget, interruptions, messy audio, no spelling, the output must sound human, and everything is a stream.6 min03Requirements and the one metricWhat the agent must do, how well it must do it, and the single number we will measure everything against: Voice-to-Voice Latency.6 min04Back-of-the-envelope estimationOne million calls a day: how many are live at once, how much audio flows, how much we store, and why the GPU bill is the real bottleneck.5 min · interactive
Part II · The five components
Audio transport, voice activity and turn detection, speech-to-text, the LLM with tools, and text-to-speech.
05High-level architectureOne call from the caller's mouth to the caller's ear: the media gateway, the orchestrator that conducts everything, three GPU services, tools, storage, and the eight messages that tie client and server together.6 min · interactive06Audio transportHow sound becomes numbers, why we cut it into 20 ms chunks, what a codec does, the three ways to carry audio over a network, and the echo that makes an agent interrupt itself.6 min · interactive07Voice activity and turn detectionIs someone speaking, and have they finished? The small model that answers the first question, the five approaches to the second, and the hardest problem in voice agents.6 min · interactive08Speech-to-textBatch versus streaming, interim and final transcripts, why STT is not the biggest part of the budget, the four things that make it hard on a real call, and the confidence score that tells the agent when to ask again.6 min · interactive09The brain: LLM with toolsWhat goes into the LLM and what comes out, what a tool is, the loop that makes an agent, streaming tokens and time to first token, the sentence overlap trick, and why a voice agent's system prompt is different.6 min · interactive10Text-to-speechStreaming TTS and time to first byte, the five marks of a good voice, text normalization with one appointment sentence, SSML, and the path the audio takes back to the caller's ear.5 min · interactive
Part III · Two ways to build it
The cascaded pipeline, the speech-to-speech model, the hybrids, and the latency budget.
11Approach 1: the cascaded pipelineThree models in a chain with text in the middle, what the old IVR bots did instead, one turn traced millisecond by millisecond to 850 ms, and the seven advantages and six disadvantages.6 min · interactive12Approach 2: the speech-to-speech modelOne model, audio in and audio out, with no text in between: audio tokens, full-duplex conversation, learned turn detection, six advantages and ten disadvantages.6 min13Approach 3: hybrids, and the comparisonFive ways production systems mix the two approaches, the full sixteen-row comparison of cascaded versus speech-to-speech, and the three rules for when to use which.6 min · interactive14The latency budgetWhere every millisecond of the 800 goes, stage by stage, and the seven rules that keep the cascaded pipeline inside the budget.6 min · interactive
Part IV · The hard parts
Interruptions, tool calling, memory, and real phone calls.
15Handling interruptionsBarge-in: the five steps when the user talks over the agent, the buffer flush that most people forget, false interruptions, and who yields when both start at once.6 min · interactive16Tool calling in a voice agentSix problems that tools bring when the conversation is spoken: slow tools, failing tools, risky actions, exact values, speech during a call, and long results.5 min17Memory and contextFour kinds of memory a voice agent needs: the history within a call and how to summarize it, the profile across calls, the knowledge base through RAG, and the session state checkpointed to Redis.6 min · interactive18TelephonyConnecting the agent to a real phone number: PSTN and SIP, inbound and outbound calls, key presses, cold and warm transfers, 8 kHz audio, and the recording announcement.5 min
Part V · Production
Scale, edge cases, observability, safety, cost, and the 45-minute interview.
19Scaling the systemServing 10,000 concurrent calls: sticky sessions, an orchestrator fleet, GPU pools that scale on their own, one full stack per region, failover that survives a crash, and what to do when the load is too much.6 min · interactive20Edge casesThe 26 things that break a voice agent in production, from a silent caller to a spoken card number, each with the fix the earlier chapters already gave us.7 min · interactive21Observability and evaluationSeven timestamps per turn, ten metrics, a test set with the names removed, a simulated caller who is annoyed, and a human who listens to a sample every day.6 min22Safety, security and privacyVoice is personal data and voices can be cloned: nine rules, from encryption and consent to who really has the authority to issue a refund.6 min23CostFive line items that add up to four to eight cents a minute, why speech-to-speech costs several times that, why a human costs many times that, and six ways to spend less.5 min · interactive24Presenting the designThe 45-minute interview plan, slot by slot, what the interviewer is really listening for, the five questions people ask most, and the whole book in one list.6 min · interactive