Presenting the design · Design a Real-Time Voice AI Agent: Novice to Ninja

In this chapter, we will learn how to walk through this whole design in a 45-minute system design interview. The post lays out six time slots, and each slot is a part of this book. We will go through them in order, see what the interviewer is actually listening for, and answer the five questions people ask most often about voice agents. By the end you will have a plan you can run against a clock, and a one-page recap of everything we built.

The 45-minute plan

The plan is six slots, from clarifying requirements to edge cases, and the three 10-minute slots in the middle go to the two approaches, the hard parts, and scale.

Minutes	Slot	What to cover
First 5	Clarify requirements	Ask about the use case (support, booking, companion), the channel (phone, app, both), whether tools are needed, the languages, and the scale. Define the key metric: voice-to-voice latency under 800 ms.
Next 5	High-level architecture	Draw the media gateway, the orchestrator, and the three model services, with tools and storage. Explain that everything is streaming.
Next 10	The two approaches	Present the cascaded pipeline and the speech-to-speech model, with the pros and cons, and choose based on the requirements. Mention the hybrid.
Next 10	Deep dive on the hard parts	Turn detection (this is where you shine), the latency budget, interruption handling with buffer flushing and history truncation, and tool calling with fillers and confirmations.
Next 10	Scale and reliability	Stateful sessions, regional deployment, GPU pools, checkpointing, failover to backup providers, graceful degradation.
Last 5	Edge cases, observability, safety, cost	Pick five or six edge cases and give the solution for each. Mention per-turn tracing, simulated callers, PII redaction, and cost per minute.

Before you move the slider, guess which slot you would run over on. Then drag through the 45 minutes and read what the plan says you should be saying at that moment.

What the interviewer is listening for

The interviewer is looking for whether you understand that a voice agent is a real-time streaming system with a strict latency budget, not a chatbot with a microphone. If you show that, along with the trade-offs between the cascaded pipeline and the speech-to-speech model, and a good list of edge cases, you will do well.

the three things that carry the interview
  1  it is a real-time streaming system with a strict latency budget
  2  cascaded pipeline versus speech-to-speech, and when to use which
  3  a good list of edge cases, each with its fix

Try it yourself

Set a timer for 45 minutes and talk through the plan out loud to an empty room, with the interactive as your clock. Note the minute at which you reach turn detection. The plan puts it at minute 20. If you are past minute 25, you spent too long on the earlier slots; the plan gives the architecture five minutes, not ten.

Common questions

Can we run a full voice AI agent on the phone itself?

No. Good STT, LLM and TTS models are big and need GPUs, and a phone cannot run them at the quality we need. So we stream the audio to the server. The only thing we can run on the device is a small Voice Activity Detection model.

Should we use WebSocket or WebRTC to stream audio from a mobile user?

We should use WebRTC for mobile apps and browsers. WebSocket runs on TCP, which waits for lost packets and causes head-of-line blocking, so a late audio chunk becomes useless. WebRTC uses UDP and comes with a jitter buffer, packet loss concealment, and echo cancellation. WebSocket is still a good fit for server-to-server audio.

How much does a voice AI agent cost per minute?

A cascaded pipeline costs roughly $0.04 to $0.08 per minute, based on rough public prices, and actual numbers vary a lot by provider. A speech-to-speech model often costs $0.10 to $0.30 per minute because audio tokens are expensive. A human agent costs around $0.50 to $1.00 per minute in many markets.

Is a bigger LLM better for a voice agent?

Not always. Bigger models reason better but are slower, and a smart answer that arrives 3 seconds late is a bad answer on a phone call. A fast and small model is often the right choice. We can route simple turns to a small model and use a bigger model only when a tool call or complex reasoning is needed.

Can a voice agent verify a caller by their voice alone?

No. We should never trust the voice alone to verify identity, because voice can be cloned. Instead, we use an OTP, a DTMF PIN, or a verified phone number. Some inputs, like a PIN, are also safer typed with keys than spoken.

Carry this: the whole book

A voice agent listens, understands, thinks, acts and talks back, and must start speaking within about 800 ms. Voice-to-voice latency is the metric.
Five components: audio transport, voice activity and turn detection, speech-to-text, the LLM with tools, text-to-speech. Everything streams in 20 ms chunks.
Two ways to build it: the cascaded pipeline (control, tools, cost, swappable parts) and the speech-to-speech model (natural, fast, emotional, less control). Hybrids mix them.
The hard parts: turn detection, the latency budget, barge-in with buffer flushing and history truncation, tools with fillers and confirmations, memory, telephony.
Production: sticky sessions, regional GPU fleets, checkpoints and failover, 26 edge cases, a per-turn trace, nine safety rules, and four to eight cents a minute.

Check yourself

1. Which slot of the plan is "where you shine", and what is in it?

Answer

The deep dive on the hard parts, minutes 20 to 30: turn detection, the latency budget, interruption handling with buffer flushing and history truncation, and tool calling with fillers and confirmations.

2. In one sentence, what is the interviewer listening for?

Answer

That you understand a voice agent is a real-time streaming system with a strict latency budget, not a chatbot with a microphone.

3. What goes in the last 5 minutes?

Answer

Five or six edge cases with their solutions, plus per-turn tracing, simulated callers, PII redaction, and cost per minute.