Approach 1: the cascaded pipeline

In this chapter, we will learn the classic way to build a voice agent: the cascaded pipeline, which is what Part II has been building. We will see what phone bots did before LLMs, trace one turn of our order example from 0 to 850 ms, and weigh seven advantages against six disadvantages. By the end you will be able to say why the text in the middle is both the strength and the weakness.

The chain

The cascaded pipeline connects three separate models in a chain, and everything in the middle is text. That is why it is called a cascaded or pipeline approach.

Audio in --> VAD + Turn Detection --> STT --> text --> LLM (+ tools) --> text --> TTS --> Audio out

Before LLMs: the old way

The old phone bots, called IVR bots, also used STT and TTS, but the brain in the middle was not an LLM. It was an intent classifier, a small model that maps "where is my order" to the intent ORDER_STATUS, plus a hand-written flow: if intent is ORDER_STATUS, ask for the order number, then look it up, then read the fixed sentence.

This was predictable and cheap, but very rigid. If the user said anything outside the script, "I ordered two things but only one arrived, and the other one was a gift", it was lost. Replacing the intent classifier and the hand-written flow with an LLM plus tools is what turned the old bot into an agent.

One turn, with timing

The clock starts the moment the user says the last word. Here is the order status example with realistic timings. Before you press Step, guess at what millisecond the caller hears the first sound.

0 ms           user finishes "Where is my order 4 5 6 7 8?"
0 to 250       VAD sees silence; semantic turn detection sees a complete question, confirms end of turn at ~250
250 to 350     STT final transcript arrives (most words were transcribed while the user spoke)
350 to 650     LLM gets the transcript, decides to call get_order_status; by our prompt rules it first
               streams a filler, "Sure, let me check that for you." First token at ~600
650 to 800     the filler goes to TTS; first audio byte at ~800
800 to 850     audio travels back; the user hears "Sure, let me..." at ~850
in background  the tool call runs (say 400 ms), the LLM writes the real answer, TTS speaks it after the filler

voice-to-voice latency   ~850 ms   right at our target

Every component must be optimized to hit this. Chapter 14 builds the full budget stage by stage.

Advantages of the cascaded pipeline

• Modular. We can pick the best STT, the best LLM and the best TTS separately, and swap any of them when a better one comes out.
• Full control. The text in the middle is visible. We can log it, add guardrails (automatic checks that block unwanted output), redact (remove) sensitive information, and enforce business rules before it becomes speech.
• Tool calling is mature. LLM tool calling is well understood and reliable.
• Easy to debug. When something goes wrong, we can see exactly which stage failed, because we have the transcript at every step.
• Cheaper. Small specialized models are cheap compared to one giant model.
• Language flexibility. Different STT and TTS models per language.
• Reuse an existing text agent. If we already have a text chatbot with prompts and tools, we can reuse its brain and just add STT and TTS around it.

Disadvantages of the cascaded pipeline

• Latency adds up. Three models plus turn detection, each adding delay. Hitting under 800 ms needs a lot of engineering.
• Information is lost. When speech becomes text, we lose the tone, emotion, hesitation, laughter, sighs and emphasis. "I am fine" said happily and "I am fine" said angrily become the same text. The LLM cannot tell the difference.
• Errors compound. If STT gets a word wrong, the LLM receives the wrong text and gives a wrong answer, and TTS speaks it confidently. Each stage trusts the previous one.
• Robotic pacing. The agent speaks with the same tone regardless of the situation. It cannot naturally laugh, whisper or sound sympathetic, because the TTS only sees text.
• Turn detection is a separate hard problem. We must build and tune it ourselves.
• Half-duplex by nature. Half-duplex means only one side speaks at a time, like a walkie-talkie. The pipeline processes one turn at a time. Natural overlapping conversation, like the agent saying "mm-hmm" while the user speaks, is hard to do.

That is the cascaded pipeline. Chapter 12 removes the text in the middle entirely.

Try it yourself

Say "I am fine" out loud three ways: happy, tired, angry. Write down the transcript each time. It is the same three words. That is the information-loss disadvantage, felt rather than read.

Common questions

Why does the LLM say "Sure, let me check that for you" first?

The tool call takes time, around 400 ms here. The filler lets the caller hear something at 850 ms instead of silence.

What is an IVR bot?

The old phone bot: STT and TTS around an intent classifier and a hand-written flow. Predictable, cheap, rigid.

Is 850 ms good or bad?

Right at the target of about 800 ms, with every stage optimized. There is no slack: a stage that slips makes the reply start to feel slow.

Carry this

• Audio in, VAD and turn detection, STT, text, LLM with tools, text, TTS, audio out. Text in the middle.
• Before LLMs: an intent classifier plus a hand-written flow. Rigid. The LLM plus tools made it an agent.
• One optimized turn: end of turn at 250, STT at 350, first token at 600, first audio at 800, heard at 850 ms.
• For: modular, full control, mature tools, easy to debug, cheaper, per-language models, reuse a text agent.
• Against: latency adds up, tone is lost, errors compound, robotic pacing, turn detection is ours, half-duplex.

Check yourself

1. In the timed walk-through, what happens between 350 and 650 ms?

Answer

The LLM receives the transcript, decides to call get_order_status, and streams the filler sentence first. The first token arrives at about 600 ms.

2. STT hears "fright" instead of "flight". Which disadvantage is this, and what does TTS do?

Answer

Errors compound. The LLM gets the wrong text and answers it, and TTS speaks that answer confidently. Each stage trusts the one before.

3. What does half-duplex mean, and why does it matter here?

Answer

Only one side speaks at a time, like a walkie-talkie. The pipeline handles one turn at a time, so natural overlapping speech such as an "mm-hmm" from the agent is hard.