Approach 3: hybrids, and the comparison

In this chapter, we will learn how real systems mix the cascaded pipeline and the speech-to-speech model, in five named patterns. Then we will put the two approaches side by side in one table, sixteen rows long, and reduce it to three rules for choosing. By the end you will be able to pick an approach for a use case and defend it, which is exactly what an interviewer wants.

Five hybrid patterns

In practice, many production systems use a mix. These are the common patterns.

Pattern 1: S2S for talking, text LLM for thinking

A speech-to-speech model is the front, so the conversation feels natural and fast. Behind it, a text-based LLM agent handles the tools, the business logic, the knowledge base lookup and complex reasoning. The S2S model is given a tool called ask_backend. Whenever the question needs real work, it calls the backend agent, gets a text answer, and speaks it. Natural conversation plus control and reasoning.

Pattern 2: cascaded pipeline with audio understanding

Keep the pipeline, but feed the audio, not just the text, to an audio-capable LLM, so it can also understand tone and emotion. The output is still text, and TTS speaks it. Better understanding, full control of the output.

Pattern 3: cascaded pipeline with expressive TTS

Keep the pipeline, but have the LLM output emotion tags along with the text, and use a TTS model that acts on those tags. Some emotional range, everything still in text.

LLM text:   [sympathetic] I am sorry to hear that. Let me fix it right away.

Pattern 4: LLM with native audio output

Keep STT in front, but use an LLM that produces audio tokens directly instead of text, so there is no separate TTS step. The input side stays in text, easy to control and log, and the output side gains natural, expressive speech with one less hop.

Pattern 5: fast model first, smart model behind

A small fast LLM produces the first sentence instantly, an acknowledgement, and a bigger model produces the real answer. The user hears something within 500 ms, and the real answer follows.

The right hybrid depends on the use case. A customer support agent with strict rules and many tools leans towards the cascaded pipeline. A companion app or a language tutor, where natural feel matters most, leans towards speech-to-speech.

Cascaded versus speech-to-speech

When to use which

• Use the cascaded pipeline when we need control, compliance, tool-heavy workflows, low cost, and the ability to swap models. This is the default for most business use cases today.
• Use speech-to-speech when the natural feel of the conversation is the product itself, when emotion matters, and when we can accept less control and higher cost.
• Use a hybrid when we want the natural feel of S2S in front and the reliability of a text agent behind.

In an interview, the best answer is to present both, explain the trade-offs, and choose based on the requirements the interviewer gives. Before you answer the questions below, pick the approach you would choose for our support line, and see whether the table agrees.

Whatever we choose, the clock in Chapter 1 is still running. Chapter 14 spends the 800 ms stage by stage.

Try it yourself

Take the five examples from Chapter 1: support line, restaurant booking, drive-thru, language tutor, appointment reminder. For each, choose cascaded, S2S or a named hybrid pattern, and write the one table row that decided it. Two of the five should come out differently from the support line.

Common questions

In pattern 1, who handles interruptions?

The S2S model in front, because that is what it learned from real conversations. The text agent behind it never hears audio; it only answers ask_backend calls in text.

Is pattern 5 a hybrid of two approaches or two sizes?

Two sizes of the same approach: a small fast LLM for the first sentence and a bigger one for the real answer, within the pipeline. It is listed with the hybrids because it mixes models to buy latency.

Why is the pipeline "the default for most business use cases today"?

Control, compliance, mature tool calling, lower cost and the freedom to swap models. The rows where S2S wins, latency and feel, matter most when the conversation is the product.

Carry this

• Five hybrids: S2S talks and a text LLM thinks via ask_backend; audio-capable LLM in the pipeline; emotion tags for expressive TTS; LLM with native audio output; fast model first, smart model behind.
• Cascaded wins on control, guardrails, tools, reasoning, debugging, cost, swapping, languages, reuse.
• S2S wins on latency (300 to 600 ms), tone in and out, turn detection, interruptions, fewer parts.
• Cascaded for support, banking, healthcare. S2S for companions, tutors, casual assistants. Hybrid for natural front, reliable back.
• In an interview, present both and choose from the requirements.

Check yourself

1. In pattern 1, what is ask_backend and who calls it?

Answer

A tool given to the speech-to-speech model. When a question needs real work, the S2S model calls it, the text-based backend agent answers in text, and the S2S model speaks the answer.

2. A bank wants a voice agent that must never state a balance it has not looked up. Which approach, and which table rows decide it?

Answer

Cascaded. Control over what is said, guardrails and redaction, mature tool calling, and easy debugging. Banking is named in the "best for" row.

3. Which pattern keeps text on the input side but removes the TTS step?

Answer

Pattern 4, an LLM with native audio output: STT in front, the LLM produces audio tokens directly, one less hop.