Approach 2: the speech-to-speech model

In this chapter, we will learn the second, fundamentally different way to build a voice agent: one model that takes audio in and gives audio out. We will see how it works with audio tokens, why it can hear a sigh and answer warmly, and why that directness makes it harder to control. By the end you will be able to list what it gains over the pipeline and what it gives up.

What it is

A Speech-to-Speech (S2S) model is a single model that takes audio in and gives audio out, directly, without converting to text in between. These are also called end-to-end voice models, audio-native models or realtime models. Examples include the models behind OpenAI's Realtime API, Google's Gemini Live, and open-source models like Kyutai's Moshi.

How it works

An LLM predicts the next token of text. A speech-to-speech model does the same thing, but with audio. There are three steps.

1  audio tokenizer (neural audio codec)   user's audio  -->  sequence of audio tokens
   each audio token is a small piece of sound, around 20 to 80 ms
   a vocabulary of sounds instead of a vocabulary of words

2  one large model, trained on huge amounts of conversational audio
   user's audio tokens  -->  predicts the agent's audio tokens as the reply

3  the same codec turns the output audio tokens back into sound

Because the model works directly on sound, it hears everything: the words, the tone, the emotion, the pause, the laughter, the background. And it produces everything: not just words, but a warm tone, a chuckle, a sympathetic pause.

Many of these models are also full-duplex, which means they listen and speak at the same time, continuously, like two humans on a phone call. The model itself learns when to start speaking, when to stop, when to say "mm-hmm", and when to yield to the user. Turn detection and interruption handling are built in, learned from real conversations.

Internally, most production S2S models also generate text alongside the audio, a transcript of what the model is saying, so we still get a transcript for logging. But the text is a by-product, not a step in the chain.

Advantages

• Very low latency. There is only one model. Voice-to-voice latency of 300 to 500 ms is possible, which is close to human.
• Emotion and tone are preserved. The model hears how the user said it and can respond with the right feeling. It can sound sorry when the user is upset.
• Natural conversation. Backchannels, overlapping speech, laughing, interrupting and being interrupted all feel natural, because the model learned them from real data.
• Built-in turn detection. No separate VAD tuning and semantic model needed.
• Handles accents and noise as one system. No compounding errors between separate stages.
• Simpler pipeline. Fewer moving parts to operate.

human reply gap           200 to 500 ms    (Chapter 1)
cascaded, optimized       ~850 ms          (Chapter 11)
speech-to-speech          300 to 500 ms    inside the human range

Disadvantages

• Less control. We cannot inspect and edit the text before it is spoken, because it is spoken directly. Adding guardrails, redaction or strict business rules is much harder.
• Harder to debug and evaluate. The transcript we get is what the model thinks it said, not a ground truth. Evaluating audio outputs is harder than evaluating text.
• Tool calling is less mature. It is supported, but newer and less reliable than in text LLMs. Complex multi-step tool workflows are harder.
• Weaker reasoning. The best reasoning models are text models. S2S models are trained for conversation and are usually weaker at complex logic, math, or long multi-step tasks.
• Expensive. Audio tokens are many more than text tokens. One minute of conversation can cost several times more than the cascaded pipeline.
• Vendor lock-in. Very few companies can train such models. We cannot swap the STT or the voice independently.
• Fewer voices and languages. Limited voice options, and language quality varies a lot.
• Hallucination in voice. Hallucination is when a model makes up a fact and states it confidently. When a text LLM hallucinates, we can catch it in text. When an S2S model hallucinates, it just says it, confidently, in a warm voice.
• Safety concerns. The model can imitate voices, and it can be pushed to produce unwanted audio. Providers add restrictions, but it is a real concern.
• Context fills up fast. Audio tokens fill up the context window, the maximum amount the model can read at once, much faster than text, so long calls are harder.

Try it yourself

Write down the last three things a support agent said to you on a call. For each, ask: would I want those words checked before they were spoken? Each yes is a place where "less control" costs something.

Common questions

If the model produces a transcript anyway, why is debugging harder?

That transcript is what the model thinks it said, not a ground truth. In the pipeline, the text is the thing that was spoken. Here it is a by-product.

What does full-duplex mean?

Listening and speaking at the same time, continuously, like two humans on a call. The pipeline is half-duplex: one side at a time, like a walkie-talkie.

Why are audio tokens so expensive?

Each covers only 20 to 80 ms of sound, so a minute of conversation is many more tokens than the same minute as text. More tokens: more cost, and a context window that fills faster.

Carry this

• One model, audio in, audio out. Also called end-to-end, audio-native or realtime models.
• A neural audio codec turns sound into audio tokens of 20 to 80 ms. The model predicts the reply's audio tokens.
• Gains: 300 to 500 ms latency, tone preserved, natural overlap, built-in turn detection, one system, fewer parts.
• Losses: less control, harder to debug, immature tools, weaker reasoning, expensive, lock-in, fewer voices, hallucination in a warm voice, safety, context fills fast.
• The transcript it produces is a by-product, not a step in the chain.

Check yourself

1. What is an audio token, and roughly how much sound does one cover?

Answer

A small piece of sound from a vocabulary of sounds, produced by a neural audio codec. Around 20 to 80 ms each.

2. Why does an S2S model not need the semantic turn detection of Chapter 7?

Answer

It learns when to start speaking, stop, say "mm-hmm" and yield directly from millions of hours of real conversations. Turn detection is built in.

3. Why is a hallucination worse in an S2S model than in a text LLM?

Answer

In text we can catch it before it is spoken. The S2S model just says it, confidently, in a warm voice, with no text step where a check could run.