Text-to-speech

In this chapter, we will learn how the reply text becomes a voice. We will see why TTS, like STT, must stream, what makes a voice good enough for a phone call, why "15/09/2026" must not be read as "fifteen slash zero nine", how we force an order ID to be spelled out, and the path the audio takes back to the caller. By the end you will know the metric for this stage and the one thing the orchestrator must keep track of as the audio plays.

What TTS is

Text-to-Speech (TTS) is a model that takes text as input and produces spoken audio as output. Just like STT, we need the streaming version. We send a sentence, and the model starts sending audio chunks within 100 to 200 ms, before it has finished generating the whole sentence. The metric here is Time To First Byte of audio (TTFB).

What makes a good TTS for a voice agent

• Naturalness. It must sound like a human, with proper rhythm (called prosody), pauses at commas, and rising tone for questions.
• Low latency. First audio byte under 200 ms.
• Stability. It must not mispronounce, skip words, or produce strange sounds.
• Consistent voice. The same voice throughout the call and across calls.
• Language and accent support.

Text normalization

Text normalization is the step that converts written form to spoken form. The LLM output is text, but text and speech are different. Consider this sentence:

LLM text:   Your appointment is on 15/09/2026 at 3:30 PM with Dr. Sharma, and the fee is $45.50.

If we send this directly, a weak TTS will say "fifteen slash zero nine slash two thousand twenty six" and "dollar forty five point fifty". We want "fifteenth September two thousand twenty six at three thirty PM with Doctor Sharma, and the fee is forty five dollars and fifty cents".

Good TTS models do most of this. But for our domain, phone numbers, order IDs, email addresses and URLs, we must add our own rules. For example, an order ID "AB12345" must be spoken letter by letter and digit by digit, "A, B, one, two, three, four, five", with small pauses.

Most TTS systems support SSML (Speech Synthesis Markup Language), a small set of tags to control pauses, spelling out, emphasis and pronunciation. We can force letter-by-letter reading like this:

<say-as interpret-as="characters">AB12345</say-as>

Before you press a button, say "$45.50" out loud the way you would on the phone. Then see what a weak TTS does with it.

Sending audio back

The TTS audio chunks go back through the same path the user's audio came in.

Orchestrator  -->  Media Gateway  -->  WebRTC or telephony  -->  user's speaker

The gateway converts the audio to the right codec and sample rate for the user's connection.

We have now seen all five components. Chapter 11 chains them into a complete design, and Chapter 12 shows the fundamentally different way to do it.

Try it yourself

Take a sentence from your own domain that contains a date, a time, a price and an ID. Write it the way an LLM would type it. Then write it the way a careful person would say it on the phone. Every place the two differ is a normalization rule you would have to add.

Common questions

Why is first-byte latency the metric and not total speaking time?

Because the caller is waiting for the first sound. Once audio is flowing, the rest streams behind it. The budget in Chapter 14 counts TTS as its time to first audio byte, 80 to 200 ms.

If good TTS models normalize text already, why write our own rules?

They handle the common cases. Our domain has order IDs, phone numbers, emails and URLs that a general model may read wrongly. Those get rules, and SSML where we need to force a reading.

Can the LLM just write "forty five dollars" itself?

The voice system prompt in Chapter 9 asks it to. Text normalization is the safety net for the times it does not.

Carry this

• Streaming TTS sends the first audio chunk 100 to 200 ms after it gets a sentence. The metric is TTFB.
• Good TTS: natural prosody, first byte under 200 ms, stable, one consistent voice, our languages.
• Text normalization turns written form into spoken form. Add our own rules for IDs, phone numbers, emails, URLs.
• SSML say-as "characters" forces letter-by-letter reading.
• Audio goes back through the gateway, which picks the codec and sample rate. The orchestrator tracks how much was played.

Check yourself

1. What does TTFB measure for TTS, and what is the target?

Answer

Time To First Byte of audio: the gap between sending a sentence and the first audio chunk coming back. Under 200 ms.

2. How should "AB12345" be spoken, and how do we force it?

Answer

Letter by letter and digit by digit, "A, B, one, two, three, four, five", with small pauses. The SSML tag say-as interpret-as="characters" forces it.

3. Why must the orchestrator know how much audio the client has played?

Answer

For interruptions. When the user barges in, the orchestrator must know which word was being spoken so it can fix the conversation history to contain only what was heard.