Why real-time voice is hard

In this chapter, we will learn why a voice agent is not "a chatbot with a microphone". We all know text chatbots: the user types a message, presses Enter, and the chatbot replies. Adding a microphone and a speaker sounds easy. But voice brings seven new problems that text never had. By the end you will be able to name all seven and say which one has no fix at all, only a guess.

Problem 1: Nobody presses Enter

In a text chatbot, we know exactly when the user has finished, because they press Enter. In voice, there is no Enter key. The user just stops speaking. But a pause can mean "I am done" or "I am thinking about what to say next". The agent has to guess. Guess too early, and it cuts off the user. Guess too late, and it feels slow. This is the one problem in the list with no clean fix, only a better guess, and Chapter 7 is about making that guess well.

Problem 2: The time budget is tiny

A text chatbot can take two to three seconds to reply and nobody complains. A voice agent has less than one second. And in that one second, we have to convert speech to text, think with an LLM, convert text back to speech, and send the audio over the network. Every step eats into the budget.

text chatbot     2 to 3 s to reply        nobody complains
voice agent      under 1 s                 speech to text + LLM + text to speech + network, all inside it

Problem 3: Users interrupt

Humans interrupt each other all the time. If the agent is reading a long answer and the user says "No, wait, I meant the other order", the agent must stop talking immediately and listen. A text chatbot never has this problem. Its reply is already on the screen, and the user can simply start typing. We handle this in Chapter 15.

Problem 4: Audio is messy

Text is clean. Audio is not. Audio has background noise, a TV playing, a child crying, a bad phone line, accents, people speaking too fast, people mumbling, two people talking at once. Every one of these can confuse the system. A typed message never arrives with a dog barking inside it.

Problem 5: Speech has no punctuation or spelling

When someone says "my order id is A B one two three", the system has to figure out whether it is "AB123" or "A B 1 2 3" or "Abe one two three". Names, numbers, emails and codes are very hard in voice. In text, the user types the exact characters. In voice, the sounds reach us and the spelling is our problem.

Caller:   my order id is A B one two three
Could be: AB123
Could be: A B 1 2 3
Could be: Abe one two three

Problem 6: The output must sound human

The agent must not only say the right words, it must say them with the right rhythm, pauses and tone. A robotic voice reading "forty-five dollars point five zero" instead of "forty-five dollars and fifty cents" sounds wrong, even though the digits are correct. A text chatbot only has to spell correctly. A voice agent has to speak correctly, which is a different skill, and Chapter 10 is about it.

Problem 7: Everything is a continuous stream

In text, one message is one request. In voice, audio is flowing in and flowing out continuously during the whole call. We must handle it as a stream, not as a request and a response. There is no moment when the system is "waiting for the next message". It is always listening and often speaking at the same time.

text    request ──► reply ──► (wait) ──► request ──► reply
voice   audio in  ────────────────────────────────────────────►  (never stops)
        audio out ────────────────────────────────────────────►  (never stops)

The seven in one table

So, a Voice AI Agent is not "a chatbot with a microphone". It is a real-time streaming system with a strict time budget. Now that we know why it is hard, the next chapter defines exactly what we are building.

Try it yourself

Say your own email address out loud, as you would to a stranger on the phone. Write down two other spellings that sound the same. Then say a price with cents. Count how many of the seven problems you just produced in ten seconds of talking.

Common questions

Which of the seven problems is the hardest?

Problem 1. The others can be engineered down. The end of a turn can only ever be a guess, because a pause truly can mean either "I am done" or "I am thinking". Chapter 7 shows how to make the guess smarter.

Can we not just wait longer, to be safe?

Then we hit Problem 2. Every extra moment of waiting is added to every reply, and over one second feels slow.

Is Problem 7 really a problem, or just a different shape?

It changes how every component is built. Everything has to work on the stream as it arrives, which is why "streaming" appears in front of each of the three models in the next chapters.

Carry this

• Seven problems: no Enter, tiny budget, interruptions, messy audio, no spelling, must sound human, a continuous stream.
• The end of a turn is a guess. Too early cuts the user off; too late feels slow.
• Under one second, shared by speech-to-text, the LLM, text-to-speech and the network.
• A voice agent is a real-time streaming system with a strict time budget, not a chatbot with a microphone.

Check yourself

1. What replaces the Enter key in a voice agent, and why is that a problem?

Answer

A pause. But a pause can mean "I am done" or "I am thinking", so the agent has to guess, and a wrong guess either cuts the user off or feels slow.

2. Name the four things that must all fit inside the voice agent's one-second budget.

Answer

Converting speech to text, thinking with an LLM, converting text back to speech, and sending the audio over the network.

3. "Forty-five dollars point five zero" has the right digits. Which problem does it break?

Answer

Problem 6. The output must sound human, with the right rhythm and wording: "forty-five dollars and fifty cents".