What is a Voice AI Agent?

In this chapter, we will learn what a Real-Time Voice AI Agent is, by taking the name apart word by word. We will meet five real examples, and we will find the single number that drives most of the design decisions in the rest of this book. By the end you will be able to say in one sentence what we are building and why "real-time" is the hard word in the name.

Three words

A Voice AI Agent is a software program that we can talk to using our voice, and it talks back to us, like a phone call with a human, except that the one on the other side is an AI. The name has three parts, and each part adds a job.

Voice AI Agent  =  Voice  +  AI  +  Agent

Voice    the input and the output are sound. We speak, it speaks. No typing, no reading.
AI       the understanding and the thinking is done by a Large Language Model (LLM),
         a model trained on a huge amount of text so it can understand language and reply sensibly.
Agent    it does not just chat. It takes actions: looks up your order, books an appointment,
         cancels a subscription, transfers the call to a human. It does this by calling tools.

Tools are the subject of Chapter 9. For now, hold on to the difference between a chatbot and an agent: a chatbot answers, an agent acts.

Five real examples

• A customer support line where you call and say "Where is my order?" and the agent checks and tells you.
• A restaurant booking agent that takes reservations over the phone.
• A drive-thru agent that takes your food order.
• A language tutor in an app that talks with you to practise English.
• An outbound agent that calls customers to remind them about an appointment.

Notice that some of these live on a phone line and some live inside an app. Notice too that the last one calls you. Both differences come back later, in Chapter 18. The support line is the example we will follow through the whole book. A caller says "Where is my order 4 5 6 7 8?" and the agent has to find out and tell them.

What "real-time" means

Real-time means the agent replies fast enough that the conversation feels like talking to a human. How fast is that? In a normal human conversation, when one person stops speaking, the other person replies within about 200 to 500 milliseconds. A millisecond is one thousandth of a second, so 500 milliseconds is half a second. That gap feels natural to us.

human reply gap        200 to 500 ms      feels natural
more than 1 second                        starts to feel slow
more than 2 to 3 seconds                  "Hello? Are you there?"  the experience is broken
our target             about 800 ms       the agent starts speaking within 800 ms after the user stops

If the reply takes more than one second, it starts to feel slow. If it takes more than two or three seconds, people say "Hello? Are you there?" and the whole experience is broken. So our target is that the agent starts speaking within about 800 milliseconds after the user stops speaking. Not finishes. Starts. The first sound is what the caller is waiting for.

Why the interpreter is a good picture

A text chatbot is like a pen pal. You write, you wait, a letter comes back. Nobody minds a pause. An interpreter cannot be a pen pal. They work while the speaker is still talking, they decide for themselves when a sentence has ended, and they are judged on the gap between the speaker's last word and their first. That is exactly our agent's job, and the next chapter lists the seven things that make it hard.

Try it yourself

Call a friend and count, in your head, how long the gap is between the end of their sentence and the start of yours. Then wait a full two seconds before replying to something they asked. Notice what they say. That reaction is what we are designing against.

Common questions

Is a voice agent just a chatbot with a microphone?

No, and the next chapter is a list of reasons. The short version: nobody presses Enter, the time budget is under a second, users interrupt, audio is messy, and everything is a continuous stream.

Why 800 milliseconds and not 500, which is the human gap?

The post picks 800 ms as "fast enough that the conversation feels like talking to a human". The budget in Chapter 14 shows why not lower: the stages add up to 600 to 1,300 ms even when optimised, so 800 is already tight.

Does the agent need a wake word like "Hey Assistant"?

Not on a phone call. The call itself is the wake-up. Always-on devices like a smart speaker need one extra small model for this, which we mention in Chapter 7.

Carry this

• Voice AI Agent = Voice (sound in, sound out) + AI (an LLM understands and thinks) + Agent (it acts by calling tools).
• Humans reply within 200 to 500 ms. Over one second feels slow. Over two or three seconds is broken.
• Our target: the agent starts speaking within about 800 ms after the user stops.
• The example we follow: a caller asks "Where is my order 4 5 6 7 8?"

Check yourself

1. What does the word "Agent" add that "Voice AI" alone does not?

Answer

Actions. The agent can look up an order, book a slot, cancel a subscription or transfer the call, by calling tools. A chatbot only talks.

2. A reply arrives 2.5 seconds after the caller stops speaking. What happens?

Answer

The caller says "Hello? Are you there?" The experience is broken. Anything over one second already feels slow.

3. Our target is 800 ms. 800 ms until what, exactly?

Answer

Until the agent starts speaking, measured from the moment the user stops speaking. The first sound, not the whole reply.