Voice activity and turn detection

In this chapter, we will learn the component that most people miss in interviews, and the one that makes or breaks the experience. It answers two questions: is a human speaking right now, and has the user finished their turn or just paused? The second is the hardest problem in voice agents, and we go through five ways to attack it. By the end you will be able to explain why "the user went quiet" is not the same as "the user is done".

Voice Activity Detection

Voice Activity Detection (VAD) is a small model that looks at each audio chunk and answers one question: is a human speaking right now, yes or no? It runs on every 20 ms chunk, takes under 1 millisecond, and is small enough to run even on the phone. There are two ways to build one.

• Energy-based VAD. Loud means someone is speaking, quiet means silence. Simple, but a slammed door or a TV in the background also looks like speech.
• Model-based VAD. A small neural network (a tiny AI model) trained to tell human speech from other sounds. This is what we use in production. It ignores background noise, keyboard clicks and music.

VAD gives us two events: speech started and speech ended. These two events drive everything else.

Where does VAD run? Both places. A VAD on the client stops us sending silence over the network, saving bandwidth and STT cost. A VAD on the server is the one we trust for turn detection and interruptions, because the server has the full picture.

Turn detection

Turn Detection, also called End-of-Turn detection or endpointing, is deciding whether the user has finished speaking and is waiting for a reply, or has just paused. VAD tells us the user stopped making sound. That is not the same as the user finished their turn.

Caller:   My order number is...        (one second pause, looking at their email)
Caller:   4 5 6 7 8
Agent:    Sorry, I did not get the order number.     (jumped in during the pause: wrong)

VAD sees a one-second silence in the middle. If we treat it as "user is done", the agent jumps in and the user gets frustrated. Here are the approaches, from simple to advanced.

Approach 1: Fixed silence timeout

Wait for a fixed period of silence, say 700 milliseconds, then the turn is over. The issue is that there is no good value. 400 ms cuts people off when they pause to think. 1,500 ms makes the agent always feel slow, because 1.5 seconds is added to every single reply.

Approach 2: Silence plus transcript signals

Use the Speech-to-Text output along with the silence. If the transcript ends with a full sentence and a question mark or full stop, and there is 300 ms of silence, end the turn quickly. If it ends with "and" or "my number is", wait longer, up to 1.5 seconds. Better, but it depends on the Speech-to-Text model producing good punctuation, which it does not always do.

Approach 3: Semantic turn detection

Semantic means based on the meaning of the words. We train a small, fast language model on one job: given the transcript so far, how likely is it that the user is finished?

"I want to book a flight to"                        not finished   (a city is coming)   wait longer
"I want to book a flight to Delhi tomorrow morning."  finished                             reply fast
"Umm, so, the thing is"                             not finished
"Yes."                                              finished

We combine this with VAD silence. The timeout becomes dynamic: around 200 ms when the model says the sentence is complete, up to 2 seconds when it says incomplete. This is how modern voice agents get both speed and patience. Some advanced versions also feed the audio itself to this model, because tone tells a lot: a rising pitch at the end usually means a question, a falling pitch usually means "I am done".

Approach 4: Speculative processing

Even with good turn detection, we still wait some milliseconds to be sure. To save this time, we start early. The moment VAD detects silence, we send the transcript to the LLM and generate a reply, but do not play it yet. If the user speaks again, we throw it away. If the turn is confirmed, the reply is ready and plays instantly. This costs extra LLM calls, sometimes thrown away, but saves 200 to 400 ms. We trade money for speed.

Approach 5: Push-to-talk

In an app, we can simply give a button. The user holds it while speaking and releases it when done. No guessing at all. This works for a walkie-talkie style translation app, but it is not possible on a phone call, and it does not feel natural.

Speech-to-Speech models (Chapter 12) learn turn detection from millions of hours of real conversations, so they need no separate turn detection module. That is one of their big advantages.

Guess first: after "Yes.", how much longer does the fixed 700 ms timeout wait than the semantic approach?

Backchannels

Humans make small sounds while listening: "mm-hmm", "okay", "right", "yeah". These are called backchannels. They do not mean the listener wants to speak, but our VAD will detect them as speech. If we treat every "mm-hmm" as an interruption, the agent stops mid-sentence for no reason. The solution is a small classifier (a model that sorts an input into one of a few labels) that looks at short speech bursts, under 500 ms, and decides: backchannel, ignore it; real interruption, stop and listen. Chapter 15 covers interruptions in detail.

Try it yourself

Overhear a phone call and count the pauses that were not the end of a turn. Most come right after "and", "so" or "my number is".

Common questions

Why not just use Approach 2, since STT already runs?

It relies on good punctuation from STT, which is not reliable. The semantic model reads the meaning, so "my number is" is unfinished whether or not a comma was produced.

Is speculative processing always worth it?

It is a trade. The LLM calls that get thrown away cost money, and the saving is 200 to 400 ms per turn. The post says to decide based on the use case: money for speed.

Carry this

• VAD answers "is a human speaking?" on every 20 ms chunk in under 1 ms. Model-based, on both client and server.
• Silence is not the end of a turn. Turn detection is the hardest problem in voice agents.
• Fixed timeout has no good value. Semantic detection makes it dynamic: about 200 ms when complete, up to 2 s when not.
• Speculative processing trades extra LLM calls for 200 to 400 ms. Push-to-talk works only in an app.
• Backchannels like "mm-hmm" are not interruptions. A classifier on bursts under 500 ms tells them apart.

Check yourself

1. VAD says "speech ended". What has it not told us?

Answer

Whether the user is finished or only paused. VAD detects sound, not turns.

2. Why is there no good value for a fixed silence timeout?

Answer

400 ms cuts people off when they pause to think; 1,500 ms adds 1.5 seconds to every reply and the agent always feels slow.

3. How does semantic turn detection get both speed and patience?

Answer

A small model judges from the transcript whether the user is finished, and the silence timeout becomes dynamic: around 200 ms when complete, up to 2 seconds when not.