Handling interruptions

In this chapter, we will learn what the agent must do when the user starts speaking while the agent is still talking. This is called barge-in, and it is a must-have. We will read the five steps as pseudocode, find the step most people forget, and see why the conversation history has to be corrected after every interruption. Then we will handle sounds that look like an interruption but are not. By the end you will be able to explain why "stop the audio" is five steps and not one.

What barge-in is

Barge-in is when the user starts speaking while the agent is still speaking. Say the agent starts reading out a long return policy, and the user says "No no, I just want a refund". The agent must stop right there and listen. A text chatbot never has this problem. A voice agent has it on every long answer.

The interruption flow

An interruption is handled in five steps: stop generating, flush the buffer, find the playback position, fix the history, then listen. Here is the flow as pseudocode, simplified for understanding.

def on_user_speech_started(session):
    if session.agent_is_speaking:
        # 1. Stop producing more audio
        session.tts.cancel()
        session.llm.cancel()

        # 2. Tell the client to drop any audio that is buffered but not yet played
        session.client.send("clear_audio_buffer")

        # 3. Find out how much the user actually heard
        played_text = session.get_text_played_until(session.playback_position)

        # 4. Fix the conversation history to reflect only what was spoken
        session.history.truncate_last_agent_turn(played_text)

        session.agent_is_speaking = False

    # 5. Now, listen to the user as usual
    session.start_listening()

Step 1: Stop generating. Cancel the TTS stream and the LLM stream, so that no new audio is produced.

Step 2: Flush the buffer. This is the step most people forget. We have already sent a few seconds of audio to the client. It is sitting in the client's buffer, waiting to be played. If we do not clear it, the agent keeps talking for two more seconds after the user interrupted. The client must be told to drop everything not yet played.

Step 3: Track the playback position. The orchestrator must know exactly which word was being spoken when the interruption happened. We get this from the playback progress the client sends back, a count of chunks played, mapped to the text.

Step 4: Fix the history. Suppose the agent's full reply was "Your order was shipped yesterday and will arrive tomorrow by 5 PM", but the user interrupted after "Your order was shipped". Then the conversation history must contain only "Your order was shipped". Otherwise, in the next turn, the LLM thinks the user already heard about the 5 PM delivery and gets confused.

Step 5: Listen. Process the new user speech normally.

Agent (full reply):   Your order was shipped yesterday and will arrive tomorrow by 5 PM.
User (cuts in):       Your order was shipped | No no, I just want a refund.
History must hold:    "Your order was shipped"   and nothing after the bar

Before you press anything, guess: with the buffer not flushed, how many of the twelve words does the user hear after they started talking? Then switch the two toggles off and on and watch what the next turn's history contains.

False interruptions

A false interruption is a sound that VAD reports as speech but that does not mean the user wants the floor. A cough, a backchannel like "okay", or a dog barking. If we stop the agent for every small sound, it will keep stopping mid-sentence, which is very annoying. So we do not interrupt immediately on VAD. We use these rules:

• Wait until the user's speech lasts at least 200 to 300 ms, or the STT gives at least one or two real words.
• Run the short burst through the backchannel classifier from Chapter 7. "mm-hmm", "okay", "right", "yeah" while the agent is speaking are acknowledgements, not interruptions.
• Once confirmed, interrupt. This adds a small delay, but it avoids a jittery agent.

Both start at the same time

When the agent and the user start speaking at the exact same moment, the agent always yields. Humans solve this by one person backing off. Our rule: if user speech is detected within the first few hundred milliseconds of the agent's reply, the agent stops and lets the user speak.

Try it yourself

Write down the five steps from memory, then open a voice assistant you have access to and interrupt it in the middle of a long answer. Count how long it keeps talking after you started. If it is about two seconds, you have just heard Step 2 missing.

Common questions

Why not just stop the TTS and be done?

Because audio already sent to the client keeps playing for about two seconds unless the client is told to drop it, and because the history would then claim the user heard words they did not. Steps 2 and 4 exist for those two reasons.

Why does the agent always yield when both start at once?

Humans solve a collision by one person backing off, and a simple fixed rule beats a negotiation. So if user speech is detected within the first few hundred milliseconds of the agent's reply, the agent stops and lets the user speak.

Carry this

• Five steps: cancel TTS and LLM, send clear_audio_buffer, read the playback position, truncate the last agent turn, listen.
• Flushing the client buffer is the step most people forget. Without it the agent talks for two more seconds.
• The history must contain only the words the user actually heard.
• Confirm an interruption first: 200 to 300 ms of speech or one or two real words, and not a backchannel.
• When both start at once, the agent yields. In S2S, the client flush is still ours.

Check yourself

1. The TTS and LLM are cancelled, but the agent keeps talking for two seconds. Which step is missing?

Answer

Step 2, the buffer flush. Audio already sent to the client is still in its buffer, and the client must be told to drop it with clear_audio_buffer.

2. The user interrupted after "Your order was shipped". What goes wrong on the next turn if the history keeps the full sentence?

Answer

The LLM thinks the user already heard "will arrive tomorrow by 5 PM" and gets confused, referring to something the user never heard.

3. The user says "mm-hmm" while the agent speaks. What should happen?

Answer

Nothing. The burst is short and the backchannel classifier marks it as an acknowledgement. The agent keeps talking.