Speech-to-text

In this chapter, we will learn how the user's speech becomes text. We will see the two ways to use a Speech-to-Text model and why only one works for us, what interim and final transcripts are, why this stage is not the biggest part of the budget, what makes it hard on a real call, and the number that tells the agent when to confirm what it heard. By the end you will be able to say what a Word Error Rate of 5% means.

What STT is

Speech-to-Text (STT), also called Automatic Speech Recognition (ASR), is a model that takes audio as input and gives the spoken words as text output. It is the first of the three AI models in our chain.

Batch versus streaming

There are two ways to use an STT model, and only the streaming one fits a real-time agent.

Batch STT       give it a complete audio file      -->  get the full transcript back
                for transcribing a recorded meeting. Accurate, but slow: it waits for the entire audio.

Streaming STT   keep feeding 20 ms chunks          -->  it keeps giving us text as the user speaks
                this is what a real-time agent needs.

Streaming STT gives two types of output.

• Interim (partial) transcripts are the best guess so far. They can change. For example: "I want to book" becomes "I want to book a fly" and then "I want to book a flight".
• Final transcripts are segments the model is confident about. Once it marks a segment as final, it will not change any more.

Interim transcripts are very useful. We can show them on screen in an app. We can feed them to the semantic turn detection model. We can even start preparing the LLM call before the final transcript arrives.

Before you press Step, guess how many times the words will change before the sentence is final.

Why streaming STT keeps the latency low

The model works on small windows of audio, a few hundred milliseconds long, and does not wait for the sentence to end. So when the user says the last word, most of the sentence is already transcribed. The final transcript usually arrives within 100 to 300 ms after the user stops speaking. This is why STT is not the biggest part of our latency budget.

while the user speaks     most words already transcribed, window by window
after the last word       final transcript within 100 to 300 ms
share of the 800 ms       small. The budget in Chapter 14 gives this stage 50 to 150 ms.

What makes STT hard in a voice agent

Word Error Rate (WER) is the standard metric: the share of words the model got wrong. If the user said 100 words and the model got 5 wrong, WER is 5%.

words said      100
words wrong     5
WER             5 / 100 = 5%

good models     under 5 to 10% on clean audio
noisy phone line with an accent     can go above 20%

Four things push that number up on a real call.

• Names, numbers and codes. "My email is amit dot shekhar at gmail dot com" or "Flight number A I 1 2 3". These are the most common failures. The solution is keyword boosting, also called custom vocabulary: we tell the STT model a list of words that are likely in our domain, like product names, city names or the company name. Many STT models also have a special mode for spelling and digits.
• Accents and languages. We must choose a model trained on the accents of our users. For India, the model must handle Indian English and code-switching, where people mix Hindi and English in one sentence.
• Noise. Noise suppression before STT helps a lot.
• Multiple speakers. If two people speak, a feature called speaker diarization tells us who said what. This is usually not needed for a one-to-one call, but it is needed for a meeting assistant.

Confidence scores

A good STT model gives a confidence score with each word: a number that tells how sure the model is. If the confidence is low for an important word, like an order number, the agent must confirm.

Caller:   Where is my order 4 5 6 7 8?
Agent:    I heard 4 5 6 7 8, is that correct?

We will use this in Chapter 20. Now we have the text. It is time to think, and that is Chapter 9.

Try it yourself

Say your own email address out loud, the way you would on a phone call, and write down what a model that only knows common words would type. Then write the words you would put in a keyword-boosting list to fix it.

Common questions

Why not use batch STT and wait for the sentence to end?

Batch waits for the entire audio before it starts, so the whole transcription time would land after the user stops. Streaming does most of the work while they are still talking.

Does a 5% WER mean the agent is wrong 5% of the time?

No. 5 words in 100 are wrong. The dangerous ones are names, numbers and codes, which is why those get confirmation.

Do we need speaker diarization for the support line?

Usually not. One caller, one agent. It is needed for a meeting assistant.

Carry this

• Streaming STT returns interim transcripts that change and final ones that do not.
• Most words are transcribed while the user speaks. The final transcript arrives 100 to 300 ms after they stop.
• WER = words wrong / words said. Under 5 to 10% on clean audio, above 20% on a noisy line with an accent.
• Names, numbers and codes fail most. Keyword boosting and a spelling mode help.
• Low confidence on an important word means confirm: "I heard 4 5 6 7 8, is that correct?"

Check yourself

1. The transcript reads "I want to book a fly". Is this interim or final, and what should the orchestrator do with it?

Answer

Interim. It may still change to "flight". The orchestrator can show it, feed it to turn detection, or start preparing the LLM call, but not treat it as final.

2. A caller said 200 words and the model got 30 wrong. What is the WER, and what kind of audio does that number suggest?

Answer

30 / 200 = 15%. Above the 5 to 10% of clean audio and heading towards the 20% or more of a noisy phone line with an accent.

3. The company sells a product called "Zentro". Which STT feature stops it becoming "central"?

Answer

Keyword boosting, also called custom vocabulary: a list of words likely in our domain, such as product names, that we give to the STT model.