The latency budget

In this chapter, we will learn where every millisecond goes in the cascaded pipeline. We will write the budget down as a table, one row per stage, add it up, and see that no stage can be ignored. Then we will learn the seven rules that keep a real system inside the budget. By the end you will be able to spend 800 milliseconds on purpose, and move them around with your own hands.

The budget, row by row

A latency budget is a table with one row per stage and a target for each, so that the sum stays under the voice-to-voice target. Here is the budget for the cascaded pipeline, with the target 800 ms.

Add the low ends and we get 590 ms, which the post rounds to 600. Add the high ends and we get 1,310 ms, rounded to 1,300, over the 1.2 s we allow at p95. The budget is very tight. There is no single stage that we can ignore. Shave 50 ms off the network, lose it again to a slow model, and we are back where we started.

all low ends    30 + 150 + 50 + 150 + 100 + 80 + 30   =   590 ms   (the post's table says 600)
all high ends   80 + 300 + 150 + 300 + 200 + 200 + 80  = 1,310 ms   (the post's table says 1,300)
target                                                     800 ms at p50, under 1.2 s at p95

Before you move anything, guess: if every stage sits at the middle of its range, are we under 800? Then try to find a way to spend the budget where one stage is at its worst and we still make it.

What p50 and p95 mean

p50 means half of the turns are faster than this number, and p95 means 95 out of 100 turns are faster than this number. The target is 800 ms at p50 and under 1.2 s at p95. Why two numbers? Because an average hides the bad turns. The user does not remember the average. The user remembers the slow turn, the one where they said "Hello? Are you there?". So we measure at p95, not average.

Seven rules to stay inside the budget

The rules are all the same idea: never wait for something to finish when you could use the start of it.

1. Stream everything. Never wait for a full transcript, a full LLM reply, or a full TTS output. Each stage hands over its first piece as soon as it has one.
2. Overlap stages. While TTS speaks sentence one, the LLM writes sentence two. This is the overlap trick from Chapter 9.
3. Co-locate. Keep the orchestrator, STT, LLM and TTS in the same datacenter. Every network hop between them adds 10 to 50 ms.
4. Warm connections. Keep the WebSocket connections to the STT and TTS services already open before the call starts. Opening a connection during a turn can cost 100 to 300 ms.
5. Pre-generate the greeting. The first thing the agent says ("Hello, thank you for calling...") is always the same. Cache its audio and play it instantly when the call connects.
6. Measure at p95, not average. The slow turn is the one that counts.
7. Fillers buy time. A quick "Sure, let me check" makes a 2-second tool call feel fine. The budget is about the first sound, and a filler is a first sound.

three hops between stages, best case    3 x 10 ms  =  30 ms
three hops between stages, worst case   3 x 50 ms  = 150 ms   as much as the whole turn-detection wait at its low end
one cold connection                     100 to 300 ms          up to 300 of the 800 ms, more than a third

Try it yourself

In the interactive, set both network rows to their worst, 80 ms. Now try to get under 800 ms by moving only the two LLM rows. Then try with only turn detection. Which stage has the most room? That is where the engineering effort should go first.

Common questions

Why is turn detection one of the biggest rows when it is mostly waiting?

Because it is a wait. We have to let some silence pass before we are sure the user is done. The semantic model in Chapter 7 makes that wait dynamic: short when the sentence is complete, long when it is not. With a fixed timeout like the 700 ms of Chapter 7, this one row would almost fill the budget.

Does the budget apply to Speech-to-Speech models too?

This table is for the cascaded pipeline, where each row is a separate system. A Speech-to-Speech model collapses the middle rows into one model, which is why Chapter 13 listed 300 to 600 ms for it. The network rows and the client jitter buffer still apply.

Carry this

• Seven rows, about 600 ms at best and 1,300 ms at worst. No row can be ignored.
• Target: 800 ms at p50, under 1.2 s at p95. Measure the slow turns, not the average.
• Stream everything, overlap stages, co-locate, keep connections warm, cache the greeting.
• A filler makes a 2-second tool call feel fine. It hides latency, it does not remove it.

Check yourself

1. Every stage is at the top of its range. What is the total, and is it acceptable?

Answer

1,300 ms. No. It is over the 1.2 s we allow at p95 and far over the 800 ms target at p50.

2. Why do we keep the WebSocket to the STT service open before the call starts?

Answer

Opening a connection during a turn can cost 100 to 300 ms. A warm connection costs nothing at the moment we need it.

3. Ten turns: nine take 700 ms, one takes 3 s. What is p50, and why is it the wrong number to report alone?

Answer

p50 is 700 ms, because half the turns are faster than that. It hides the 3-second turn, which is the one the user remembers. Report p95 too.