Scaling the system

In this chapter, we will learn how to serve 10,000 calls at the same time. A voice call is nothing like a web request: it lasts minutes and holds state and open connections the whole time. That changes how we spread the load, where we put the servers, and how we survive a crash. By the end you will be able to size the orchestrator fleet and say what happens to a call when its server dies.

Sessions are stateful and long-lived

A web request lasts 100 ms. A voice call lasts 5 minutes, and the whole time the orchestrator holds state and open connections. So we cannot treat the orchestrator like a normal stateless web server, where any server can answer any request. We must use sticky sessions: all audio for one call goes to the same orchestrator instance for the entire call.

The orchestrator fleet

One orchestrator server can handle a few hundred calls, because the orchestrator itself is light and the heavy lifting is on the GPU services. We run many such servers behind a load balancer, a server that distributes incoming calls across our servers. It assigns each new call to the server with the least load and keeps it there for the whole call.

Before you move anything, guess: at 10,000 concurrent calls and a few hundred calls per server, how many orchestrator servers is that? Then switch the user to another continent and watch the latency tile.

GPU services

STT, LLM and TTS run as separate services, each with its own pool of GPUs, each autoscaled independently. Autoscaled means GPUs are added or removed automatically based on load. STT and TTS models are small and can serve many streams per GPU. The LLM is the biggest cost. For it we use techniques like continuous batching, where many calls share one GPU efficiently, and a fast inference engine, the software that runs the model.

load balancer  -->  orchestrator 1 .. N   (light, a few hundred calls each, sticky)
                          |        |        |
                          v        v        v
                      STT pool   LLM pool   TTS pool     (GPUs, each pool autoscales on its own)
                      small,     biggest    small,
                      many       cost       many
                      streams               streams

Regional deployment

Sound travels around the world at network speed, and a user in India talking to a server in the US adds 200 to 300 ms round trip. That kills our budget. So we deploy the full stack (gateway, orchestrator, STT, LLM, TTS) in multiple regions, and route each call to the nearest one. The full stack, because of the co-location rule from Chapter 14.

budget                                800 ms
India to US round trip                200 to 300 ms
left for everything else              500 to 600 ms     below the budget's own best case of 600 ms

Failover

If an orchestrator server crashes, the calls on it are lost, unless the session state is checkpointed. We checkpoint to Redis after every turn, as in Chapter 17. The media gateway can then reconnect the audio stream to a new orchestrator, which loads the state and continues. The user hears a small pause, not a dropped call.

If the STT or TTS provider is down, we fail over to a second provider. This is one of the biggest advantages of the cascaded approach: the pieces are separate, so each piece can have a spare.

Graceful degradation

Under heavy load, instead of dropping calls, we degrade. We can switch to a smaller LLM, reduce the context, or, in the worst case, play "All our agents are busy, please hold" and queue the call.

Rate limits

If we use external providers, they have rate limits: calls per minute, concurrent streams. We must track our usage and reserve enough capacity, and have a backup provider.

Try it yourself

In the interactive, set the headroom to zero and ask: one server dies at peak, where do its few hundred calls go? Then set it to 25 percent and answer again. Then write one sentence on why the LLM pool, not the orchestrator fleet, is where the money goes.

Common questions

Why can the orchestrator not be stateless like a web server?

Because a call lasts minutes and the orchestrator holds the session state and the open connections the whole time. The fix is sticky sessions plus a checkpoint in Redis, so that state survives a crash without every server needing to hold every call.

Why deploy the whole stack per region and not just the gateway?

Because every hop between the orchestrator and a model adds 10 to 50 ms, and a hop across an ocean adds 200 to 300 ms round trip. The models have to be next to the orchestrator, and the orchestrator next to the user.

Carry this

• Calls are stateful and long-lived: sticky sessions, one orchestrator instance per call for the whole call.
• The orchestrator is light, a few hundred calls per server behind a load balancer. The GPU pools do the heavy work and each autoscales on its own.
• Deploy the full stack in every region. India to US adds 200 to 300 ms and kills the budget.
• Checkpoint to Redis after every turn; a crash becomes a small pause. Keep a second provider for STT and TTS.
• Under load, degrade before you drop: smaller model, shorter context, then "please hold".

Check yourself

1. Why must all audio for one call go to the same orchestrator instance?

Answer

Because the session is stateful and long-lived: the orchestrator holds the call's state and open connections for the whole call. That is what sticky sessions guarantee.

2. An orchestrator server dies with 300 calls on it. What do those callers hear?

Answer

A small pause. The media gateway reconnects each audio stream to a new orchestrator, which loads the session state checkpointed to Redis after the last turn.

3. Load spikes beyond what the GPU pools can serve. List the three steps of graceful degradation in order.

Answer

Switch to a smaller LLM, reduce the context, and in the worst case play "All our agents are busy, please hold" and queue the call.