Back-of-the-envelope estimation

In this chapter, we will learn how big the system is, with simple arithmetic that fits on the back of an envelope. We will take a company that gets 1 million calls per day, with an average call length of 5 minutes, and work out four things: how many calls are live at once, how much audio flows through the network, how much we store, and how much computing we need. By the end you will be able to redo every step with your own numbers.

Concurrent calls

Concurrent calls are the calls that are going on at the same moment. We get there in three steps: calls per second, then calls in progress, then the peak.

calls per day            1,000,000
calls per second         1,000,000 / 86,400 s     =  around 12 per second
call length              5 minutes                =  300 seconds
live at any moment       12 x 300                 =  3,600 calls
peak (3x the average)    3,600 x 3                =  around 10,000 concurrent calls

Calls are not spread evenly across the day, so at peak we assume 3 times the average: around 10,000 concurrent calls. This 10,000 is the number the rest of the book designs for.

Audio bandwidth

Sound is captured as numbers. A phone call is usually captured 8,000 times per second (we call it an 8 kHz sample rate). A good quality voice for AI models is 16,000 times per second (16 kHz). Each number is stored in 2 bytes.

16 kHz audio             16,000 x 2 bytes          =  32,000 bytes per second
in bits                  32,000 x 8                =  256,000 bits per second  =  256 kbps  (raw PCM)

This raw form is called PCM (Pulse Code Modulation). We do not send raw audio over the network. We compress it using a codec, a program that shrinks audio. A common codec called Opus can bring it down to around 32 kbps with good quality. Phone networks use a codec called G.711 at 64 kbps.

10,000 calls, both ways  10,000 x 2 x 64 kbps     =  1,280,000 kbps  =  1.28 Gbps

1.28 gigabits per second is easily handled by modern servers. Bandwidth is not the bottleneck. Chapter 6 goes deeper into samples, chunks and codecs.

Storage for recordings

If we record every call, the recordings are the big item; transcripts are tiny.

call-minutes per day     1,000,000 x 5 min        =  5,000,000 call-minutes
one minute of Opus       32 kbps                  =  around 240 KB
per day                  5,000,000 x 240 KB       =  1.2 TB per day
one 5-minute transcript  text only                =  around 5 to 10 KB

1.2 terabytes a day if we record everything. A 5-minute transcript, being text, is around 5 to 10 KB, so transcripts are small in comparison.

Compute

Compute is the real bottleneck. Every concurrent call needs three things at once:

• One streaming Speech-to-Text session.
• One LLM session generating tokens.
• One streaming Text-to-Speech session.

All three run on GPUs, the special processors that run AI models. So, for 10,000 concurrent calls, we need a large GPU fleet, and our cost is mainly GPU cost. This is why we will care a lot about using small and fast models wherever possible, a theme that returns in Chapter 9 and Chapter 23.

Move the numbers

Before you move anything, guess: if the average call were 10 minutes instead of 5, how many peak concurrent calls would we have? Then check.

Notice what moves what. Doubling the call length doubles the concurrent calls and the storage, but the calls per second stay the same. Switching the codec changes the bandwidth on the wire and nothing else, because recordings are always stored as Opus. Nothing on the sliders changes the compute need, except the concurrent count itself, because each live call needs its three GPU sessions whatever the codec.

Try it yourself

Redo the estimation for a small business: 1,000 calls a day, 3 minutes each. Work it out on paper first, then set the sliders. You should find that the peak is a handful of calls, which changes the whole scaling story in Chapter 19.

Common questions

Why multiply the average by 3 for the peak?

Because calls bunch up. A support line is busy in the morning and quiet at night. 3 times the average is the article's working assumption; your own traffic may be flatter or spikier.

Why is the bandwidth calculated with G.711 at 64 kbps and not Opus?

It is the worse case of the two, which is the safe way to size a network. Even at 64 kbps both ways, 10,000 calls is 1.28 Gbps, which modern servers handle easily.

If bandwidth and storage are fine, what are we actually paying for?

GPUs. Three model sessions per live call, times 10,000 live calls. Everything in the later chapters about small models, caching and ending dead calls quickly is about this one line.

Carry this

• 1 million calls a day is about 12 per second; times 300 seconds is 3,600 live; times 3 for peak is about 10,000 concurrent calls.
• 16 kHz, 2 bytes per sample is 256 kbps raw PCM; Opus brings it to about 32 kbps, G.711 is 64 kbps.
• 10,000 calls both ways at 64 kbps is 1.28 Gbps: easy. Recording everything is 1.2 TB a day.
• Each live call needs an STT, an LLM and a TTS session on GPUs. Compute is the bottleneck and the cost.

Check yourself

1. Walk from 1 million calls a day to 10,000 concurrent calls in three steps.

Answer

1 million a day is about 12 per second. Each lasts 300 seconds, so 12 x 300 = 3,600 are live at any moment. Peak is 3 times that, about 10,000.

2. Where does 256 kbps come from?

Answer

16,000 samples per second times 2 bytes is 32,000 bytes per second. Times 8 bits per byte is 256,000 bits per second, raw PCM.

3. Which of network, storage and compute is the real bottleneck, and why?

Answer

Compute. Every concurrent call needs a streaming STT session, an LLM session and a streaming TTS session, all on GPUs. 10,000 calls means a large GPU fleet, and that is where the cost is.