Part II · The five components
Chapter 6
Audio transport
How sound becomes numbers, why we cut it into 20 ms chunks, what a codec does, the three ways to carry audio over a network, and the echo that makes an agent interrupt itself.
On this page
In this chapter, we will learn how sound travels from the user's mouth to our server, before any AI model touches it: how a microphone turns air into numbers, why we cut them into 20 millisecond chunks, how a codec shrinks them, and which of three network transports to use. We will also meet the echo that breaks interruption handling. By the end you will be able to say why WebRTC and not WebSocket for a phone in a pocket.
How sound becomes data
A microphone measures air pressure thousands of times per second and gives a number each time. These numbers are called samples. How many samples we take per second is the sample rate.
| Use | Sample rate | Note |
|---|---|---|
| Phone calls | 8,000 per second (8 kHz) | This is why phone voices sound a bit muffled |
| Speech AI models | 16,000 per second (16 kHz) | Good enough for speech |
| Music | 44,100 or 48,000 per second | Overkill for voice |
Each sample is stored as a 16-bit number, 2 bytes. This raw list of numbers is called PCM (Pulse Code Modulation), the simplest form of digital audio.
Chunks
We do not send audio one sample at a time, and we do not wait for the whole sentence either. We cut the audio into small chunks, usually 20 milliseconds each. These chunks, also called frames, are sent continuously. Every component in our system works on them as they arrive. This is what makes real-time possible.
samples per chunk 16,000 per second x 0.020 s = 320 samples chunks per second 1,000 ms / 20 ms = 50 chunks per second
Codecs
Raw PCM is big. A codec compresses the audio before sending and decompresses it after receiving. The most common one for real-time voice on the internet is Opus. It is designed for speech, works well even when some packets are lost, and gives good quality at 24 to 32 kbps. Phone networks use older codecs like G.711.
Before you move anything, guess how many samples a 20 ms chunk has at 8 kHz, the phone rate.
Transport options
There are three main ways to send the chunks over a network.
Option 1: WebSocket
A WebSocket is a long-lived two-way connection between client and server, built on top of TCP. TCP is the internet protocol that guarantees every packet (one small parcel of data on the network) arrives, in the correct order. That sounds good, but for real-time audio it has a problem. If one packet is lost, TCP stops and waits for it to be re-sent before delivering the later packets. This is called head-of-line blocking. In a voice call, a 20 ms chunk that arrives 500 ms late is useless. We would rather skip it and keep going.
WebSocket is simple and good for server-to-server audio, for example our orchestrator talking to the STT service inside our datacenter, where the network is reliable. It is not the best choice for a user on a mobile network.
Option 2: WebRTC
WebRTC is built into every modern browser and mobile platform specifically for real-time audio and video calls. It uses UDP instead of TCP. UDP does not wait for lost packets, it just keeps going. That is exactly what we want for voice. WebRTC also comes with a full toolbox:
- Jitter buffer. Packets arrive at uneven timing over the internet. The jitter buffer holds them for a few milliseconds and plays them smoothly.
- Packet loss concealment. If a chunk is lost, it fills the gap with a guess based on the neighbouring chunks, so we hear a tiny glitch instead of a silence.
- Acoustic Echo Cancellation (AEC). Very important, discussed just below.
- Noise suppression and automatic gain control, which keeps the volume steady.
- Encryption by default.
For mobile apps and browsers, WebRTC is the right choice.
Option 3: SIP and RTP (telephony)
When the user calls from a normal phone, the audio comes through the telephone network, the PSTN (Public Switched Telephone Network). A telephony provider receives the call and forwards the audio to us using SIP (Session Initiation Protocol, for setting up and ending the call) and RTP (Real-time Transport Protocol, for carrying the audio). Chapter 18 is about telephony.
Acoustic Echo Cancellation
When our agent speaks, its voice comes out of the user's speaker, and the user's microphone sends it back to our server. Our system hears the agent's own voice and thinks the user is speaking. The agent interrupts itself. This is called echo.
Acoustic Echo Cancellation solves this. The system knows exactly what audio it is playing on the speaker, so it subtracts that from what the microphone hears. What remains is only the user's real voice. WebRTC does this automatically on the client. On phone calls, the phone network handles most of it. If we build our own client, we must make sure AEC is on, otherwise interruption handling will never work correctly.
Why send audio to the server at all?
Why not run everything on the phone? Because good STT, LLM and TTS models are big and need GPUs. A phone cannot run them at the quality we need. So we stream the audio to the server. The only thing we can run on the device is a small Voice Activity Detection model, the subject of Chapter 7. The post points to its separate blog on cloud versus on-device model deployment for the general rule.
Try it yourself
Put a video call on speaker and turn the volume up. If the other side hears themselves, that is echo. Now imagine our agent hearing itself: it would stop mid-sentence to "listen" every time it spoke.
Common questions
If WebSocket is bad for audio, why does the orchestrator use it to talk to STT?
Inside the datacenter packets are rarely lost, so head-of-line blocking rarely happens. WebSocket is simple, and simple wins there.
Does echo cancellation matter if we never let the user interrupt?
Interruption is a requirement from Chapter 3. Without AEC the agent hears its own voice as the user and stops itself.
Carry this
- Samples at 8 kHz for phones, 16 kHz for speech models, 2 bytes each: raw PCM.
- 20 ms chunks, 320 samples at 16 kHz, 50 per second. Everything works on chunks as they arrive.
- Opus at 24 to 32 kbps on the internet; G.711 on phone networks.
- WebSocket inside the datacenter, WebRTC for apps and browsers, SIP and RTP for phones.
- AEC subtracts the agent's own voice from the microphone. Without it, interruptions never work.
Check yourself
1. What is head-of-line blocking, and why does it hurt a voice call?
Answer
TCP waits for a lost packet before delivering later ones. A chunk that arrives 500 ms late is useless, so we would rather skip it.
2. How does Acoustic Echo Cancellation know what to remove?
Answer
The system knows exactly what audio it is playing on the speaker, so it subtracts that from what the microphone hears. Only the user's real voice remains.
3. How many samples are in a 20 ms chunk at 16 kHz, and how many chunks arrive per second?
Answer
320 samples, and 50 chunks per second.