High-level architecture

In this chapter, we will learn the shape of the whole system before we open any one part. The best way is to follow one call. A user calls our support number and says, "Where is my order 4 5 6 7 8?" We will trace the six steps that must happen, draw the boxes they pass through, and list the small set of messages that the client and the server send each other. By the end you will be able to draw the architecture from memory and name the two messages that make interruptions work.

Six steps for one question

Every turn of a voice call passes through the same six steps.

1. The user's voice is captured and streamed to our server as small audio chunks.
2. The system detects that the user is speaking, and later detects that the user has stopped.
3. The audio is converted to text: "Where is my order 45678?"
4. The LLM reads the text, decides it needs to look up the order, calls the get_order_status tool, gets the result, and writes a reply: "Your order 45678 was shipped yesterday and will arrive tomorrow."
5. The reply text is converted to speech.
6. The speech audio is streamed back to the user.

The boxes

   User (Phone / Mobile App / Browser)
                 |
                 |  audio stream in both directions
                 |  (SIP/RTP for phone, WebRTC for app and browser)
                 v
   +-----------------------------------+
   |        Media Gateway (Edge)       |
   |  handles call setup, codecs,      |
   |  echo cancellation, jitter buffer |
   +-----------------------------------+
                 |
                 |  clean audio chunks (20 ms each)
                 v
   +-----------------------------------+
   |    Voice Agent Orchestrator       |
   |     (one session per call)        |
   |                                   |
   |  Voice Activity Detection (VAD)   |
   |  Turn Detection                   |
   |  Speech-to-Text (streaming)   --> |---> STT Service (GPU)
   |  LLM + Tools (streaming)      --> |---> LLM Service (GPU)
   |  Text-to-Speech (streaming)   --> |---> TTS Service (GPU)
   |  Interruption Handling            |
   +-----------------------------------+
          |                  |
          v                  v
   +--------------+   +---------------------------+
   | Tools / APIs |   | Storage                   |
   | (order DB,   |   | (recordings, transcripts, |
   |  booking,    |   |  logs, metrics)           |
   |  CRM, etc.)  |   +---------------------------+
   +--------------+

Media Gateway. This is the entry point. It talks to the phone network or to the app, accepts the call, and converts audio into a clean and standard format that our system understands. It also does audio clean-up like echo cancellation and noise suppression. Each of these is explained in Chapter 6.

Voice Agent Orchestrator. This is the heart of the system. For every call, one orchestrator session is created. A session is the living record of one call, with its state and open connections. The orchestrator receives audio chunks, decides when the user has finished speaking, sends audio to Speech-to-Text, sends text to the LLM, sends the LLM's reply to Text-to-Speech, and streams the audio back. It also handles interruptions. It is the conductor: it plays no instrument, but it makes everyone play at the right time.

STT, LLM, TTS Services. These are the three AI models. They run on GPUs and are shared by many calls. Each gets its own chapter: 8, 9 and 10.

Tools / APIs. The real business systems, like the order database, the booking system, or the CRM (the customer records system). An API is simply a way for one program to ask another program for something.

Storage. Recordings, transcripts, logs, and metrics for debugging, compliance, and improvement.

The messages between the client and the orchestrator

The client (the app or the media gateway) and the orchestrator talk using a small set of messages. They will come up again and again, so here they are in one table.

Keep playback_progress and clear_audio_buffer in mind especially. They are the two messages that make interruption handling work, as Chapter 15 shows.

Step through the call

Before you press Step, guess which of the six steps sends the most messages between client and orchestrator, and in which direction.

Two things to notice. The orchestrator is active in every step, even though it never converts a sound or thinks a thought. And audio chunks are flowing in during the whole walk, not only in step 1, because the stream never stops.

Try it yourself

Close this page and draw the boxes on paper: the user, the gateway, the orchestrator with its six jobs, the three GPU services, tools and storage. Then draw the eight messages as arrows between the client and the orchestrator. This drawing is the first ten minutes of the interview in Chapter 24.

Common questions

Why is the orchestrator a separate box from the three models?

It holds the state of one call and coordinates. The models are heavy workers on GPUs, shared across many calls. Keeping them apart lets each scale on its own, which is Chapter 19.

What is "one session per call"?

A session is the living record of one call: its state and its open connections to STT, LLM and TTS. It is created when the call starts and lives until the call ends.

Why does the client report playback progress?

Because the orchestrator sends audio ahead of time and the client buffers it. Only the client knows how much the user has really heard. When the user interrupts, that number tells us where to cut the conversation history.

Carry this

• Six steps: capture, detect speech and its end, speech to text, LLM with a tool call, text to speech, stream back.
• Media gateway at the edge; one orchestrator session per call; STT, LLM and TTS on shared GPUs; tools; storage.
• The orchestrator is the conductor: it plays no instrument, it makes everyone play at the right time.
• Eight messages. Remember playback_progress and clear_audio_buffer: they make interruptions work.

Check yourself

1. What does the media gateway do before audio reaches the orchestrator?

Answer

It accepts the call from the phone network or the app, converts the audio into a clean standard format, and does clean-up like echo cancellation and noise suppression.

2. Name the six jobs listed inside the orchestrator box.

Answer

Voice Activity Detection, turn detection, streaming speech-to-text, LLM with tools, streaming text-to-speech, and interruption handling.

3. Which two messages make interruption handling work, and which way does each travel?

Answer

playback_progress goes from client to orchestrator; clear_audio_buffer goes from orchestrator to client.