Requirements and the one metric

In this chapter, we will learn what exactly we are building, written as a list of requirements. In a system design interview, we must always start by clarifying the requirements, so we will do that here: seven things the agent must do, seven ways it must behave, and then the one metric that matters more than any other. By the end you will be able to define Voice-to-Voice Latency in one sentence.

Functional requirements

A functional requirement says what the system must do. Here is our list.

• The user speaks, and the agent listens, understands, and replies by voice.
• The agent holds a multi-turn conversation, which means it remembers what was said earlier in the same call.
• The agent can take actions using tools, like looking up an order, booking a slot, or sending a confirmation message.
• The user can interrupt the agent while it is speaking, and the agent must stop and listen.
• The agent works over a normal phone call and also inside a mobile app or a web browser.
• The agent can transfer the call to a human when needed.
• The agent supports multiple languages. We can keep this one as a stretch goal.

Each line is a chapter waiting to happen. Tools are Chapter 9 and Chapter 16. Interruptions are Chapter 15. Phone calls are Chapter 18. Remembering the call is Chapter 17.

Non-functional requirements

A non-functional requirement says how well the system must do it. These are the numbers and the promises.

Notice the word "never" in the latency line and in the reliability line. A number that is usually good is not enough. The caller remembers the one slow turn and the one sentence that stopped halfway. Chapter 14 turns this into a rule about how to measure.

The most important metric

Voice-to-Voice Latency is the time between the moment the user stops speaking and the moment the user hears the first sound of the agent's reply. For a voice agent, this is the most important metric. We will keep coming back to this number. Everything we design is to keep this number small while keeping the answer correct.

user's last word                                  first sound of the reply
      |                                                     |
      |◄──────────── Voice-to-Voice Latency ───────────────►|
      |                                                     |
   user stops                                        user hears
   speaking                                          the agent

Three details in that definition are easy to miss.

• It starts when the user stops. Not when our server notices they stopped. The time we spend deciding that they are done is inside the metric.
• It ends at the user's ear. Not when our server sends the first audio chunk. The network on the way back is inside the metric too.
• It ends at the first sound. Not the end of the reply. A long answer is fine as long as it begins quickly.

Because the clock starts and stops at the user, the only honest place to measure it is on the user's device. We will see in Chapter 21 that this is exactly where production systems measure it.

Why one metric

We have fourteen requirements and one metric. That is on purpose. When two good designs disagree, we need a tie-breaker that everyone agrees on. Every later chapter will make a trade: a bigger model or a faster one, a safer wait or a quicker reply, a cheaper transport or a smoother one. The tie-breaker is always the same question: what does it do to Voice-to-Voice Latency, and is the answer still correct?

Try it yourself

Pick one of the five examples from Chapter 1, say the drive-thru agent. Go through the seven functional requirements and decide which ones it needs and which it does not. Then pick the non-functional one that would hurt the business most if it failed. There is no single right answer, which is why interviews start with this step.

Common questions

Why is "multiple languages" only a stretch goal?

Because it multiplies the work: the post uses different speech-to-text and text-to-speech models per language, and the LLM must be told which language is in use. We design for one first and keep the door open. Chapter 20 covers a caller who switches language mid-call.

Is 99.9% availability a lot?

It means the system may be down for about one part in a thousand of the time. The point of the requirement is the sentence after it: a dropped call is a very bad experience, worse than a slow page would be for a website.

Why measure latency at the user and not at the server?

Because the definition starts when the user stops speaking and ends when the user hears the reply. A server clock misses the network in both directions, and the network is part of what the user feels.

Carry this

• Functional: listen and reply by voice, multi-turn, tools, interruptions, phone and app, transfer to a human, languages as a stretch goal.
• Non-functional: 800 ms and never more than 1.5 s, 99.9% up, thousands of calls at once, never go silent, low cost per minute, protect voice data, see every call step by step.
• Voice-to-Voice Latency: from the moment the user stops speaking to the moment they hear the first sound of the reply.
• Every design decision is judged against that one number, while keeping the answer correct.

Check yourself

1. Define Voice-to-Voice Latency without looking.

Answer

The time between the moment the user stops speaking and the moment the user hears the first sound of the agent's reply.

2. Our latency requirement has two numbers. What are they, and what does each mean?

Answer

800 milliseconds is the target for starting to speak after the user stops. 1.5 seconds is the ceiling that must never be crossed.

3. Which non-functional requirement says "never leave a sentence half-finished"?

Answer

Reliability. The agent must never leave a sentence half-finished or go silent without reason.