Part IV · The hard parts
Chapter 18
Telephony
Connecting the agent to a real phone number: PSTN and SIP, inbound and outbound calls, key presses, cold and warm transfers, 8 kHz audio, and the recording announcement.
On this page
In this chapter, we will learn what it takes for the agent to answer a normal phone number, and to dial one. Most business voice agents must work on the phone, and the phone network is older and lower quality than an app. We will meet the providers that bridge the two worlds, the two extra problems of calling people, the key presses that still matter, the two kinds of transfer, and the legal sentence at the start of the call.
PSTN and SIP
The PSTN (Public Switched Telephone Network) is the traditional phone network, and to connect it to our software we use a telephony provider. Companies like Twilio, Vonage and Plivo do this, or we can use a direct SIP trunk, which is a direct connection to the phone network. When someone dials our number, the provider receives the call and opens a media stream to our media gateway using SIP (for call control) and RTP (for audio).
caller's phone --> PSTN --> telephony provider --SIP (call control)--> media gateway --> orchestrator
--RTP (audio)-------->
Inbound and outbound
Inbound is when the user calls us. Outbound is when our agent calls the user, for example for appointment reminders. Outbound has two extra problems that inbound never has.
First, answering machine detection. If a voicemail picks up, the agent must detect it and either leave a message or hang up, instead of talking to a recording for two minutes.
Second, who speaks first. On an outbound call, the agent must wait for the person to say "Hello" before starting, just like a human caller does.
Inbound: [call connects] Agent: Hello, thank you for calling... Outbound: [call connects] ...silence... Person: Hello? Agent: Hi, this is a reminder about... Outbound: [call connects] Voicemail greeting detected --> leave a message, or hang up
DTMF: the key presses
DTMF (Dual-Tone Multi-Frequency) tones are the key press tones, as in "Press 1 for sales". A voice agent must still support them, for two reasons. Some users prefer keys. And some inputs, like a PIN, are safer typed than spoken. The card number case in Chapter 20 uses the same idea: collect it via DTMF keys and keep it out of the recording and the logs.
Transfer to a human
When the agent cannot help, or the user asks for a human, we transfer, and there are two kinds. A cold transfer just connects the user to a human queue. A warm transfer first briefs the human agent with a summary of the conversation so far, so the user does not have to repeat everything. The summary is generated by the LLM and sent to the human agent's screen.
| Transfer | What happens | What the user feels |
|---|---|---|
| Cold | Connect the user to a human queue | "Let me start again from the beginning..." |
| Warm | LLM writes a summary, it appears on the human's screen, then the user is connected | The human already knows the order number and the problem |
The transfer itself is a tool, transfer_to_human, one of the call-control tools from Chapter 9.
Audio quality
Phone audio is 8 kHz and uses old codecs, so it is lower quality than app audio. Our STT model must be trained or tested on phone audio, and we may upsample, converting 8 kHz to 16 kHz, before STT, because 16 kHz is the sample rate speech models expect, as we saw in Chapter 6.
phone line 8,000 samples per second old codecs such as G.711, a bit muffled speech AI models 16,000 samples per second what STT expects upsample 8 kHz --> 16 kHz before STT; test the STT on real phone audio
Call recording consent
In many countries, we must announce "This call may be recorded" at the start. This announcement is a pre-generated audio file, the same trick as the cached greeting in Chapter 14.
Try it yourself
Call any large company's support line and listen for four things from this chapter: the recording announcement, a "press 1" menu, the moment it asks for a PIN, and what happens when you ask for a person. Count how many of the four are there. Then imagine the same call as an outbound reminder and write down the one extra thing the agent must wait for.
Common questions
Do we have to use a provider? Can we connect to the phone network ourselves?
The article lists both: a provider such as Twilio, Vonage or Plivo, or a direct SIP trunk, which is a direct connection to the phone network. Either way the call reaches our media gateway as SIP for control and RTP for audio.
Why does the agent wait on an outbound call but not on an inbound one?
Because that is what a human caller does. The person who was called says "Hello" first. On an inbound call, the caller is waiting for us to speak.
Carry this
- A telephony provider (or a direct SIP trunk) bridges the PSTN to our media gateway: SIP for call control, RTP for audio.
- Outbound calls add two problems: answering machine detection, and waiting for the person's "Hello".
- Keep DTMF. Some users prefer keys, and a PIN is safer typed than spoken.
- Warm transfer: the LLM writes a summary for the human's screen so the user does not repeat everything.
- Phone audio is 8 kHz with old codecs: test STT on it, upsample to 16 kHz. The recording announcement is a pre-generated file.
Check yourself
1. An outbound reminder call connects. What must the agent do before speaking?
Answer
Wait for the person to say "Hello", like a human caller would, and check whether a voicemail picked up instead, in which case leave a message or hang up.
2. What makes a transfer warm rather than cold?
Answer
The LLM generates a summary of the conversation so far and it is sent to the human agent's screen before the user is connected, so nothing has to be repeated.
3. Why might we ask for a PIN by key presses rather than by voice?
Answer
Some inputs are safer typed than spoken, and a PIN is one of them. The card number case does the same: collect via DTMF keys and keep it out of the logs.