Part IV · The hard parts
Chapter 16
Tool calling in a voice agent
Six problems that tools bring when the conversation is spoken: slow tools, failing tools, risky actions, exact values, speech during a call, and long results.
On this page
In this chapter, we will learn the special problems that tools bring when the agent is on a phone call rather than in a text box. There are six of them. Each one has a solution, and most of the solutions are a sentence in the system prompt, a rule in the orchestrator, or both. By the end you will be able to name the six and say which ones the prompt can fix and which ones only code can fix.
Problem 1: Tools are slow
A database lookup takes 100 ms, but an external API can take 2 to 5 seconds. Silence for 3 seconds on a phone call feels like the call dropped. The agent must say something before the tool runs: "Let me pull up your account, one moment." For very long tools we can even play light hold music or a soft "typing" sound. And we must set a timeout. If the tool takes more than, say, 8 seconds, the agent says "This is taking longer than usual, please stay with me" and keeps waiting, or gives up gracefully.
database lookup about 100 ms fast external API 2 to 5 s silence feels like a dropped call: say a filler first timeout 8 s "This is taking longer than usual, please stay with me"
Problem 2: Tools fail
The API returns an error, and the agent must not read the error out loud. Never "Error 500 internal server error". The system prompt must tell it to say "I am unable to fetch that right now. Would you like me to try again or connect you to a person?"
Problem 3: Risky actions need confirmation
Before cancelling an order or making a payment, the agent must confirm and wait for a clear
yes. "Just to confirm, you want me to cancel order 4 5 6 7 8, is that right?" We can enforce
this in code, not just in the prompt, by marking certain tools as requires_confirmation.
Then the orchestrator refuses to run the tool until a confirmation turn has happened, whatever the LLM
decides.
Problem 4: Tools need exact values
The tool needs order_id = "45678", but STT may have given "four five six seven eight" or "45 678". The LLM must normalize it into the exact value the tool expects. For critical values, the agent reads it back for confirmation before calling the tool, which also catches an STT mistake before it becomes a wrong lookup.
STT heard: "four five six seven eight" or "45 678" Tool needs: order_id = "45678" Agent: I heard order four five six seven eight, is that correct? User: Yes. Then: get_order_status(order_id="45678")
Problem 5: The user speaks during the tool call
If the user says "Actually, never mind" while the tool is running, the orchestrator must cancel the tool call, or ignore its result, and process the new input. This is the same yield rule as Chapter 15, applied to a tool instead of to speech. A result that arrives after the user changed their mind must not be spoken.
Problem 6: Long tool results
A tool returns a list of 20 items, and the agent must not read all 20. The prompt must say to summarize and offer: "I found 20 orders. The most recent one is from yesterday. Do you want to hear about that one?"
Prompt or code?
| Problem | Fixed by |
|---|---|
| Slow tools | Prompt (say a filler first) and code (timeout at 8 s, hold sound) |
| Failing tools | Prompt (friendly message, offer to retry or transfer) |
| Risky actions | Code (requires_confirmation on the tool) and prompt (read it back) |
| Exact values | LLM normalizes, prompt (read back critical values) |
| Speech during a tool call | Code (cancel or ignore the result) |
| Long results | Prompt (summarize and offer) |
Try it yourself
Take a tool you would give the order agent, say cancel_order. Write the one sentence the agent must say before calling it, the one sentence it says if the call fails, and whether you would mark it requires_confirmation. Then do the same for get_order_status. Notice that the second one still gets the filler, as in the Chapter 11 walk-through, but needs no confirmation.
Common questions
Why can the filler not wait until we know the tool is slow?
Because by then the silence has already started. The prompt rule says the filler comes first, before the tool runs, so the user hears something within the latency budget whatever the tool does.
Who should normalize "four five six seven eight", the STT or the LLM?
The post puts it on the LLM: it must normalize the value into what the tool expects. STT spelling and digit modes from Chapter 8 help, and reading the value back is the safety net for critical ones.
Carry this
- Say something before a slow tool. External APIs take 2 to 5 s; time out at about 8 s.
- Never read an error aloud. Offer to retry or connect to a person.
- Risky tools carry requires_confirmation in code, and the agent reads the details back.
- Tools need exact values. Normalize, and confirm critical ones before the call.
- Speech during a tool call cancels it. Long results get summarized and offered, never read in full.
Check yourself
1. An external API usually takes 4 seconds. What does the caller hear during those 4 seconds, and why?
Answer
A filler like "Let me pull up your account, one moment", possibly followed by light hold music. Three seconds of silence on a phone feels like the call dropped.
2. Why mark cancel_order as requires_confirmation in code when the prompt already says to confirm?
Answer
The prompt is a request the LLM might not follow. The code flag makes the orchestrator refuse to run the tool until a clear yes has been given.
3. A tool returns 20 orders. What does the agent say?
Answer
"I found 20 orders. The most recent one is from yesterday. Do you want to hear about that one?" It summarizes and offers, instead of reading all 20.