The brain: LLM with tools

In this chapter, we will learn about the brain of the agent: the Large Language Model and its tools. We will read one tool definition and one tool call, meet the loop that turns a chatbot into an agent, see why we stream tokens and chunk them into sentences, and read the seven rules of a voice system prompt. By the end you will be able to explain why the user hears sentence one while the model is still writing sentence three.

Input and output

The LLM (Large Language Model) is the brain of the agent: it reads the conversation so far and decides what to say next or what action to take.

in    a system prompt            who the agent is, what it can do, how to speak
      the conversation history   everything said so far in this call, by user and agent
      the latest user transcript what the user just said (the final transcript from Chapter 8)
      a list of tools            functions the LLM is allowed to call

out   a text reply, or a tool call, or both

What a tool is

A tool is a function that we write and describe to the LLM, so that the LLM can ask us to run it. We give it a name, a plain description, and the input it needs. The LLM reads the description to decide when to use it.

{
  "name": "get_order_status",
  "description": "Get the current status of a customer order",
  "parameters": {
    "order_id": "string"
  }
}

When the user says "Where is my order 45678?", the LLM does not know the answer, but it knows there is a tool for it. So it outputs a tool call:

{
  "tool": "get_order_status",
  "arguments": { "order_id": "45678" }
}

The orchestrator runs the actual function that queries the order database, gets the result ("Shipped, arriving tomorrow"), and sends it back to the LLM, which writes the final reply: "Your order 45678 was shipped and will arrive tomorrow."

LLM  -->  tool call  -->  tool result  -->  LLM  -->  reply

This loop is what makes it an agent instead of a chatbot.

A voice agent also needs special tools that control the call itself: end_call, to hang up politely when the work is done; transfer_to_human; and send_sms, to send a link or a confirmation to the user's phone. Without end_call, the agent never knows when to say goodbye.

Streaming tokens

An LLM generates its reply one token at a time, and we must not wait for the full reply before sending it to Text-to-Speech. A token is a small piece of text, roughly a word or part of a word. We stream the tokens as they are generated. The number that matters most is Time To First Token (TTFT): how long the LLM takes to produce the first token after we send the request. For a voice agent, we want this under 300 ms.

Sentence chunking

Text-to-Speech works best on complete sentences, because it needs to know where the sentence ends to get the rhythm right. So the orchestrator collects streamed tokens until it sees a sentence boundary (a full stop, a question mark, or a comma after a long phrase) and sends that sentence to TTS. While TTS speaks the first sentence, the LLM generates the second. This overlap is the key trick that keeps the latency low.

Time      ----------------------------------------------------->
LLM       [ sentence 1 ][ sentence 2 ][ sentence 3 ]
TTS                    [ audio 1    ][ audio 2    ][ audio 3    ]
Playback                     [ play 1      ][ play 2      ][ play 3      ]

The user hears sentence 1 while the LLM is still writing sentences 2 and 3. Without the overlap, the user would wait for the whole reply to be written and converted before hearing anything, adding seconds to every turn. Guess how much later the first sound arrives with the overlap off, then check.

The system prompt for a voice agent is different

The LLM was trained mostly on text, so by default it writes like a text chatbot: long paragraphs, bullet points, markdown, tables. All of that is terrible spoken out loud. So the system prompt must include rules like these.

• Reply in one or two short sentences. This is a phone call, not an essay.
• Never use bullet points, numbered lists, markdown, or emojis.
• Say numbers in words as they are spoken. Say "forty five dollars" not "$45".
• Ask one question at a time.
• If you need to call a tool that takes time, first say a short phrase like "Sure, let me check that for you."
• Confirm important details like order numbers, dates, and names by repeating them back.
• If you do not know something, say so and offer to transfer to a human.

Choosing the model

Bigger models reason better but are slower, and a smart answer that arrives 3 seconds late is a bad answer on a phone call. So a fast and small model is often the right choice. These techniques help.

• Prompt caching. The system prompt and tool definitions are the same every turn. Most LLM providers can cache them so only the new part is processed. This cuts TTFT a lot.
• Keep the context short. Fewer tokens in, faster the response.
• Model routing. A small model for simple turns ("Yes", "Thank you", "Hold on") and a bigger model only when a tool call or complex reasoning is needed.
• Multiple specialized agents. One agent for billing, one for technical support, one for bookings, each with a short prompt and a few tools, and a router that hands the call to the right one. Short prompts mean faster and more accurate replies.
• Co-location. Run the LLM in the same datacenter as the orchestrator to avoid network hops.

Now we have the reply text. Chapter 10 turns it into voice.

Try it yourself

Ask a text chatbot "What is your return policy?" and read its answer out loud. Then rewrite it under the seven rules above.

Common questions

Why does the LLM need a tool for the order?

It does not know our order database. Only a tool that queries it can tell the LLM the status.

Is a bigger LLM better for a voice agent?

Not always. Bigger reasons better but slower. A fast small model is often right, with routing to a bigger one only for tool calls or complex reasoning.

Carry this

• In: system prompt, history, latest transcript, tools. Out: text, a tool call, or both.
• LLM, tool call, tool result, LLM, reply: this loop makes an agent.
• Stream tokens, TTFT under 300 ms, chunk into sentences so TTS speaks one while the LLM writes the next.
• Voice prompt: short, no lists, numbers in words, one question, fillers, confirm details, offer a human.
• Small fast model, prompt caching, short context, routing, specialized agents, co-location.

Check yourself

1. Name the three call-control tools. Why does the first matter?

Answer

end_call, transfer_to_human and send_sms. Without end_call, the agent never knows when to say goodbye.

2. What is TTFT and what is our target?

Answer

Time To First Token, from sending the request to the first token back. Under 300 ms.

3. Why does prompt caching cut TTFT?

Answer

The system prompt and tool definitions are the same on every turn, so the provider caches them and processes only the new part.