Memory and context

In this chapter, we will learn how a voice agent remembers. There are four things to remember, in four places: what was said earlier in this call, who the caller is from earlier calls, what the company knows, and where we are in the workflow right now. By the end you will be able to say what gets sent to the LLM on turn forty of a long call, and why it is not everything.

Within a call: the conversation history

The conversation history is kept in the orchestrator's session memory and sent to the LLM on every turn. For a long call, say 30 minutes, this history becomes long, which makes the LLM slow and expensive. The LLM has no memory of its own between turns, so everything it should know about this conversation must be in the prompt, every single turn.

The solution is summarization. After every 10 to 15 turns, we ask a small model to summarize the older part of the conversation into a few lines and replace the old turns with the summary. We keep the recent turns as they are, word for word, because the next reply depends on them most.

turn 1 .. turn 12      -->  [summary: caller is asking about a delayed order, number confirmed]
turn 13 .. turn 24     -->  [summary: ...]
turn 25 .. turn 30     kept word for word
                       all of the above is the prompt for turn 31

Before you move anything, guess: on turn 40 of a call, with 150 tokens per turn (an illustrative number, not the post's), how many tokens would the history be with no summarization at all? Then turn summarization on and watch the line.

Across calls: the customer profile

When a known customer calls again, the agent must know who they are. This is done by looking up the customer profile, by phone number or by authentication, and putting a short summary in the system prompt. For example: "The caller is Amit, a premium customer, last called 3 days ago about a delayed order 45678, which is now delivered." One sentence, and the agent can open with "Hi Amit, is this about order 45678?" instead of "Please tell me your name".

The post points to its separate blog on AI Agent Memory for short-term and long-term memory in depth.

Knowledge: the base we search, not paste

The agent must answer questions from a knowledge base, and we do not put the whole knowledge base in the prompt. We use Retrieval-Augmented Generation (RAG): a search_knowledge_base tool finds the few relevant paragraphs and gives them to the LLM. The LLM then answers from those paragraphs, not from its own guesses.

In voice, the latency of this search matters. The search index must be fast, under 100 ms, so that the lookup fits inside the budget of Chapter 14.

whole knowledge base in the prompt     every turn pays for every page, slow and expensive
RAG, few paragraphs per question       search under 100 ms, then only those paragraphs in the prompt

Session state: where we are right now

Along with the conversation, the orchestrator keeps structured state: the verified customer ID, the order being discussed, and which step of a workflow we are in. This is not prose for the LLM. It is fields that code reads. It is stored in memory, and checkpointed, meaning a copy is saved, to a fast store like Redis after every turn. If the orchestrator server crashes, another server can pick up the call with the state intact. Chapter 19 shows the failover that makes use of this checkpoint.

Try it yourself

Write the profile line the agent would get for you, if you called your own favourite shop. One sentence: name, status, the last thing you called about, how it ended. Then write the three fields of session state at the moment you say "yes, cancel it". Those two short pieces of text are most of what "memory" means for a voice agent.

Common questions

Why a small model for the summary and not the main LLM?

Because the job is simple, a few turns compressed into a few lines, and a small model is fast and cheap. The post's rule for the whole design applies here too: use small and fast models wherever possible.

Does summarizing lose information?

Some, by design. The summary keeps what was agreed and drops the exact wording. The recent turns stay word for word, so the part that matters most for the next reply is never lost.

Why checkpoint to Redis after every turn and not at the end?

Because a crash happens in the middle of a call, not at the end. With a checkpoint after every turn, the new server loses at most the turn in progress.

Carry this

• Within a call: the history is sent on every turn. Every 10 to 15 turns, a small model summarizes the old part and the recent turns stay intact.
• Across calls: a one-line customer profile in the system prompt, found by phone number or authentication.
• Knowledge: RAG through a search_knowledge_base tool, with an index that answers in under 100 ms.
• Session state: verified customer ID, current order, workflow step, checkpointed to Redis after every turn.

Check yourself

1. A call reaches turn 30. What does the LLM receive as history?

Answer

A few lines of summary for the older turns, produced by a small model every 10 to 15 turns, plus the recent turns word for word.

2. Why does the return policy go through a tool instead of into the system prompt?

Answer

The whole knowledge base is too big to send on every turn. RAG searches it and sends only the few relevant paragraphs. The search must take under 100 ms.

3. The orchestrator server crashes mid-call. What makes it possible for another server to continue?

Answer

The session state was checkpointed to Redis after every turn. The new server loads it and carries on with the verified customer, the order and the workflow step intact.