Part III · Journeyman
Chapter 12
Giving the model what it lacks
The weights are frozen, so everything new must arrive through the prompt. Repair three wrong answers of the last chapter by three methods: documents in the prompt, examples in the prompt, and a calculator the model can ask for.
On this page
In this chapter, we will learn how applications make a language model useful for real work. The model cannot be changed from outside, so all the methods do one thing: they put the right text into the prompt. We will repair three wrong answers from the last chapter, with retrieval and with a tool. With examples we will repair a fourth fault: an answer in the wrong form.
Two places where knowledge can be
A model can use only two kinds of knowledge: what is in its weights, and what is in the prompt. There is no third place.
| In the weights | In the prompt | |
|---|---|---|
| Where it comes from | training | the application, on every request |
| How much | a large part of what was written in public | as much as the context window holds |
| How fresh | ends at the knowledge cutoff | as fresh as this second |
| Private facts | none | whatever the application supplies |
| How exact | blurred, spread over many weights | exact, word for word |
| Cost of a change | a new training | a few tokens |
The right column wins on almost every line. This is why so much of the craft of building with language models is the craft of filling the prompt. That craft is now often called context engineering: deciding what goes on the desk.
Repair 1: put the facts in the prompt
Retrieval means finding the pieces of text that fit a question and placing them into the prompt, in front of the question. The full name of the method is retrieval-augmented generation, or RAG: generation, helped by retrieval.
In Chapter 11 the model did not know the name of Maya's cat and answered "Whisker". We asked again, with a few notes about Maya in the system prompt and one instruction.
System: Answer only from the notes. If the notes do not say, reply: I do not know.
Notes about Maya: Maya is vegetarian and has a serious peanut allergy. She lives in
Paris. Her cat is called Miso. Her partner Sam has a birthday on June 18.
For the question about the date, the system prompt held one line instead: "Today is Monday, 28 September 2026."
(b) Repair 1: put the facts in the prompt (retrieval)
the same private question, with her notes pasted in
asked "What is the name of Maya's cat? Reply with the name only."
answered 'Miso'
a question the notes cannot answer
asked "What is the name of Maya's dentist? Reply with the name only."
answered 'I do not know.'
today's date, with the date given
asked "What is today's date? Reply with the date only."
answered '28 September 2026'
Three answers, three right. The second is the most valuable. Without the instruction the model would have invented a dentist, as it invented a cat. With the instruction and the notes, it has a likely way to say that it does not know. The third repairs the wrong date of Chapter 11. Most assistants have the date in their system prompt for this reason.
In a real application nobody pastes notes by hand. The notes of a company may be a hundred thousand pages, and they do not fit into the prompt. So a search runs first.
the question ---> search the documents ---> the 4 passages that fit best
|
v
+-------------------------------------------------------------+
prompt = | instructions | the 4 passages | the question | ---> the model
+-------------------------------------------------------------+
The search is usually a search by meaning. An embedding model turns every passage into a vector, and the question too. The passages whose vectors point most nearly the same way as the question are taken. That is the cosine similarity of Chapter 5, used on whole passages. How such a search is built and made fast for a hundred million passages is the subject of Thinking in Vectors.
Repair 2: show examples
Few-shot prompting means placing a few solved examples into the prompt, so that the model continues the pattern. Each example is called a shot.
We wanted the feeling of a restaurant review as one of three words: positive, negative or neutral. We
sent each example as two messages, a review with the role user and its label with the role
assistant, as if the conversation had already taken place.
(c) Repair 2: show examples in the prompt (few-shot)
no examples
asked 'Classify the feeling of this review as one word. Review: The soup was cold and the waiter was rude.'
answered 'The feeling of this review can be classified as **"disappointed"**.'
example 'I loved every bite.' -> 'positive'
example 'Never again.' -> 'negative'
example 'It was fine, nothing special.' -> 'neutral'
the same review, after three examples
asked 'Review: The soup was cold and the waiter was rude.'
answered 'negative'
prompt tokens with the examples 95, without 41: the three examples cost 54 tokens
The first answer is not wrong. It is useless for a program, which expected one of three words and received a sentence with a fourth. After three examples the answer is the single word "negative".
We gave no rule. We did not write "use only these three words". The model saw a pattern three times and continued it. Remember what the model is: a machine that continues text. An example is a request in the language it knows best.
Examples are not free, because they travel with every request. We price them with the input price of gpt-6-luna from Chapter 10, and with 100,000 requests a day, a number made up for illustration.
three examples, as messages 54 tokens, measured sent with every request 54 tokens x 100,000 requests a day = 5,400,000 tokens a day at $0.10 per million 5.4 x $0.10 = $0.54 a day
Repair 3: give the model a tool
A tool is a function of ordinary software that the model can ask for. The model writes the request. The application carries it out. This is also called function calling.
In Chapter 11 the model multiplied 4837 by 9216 and got a number about ten times too large. We sent the
same question again, together with the description of one tool. The description is written in JSON and
travels in the request, in a field called tools. It gives the name of the function, what it
does, and the values it needs.
{
"type": "function",
"function": {
"name": "multiply",
"description": "Multiply two whole numbers exactly.",
"parameters": {
"type": "object",
"properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
"required": ["a", "b"]
}
}
}(d) Repair 3: give the model a tool
step 1 we send the question and the description of one tool: multiply(a, b)
step 2 the model does not answer. It writes a request: multiply({"a": 4837, "b": 9216})
step 3 OUR code runs the tool: 4837 x 9216 = 44577792
step 4 we send the result back, and the model writes: 'The product of 4837 and 9216 is 44,577,792.'
finish_reason of step 2: 'tool_calls'
Read the four steps carefully, because there is a common misunderstanding here.
application model
| question + description of the tool |
| ----------------------------------------> |
| | writes TEXT: multiply(4837, 9216)
| <---------------------------------------- |
| runs the function: 44,577,792 |
| question + request + result |
| ----------------------------------------> |
| | writes the answer, using the result
| <---------------------------------------- |
The model never runs anything. It writes a piece of text in an agreed form, which says
which function it would like and with which values. The application reads that text, decides whether to
obey, runs the function, and sends the result back as one more message, with the role tool.
The model did what it is good at:
it recognised that this question needs a calculator, and it found the two numbers in the sentence.
Tools are how an assistant searches the web, reads a calendar, queries a database or sends a message. In each case the model writes a request, and ordinary software does the work.
The agent: a loop around the loop
An agent is a program that lets a model call tools again and again, until the task is done.
the task
|
v
+-----------------------------------------------+
| the model reads everything so far | <------------------+
| it writes EITHER a request for a tool | |
| OR the final answer | |
+----------------------+------------------------+ |
| |
a request for a tool |
v |
the application runs the tool ---> the result is added to the prompt
|
the final answer
v
done
Chapter 1 had a loop that adds one token at a time. This is a second loop around it, which adds one result at a time. With every round the prompt grows, and so does the bill of Chapter 10. An agent that needs thirty rounds for a task sends the growing prompt thirty times.
Packing the prompt
The prompt of a real application is put together from many blocks, and the question of the user is one of the smallest. Instructions, examples, passages, tools and the conversation so far all arrive together.
Switch the blocks on, one at a time. Guess first: which share of the full prompt is the user's own question?
everything switched on system prompt 200 examples 150 retrieved documents 1,200 memories 75 tool descriptions 300 conversation so far 1,500 the question 20 total 3,445 tokens share of the question 20 / 3,445 = 0.6 %
The sizes are typical values that we chose for illustration. The proportion is the point: more than 99% of what the model reads was put there by the application.
And memory?
The block called "memories" holds facts about the user that were learned in earlier conversations, such as "Maya is vegetarian". The model did not store them, because it stores nothing. A separate system wrote them down after earlier conversations and finds the right three for today's question. How to build that system is the whole subject of the first book on this shelf, AI Agent Memory: Novice to Ninja.
When to change the weights instead
Fine-tuning changes the weights by further training, and it is the right tool when a behaviour is wanted that no prompt achieves. It is slower and dearer than the three repairs above, so it is the last thing to try.
| You need | Reach for |
|---|---|
| facts that are private, recent or too many to remember | retrieval |
| a fixed form of answer, or the style of your team | examples, then a clear system prompt |
| exact calculation, live data, actions in other systems | tools |
| facts about this one user, across conversations | memory |
| a special skill or tone, on thousands of requests, that examples do not reach | fine-tuning |
Try it yourself
Do retrieval by hand. Open an assistant and ask a question about yourself that it cannot know, such as "what should I cook for my guests on Saturday?". Note the answer.
Then open a new conversation. Paste in five lines of notes first: who is coming, who does not eat meat, who is allergic to what, how much time you have, what is in the kitchen. Ask the same question.
You have just been the search system. The model was the same, and only the desk was different.
The program behind this chapter is 05_wrong_and_fixed.py, parts (b), (c) and (d).
Common questions
Why not put everything into the prompt, now that windows hold a million tokens?
For three reasons. Every token is paid for on every request. A longer prompt is slower to read. And models make better use of a short prompt with the right passages than of a long one in which the right passage is buried. Retrieval is a way of choosing, and choosing stays useful however large the window is.
Is a prompt with instructions a kind of program?
In its effect, yes: it decides what the machine does. In its nature, no. A program is carried out exactly. A prompt shifts the chances of the tokens. A good instruction makes the wanted behaviour very likely and never certain, so an application must check what comes back.
How many examples are enough?
Often two to five. More examples help with difficult forms, and they cost tokens on every request. The examples should differ from each other, or the model will copy what they have in common, even when that was an accident.
What is MCP?
The Model Context Protocol is an open standard for describing tools and sources of data in one agreed way. A tool written once can then be used by many assistants. It standardises the description and the call. The principle is the one of this chapter: the model writes a request, and software carries it out.
Does retrieval stop the model from making things up?
It lowers the rate a great deal, and it makes checking possible, because the answer can name the passage it rests on. It does not reach zero. The model can still misread a passage or add something of its own. The instruction "answer only from the notes" helps, as our dentist showed.
Can the model decide by itself which tool to use?
Yes, that is its part. It reads the descriptions of the tools and the question, and writes a request for the tool that fits, or none. The quality of the descriptions matters as much as the quality of any other part of the prompt.
Carry this
- Knowledge is in the weights or in the prompt. The weights are frozen, so everything new comes through the prompt.
- Retrieval: search first, and put the passages that fit into the prompt. Allow the answer "I do not know".
- Few-shot: show a few solved examples, and the model continues the pattern.
- Tools: the model writes a request, and the application runs it and returns the result. The model never runs anything.
- An agent is a loop of such requests. In our packed prompt the user's question was 0.6% of the tokens.
Check yourself
1. A support assistant must answer from a handbook of 3,000 pages. Which method do you use, and why not the others?
Answer
Retrieval. The handbook is too large to send with every request and it changes over time. Examples teach a form and carry no facts. A tool could do it if the handbook had a search function, and that would be retrieval by another name. Fine-tuning would blur the facts into the weights and would have to be repeated after every change.
2. In the tool example, who multiplied 4837 by 9216?
Answer
Our own program, in step 3. The model wrote the request in step 2 and the final sentence in step 4. It never calculated.
3. A prompt has a system prompt of 300 tokens, retrieved passages of 2,000 tokens, and a question of 25 tokens. What share is the question?
Answer
total 300 + 2,000 + 25 = 2,325 tokens share 25 / 2,325 = 1.1 %