Monitoring and evaluation

In this chapter, we will learn how to know whether a memory system is working. We will choose the numbers to watch in production and build a small exam called a golden set. We will learn a routine that takes any wrong answer and names the part that caused it. We will also learn how much a score can move by luck, so that we do not celebrate noise.

Why memory fails quietly

When a web page breaks, we see an error. When memory breaks, we see a perfectly polite answer that happens to ignore what the user said last month. Nothing crashes. The secretary's mistakes are invisible and they are permanent, because a wrong card stays in the box until someone finds it.

So we cannot wait for errors. We have to go looking.

The dashboard: eight metrics

A metric is a number we record continuously, so that a change in it tells us something changed in the system. The class lists eight for a memory layer.

A healthy system has a shape. Here is what the write mix should roughly look like for a user we already know well.

heavy user, 2 sessions/day x 3 candidates      = 6 candidates per day
two years                  6 x 730             = 4,380 candidates
planned store                                    200 memories
absorbed by NOOP and UPDATE   1 - 200/4,380    = about 95% of candidates

This quick estimate ignores that events expire. Chapter 12 counts the expiry and arrives at about 93 percent. Either way, more than nine in ten.

For a new user almost everything is an ADD. After a year, most candidates are things we already know. If the ADD share stays high for old users, the store is filling with duplicates and stale facts, the problem of Chapter 12.

The garage: two layers of evaluation

Evaluation is running the system on fixed inputs with known correct results. The class splits it into two layers, one for each path.

layer 1   did the right memory reach the prompt?            READ PATH
          golden set: (question, memory that MUST be retrieved)
          no LLM needed: deterministic, fast, nearly free

layer 2   did extraction and consolidation do the right ops?  WRITE PATH
          golden set: (session transcript, expected ADD / UPDATE / NOOP list)
          replayed on every extractor or prompt change

Layer 1: did the right card reach the desk?

We store a fixed set of memories, ask a fixed question, and check whether one particular memory is in the top results. Either it is or it is not. No model judges anything, so the test gives the same result every time and costs almost nothing.

The widget below runs 8 such pairs over 5 questions. Here k is the number of memories placed in the prompt. With k = 3, 7 of the 8 pass: the dinner question finds the vegetarian card and misses the peanut allergy card, which is ranked fourth. Change k and the extra weight, and watch which pairs pass and what the tokens cost.

Layer 2: did the secretary do the right thing?

Here we replay a whole session through the real write path and check what happened to the store.

This layer calls the extractor, so it costs a little. Our 23-check set takes about ten model calls and about one cent. It runs every time the extraction prompt, the extractor model or the policy changes.

How to build a golden set

1. Start with 20 pairs, by hand. Use the questions your users really ask. Twenty is enough to catch most regressions. A regression is something that used to work and no longer does.
2. Add every real failure. When a user corrects the assistant, turn that moment into a new pair. The set grows exactly where the system is weak.
3. Cover each kind of difficulty. A changed fact, a question about the past, a date ("last March"), a rare name, a fact that needs two memories, and a question with no answer at all.
4. Include things that must not happen. Smalltalk that must not be stored, a secret that must be refused, another user's fact that must not appear.
5. Freeze it. Changing the exam and the student in the same week tells us nothing.

Classifying a wrong answer

Every wrong answer has an address. When an answer is wrong, we do three lookups and give it one of five labels. We used this routine after every benchmark run.

Sample twenty wrong answers, label each, and count. The biggest pile is the next piece of work. "Memory feels off" starts an argument. "Twelve of twenty are IGNORED" starts a fix.

Evidence recall: a free metric

Evidence recall is the share of the needed source messages that appear among the sources of the retrieved memories. Benchmarks mark which turns hold the answer. Every hanumemAI memory remembers the turns it came from. Comparing the two lists needs no model call, so it costs nothing.

LoCoMo, 385 questions, free sweep over k (no answers generated, $0)

k = 15     evidence recall 83.0%      875 tokens per question
k = 30     evidence recall 87.6%    1,489 tokens per question
k = 50     evidence recall 90.5%    2,324 tokens per question
k = 100    evidence recall 93.3%    4,431 tokens per question

More recall is not always a better answer. Experiment 0008 answered the questions at k = 30 and at k = 50. It ran after a change that had shortened the prompt, so its token counts are lower than in the sweep.

                  k = 30     k = 50
evidence recall   87.3       90.2      90.2 - 87.3    = +2.9 points
tokens            1,250      2,092     2,092 / 1,250  = 1.67, so 67% more
score             91.17      90.91     90.91 - 91.17  = -0.26 points

A small reader is distracted by a crowded desk. The free metric tells us what reached the desk. Only an answered exam tells us what was used.

The improvement loop

users chat
   |
   v
signals: corrections, memory edits, deletions, thumbs on personalized answers
   |
   v
triage: the worst memories, every week
   |
   v
which layer failed?
   |-- bad extraction  --> fix the extractor prompt
   |-- bad retrieval   --> fix scoring or k
   |-- bad update      --> fix consolidation
   |
   v
replay the golden sets  -- no regression -->  ship  --> users chat

The step that people skip is the replay. A fix for one failure often breaks something that used to work. The golden sets are how we find out before the users do.

Noise: how much can a score move by luck?

Noise is the amount a score changes when nothing in the system changed.

We ran the same configuration twice on 385 LoCoMo questions, with fresh model calls, at temperature 0. Temperature 0 is the setting that asks a model for its most likely answer every time.

run 1                     93.51
run 2, same code          92.47
difference                93.51 - 92.47 = 1.04 points
answers that changed      63 of 385     = 16%

Sixty-three answers came back different from an unchanged system. Hosted models are not perfectly repeatable. So a gain of one point is not evidence of anything. Our rules became two. Keep a change that gains 2 points or more. Test any gain between 1 and 2 on a second set of questions, a confirmation split, where it must point the same way.

Small test sets are worse. With 50 questions, one question is 2 points. A rule that looked like +5 on 100 questions lost 5 on another set (A0002). And a 100-question sample of LongMemEval scored 91 while the full 500 scored 81 with the same code, because the sample held fewer of the hardest questions.

The public benchmarks

Our own golden set tells us about our users. Public benchmarks let us compare with other systems. There are three, and Chapter 18 is about them.

# from the root of the hanumem research repository
export OPENROUTER_API_KEY=sk-or-...
uv run python scripts/golden.py        # prints what was stored, then PASS or FAIL per check

from hmem import Memory, Config

m = Memory(Config(db_path="quickstart.sqlite"))
golden = [
    ("Any good bakeries near me?",          "Paris"),
    ("What should I get Sam for his birthday?", "June 18"),
]
passed = 0
for question, must_contain in golden:
    rows = m.search(question, user_id="maya", k=5)
    ok = any(must_contain in r.text for r in rows)
    passed += ok
    print("PASS" if ok else "FAIL", question)
print(f"layer 1: {passed} of {len(golden)}")

one question   100 / 50      = 2 points
the gain       (82 - 78) / 2 = 2 questions