Benchmarks are claims

In this chapter, we will learn how memory systems are measured. We will meet the three public exams and see how a second AI model grades the answers. We will learn why the same system can score 88 or 94 without changing a single line of code. Then we will read hanumemAI's own results, including the places where other systems score higher.

Why we need an exam at all

A benchmark is a fixed set of conversations and questions with known correct answers. Because the set is fixed, two systems can be compared on the same work, and one system can be compared with itself before and after a change.

Chapter 17 built our own small golden sets. Those tell us whether our system works for our users. Public benchmarks answer a different question: how does this design compare with everybody else's? Three of them matter, and each one tests a different weakness.

LoCoMo: does memory survive a long conversation?

LoCoMo is a set of ten very long conversations between two people, with about 1,540 questions that are normally scored. It comes from the paper "Evaluating Very Long-Term Conversational Memory of LLM Agents" (2024).

Each conversation has about 300 turns spread over up to 35 sessions. Every session has a date. The questions fall into five categories.

Almost everyone reports the first four categories. In the paper, humans scored about 88 and GPT-4 Turbo with the whole conversation on its desk scored about 52. Temporal questions were the biggest gap.

LongMemEval: does memory stay correct when facts change?

LongMemEval is a set of 500 questions, each hidden inside its own haystack, the paper's word for a pile of chat sessions. It comes from the paper "Benchmarking Chat Assistants on Long-Term Interactive Memory" (2024). We use the S version: about 115,000 tokens and about 40 sessions for every question.

This is the exam for Chapter 11. A system that overwrites old facts, or that cannot tell which of two facts is newer, loses the knowledge-update questions.

BEAM: does memory hold at extreme length?

BEAM is a set of 100 conversations at four sizes: about 128,000 tokens, 500,000 tokens, 1 million tokens and 10 million tokens. It comes from the paper "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs" (ICLR 2026). It has 2,000 questions over ten abilities. The dataset calls its smallest size the 100K split, so our tables say BEAM-100K.

information extraction     multi-session reasoning     knowledge update
temporal reasoning         summarization               preference following
abstention                 contradiction resolution    event ordering
instruction following

BEAM is graded differently. Every question has a rubric, a short list of points that a good answer must contain. The judge checks the points one at a time and gives each one 0, 0.5 or 1. In our tables, a question counts as passed when its mean rubric score is 0.5 or more, and we print the mean rubric score next to the pass rate. In the paper, models that read the whole conversation with no memory system score about 0.30 to 0.36 on a scale of 0 to 1. That is 30 to 36 on our scale of 0 to 100.

How a machine grades an answer

An LLM judge is a second model that reads the question, the correct answer and the system's answer, and decides whether they agree.

We need a judge because exact matching is too strict. The known correct answer is called the gold answer. If the gold answer is "7 May 2023" and the system says "on May 7th, 2023", a string comparison marks it wrong. A judge marks it right.

question  ─┐
gold      ─┼─►  judge model + judge prompt  ─►  CORRECT or WRONG
answer    ─┘

But the judge is a model too, with its own habits. One judge accepts a loose paraphrase. Another wants the exact detail. This brings us to the most important lesson of the chapter.

We repeated the double grading on our final run. The widget below shows that pair.

earlier run    94.0  - 88.5  = 5.5 points
final run      94.22 - 87.73 = 6.49 points
open-domain    80.21 - 55.21 = 25 points, the largest gap of any category

Look at the two judges side by side, category by category. Watch which category moves most.

The five dials behind every score

A benchmark score is produced by a whole pipeline, and only one part of that pipeline is the memory system. At least five dials move the number without touching memory at all.

Samples flatter

Our 100-question sample of LongMemEval scored 91. The full 500 scored 81.0 with the same code. The sample happened to contain fewer questions of the two hardest types. In the full set, multi-session has 121 questions and we scored 65 percent on them. Temporal has 127 questions and we scored 72 percent. A score must say how many questions it covers.

Noise is about one point

This is the noise test of Chapter 17. We ran one configuration twice with fresh answers, and 63 of the 385 answers came back different.

run 1                       93.51
run 2, same configuration   92.47
difference                  93.51 - 92.47 = 1.04 points, from nothing at all

So on that set, any difference under one point is noise. Any gain under two points needs a second, separate set of questions to confirm it.

What an honest score looks like

A score is a claim until it names its answer model, its judge, its question count and its context tokens per question. With those four, a reader can decide what the number is worth.

claim     "We score 92.5 on LoCoMo."

report    LoCoMo, all 1,540 questions, categories 1 to 4
          memory layer   qwen3.7-flash extraction, qwen3-embedding-4b
          answer model   qwen3.7-flash
          judge          gpt-4o-mini
          context        about 1,470 tokens per question
          score          94.2

hanumemAI's results

Here is every headline run. The memory layer is the same in all of them: qwen3.7-flash writes the memories and qwen3-embedding-4b makes the vectors. "Flash" means qwen3.7-flash. The other answer model, qwen3.8-flash, is a stronger and more expensive model.

The BEAM rows do not cover the same share of the benchmark. BEAM-100K is 20 conversations and BEAM-500K is 35. BEAM-1M is the first 10 conversations only. Each conversation has 20 questions.

The full LoCoMo run, by category, under both judges. Evidence recall is the free retrieval metric of Chapter 17.

And LongMemEval, all 500 questions, by type:

Next to other systems

These numbers come from the other systems' own papers and posts. We did not run them. They use other answer models, other judges and other budgets, so this is context and not a leaderboard. "Self-reported" marks a number that a vendor published about its own product.

Where others score higher

On LongMemEval, hanumemAI is not the top number. Agent Zero reports 95.6, the Mem0 platform reports 94.4 with gpt-5, SodaMem reports 92.8 with about 18,000 tokens of context, and HydraDB reports 90.8 with Gemini 3 Pro. Our best is 88.6. Our evidence recall on that set is 98.1 percent, so the right memories do reach the desk. The misses are reading mistakes on multi-session and temporal questions. They are made by a small answer model in a single pass: one model call per question, with no second look.

Where hanumemAI leads

On LoCoMo, under the same judge model the Mem0 paper used, hanumemAI scores 94.2 with a flash answerer and 96.0 with qwen3.8-flash. It does so with about 1,470 tokens per question against about 7,000. The answer models differ, so even this is not a like-for-like race.

context per question     hanumem 1,470      Mem0 top-200  about 7,000
ratio                    7,000 / 1,470  =  about 4.8 times less context

On BEAM at one million tokens, hanumemAI's 64.0 is level with Mem0's self-reported 64.1, with about half the context (3,500 tokens against 6,700).

The honest framing

hanumemAI's claim is this: the best score per context token, with small and cheap models. It is not "the highest number on every exam". Every point in the chart below is a score next to its cost. Choose a benchmark, hover a point to see its model stack, and read the table under the chart. Systems that publish no token count are in the table only.

benchmark and version        ....................
questions scored             ....... of .......
answer model                 ....................
judge model and prompt       ....................
context tokens per question  ....................
who ran it                   the vendor / a third party
per-category scores shown    yes / no

scripts/bench.sh --free      # evidence recall only, no answer or judge calls, $0
scripts/bench.sh             # LoCoMo, 385 questions, a few cents with Qwen flash
scripts/bench.sh --full      # all 1,540 questions

A   2,400,000 x 7,000 = 16,800,000,000 tokens   x $3 / 1,000,000 = $50,400 per day
B   2,400,000 x 1,500 =  3,600,000,000 tokens   x $3 / 1,000,000 = $10,800 per day