Evaluators

In this chapter, we will learn the Evaluate box, the one arrow on the whiteboard that points backwards. We will build the three evaluators agl ships, check, judge and all_of, learn to write feedback the model can act on, see why a rubric is code and must be tested like code, and learn the one rule that keeps judges honest: the judge is never the model that wrote the answer. By the end you will know when a regex is enough, when you need a second model, and how to order them.

What an evaluator is

An evaluator is a function that takes the model's answer and returns a verdict: passed or not, with feedback and an optional score.

Verdict = dict        # {"passed": bool, "feedback": str, "score": float | None}
Evaluator = Callable[[str, Run], Verdict]

The loop calls it only when the model has stopped calling tools and produced a reply. Pass means that reply is the final answer. Fail means the loop appends "Your answer did not pass review. Feedback: ... Revise and answer again" and goes round once more, up to max_fails times. The feedback string is the whole value of a fail: a fail without a reason sends the model round blind.

check: deterministic, free, honest

Use check whenever the answer has a shape you can test in code.

def check(fn, feedback="answer failed the check"):
    """fn returns True (pass), False (fail) or a str (fail with that feedback)."""
    def evaluate(answer, run):
        r = fn(answer)
        if r is True:
            return {"passed": True, "feedback": "", "score": 1.0}
        return {"passed": False, "feedback": r if isinstance(r, str) else feedback, "score": 0.0}
    return evaluate

Three idioms cover most of what you will need:

# a shape
check(lambda a: a.strip().isdigit() or "reply with only the number")

# a count, with the number in the feedback
def outline_ok(answer):
    n = len(parse_json(answer)["chapters"])
    return True if n == 3 else f"the plan has {n} chapters; it must have exactly 3"
check(outline_ok)

# a contract with many clauses, each a precise complaint
check(validate_chapter)      # returns "Fix these and resend the whole chapter: ; "

# the strongest evaluator there is: run the real tests or the real build
check(lambda a: run_tests() or "tests failed: " + last_failure())

Write the feedback for the model

Feedback is a prompt. The model reads it and acts. "the plan has 1 chapters; it must have exactly 3" produces a three-chapter plan on the next turn. "answer failed the check" produces a guess. Three rules for feedback that works:

name the fact       "2,900 words"           not "too long"
name the target     "the range is 1,400 to 2,400"
say what to do      "resend the whole chapter"   not "please fix"

When a validator has many rules, return all the failures in one message. A model that gets one complaint per round takes five rounds; a model that gets five complaints fixes them in one.

judge: a second model with a rubric

Use judge when the answer is prose, a plan, a chapter: anything without a testable shape.

def judge(llm, rubric, threshold=0.7, model=None):
    def evaluate(answer, run):
        prompt = (f"You are a strict reviewer.\n\nTask:\n{run.task}\n\nAnswer:\n{answer}\n\n"
                  f"Rubric:\n{rubric}\n\nReply with JSON only: "
                  '{"score": <0..1>, "passed": , "feedback": ""}')
        reply = llm.chat([{"role": "user", "content": prompt}], json_mode=True, model=model, temperature=0)
        v = parse_json(reply.text)
        score = float(v.get("score", 0))
        return {"passed": bool(v.get("passed", score >= threshold)) and score >= threshold,
                "feedback": str(v.get("feedback", "")), "score": score}
    return evaluate

Three decisions hide in those lines. Temperature zero, so the same answer gets the same verdict as often as a model allows.

JSON mode, so the verdict parses. And a model argument, so the judge runs on a different model from the worker.

The judge is not the worker

A model grading its own prose shares its blind spots, and a rubric that changes between runs makes scores incomparable. So two rules. The judge runs on a different model, named by AGL_MODEL_JUDGE. And once you start comparing runs, the judge model and the rubric are frozen; changing either means starting the comparison again.

Rubrics are code

The judge is a literal reader. A rubric that says "one concrete example" will fail an article for having two. This is the most common evaluator bug and the easiest to miss, because the score looks like a judgment and the feedback looks like an opinion. It is neither: it is the rubric being applied exactly as written. Two habits prevent it.

test the rubric     run the judge on three answers you have already graded yourself; if its verdicts differ from yours,
                    the rubric is wrong, not the answers
say what you mean   "at least one concrete example"; "judge the whole, do not fail it for one clause";
                    "a number without a source is a fail" (if that is what you mean)

Read the feedback, not the score. A judge that fails the same answer three times with the same feedback is telling you what its rubric says. Fix the rubric with as few words as possible, and keep the old wording in version control so you know what changed.

all_of: cheap checks first

all_of runs evaluators in order and stops at the first fail, so the judge only reads answers that passed the checks.

def all_of(*evaluators):
    def evaluate(answer, run):
        scores = []
        for ev in evaluators:
            v = ev(answer, run)
            if not v.get("passed"):
                return v                      # the first fail wins; nothing after it runs
            scores.append(v.get("score") or 1.0)
        return {"passed": True, "feedback": "", "score": sum(scores) / len(scores)}
    return evaluate

evaluate = all_of(check(validate_chapter),                                       # free
                  judge(llm, CONTENT_RUBRIC, threshold=0.7, model=JUDGE_MODEL))  # one call, only for chapters that passed

In a chapter-writing loop with this evaluator, a chapter with a missing section never reaches the judge; the checklist bounces it for free. A chapter that reads the source wrongly passes the checklist and is caught by the judge. Each fail is specific, and the judge is paid for only when it can add something a regex cannot.

What a fail looks like to the model

user:       How many lines does agl/loop.py have, times 7? Reply with only the number.
assistant:  The file has 217 lines, so the answer is 1519.
user:       Your answer did not pass review. Feedback: reply with only the number
            Revise and answer again.
assistant:  1519

Nothing clever. One message was added and the model was called again. It has seen its own earlier answer and the complaint, which is all it needs.

Choosing an evaluator

Priya's evaluators

Priya's chapter writer gets all_of: her checklist, then a content judge. Her checklist pins the chapter title and part label to the outline's exact strings, because a writer left alone drifts into "(Part I)" suffixes. Her content rubric says "every claim agrees with the source material in the task; definitions are not inverted; nothing is invented". She tests the rubric on three chapters she graded herself before trusting it, and finds the first draft of it too strict about length, which the checklist already handles. She removes the length clause from the rubric: one job per evaluator.

Try it yourself

Write a rubric for "a good commit message" in three sentences. Give it to judge and run it on five real commit messages from any repository. Read the feedback. Change one word in the rubric and run again. Count how many verdicts changed. That number is how much your rubric is code.

Common questions

Why not let the worker grade itself? It is cheaper.

It shares its own blind spots, and it is inclined to say its answer is fine. A second model with a fixed rubric is the cheapest independent reader you can buy.

Is a threshold of 0.7 right?

It is a starting point. Judge scores cluster; look at the scores on answers you have read yourself and put the line where your own verdicts flip.

What if the judge returns something that is not JSON?

The verdict is a fail with feedback "judge returned unparseable output". JSON mode makes this rare, and the parser strips code fences and prose around the object first.

Can the evaluator see the tools the model used?

Yes. It receives the run, which holds the whole history. An evaluator can fail an answer because the model never ran the tests.

How many retries?

Two by default. Most fixable fails are fixed on the first retry; a third usually means the task or the rubric is wrong.

Should evaluators be strict or lenient?

Strict on shape, measured on content. A shape check that lets a malformed answer through wastes the judge's call; a judge that fails good answers for a clause you did not mean wastes retries. Test both on answers you graded yourself.

Carry this

• An evaluator returns passed, feedback and a score. The feedback is for the model: name the fact, the target, the action.
• check for anything with a testable shape; running the real tests is the strongest check there is.
• judge for prose, on a different model, at temperature 0, with a rubric tested on answers you graded.
• all_of(check, judge): cheap checks first, so the judge only reads what passed.
• Rubrics are code. Read the feedback, not the score, to find their bugs.

Check yourself

1. What happens in the loop when an evaluator returns passed: false?

Answer

The loop appends a user message with the feedback and asks the model to revise, then calls it again; after max_fails fails it stops with status "fail".

2. Why should the judge run on a different model from the worker?

Answer

A model grading its own output shares its blind spots; a second model with a fixed rubric is an independent reader, and freezing it keeps runs comparable.

3. A judge fails the same good answer three times with the same feedback. What is wrong?

Answer

The rubric. The judge is applying it literally; fix the wording with as few words as possible and test it on answers you have graded yourself.