What is an LLM-as-judge and why does it need validating?

An LLM-as-judge is a language model scoring another model's output against written criteria, used because human review does not scale to every prompt change and model swap. It needs validating because the judge is a language model too, with the same failure mode as the thing it grades: it produces a confident, fluent, well-formatted score that can be wrong, and nothing crashes when it is. The way to validate it is to have a person label a sample of cases first, run the judge over the same cases, and measure how often the 2 agree. Until that number exists, an eval suite is not a measurement, it is a second opinion from a system nobody checked.

How do you build a golden dataset for an AI product?

Sample it from real traffic rather than writing it from imagination, because the point of the dataset is to represent the distribution of inputs users actually send. Pull real questions, keep the messy ones, and deliberately include the categories a team would rather forget: half-configured accounts, contradictory data, questions the product cannot answer, and questions phrased in a way nobody on the team would phrase them. A small set drawn from production beats a large set invented in a planning document, because the invented set encodes what the team assumed users would do.

How do you write eval criteria that are objectively assessable?

Replace judgments of quality with checks a second person would apply identically. Does it sound helpful is not assessable. Whether every figure in the answer appears in the account's own data, whether the answer refuses when the data required is missing, and whether it names the time period it used are all checkable by inspection. The test for a criterion is whether 2 reviewers working separately would reach the same verdict. If they would not, the criterion is taste, and taste cannot be automated, only argued about later at the worst possible moment.

Who Grades the Grader: Validating an AI Eval Suite

Your eval judge is a model too. Once a suite grows past a few dozen cases, nobody reads the outputs by hand, so a language model scores them against written criteria. That model has exactly the failure mode of the thing it is grading: a confident, fluent, well-formatted verdict that is wrong, with nothing crashing to tell you.

Which makes the question every eval deck skips the only one that matters. How do you know the grader is right? If you have never measured your judge against human labels, you do not have a measurement. You have a second opinion from a system nobody checked.

The argument for evals is settled. I have made it myself: an AI product fails politely rather than loudly, so the suite belongs in the release path as a gate. Teams have largely accepted this. What almost nobody has done is the next layer down, which is where an eval suite either becomes trustworthy or becomes theater.

3 parts decide it. The set you grade against, the criteria you grade by, and the grader itself.

The set has to come from production

The instinct is to write the test cases in a planning document. Sit down, imagine the questions users will ask, write 50 of them, feel organized.

That set encodes what your team assumed users would do, which is the one thing you already know. Real traffic is different in a specific way: it is messier, more ambiguous, and heavily weighted toward situations the product handles badly. Users arrive with half-connected integrations, contradictory records, and questions phrased in ways nobody in the building would phrase them.

So sample the set from what actually arrives. Keep the awkward ones. Deliberately over-represent the categories a team would rather forget: accounts missing the data required to answer, questions the product genuinely cannot answer, and the ones where the honest response is a refusal. A small set drawn from production beats a large one invented in a doc, every time.

This is also why fixtures do not work for the final gate. A fixture is a clean account, and clean accounts are the rarest thing in any real customer base.

The criteria have to be checkable by a stranger

Here is where most suites quietly fail. Someone writes a rubric with a line like the answer should be helpful and accurate, and the whole thing becomes unfalsifiable.

The working test for a criterion: would 2 reviewers, working separately, reach the same verdict? If not, it is taste. Taste is real and it matters, but it cannot be automated, and pretending otherwise means you will discover the disagreement at the worst possible moment, when a customer is on the phone.

What passes that test looks unglamorous and mechanical. Does every figure in the answer appear in that account's own data. Does the answer refuse when the data needed to answer is absent. Does it state the time period it used. Does it avoid asserting a cause when it can only see a correlation. Each of those is checkable by inspection, which means it is checkable by a machine, which means it can run on every prompt change without a person in the room.

Writing that rubric is product work, not QA work. It is the same artifact as a metric definition: a decision about what good means, made once, in advance, in writing. The teams that struggle are not short of tooling. They have never been forced to say precisely what a correct answer is.

Then validate the grader itself

This is the step that gets skipped, and it is not optional.

Label a sample by hand first. Have a person, ideally 2, go through 50 or 100 real cases and mark each one pass or fail against the rubric. Where the 2 humans disagree, the rubric is ambiguous and needs rewriting before any model sees it. That disagreement is a gift, because it surfaces a definitional problem while it is still cheap.

Then run the judge over the same cases and compare. You are looking for how often the judge and the humans reach the same verdict, and more importantly where they diverge. Judges tend to fail in patterns rather than at random. A common one is rewarding fluency, marking a well-written wrong answer as a pass, which is precisely the failure your suite exists to catch. A judge with that bias will certify the exact defect you built it to prevent.

Only once you know the agreement rate does a suite score mean anything. Before that, a green dashboard is a model agreeing with another model, and the two of them share a training distribution.

Re-check the judge whenever the underlying model changes. Its behavior is not fixed, and the day the provider ships an update is the day your grading silently moves.

What this costs, honestly

A few days of a person reading outputs and arguing about a rubric, then a recurring cost each time the model underneath changes. That is the real bill, and it is why it gets deferred.

The comparison worth making is not against zero. It is against the alternative, which is shipping a change to a system that talks to customers with no evidence beyond a demo that went well. Pilots stall before production for exactly this reason: a pilot needs an agent that works once, and production needs something that decides on its own whether each run was acceptable.

The line to hold

When an answer misses the standard, the reflex is to fix that answer. Adjust the prompt, patch the case, move on.

Do the other thing. Add the case to the set, and if the rubric did not catch it, fix the rubric. The suite is the only asset in an AI product that compounds, because every failure you encode is a failure that cannot ship twice. Everything else, the prompts, the model, the scaffolding, will be replaced within a year. The set of things your product has agreed never to get wrong again is the part you keep.

What the grader is shown matters as much as how it grades, and deciding that is its own discipline: Context Engineering for PMs.

Grading a single answer is the easy half. Grading a loop that runs without you watching is the harder one, and I worked through where the gates go in You Can Only Loop What You Can Grade.