LLM as a Judge: How to Calibrate Model Graders for Agent Evals

Yes, an LLM can grade another LLM. No, you should not trust it until you have checked it against people.

An LLM as a judge is a model that reads an agent’s output and scores it against a rubric. It is fast and cheap. It is also biased in known ways, and it can look accurate while being wrong.

My recommendation: grade 50 outputs by hand, run the judge on the same 50, and measure where they disagree. This post shows how.

What a judge does

A judge is a model call with three inputs: the task, the agent’s output, and a rubric. It returns a verdict, usually pass or fail, sometimes a score, plus a short reason.

You use one when code cannot check the answer. A script can verify that an invoice total matches. It cannot tell whether a summary kept the key facts or whether a reply sounded appropriate. Anthropic’s agent eval guide (January 2026) splits graders into three types: code-based, model-based, and human. Code graders are fast and reproducible but brittle. Model graders handle open-ended work but are non-deterministic and “require calibration.” Human graders are the gold standard but slow and costly.

The research backs the idea. A 2023 study from the LMSYS team found that strong judges like GPT-4 reached over 80% agreement with human preferences, about the level humans agree with each other. That is a good start. It was on chat-style comparisons, not your agent’s tasks. Your numbers may differ, which is why you calibrate.

Use code checks first. Use a judge for what is left. For the wider picture, see our agent evaluation guide.

Write a rubric the judge can follow

Most bad judges are bad rubrics. “Rate the quality from 1 to 10” gives you noise.

A good rubric has these traits:

• One criterion per judge. Check “did it cite the right policy” in one call and “is the tone appropriate” in another. Bundled criteria hide failures.
• Binary where you can. Pass or fail is easier to calibrate than a 10-point scale. If you need a scale, define each level with an example.
• Concrete definitions. Replace “accurate” with “every figure in the answer appears in the source document.”
• A way out. Anthropic’s guide advises giving the model an exit such as returning “Unknown” when it lacks enough information. That cuts made-up verdicts.
• Reason first, then verdict. Ask for a short explanation before the label. It makes failures easier to review.
• Grade the outcome, not the path. The same guide warns that checking exact steps is brittle, because agents find valid routes you did not expect.

Here is a template:

You are grading an AI agent's output.

Task given to the agent: {task}
Agent output: {output}
Source material: {context}

Criterion: Every factual claim in the output is supported by the source material.

Pass if: all claims are supported.
Fail if: any claim is unsupported or contradicts the source.
Unknown if: the source is too incomplete to decide.

Give a one-sentence reason, then the verdict on its own line:
VERDICT: PASS | FAIL | UNKNOWN

Keep the judge’s temperature low and the prompt versioned. Our post on prompt change control explains why, since a changed judge prompt changes every score after it.

Calibrate against human grades

This is the step people skip. It takes an afternoon.

Step 1: Pull 50 real cases. Use real agent outputs, not invented ones. Include easy cases, hard cases, and known failures. If failures are rare in production, oversample them. A set of 50 passes tells you nothing.

Step 2: Grade them by hand. Use the same rubric the judge gets. Ideally two people grade, so you can see how much humans disagree. Settle ties by discussion and fix the rubric if the same confusion keeps appearing.

Step 3: Run the judge on the same 50. Same rubric, same inputs.

Step 4: Compare. Build a simple table of human verdict against judge verdict.

Step 5: Measure agreement with the right number. Percent agreement is the obvious metric. It can mislead. Thakur et al. found that judges with high percent agreement can still assign very different scores, and they recommend alignment metrics beyond raw percent.

Here is why. Say 45 of your 50 cases are real passes. A lazy judge that passes almost everything will agree with humans about 90% of the time. Cohen’s kappa corrects for chance: it runs from -1 to 1, and 0 means no agreement beyond chance. In a worked example I ran (hypothetical data: humans pass 45 of 50, judge passes 46, they disagree on 5 cases), percent agreement is 90% but kappa is only 0.39. The judge looks fine and is weak.

Track these for every judge:

Metric	What it tells you
Percent agreement	Rough overall match
Cohen’s kappa	Agreement beyond chance
False pass rate	Judge passes what humans fail (the dangerous one)
False fail rate	Judge fails what humans pass (the annoying one)
Unknown rate	How often the judge cannot decide

For most agent work, false passes cost more than false fails. A false pass ships a bad output. Set your target accordingly.

Then read every disagreement. Most come from a vague rubric line. Fix the rubric, rerun, and repeat until the numbers settle. Do not tune on the 50 forever. Hold 20 back as a test set so you do not overfit the rubric to your own sample.

How many samples do you need?

More than 50 if you want tight numbers. Here is the arithmetic, using a simple normal approximation for a 95% interval around an 80% agreement rate (my calculation, not from a paper):

Cases	Roughly plus or minus
50	11 points
100	8 points
200	5.5 points
400	4 points

So 44 of 50 agreements (88%) really means somewhere between about 76% and 94%. That is good enough to find big problems and decide if a judge is usable at all. It is not enough to claim 88%. Evals are experiments, and the paper Adding Error Bars to Evals makes that case with formulas for sizing them. Start at 50, grow toward 200 for any judge that gates releases.

Bias traps

Judges have known habits. Test for each one on your own data.

Position bias. In pairwise comparisons, the judge favors an answer because of where it appears. Wang et al. showed that swapping the order of two answers could flip the outcome, with one model beating another on 66 of 80 queries just through ordering. Fix: run every comparison twice with the order swapped, and count it only when both runs agree.

Verbosity bias. Judges tend to prefer longer answers. Researchers behind Length-Controlled AlpacaEval corrected for length statistically and raised the benchmark’s correlation with Chatbot Arena from 0.94 to 0.98. Fix: tell the rubric that length earns nothing, and test by padding good answers with filler to see if scores rise.

Self-preference. The 2023 LMSYS paper lists self-enhancement as a bias: a model may favor its own style. Fix: do not grade a model’s output with the same model if you can avoid it.

Leniency. Thakur et al. also report a tendency toward leniency and sensitivity to prompt complexity and length. Fix: watch the false pass rate, and keep rubric prompts short.

Limited reasoning. A judge can miss errors it could not have solved itself. Fix: give it the source of truth to check against, not just the answer.

A quick bias test costs little. Take 20 outputs. Pad them, swap their order, and rerun with a different judge model. If scores move, you found a trap.

One more option: use several judges. A study on judge panels found a panel of smaller models from different families beat a single large judge, showed less bias, and cost over seven times less. Worth testing if your volume is high.

Pairwise versus score

Two ways to grade:

Method	Good for	Watch out
Pass/fail on a rubric	Release gates, regressions	Rubric must be sharp
Score (1 to 5)	Tracking trends	Scores drift, levels blur
Pairwise (A vs B)	Choosing between two versions	Position bias; needs swaps

My rule: use pass/fail for gates and pairwise for choosing between two prompts or models. Use scores only with written anchors for each level. If your team cannot explain the difference between a 3 and a 4, drop the scale.

When humans still grade

Keep people in the loop for:

• New failure types. When production shows a pattern your rubric never imagined, a human defines the rule first. Then the judge scales it.
• High-stakes outputs. Money moves, legal text, health advice. Sample and review by hand, always.
• Low-agreement areas. If kappa stays low on a criterion after rubric fixes, that criterion is not judge-ready.
• Unknown verdicts. Route them to a person.
• Periodic spot checks. See the next section.

Wang et al. propose a similar idea: measure how unstable a judgment is across orderings and send the hard ones to humans. You can do the same with disagreement between two judge runs.

Monitoring judge drift

A calibrated judge today is not a calibrated judge forever. Models get updated, prompts change, and your agent’s outputs shift.

Do this:

1. Freeze a golden set. Keep your human-graded cases. Rerun the judge on them weekly, and after any change to the judge model or prompt. If kappa drops, stop and investigate.
2. Pin versions. Record the judge model version and prompt version with every score.
3. Sample live traffic. Each week, have a person grade 10 to 20 fresh cases and compare. Log it.
4. Watch the distribution. A sudden jump in pass rate with no agent change often means the judge moved.
5. Refresh the set. Add new failure cases as they appear.

Feed these checks into your tracing setup. Our observability tools post covers where judge scores fit next to traces, and the testing strategies guide covers the rest of the test stack.

Bottom Line

An LLM judge is a measuring tool. Calibrate it like one.

Grade 50 cases by hand. Compare with the judge using kappa and false pass rate, not only percent agreement. Test for position and length bias. Hold out a test set, freeze a golden set, and keep humans on high-stakes and low-agreement work.

Next step: pick one criterion your agent fails today, write a one-line pass/fail rubric, and run the 50-case check this week.

FAQ

Can an LLM grade another LLM? Yes, for many tasks. A 2023 study found strong judges matched human preferences over 80% of the time on chat comparisons. But accuracy varies by task, and judges have known biases. Check yours against human grades before you rely on it.

How do I calibrate a judge? Grade 50 or more real outputs by hand, run the judge on the same cases, and compare. Track percent agreement, Cohen’s kappa, and the false pass rate. Read every disagreement, fix the rubric, and test on a held-out set.

When do I need human graders? For new failure types, high-stakes outputs, criteria where agreement stays low, and any case the judge marks Unknown. Also use them for weekly spot checks to catch drift.

What biases should I check? Position bias in pairwise tests, verbosity bias, self-preference, and leniency. Swap answer order, pad answers with filler, and try a different judge model to see whether scores move.

How many samples do I need? Start with 50 to find big problems. At 50 cases the 95% range on an 80% agreement rate is about plus or minus 11 points, so move toward 200 for any judge that blocks releases.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect