LLM as a Judge
Quick Answer: LLM as a judge is a method where one language model scores or compares the outputs of another model or agent against criteria you define.
LLM as a judge means using a language model to grade the output of another model or agent. You give the judge the task, the answer, and a rubric. It returns a score or a pass/fail with a reason.
It exists because many agent outputs cannot be checked with an exact-match test. “Was this reply polite, accurate, and on policy?” has no string to compare against. A human can answer it. A judge model can answer it too, at a much lower cost per check.
The research
The method was formalized in a 2023 paper by Lianmin Zheng and colleagues, who introduced MT-Bench and Chatbot Arena. They found that strong judges like GPT-4 reached over 80% agreement with human preferences, the same level of agreement humans have with each other. They also documented known problems: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability.
Those problems are still the main reason to be careful.
How it works
There are three common setups:
- Single-answer grading. The judge scores one output against a rubric.
- Pairwise comparison. The judge sees two outputs and picks the better one. Useful when you compare a new prompt against the old one.
- Reference-guided grading. The judge sees a known-good answer and checks whether the new output matches its meaning.
Anthropic’s engineering team describes model-based graders as one of three grader types for agent evals, alongside code-based graders and human graders. Each has a role. Code graders are fast and exact. Humans are the ground truth. Model graders cover the middle: flexible and cheap, but they need checking.
The known biases and fixes
| Bias | What happens | Fix |
|---|---|---|
| Position | The judge favors the first or second answer | Run both orders and keep only consistent verdicts |
| Verbosity | Longer answers score higher | Tell the judge to ignore length, and test with padded answers |
| Self-enhancement | A model favors its own style | Use a different model family as judge |
| Weak reasoning | The judge misses math or logic errors | Use code checks for anything computable |
The first three come from the Zheng paper. The fixes are standard practice and my own recommendations.
Judges are not the only way models show up in evals. The tau-bench benchmark uses a language model to simulate the user, then grades the agent by comparing the final database state to the goal state. That is a good pattern: let the model play a role, but let a system of record decide the outcome.
How I would set one up
- Write binary criteria. “Does the reply cite an order number from the context? Yes or no.” Narrow yes/no checks beat a vague 1-to-10 scale.
- Ask for reasoning before the verdict. It makes errors easier to review.
- Label a sample by hand. Have a person grade a set of real outputs first. Fifty is a workable start for a narrow criterion. That is my rule of thumb, not a measured threshold.
- Compare judge to human. Where they disagree, fix the rubric, not the data.
- Re-check on a schedule. Models change. A judge that agreed last quarter may drift.
- Keep humans on high-stakes calls. The judge triages. It does not decide refunds or compliance questions.
Why it matters for production agents
Agents fail in ways unit tests miss. They pick the wrong tool, drop a required disclaimer, or answer fluently from the wrong document. A judge lets you grade hundreds of transcripts after every prompt or model change. That turns agent evaluation from a manual chore into a routine check. Without it, teams ship changes on vibes.
A short example
Say a support agent drafts refund replies. The team writes four yes/no criteria: correct policy cited, no promise outside policy, correct customer name, tone polite. A judge model grades each draft against the criteria and the policy text. A code check separately confirms the refund amount matches the order. Any draft with a failed criterion goes to a reviewer. The team tracks the pass rate across prompt versions.
Common mistakes
- Trusting the judge without comparing it to human labels.
- Using the same model as agent and judge.
- Grading with a vague scale like “quality 1 to 10.”
- Using a judge for things code can verify.
- Letting the judge see which version produced the answer.
When not to use a judge
Do not use a judge for anything code can check exactly. Totals, dates, IDs, schema validity, and whether a record exists in a database are all better done with plain tests. They are faster, cheaper, and never disagree with themselves.
Also avoid a judge as the only check on high-stakes output. A judge can approve a confident, well-written, wrong answer, especially when it shares blind spots with the model that wrote it. For legal, financial, or safety-related decisions, keep a person in the path and use the judge to prioritize their time.
Related terms
Agent evaluation is the broader practice. Prompt engineering matters because the judge prompt is a prompt like any other. For how evals fit in a short pilot, see the one-week AI pilot methodology.