LLM-as-Judge with a Calibrated Rubric
Also known as: model-graded evaluation, automated scoring, AI grader
Score subjective quality with a model, but only against an explicit rubric that has been calibrated against human labels.
The problem
Most of what matters about LLM output — is it helpful, is it grounded, does it follow the tone guide — cannot be checked with string matching. Human grading is accurate and far too slow to run on every change. The obvious shortcut, asking a model to "rate this 1–10", produces numbers that look rigorous and mean nothing: they cluster around 7–8, they move with irrelevant surface features like length, and they have never been checked against anything.
The pattern
Give the judge an explicit rubric with named criteria and concrete anchors for each score level, and require per-criterion reasoning before the score. Then calibrate: grade a sample by hand, measure agreement with the judge, and iterate on the rubric until agreement is high enough to trust. An uncalibrated judge is a random number generator with a confident interface.
Use when
- Quality dimensions are subjective and deterministic checks cannot capture them
- You need to score more output than humans can review
- You have or can create a few hundred human-labelled examples for calibration
- The judge is scoring output from a different model or a weaker tier
Avoid when
- A deterministic check would work — always prefer it; it is free and never drifts
- You cannot produce human labels, so calibration is impossible and the score is unfounded
- The judged property requires domain expertise the model does not have
Tradeoffs
GainSubjective quality becomes measurable continuously and at scale
CostEvery evaluation run costs judge calls, which on large suites is significant
GainA written rubric forces the team to define what good actually means
CostRubric design is real work and needs revision as the product changes
GainA calibrated judge agrees with humans closely enough to gate on
CostJudges carry biases — length, verbosity, self-preference — that must be tested for
The difference between a useful judge and a useless one is entirely in the rubric and the calibration. The model is the easy part.
Anchor every score level
A rubric without anchors is a vibe check. Define what each score means concretely enough that two different people reading it would grade the same output the same way — and score criteria separately rather than producing one overall number, because a single score hides which dimension actually moved.
GROUNDEDNESS = """Score how well the answer is supported by the provided sources.
5 — Every factual claim is directly supported by a cited source. No outside knowledge.
4 — All claims supported; one or two lack an explicit citation but are clearly present.
3 — Main claims supported; a minor detail is unsupported or drawn from outside knowledge.
2 — A significant claim is unsupported, OR a citation points to text that does not support it.
1 — Substantially unsupported, or contradicts the sources.
A correct refusal ("the sources do not cover this") scores 5.
Do not reward or penalize length, confidence, or writing style. Score support only."""
class CriterionScore(BaseModel):
reasoning: str = Field(description="Cite the specific text driving this score.")
score: Literal[1, 2, 3, 4, 5]
class Judgement(BaseModel):
groundedness: CriterionScore
completeness: CriterionScore
tone: CriterionScore"A correct refusal scores 5" is the kind of line rubrics need. Without it judges punish refusals as unhelpful, and you end up optimizing your system away from the honest behaviour you deliberately built.
Calibrate against humans, then keep checking
Calibration is what converts a plausible number into a usable one. Have humans grade a sample with the same rubric, measure agreement, and revise the rubric where the judge and the humans diverge. Track exact agreement and within-one agreement; the second matters more, since a judge that is never off by more than a point is trustworthy for gating.
def calibrate(sample: list[Example], human_scores: dict[str, int], criterion: str) -> Calibration:
judged = {ex.id: judge(ex, criterion).score for ex in sample}
exact = mean(judged[i] == human_scores[i] for i in human_scores)
within_one = mean(abs(judged[i] - human_scores[i]) <= 1 for i in human_scores)
bias = mean(judged[i] - human_scores[i] for i in human_scores) # systematic lenience
return Calibration(
criterion=criterion,
exact_agreement=exact,
within_one=within_one,
bias=bias,
worst_disagreements=sorted(human_scores, key=lambda i: -abs(judged[i] - human_scores[i]))[:10],
usable=within_one >= 0.90 and abs(bias) < 0.35,
)Test the judge for known biases
Judges have measurable, reproducible biases: they prefer longer answers, they prefer output from their own model family, and in pairwise comparison they favour whichever option is presented first. Test for each rather than assuming. Position bias in particular is easy to neutralize — run both orderings and treat disagreement as a tie.
def pairwise(question: str, a: str, b: str) -> Literal["a", "b", "tie"]:
first = compare(question, a, b) # a presented first
second = compare(question, b, a) # b presented first
if first == "first" and second == "second":
return "a"
if first == "second" and second == "first":
return "b"
return "tie" # order changed the verdict: not a real preference
def length_bias_check(pairs: list[tuple[str, str]]) -> float:
"""Correlation between preferred-answer length and the verdict. Near 0 is healthy."""
verdicts = [pairwise(q, a, b) for q, a, b in pairs]
return correlation(
[len(a) - len(b) for _, a, b in pairs],
[1 if v == "a" else -1 if v == "b" else 0 for v in verdicts],
)Never let a model judge its own output in a comparison it can identify. Self-preference is well documented and large. Use a different model family as judge, and prefer pairwise comparison to absolute scoring — models are considerably better at ranking than at scoring.
- Anchor every score level concretely; score criteria separately, never one overall number
- Require per-criterion reasoning before the score
- Calibrate against human labels and re-check after any rubric or model change
- Neutralize position bias by running both orderings; test for length and self-preference bias
- Use a different model family for judging than for generating
The worst disagreements from calibration are the most valuable output of the whole exercise. Read them — they usually reveal an ambiguity in the rubric rather than a failure of the judge.