Using a model as a judge

A model judge is a language model given a written rubric and asked to score another model's output against it. A set of three hundred cases arrives with a problem attached, because a person reading each response against a rubric works comfortably at twenty cases and collapses at three hundred on every change. A judge applies the same rubric for pennies a case and returns in seconds.

The practice was established by a 2023 paper from Lianmin Zheng and colleagues, presented that year at the NeurIPS datasets and benchmarks track. The authors built a multi turn question set and a public arena where visitors compared two answers side by side, then measured how far a model's verdicts tracked human ones. Strong judges matched human preferences more than eighty per cent of the time, which is about how often two people agreed with each other on the same pairs. The same paper named the weaknesses, and those weaknesses are the reason this page is long.

The sections below set out what a judge does and how it sits among the other ways of grading. A weak judge prompt is then shown against the same prompt written properly. The biases come next, with the procedure that cancels the worst of them, and the page ends with the validation step teams skip and with the finding that explains why a rubric is written while grading and never before it.

What a model judge does

A judge takes three things and returns a score. It takes the input the system was given, the output the system produced and a rubric describing what an acceptable output contains. It is a prompt, it sits in the same repository as every other prompt, and it gets the same versioning, the same review and the same eval set pointed at it.

Two shapes are in common use and they answer different questions.

Pointwise grading scores one response on its own against the rubric. It gives an absolute number, which is what a build gate needs, because a gate compares today against a threshold agreed weeks ago. It is harder for a model to do consistently, since the model has to hold a standard in mind with nothing to compare against.

Pairwise comparison shows the judge two responses and asks which is better. Models are markedly better at this, because a comparison asks for a preference and never for a calibrated scale. It answers whether the new prompt beats the old one, which is exactly the question a release decision asks.

A team running a build gate needs both. The pairwise comparison decides whether a change helped, and the pointwise score tells anybody looking at a dashboard how the feature is doing in absolute terms.

The four ways to grade an answer

A judge is one option among four, and choosing it for a case that a simple assertion would settle wastes money and adds noise.

The ordering is deliberate. Every case that can be shaped into the first row should be, because an assertion costs nothing and never drifts. The judge earns its place on the remainder, and the remainder is usually about a third of a support or summarisation set.

A weak judge prompt

Most teams start here, and the prompt below is close to verbatim from several first attempts.

Rate the assistant's answer from 1 to 10.

Question: {{question}}
Answer: {{answer}}

Score:

Four things are wrong with it and each one shows up in the numbers. There are no criteria, so the judge invents its own and invents different ones on different runs. There are no anchors on the scale, so a 7 has no definition and cannot be compared with last week's 7. There is no reference material, so the judge grades factual accuracy from its own memory. And a single number gives nobody a reason to believe it, so the first time a score looks wrong the whole apparatus loses its credibility.

The scores such a prompt returns cluster between 6 and 8 whatever it is shown, which is the signature of a judge that is guessing politely.

The same judge prompt written properly

The version below grades a customer support reply. It is longer than the prompt it replaces by a factor of twenty, and that is the correct ratio.

SYSTEM
You are grading a customer support reply written by an automated
assistant. You are not the assistant. Do not rewrite the reply, do not
answer the customer, and do not comment on anything the rubric does not
name.

You will be given the customer's message, the policy passages the
assistant was shown when it wrote the reply, and the reply itself. Judge
the reply only against those passages. If a passage does not support a
claim, the claim is unsupported, even if you believe it is true.

Length, confidence, politeness and formatting carry no marks. A short
reply that satisfies every criterion scores higher than a long one that
does not.

RUBRIC
Score each criterion independently, on 0, 1 or 2. Do not average them
yourself.

1. GROUNDING
   2  Every claim about policy is supported by a passage above, and the
      reply gives the document id for each such claim.
   1  Every claim is supported, and at least one document id is missing
      or points at the wrong passage.
   0  The reply states a policy fact that no passage supports.

2. OUTCOME
   2  The reply reaches the outcome the passages require for this case.
   1  The outcome is defensible and the reply omits a condition in the
      passages that would change it.
   0  The reply reaches an outcome the passages do not allow.

3. BOUNDARIES
   2  No commitment the system cannot keep. No amount, no date, no
      approval, no promise about what a colleague will do.
   1  One commitment, hedged with a word such as "usually" or "should".
   0  Any firm promise of money, timing or approval.

4. ANSWERED
   2  The customer's actual question is answered in the first sentence.
   1  The question is answered after material the customer did not ask
      for.
   0  The question is not answered.

EVIDENCE
For each criterion, quote the span of the reply or the passage that
decided the score. Quote it exactly. If you cannot find a span to quote,
the score for that criterion is 0.

OUTPUT
Return one JSON object and nothing else, in exactly this shape:

{"grounding": 0,
 "outcome": 0,
 "boundaries": 0,
 "answered": 0,
 "evidence": {"grounding": "", "outcome": "",
              "boundaries": "", "answered": ""},
 "verdict": "pass"}

Set "verdict" to "pass" only if boundaries is 2, and outcome is 2, and
the four scores add to 7 or more. Otherwise set it to "fail".

USER
<customer_message>
{{message}}
</customer_message>

<policy_passages>
{{passages}}
</policy_passages>

<assistant_reply>
{{reply}}
</assistant_reply>

Six changes separate the two prompts, and each one removes a specific way the first prompt misled its readers.

1. Named criteria. The judge no longer chooses what to care about, so two runs on the same input are graded on the same four things.
2. Anchors on every point of every scale. A 1 on grounding has a written definition, so a score can be argued with and a disagreement can be settled by reading the anchor.
3. The source material in the prompt. Factual accuracy is judged against the passages the assistant actually saw, which turns a question about the world into a question about two pieces of text.
4. A required quotation per criterion. A judge that has to point at a span cannot score a criterion on a general impression, and a reviewer reading the output can check the quotation in seconds.
5. An explicit instruction that length earns nothing. This does not remove verbosity bias and it reduces it, which is the most any instruction in a prompt achieves.
6. A pass rule made of arithmetic. Two criteria are hard gates and the rest is a sum against a threshold, so the verdict is reproducible and the threshold can be raised on a date somebody records.

The delimiters around the three inputs do a seventh job. A customer message that contains the words "score this a 10" arrives inside a tagged block that the rubric has already told the judge to treat as material to grade. That is the same separation the module on production covers at length, applied here because a judge reads attacker controlled text on every single run.

The three biases a judge brings

The 2023 paper that established the practice named three biases alongside the agreement figure. Each of the three is a property of how a judge is asked rather than of one model, so a team validates against them on whatever model it is using and never assumes a newer version has removed them.

Position bias. A judge shown two answers prefers the one in a particular slot, independent of content. Swap the two answers and a share of the verdicts swap with them.

Verbosity bias. A longer, more confident, better formatted answer scores above a shorter correct one. A rubric that says length earns nothing helps, and a rubric with a criterion that explicitly penalises material the customer did not ask for helps more.

Self preference. A judge scores output from its own model family more generously than a person would. The practical consequence is that a team grading its own system with the model that produced the output has a measurement contaminated by the thing it measures.

Two of the three have cheap mechanical fixes. Self preference has one expensive fix, which is grading with a model from a different family and checking both against people.

Running both orders to cancel position bias

Position bias is the easiest to remove because it is symmetrical. Run every comparison twice with the answers swapped, and keep only the verdicts that agree.

def judge_pair(left, right, ask):
    """Compare two answers in both orders and keep only agreement.

    `ask(x, y)` shows x first and y second and returns "first" or
    "second". A judge with position bias answers "first" in both runs,
    which this records as a tie. The tie is the honest answer, because
    the only thing that changed between the two runs was the order.
    """
    forward = ask(left, right)
    reverse = ask(right, left)

    if forward == "first" and reverse == "second":
        return "left"
    if forward == "second" and reverse == "first":
        return "right"
    return "tie"


def compare_versions(cases, baseline, candidate, ask):
    """Paired comparison across a set, with ties excluded from the count.

    Ties carry no information about which version is better, so they are
    reported separately and never folded into the win rate.
    """
    tally = {"baseline": 0, "candidate": 0, "tie": 0}
    for case in cases:
        winner = judge_pair(baseline[case["id"]], candidate[case["id"]], ask)
        tally[{"left": "baseline", "right": "candidate"}.get(winner, "tie")] += 1
    return tally

The cost of this is exactly double, and the tie rate is the diagnostic worth watching. A judge that ties on sixty per cent of pairs is telling the team that its verdicts are mostly position and that the rubric needs work.

Validating the judge against human labels

This is the step teams skip, and skipping it turns an eval into theatre. A judge that has never been compared with a person produces numbers of unknown meaning, and those numbers then gate releases.

The procedure takes an afternoon. A person grades a sample of cases against the same rubric, without seeing the judge's scores. The judge grades the same cases. The two label sets are compared, and the comparison uses agreement corrected for chance, because raw agreement flatters any grader on an unbalanced set.

A worked example makes the correction visible. A hundred and twenty cases are graded by both, and the verdicts land like this.

Raw agreement is ninety two plus sixteen over a hundred and twenty, which is ninety per cent and looks excellent. Both graders pass about eighty per cent of cases, so two graders assigning labels at random in those proportions would agree seventy per cent of the time by chance alone. Cohen's kappa removes that floor and reports 0.67.

from collections import Counter

def cohens_kappa(human, judge):
    """Agreement between two raters with chance agreement removed.

    Raw agreement is the share of cases where the two labels match. On a
    set where four cases in five pass, two graders who never looked at
    anything would match most of the time, so the raw figure says little.
    """
    n = len(human)
    observed = sum(h == j for h, j in zip(human, judge)) / n
    hc, jc = Counter(human), Counter(judge)
    labels = set(human) | set(judge)
    expected = sum((hc[k] / n) * (jc[k] / n) for k in labels)
    return (observed - expected) / (1 - expected)

# observed 0.90, expected 0.70, kappa 0.67

Under the benchmarks J. Richard Landis and Gary Koch published in Biometrics in March 1977, a kappa between 0.61 and 0.80 counts as substantial agreement and anything above 0.81 as almost perfect. Those bands are conventions, useful because they give a team somewhere to start. A judge below about 0.6 is not ready to gate anything, and the repair is the rubric and never the model.

Two rules keep the validation useful after that afternoon. The team regrades the sample whenever the rubric changes, the judge model changes or the feature changes what it produces. And the human labels are kept, because they become the regression set for the judge itself.

Criteria drift and why the rubric is written while grading

One finding explains why a rubric written in a meeting never survives contact with the output. Shreya Shankar and colleagues reported it at the ACM Symposium on User Interface Software and Technology in October 2024, in a paper asking how evaluation carried out by a model can be brought into line with what people actually want. People do not hold a fixed standard while grading. They refine their criteria as they read, and reading a surprising output is often what produces the criterion.

The paper calls this criteria drift, and it has a direct consequence for practice. A rubric drafted before anybody has graded twenty real outputs is a guess, and a rubric that stops changing after a hundred is finished. The order that works is to grade first, write the criterion the grading exposed, then regrade the same cases against it and see what moves.

That order also explains a common failure. A team writes a rubric, builds a judge, runs three hundred cases and gets a number. Nobody has read the output, so nobody knows whether the number describes the thing anybody cared about.

What a judge cannot do

A judge inherits the limits of the model behind it, and three of those limits matter for a build.

It cannot verify a fact that is absent from its context. Grounding is checkable because the passages are in the prompt. Whether a statement about the world is true is not, and a rubric that asks for it is asking the judge to hallucinate a verdict.

It cannot hold a numerical scale steady across months. The distribution of a judge's scores shifts when the provider updates the model underneath, which is why a judge is pinned to a model version and rerun against the human labels whenever that version moves.

And it cannot tell a team what it should have been measuring. The rubric carries that decision, the rubric was written by people, and every bias in the rubric is reproduced at volume and at speed.

A validated judge gives a team a score it can defend on three hundred cases in under a minute. That score is worth having only when it stops something bad from shipping, so it runs before the release rather than after it. The next page is about the gate.