LLM-as-Judge: Using AI to Evaluate AI Output

TL;DR: LLM-as-judge means using one AI model to score or compare another model's output against a rubric, instead of relying only on human review. Research on GPT-4 judges found over 80% agreement with human evaluators on open-ended answers — but also documented position bias, verbosity bias, and an inconclusive self-enhancement effect. Use it to triage volume, not to replace a human on anything you can't afford to get wrong.

What does "LLM as judge" actually mean?

LLM as judge is exactly what it sounds like: you give a model an output (a chatbot reply, a piece of generated code, a summary, a marketing draft) plus a rubric, and you ask it to score that output, or pick the better of two candidates, instead of a human doing that comparison by hand.

It exists because human evaluation doesn't scale. If you're running a prompt through a hundred test cases every time you tweak it, or comparing two models across a thousand generated answers, you cannot have a person read all of it every time. An LLM judge turns evaluation into something you can run in a loop, the same way a test suite runs in a loop. The difference is that a rubric replaces the assertion, and the check is fuzzy instead of exact — which is exactly why the rest of this piece is about keeping that fuzziness under control.

That framing matters because it sets the right expectation from the start: a judge model is a triage tool, not an oracle. It tells you where to look. It does not, on its own, tell you the truth.

How does an LLM-as-judge prompt actually work?

There are two common shapes, and picking the right one matters more than most people expect.

Pairwise comparison shows the judge two candidate answers to the same input and asks which is better, or whether it's a tie. This is the shape used for head-to-head model comparisons and for A/B-testing a prompt change against the version it's replacing.

You are evaluating two responses to the same user request.

User request:
{user_request}

Response A:
{response_a}

Response B:
{response_b}

Compare the two responses on: accuracy, completeness, and clarity.
Ignore length — a longer response is not automatically better.
State which response is better, or "tie" if genuinely equal, then
give a one-sentence reason. Do not explain before your verdict.

Single-answer grading shows the judge one answer and a rubric, and asks for a score, with no second answer to compare against. This is the shape you want for a regression suite, where you need an independent score per output rather than a relative ranking.

You are scoring one response against a rubric. Score each dimension
1-5 and give a one-line reason per score. Do not average the scores
yourself — list them.

Rubric:
1. Factual accuracy — does every claim hold up against the provided source?
2. Completeness — does it address every part of the request?
3. Format compliance — does it follow the requested structure exactly?

Request:
{user_request}

Source material (if applicable):
{source_material}

Response to score:
{response}

Pairwise comparison tends to be more reliable for close calls, because relative judgments are usually easier for a model than absolute ones. Single-answer grading scales better when you're scoring outputs independently and don't have a natural "opponent" answer to compare against.

In practice, most real evaluation setups end up using both, at different stages. Single-answer grading runs cheaply and continuously, as a first-pass filter across every output a system produces, flagging anything that scores low on any rubric dimension. Pairwise comparison then gets reserved for the smaller, higher-stakes set of decisions: is this prompt change actually better than what it's replacing, or is this new model version worth switching to. That split matters because pairwise comparisons require you to already have two candidates worth comparing, which is a much smaller and slower-moving set than "every output the system generates today."

What does the actual research say about LLM-as-judge accuracy?

The research most people gesture at without reading is Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023 Datasets and Benchmarks Track; arXiv:2306.05685). It's worth reading directly, because it's more specific, and more limited, than the way it usually gets cited.

The setup: the authors generated answers to 80 questions (MT-Bench) from six models: GPT-4, GPT-3.5, Claude-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B. They then had both LLM judges and around 58 expert human labelers score those answers, either head-to-head or on an absolute scale. They also ran a second, larger comparison on Chatbot Arena, a crowdsourced battle platform. The headline result, in the paper's own words: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans." In plain terms, a GPT-4 judge disagreed with a human rater about as often as two different human raters disagreed with each other.

That's a genuinely strong result for open-ended, subjective answers: tone, helpfulness, structure. It is not a blanket result, and treating it as one is the single most common way this paper gets misquoted. The same paper is explicit that LLM judges have limited capability in grading math and reasoning questions, because a judge that cannot solve a problem itself has no reliable way to check whether a proposed solution is actually correct; it can only judge how convincing the explanation sounds.

The distinction the "self-enhancement bias" citation usually gets wrong

This is the part worth slowing down on, because it's the single most commonly overstated claim about LLM-as-judge, and the source of the overstatement is a real, careful paper being cited past what it actually tested.

The paper does document something it calls self-enhancement bias, defined as "the effect that LLM judges may favor the answers generated by themselves." Here's exactly what they found: "GPT-4 favors itself with a 10% higher win rate; Claude-v1 favors itself with a 25% higher win rate. However, they also favor other models and GPT-3.5 does not favor itself." Their own conclusion about that data is more cautious than the finding usually gets repeated elsewhere: due to limited data and small differences, they say the study cannot determine whether the models exhibit a self-enhancement bias.

Now look closely at what that test actually was. Six models each answered the same set of questions, independently, in six separate generation runs, not in the same conversation as the judging. The judge then saw two of those answers, anonymized, with no label saying which model wrote which, and picked a winner. Only after aggregating thousands of these blind battles could the researchers check, statistically, whether the answer that happened to come from the same model as the judge won more often than a human would have picked it.

That is a materially easier case than the one most people actually mean when they ask whether a model can grade its own work: writing something, then in the same conversation, asking the model to grade what it just wrote. That scenario combines whatever self-preference exists with something the MT-Bench test structurally can't measure: the model isn't looking at an anonymized transcript it has no memory of; it's looking at its own immediately-prior turn, with every reason to stay consistent with what it just asserted. The paper never tests that setup, and its "cannot determine" finding about the much easier, blind, cross-model comparison should not be read as reassurance about the harder, same-conversation one.

If you need a model to check its own output, the cheapest mitigation the paper's own design implies is structural: run the grading as a separate call, without the drafting conversation in context, so the judge is scoring a transcript rather than defending a turn it just took.

The other two biases you need to prompt around

Self-enhancement isn't the only documented failure mode, and the other two are easier to trigger by accident.

Position bias. The paper defines this plainly: "Position bias is when an LLM exhibits a propensity to favor certain positions over others." In their worked example, GPT-4 judged two answers to the same question from two different models, and whichever answer it saw first, it called more detailed and superior. Then, when the two answers simply swapped positions with nothing else changed, GPT-4's verdict flipped to favor the other one. The practical fix is mechanical: run every pairwise comparison twice, with the order swapped, and only trust a verdict that holds both times. Anything that flips is a tie you should treat as a tie, not a real preference.

Verbosity bias. Also defined directly in the paper: "Verbosity bias is when an LLM judge favors longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives." The researchers tested this with what they called a "repetitive list" attack: taking a correct, concise answer and padding it with restated, non-additive content, then checking whether judges still preferred the original. Judges were fooled a meaningful share of the time. The fix belongs in the rubric itself: tell the judge explicitly to ignore length and to penalize restated or padded content, the way both prompt templates above already do.

A rubric-based judge prompt you can adapt

The pattern that holds up best across both papers and practice is: name the exact failure modes you're guarding against, in the rubric itself, rather than asking for a vague "quality" score. Opening with "you are an evaluation judge, not the original assistant" is a small persona prompt doing real work: it puts distance between the model's identity in this call and whatever it wrote in a different one.

You are an evaluation judge, not the original assistant. You did not
write either response and have no attachment to either one.

Task: {task_description}

Response A:
{response_a}

Response B:
{response_b}

Score each response 1-5 on:
1. Correctness — every factual or computational claim checks out.
2. Completeness — every part of the task is addressed.
3. Concision — no restated content or padding; ignore total length.

Then state an overall winner or "tie." If your verdict would change
by swapping which response is A and which is B, say so and mark the
result "position-sensitive — needs a second pass."

Running the same judge prompt more than once, and checking whether the verdict holds (swapped order, or a second independent sample), borrows the same idea as self-consistency prompting for reasoning tasks: one pass from a probabilistic system is a sample, not a certainty, and a verdict that only shows up once is weaker evidence than one that repeats.

Where to actually stop trusting the judge

For anything with a single objectively correct answer that the judge itself cannot independently verify (arithmetic, exact code output, a factual lookup), an LLM judge is the wrong tool, or at best a second-tier one behind a deterministic check. The MT-Bench findings on math and reasoning grading are a specific instance of a general problem: a judge that cannot solve the problem is scoring how convincing the answer sounds, which correlates with quality but is not the same thing as correctness. If you're evaluating code, run the tests; if you're evaluating a calculation, check the number. Save the judge for the genuinely open-ended half of the task: tone, structure, completeness, whether the response actually answers what was asked.

That's also where a well-built prompt template earns its keep: the rubric above works far better as a saved, reused evaluation prompt than as something rebuilt from memory each time, because the exact wording of "ignore length" and "state if position-sensitive" is what keeps the biases in check; drop either line and the old failure modes come back. If you're building judge prompts for a real workflow rather than a one-off check, the same discipline that makes any prompt reliable (explicit constraints, saved and reused rather than retyped) is covered in our guide to what prompt engineering actually is and in the prompt engineering cheat sheet. If the output you're judging is code rather than prose, our companion piece on prompting for defensive security review covers the adjacent case of using AI to flag problems in your own code — with the same caveat that its output is a lead list for a human, not a clearance.