Don't Trust a Single Judge
The case for grading AI with a jury
How AI quality assurance borrowed the courtroom jury, and why the winning panel is exactly three.
When you need to know whether an AI's answer is any good, the reflex is to ask a better AI. Point your strongest model at the output, hand it a rubric, let it score. It has a name now, LLM-as-a-Judge, and it works well enough that most teams have quietly made it the default. That is the problem.
Here's what nobody tells you: the model holding the gavel has a house style, and it marks up its own reflection. The paper that made LLM-as-a-Judge respectable, Zheng and colleagues in 2023, also listed its tells. A strong model agrees with humans more than 80% of the time, about as often as two humans agree with each other. Reassuring, until the next line: the same model favours its own phrasing, rewards the longer answer, and trusts whatever it read first. Give one model the bench and you don't get an impartial verdict. You get one model's blind spots, at scale, with a confident number attached.
Why you swear in a jury
We settled this a long time ago, and not in a lab. No court convicts on the word of one juror. When a judgement is subjective and the stakes are real, you don't hunt for a cleverer individual, you swear in a panel of different people and take the majority. The reliability lives in the disagreement, not in any single head. There is a reason juries run to an odd number: someone has to break the tie.
Swap the jurors for AI models and the logic holds, with one condition you cannot skip. In 2024, Verga and colleagues tested it directly, in a paper with the honest title "Replacing Judges with Juries". A panel of smaller models from different families beat a single large judge, showed less bias, and cost over seven times less to run. The word doing the work is families. Three models from the same lineage are not a jury; they share the same blind spots, so their errors stack instead of cancelling. Three different lineages point their mistakes in different directions, and mostly cancel out.
The lineage
- Before 2023 Human juries The wisdom of crowds, long before AI: use an odd number of raters and take the majority verdict. Never trust a single annotator. Inter-rater reliability: Cohen's kappa, Krippendorff's alpha
- June 2023 LLM-as-a-Judge A strong model can grade another. GPT-4 matches human preference more than 80% of the time, but favours its own text, longer answers, and whatever it reads first. Zheng et al., "Judging LLM-as-a-Judge", NeurIPS 2023
- April 2024 Judges become juries Swap the lone judge for a panel of different model families. It beats a single large judge, shows less bias, and costs over seven times less. Verga et al., "Replacing Judges with Juries", 2024
- 2025 to today Gated in the pipeline Calibrate each judge against known-good and known-bad cases. If the panel drifts after a model update, the release stops. No spend on unreliable runs. Continuous evaluation practice: LangSmith, Arize, Braintrust
So how does that panel actually work?
The panel at work
One output. Three lineages. Majority rules.
AI output under test
- Judge 1 Claude Anthropic lineage Binary verdict Yes
- Judge 2 GPT OpenAI lineage Binary verdict Yes
- Judge 3 Gemini Google lineage Binary verdict No
Majority vote
Verdict: pass
Two yes, one no. The majority carries it.
Ask for facts, not moods
The questions you hand the jury matter as much as who sits on it. Ask "is this response helpful?" and you are asking for a mood; three models will give you three. Ask "did the reply tell the user their account was locked, yes or no?" and you are asking for a fact. Binary questions are what let you add three verdicts into one honest number, and see exactly where a feature broke rather than watching a vague score sag.
Test the jury before you trust it
One more thing the courtroom gets right: you vet the jury before you rely on it. Models drift when a provider updates its backend, quietly changing how your judge behaves. So you calibrate the panel against cases you already know the answer to, and if it cannot mark the answer key, it does not get to mark your product. Wire that check into your pipeline and a drifting judge stops the release instead of lying to you politely.
What to do
This applies wherever a model's verdict decides what happens next. Marking the quality of an AI's output is one case. An agent deciding whether to approve, escalate or stop is another.
- Find the decisions. List every point where a model's verdict decides what happens next: a quality score, an approval, an escalation, a release.
- Use three models from different makers. Same maker, same blind spots.
- Ask yes or no questions. One fact per question.
- Take the majority. When the vote splits on something that matters, send it to a person.
- Test the panel first. Run it against cases a person has already decided, before every release and after every model update. If it fails, nothing ships.
If code can check the answer, let code check it. A jury is for judgement calls.
The test
Would three different models, asked the same yes or no question, reach the same verdict?
If you only ever asked one, you do not actually know.
Three is the number worth remembering. Enough difference to cancel the bias, few enough to keep the bill and the wait reasonable. Add a fourth or a fifth and you are mostly paying for a rounding error.
The surprise was never that AI can grade AI. It is that the trustworthy verdict turned out to be the one three different minds could agree on, not the one the cleverest model handed down alone.
Sources
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023. arxiv.org/abs/2306.05685
- Verga et al. (2024), Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arxiv.org/abs/2404.18796
Take this with you
The diagram as a one-page PDF: the lineage from human juries to model panels, and how three judges reach a verdict.