Why single-judge LLM scoring is structurally unreliable, and what to do about it
Your evaluation ran. The scores came back. Model A scored 4.3 on helpfulness. Model B scored 3.9. You picked Model A.
Here's the uncomfortable question: run the same evaluation again right now, same models, same scenarios, same rubric — will you actually get the same result?
In practice, often not. And the gap between those two runs is not a quirk or configuration bug; it's baked into how LLM-as-a-judge scoring works.
1. The variance problem, explained plainly
LLM evaluation has a compounding noise problem that most teams discover too late.
The model under test is stochastic. Give it the same prompt twice, and it generates two different outputs. That is expected and well understood. What is less understood is what happens next: the judge model that scores those outputs is also stochastic. It reads a response and assigns a score, but if you ask it again, it may assign a different one. Not wildly different, but different enough to matter when you are trying to distinguish between models that are close in quality.
Stack those two sources of variance, and a single evaluation run gives you one sample from a distribution, not a reliable estimate of where that distribution actually sits. Presenting that sample as "Model A scored 4.3" implies a precision you do not have.
We saw this repeatedly, building evaluation infrastructure at Plumloom. A team would run an evaluation, get a clear winner, ship the decision, then re-run a week later on the same scenarios and watch the rankings shift. The underlying scenarios, models, and rubric hadn't materially changed. What changed was the particular roll of the dice on that run.
This is not a vendor problem or a tooling problem. It is a structural property of using probabilistic systems to score probabilistic outputs.
2. Why this matters more than most teams realize
The obvious cost is bad model selection: you pick the wrong model because the noise in your evaluation favored it that day.
The less obvious cost is false confidence in the process itself. When evaluation infrastructure returns a number, it implicitly claims reliability. Dashboards, leaderboards, and release gates all communicate certainty. If the underlying measurement is noisy, every downstream decision inherits that noise without realizing it.
There is also a subtler problem in high-stakes evaluation scenarios: borderline cases. When two models score close together, the noise is often larger than the gap you are trying to measure. A single-judge run in that situation is not telling you which model is better. It is telling you which model got lucky on that pass, relative to the judge's internal randomness.
3. The k-judge idea
The fix draws on a simple principle: if a single measurement is noisy, take multiple independent measurements and aggregate them.
This is not a new idea in measurement generally. Clinical trials do not rely on a single patient's response. Polling aggregators average across polls rather than relying on a single survey. Peer review uses multiple reviewers precisely because a single reviewer's judgment is unreliable in isolation.
Applied to LLM evaluation, the same logic holds. Instead of calling your judge model once per scenario and treating the result as ground truth, you call it K times independently and aggregate the scores. Each call is a separate draw from the distribution. Aggregating across K draws gives you an estimate that is more stable than any individual call.
We call this k-judge scoring and treat it as the default for decisions that actually matter.
How the distributions tighten as K grows:
- K = 1 — One pass. One roll of the dice. The curves overlap heavily and rankings can flip on noise alone.
- K = 3 — Noise starts to average out. Borderline calls become more trustworthy and the curves begin to separate.
- K = 5 — The gap between models starts to outweigh the variance. Most high-stakes comparisons are reliable here.
- K = 9 — Strong signal. The curves are clearly separated and wrong-pick risk drops to a fraction of K=1 levels.
- K = 15 — Near-deterministic. Variance has been squeezed out and the ranking reflects true model quality.
The mechanics are straightforward: run the same judge prompt K times against the same model output, collect K independent scores, and aggregate. Majority vote works well for categorical judgments. Averaging with variance tracking works well for continuous scores. The key is that each call is genuinely independent — not a retried API error, but a fresh judgment.
4. What k-judge actually changes
The most direct benefit is score stability. A single judge call on a borderline scenario might return a 3 one time and a 4 the next. K independent calls with aggregation gives you a distribution: "this scenario scores around 3.2, with a spread that tells you how confident to be in that number."
That spread is itself useful information. Tight agreement across K calls indicates the score is reliable. Wide disagreement suggests the rubric may be ambiguous for this scenario type, or that the scenario is genuinely hard for the judge to score consistently. Either way, you learn something a single-judge setup would have hidden completely.
Model comparisons become more trustworthy because you can see whether the gap between two models is larger or smaller than the noise. If Model A scores 4.3 and Model B scores 3.9 on a single run, you cannot know if that 0.4 gap is real. If K independent scoring passes consistently place Model A above Model B with low variance, the comparison is credible. If the results flip across passes, the gap is not meaningful.
Release gates and regression checks also get more reliable. A single-judge gate that flips on noise produces false alarms and missed regressions. A k-judge gate that requires consistent evidence across multiple scoring passes is more conservative, which is the right property for a gate that controls what ships.
5. When K=1 is fine and when it is not
K-judge is not free. Each additional scoring pass adds cost and latency. Running K=3 triples your judge budget compared to K=1. That tradeoff is worth being deliberate about.
For quick, continuous spot checks where you want a directional signal rather than a precise measurement, K=1 is often sufficient. The cost of being slightly wrong on any individual check is low, and you are not making a high-stakes decision from a single run.
For decisions that actually matter, the calculation changes. Comparing models before a production deployment, running a regression gate on a prompt change, or establishing a baseline for a new evaluation suite: these are the cases where measurement error has real consequences. Spending more on judge calls in these scenarios is cheap compared to shipping the wrong model or missing a regression.
Borderline scores are the clearest signal that K=1 is not enough. Treat near-threshold results as a cue to spend more on judgment.
6. What to watch for in your own setup
If you are running LLM evaluation today, a few checks reveal whether you have a variance problem worth addressing.
Run the same evaluation twice, without intentionally changing anything, and compare the results. If rankings shift or scores move by more than a small margin, your single-judge pass is capturing noise as signal.
Look at your borderline decisions specifically. Any model that landed near a threshold on a single run deserves scrutiny. Re-running with multiple scoring passes on those cases costs little and catches the cases where noise is most likely to have changed the outcome.
Track judge agreement explicitly when you run multiple passes. High agreement is evidence the rubric is working well. Low agreement on specific scenario types is a signal that the rubric needs sharpening, not that one model is better than another.
7. The underlying shift
Single-judge scoring treats a probabilistic system as if it were a deterministic instrument. It returns a number, which is treated as a fact.
K-judge treats the judge for what it actually is: a noisy estimator. Multiple draws, aggregated, give you something closer to a measurement and a sense of how stable that measurement is.
The teams that build the most reliable evaluation infrastructure are the ones who instrument for uncertainty rather than hiding it behind point estimates. K-judge is one way to do that.
The k-judge approach described here is one of the patterns that inform how we design for evaluation reliability at Plumloom. If you are running evaluations in production and would like to discuss what this means for your setup, we'd like to hear from you.