Evaluation Reliability Benchmark
Testing whether an evaluation system responds correctly to score variability, and whether repeated evaluation produces more stable model choices.
Under one-shot evaluation, the selected model changed across trials in 11 of 45 single-turn chat cases (24.44%). With repeated evaluation, it changed in 3 of 45 cases (6.67%). This represents a 17.78 percentage-point absolute reduction and a 72.73% relative reduction.
Resources for this report
Primary question
Can an evaluation system recognize when it has enough reliable evidence, and does repeating an evaluation produce a more stable model choice?
Evaluation reliability, in the sense used here, is the property that an evaluation procedure produces the same qualitative conclusion when repeated under equivalent conditions. It is distinct from the accuracy of any single judgement: an evaluation can be locally correct on every call and still reverse its overall verdict from one execution to the next.
A conclusion drawn from a single evaluation is a point estimate with unreported variance. Whether that estimate is trustworthy depends on how often the same procedure, given the same inputs, would land on a different model choice.
The two questions below are linked. The first asks whether the system makes the right continue-or-stop decision as scores stabilise. The second asks whether the model-selection conclusions survive repetition once execution noise is introduced.
Question 1
Knowing When Evaluation Evidence Is Reliable
Does the evaluation system continue collecting evidence while scores remain variable and stop when the evidence becomes sufficiently consistent?
Question 2
Making Model Selection More Repeatable
Does repeated evaluation reduce how often the selected model changes across fresh trials?
The headline result is a 72.73% relative reduction in unstable model choices, from 24.44% under one-shot evaluation to 6.67% under repeated evaluation. That is a 17.78 percentage-point absolute reduction.
Subjects
Gemini 2.5 Pro vs. Gemini 2.5 Flash-Lite
Independent judge
Gemini 3.1 Pro Preview
Single-turn chat cases
45 controlled, held-fixed
Model-comparison evaluations
16 (8 single, 8 repeated)
Executions per repeated evaluation
10 (every treatment reached the ceiling)
Winner rule
Model-winner change · practical-equivalence transition tracked separately
- 38 measurement-reliability lifecycles
- 45 controlled model-comparison single-turn chat cases
- 16 model-comparison evaluations
- 8 single-evaluation trials
- 8 repeated-evaluation trials
- Gemini 2.5 Pro compared with Gemini 2.5 Flash-Lite
- Gemini 3.1 Pro Preview as the independent judge
- Every completed repeated-evaluation treatment reached 10 executions
Relative Reduction in Model-Choice Instability
Running multiple iterations per evaluation eliminates phantom variance, protecting product teams from choosing the wrong model based on run-to-run noise.
Nearly 1 in 4 runs flipped winner outcomes due to minor API fluctuations.
Systematically flattens out outlier responses to lock in ground truth.
The evidence supports repeated execution and score aggregation for the tested model pair, judge, single-turn chat-case domain, and evaluation configuration. Every completed repeated-evaluation treatment reached ten executions, so the study does not isolate a separate causal effect from adaptive early stopping.
- Two model subjects and one judge. The reliability numbers describe this triple; generalisation to other pairings requires replication.
- 45 single-turn chat cases sample a curated task distribution across four workspaces, not an unweighted universe of real user prompts.
- Every completed repeated-evaluation treatment reached ten executions. The plateau observed by the tenth execution is consistent with — but does not prove — that further repetition would not change the conclusion, and the study does not isolate a separate effect from adaptive early stopping.
- Prompts, rubrics, and raw model outputs remain private for integrity reasons; aggregates and per-case indicators are released.