A failed answer tells you that the task failed. A useful evaluation tells you where the failure occurred, whether the evaluator can detect it, and whether the resulting signal is reliable enough to support a release decision.
A document agent returns an empty answer to a question the source can answer.
The product team already knows the outcome is bad. The next decision requires more evidence.
Did retrieval miss the required evidence? Did the agent stop too early? Did generation receive enough context? Will the same behavior happen again? Does the evaluator itself produce a reliable signal?
A final-answer score cannot answer those questions by itself.
The evaluation needs to follow the execution.
Release evidence starts where the final answer ends.
1. The first job is to locate the failure
Our benchmark contained 20 corpus-verified cases for a document-grounded RAG agent.
One case asked for three definitions that all existed in the source.
The agent completed four search rounds. It retrieved one definition, missed the other two, reached its search limit, and returned no substantive answer.
A deterministic grounding check still passed because the empty response contained no unsupported claims.
That check captured one property correctly: the answer did not invent unsupported content.
The trace captured the broader workflow failure.
Retrieval never acquired enough evidence to complete the task before the agent stopped.
That changes the diagnosis.
"The answer failed" describes the outcome.
"Retrieval missed two required definitions before stopping" identifies where the workflow broke.
A failed answer gives you an outcome. The trace gives you a failure location.
This distinction matters because engineering work starts from the failure location.
A generic low score leaves several parts of the system under suspicion. A trace showing incomplete retrieval and a stopping limit narrows the investigation.
2. The evaluator also needs to prove that it responds to quality
Once a team evaluates the trace, another question appears: does the measurement move when quality changes in a known direction?
We created eight matched honest and fabricated pairs.
Each fabricated version preserved the original retrieval trajectory and added unsupported claims to the final answer.
The case stayed the same. The retrieval path stayed the same. The evidence stayed the same.
The answer quality changed in a controlled way.
Under DeepSeek V4 Pro, all eight fabricated versions scored below their honest counterparts.
The overall gaps ranged from 0.1 to 1.9 points. Factuality and Relevance showed the largest average separation.
That is stronger evidence than comparing averages from unrelated "good" and "bad" examples.
Each execution becomes its own control.
The full matched-pair data and judge results are available in the Hugging Face dataset, with the underlying experiment artifacts in the GitHub repository.
A useful evaluator should move when the behavior you care about moves.
3. One judge is another source of uncertainty
LLM-based evaluation introduces another model into the measurement system.
That means a strong result deserves a reliability check of its own.
We froze the 16 matched artifacts and evaluated them again with a second judge. The traces, expected outcomes, rubric, metrics, and evaluation setup remained fixed.
Only the judge changed.
The central result held.
All eight fabricated answers again scored below their matched honest answers.
Across both judges, the experiment produced 16 out of 16 honest-over-fabricated comparisons.
Factuality remained the strongest separating metric. Relevance remained second.
Absolute scores moved between judges. The decision-relevant conclusion remained stable.
That is the useful signal.
Judge replication is about preserving the conclusion, rather than reproducing identical numbers.
For release work, this matters because one judge score can reflect the evaluated system, the rubric, the judge, or an interaction among them.
Frozen-artifact replication helps separate those effects.
4. A trace explains one execution
A frozen trace is detailed evidence of one real run.
It shows retrieval, evidence use, stopping, citations, and the final outcome.
Release decisions often need another layer of evidence: how consistently does the relevant behavior appear when generation and judging are repeated?
We used the retrieval failure to create a focused Scenario evaluation.
One information state contained the passages reached by the original RAG execution. It contained only part of the evidence required for the answer.
The second contained the relevant glossary evidence for all three requested definitions.
The task, expected outcome, generation setup, evaluation criteria, and judge remained fixed. The supplied information changed.
With the retrieved-only evidence, the model produced a faithful partial answer and identified the missing coverage.
With the glossary-focused evidence, it produced the complete answer.
Both conditions met the configured reliability criterion. Fresh evaluations reproduced their aggregate means closely, within 0.003 for one condition and 0.068 for the other.
The confidence intervals overlapped, so the experiment does not support a causal claim about one isolated context change.
It does support a different conclusion: each controlled information state produced a repeatable aggregate behavior.
That gives the team something more useful than one successful replay.
5. Failure diagnosis becomes a reusable evaluation asset
This is where the pieces connect.
The trace identifies where the real execution failed.
The matched controls test whether the evaluator detects a known degradation.
The second judge tests whether the conclusion survives evaluator variation.
The repeated Scenario evaluation measures whether the isolated behavior produces a consistent signal.
Each layer answers a different release question.
Together, they produce evidence that can be inspected, challenged, repeated, and reused when the prompt, model, retriever, or agent changes.
The goal of evaluation is evidence strong enough to support a release decision.
That is a different standard from "the demo looked good" or "the average score went up."
It asks whether the team can explain the failure, verify the evaluator, reproduce the finding, and carry the result forward as a regression test.
6. What a stronger RAG release packet looks like
A useful release packet can include the corpus contract, verified evaluation cases, frozen traces, matched negative controls, judge replication, and repeated Scenario results.
Those artifacts answer different questions.
The case suite tells you what behavior the product is expected to produce.
The trace tells you how the system actually behaved.
The matched pair tests whether the evaluator detects a known regression.
The second judge tests whether the conclusion depends on one evaluator.
The repeated Scenario run measures whether the isolated behavior is stable enough to support a decision.
This gives product and engineering teams a stronger basis for deciding whether to ship, constrain the release, or keep iterating.
The full Plumloom RAG Evaluation Benchmark, including methodology and results, is available at plumloom.ai/research/rag-evaluation-benchmark.
Plumloom is building evaluation infrastructure for teams that need reliable evidence before shipping RAG and agent products.