A document assistant can give a correct-looking answer while relying on information outside the document it was asked to use.
RAG evaluation usually starts with a reasonable question: did the system produce the right answer?
For enterprise document agents, that leaves out a critical part of quality. The answer also needs to come from the information the application was expected to use.
Consider a security document agent asked to define risk.
A capable model already knows several common definitions. The approved source in our benchmark defines risk differently.
| Answer | |
|---|---|
| Question | What is risk? |
| Plausible LLM output | Risk = Threat × Vulnerability × Impact |
| Document truth | “The effect of uncertainty on objectives.” |
The first answer sounds expert. It belongs to the domain. A reviewer could easily approve it.
It still fails the document task.
The system answered from adjacent knowledge instead of following the source it was given.
A correct-looking answer does not prove that retrieval did its job.
1. The blind spot in standard RAG evaluation
Most RAG evaluation starts with answerable questions and measures whether the generated response resembles an expected answer.
That tells you something useful about answer quality.
It tells you much less about whether retrieval actually supplied the knowledge behind the answer.
A foundation model already carries broad knowledge about security, finance, law, policy, products, and common enterprise workflows. Retrieval adds source-specific evidence, while generation still has access to everything the model learned before the document ever arrived.
When those sources agree, the distinction disappears.
When they diverge, the model can quietly fill gaps from memory and produce an answer that looks completely reasonable.
We call this adjacent-knowledge bleed: the model supplies plausible domain knowledge that the retrieved evidence does not support.
As models become more capable, this problem becomes harder to spot by reading the final response. Fluency makes provenance less visible.
A RAG evaluation therefore needs to measure more than whether an answer sounds correct. It needs to test whether the system stayed inside the knowledge boundary the product promised to use.
2. The corpus is an evaluation contract
We designed the benchmark from the corpus outward.
Before writing evaluation cases, we checked which concepts were clearly present, which were absent, which had partial coverage, and which had document-specific meanings that differed from common industry language.
That establishes the expected behavior.
A fully supported concept should produce a grounded answer. Partial coverage should produce a qualified answer. Missing information should produce a refusal. A familiar term with a document-specific meaning should follow the document.
The corpus defines the information contract. The evaluation should test its boundaries.
That contract matters in enterprise systems because the source often carries authority.
A policy assistant needs to follow the approved policy. A compliance agent needs to follow the applicable standard. A contract assistant needs to stay faithful to the contract. A support agent may need to answer from the company’s current documentation rather than from generic product knowledge.
A plausible answer from the wrong knowledge source can create policy violations, incorrect customer guidance, or decisions that become difficult to defend later.
The evaluation needs to expose that failure before release.
3. Four ways to make knowledge bleed visible
We treated corpus-boundary evaluation as an OOD-style test design problem.
1. Audit the corpus before writing the tests
We first probed the corpus for concepts that were present, absent, sparsely covered, or especially likely to trigger outside model knowledge.
This tells you where the interesting boundaries are.
An absent concept can test refusal. Sparse evidence can test partial coverage. A common term with specialized document wording can test whether prior model knowledge overrides retrieval.
2. Build cases around behavior, not only questions
The 20-case benchmark included direct questions, multi-part retrieval, refusals, partial coverage, adjacent-knowledge cases, and search-trajectory cases.
Across expected outcomes, the suite contained 13 answers, five refusals, and two partial answers.
That mix matters because a production RAG agent has a broader job than answering supported questions.
It also needs to recognize incomplete evidence, keep searching when appropriate, qualify an answer when coverage is partial, and stop when the corpus cannot support the request.
3. Target definition shifts
Definition shifts are especially useful because they create a conflict between what the model probably knows and what the document actually says.
Risk was one example.
A normal factual question provides weak evidence of retrieval when the model already knows the answer. A source-specific definition creates a stronger test.
The evaluation can now ask whether the system followed its retrieved evidence or substituted a familiar answer from prior knowledge.
4. Verify the tests against the corpus
Every benchmark case was checked against the extracted source before the agent ran.
Supported claims had to exist. Absence conditions had to be real. Page references and expected content had to align.
This protects the validity of the evaluation itself.
A broken test can make a healthy system look wrong. Corpus verification separates benchmark-authoring errors from actual RAG failures.
The full 20-case benchmark and public result data are available in the Hugging Face dataset, with the implementation and experiment artifacts in the GitHub repository.
4. The final answer only shows the endpoint
The RAG agent in the experiment used iterative retrieval. It searched the document, assessed whether the evidence was sufficient, refined queries when coverage remained incomplete, and produced cited answers.
Each execution also preserved the trace.
Across the 20 cases, 19 produced the expected answer, refusal, or partial-answer class. All five expected refusal behaviors were observed.
One failure shows why execution evidence matters.
The agent was asked for three definitions that all existed in the source. Retrieval found one, missed two, exhausted its search rounds, and returned no substantive answer.
The final response showed that the task failed.
The trace showed where the workflow broke: retrieval never assembled enough evidence before the agent stopped.
A deterministic grounding check still passed because the empty response contained no unsupported claim. The trace revealed the larger task failure.
A failed answer gives you an outcome. The trace gives you a failure location.
That difference changes what an engineering team can do next.
A generic low score says the system performed poorly. A trace showing that retrieval missed two required definitions before stopping points directly toward the part of the workflow that needs investigation.
5. Test whether the evaluator detects a known regression
Evaluating the RAG is only half the problem.
The evaluation itself also needs evidence that it responds when quality changes.
We created eight matched pairs.
Each pair started with a real RAG execution whose answer was supported by the retrieved evidence. We then created a controlled twin that preserved the retrieval trajectory while adding unsupported claims to the final answer.
The retrieval path stayed fixed. The answer became worse in a known way.
Under the first judge, every fabricated answer scored below its matched honest answer: 8 out of 8.
We repeated the evaluation with a second judge on the same frozen artifacts.
Again: 8 out of 8.
Across both judges, fabricated answers scored below their matched honest versions in 16 out of 16 paired comparisons.
Factuality and Relevance carried the strongest separation under both judge configurations.
The public result tables and matched-pair data are available in the dataset, and the underlying evaluation artifacts are available in the repository.
The matched design matters because every execution has its own control.
Instead of asking whether a collection of “bad” examples scores lower on average than a collection of “good” examples, you can make a known degradation to the same execution and see whether the evaluation moves with it.
A useful evaluator should move when the behavior you care about moves.
6. RAG evaluation needs to test the information boundary
The lesson from this experiment goes beyond one corpus.
A strong RAG evaluation set covers the information contract of the product.
It tests supported answers, partial coverage, missing information, source-specific definitions, retrieval behavior, citations, and controlled negative examples.
It also preserves the execution evidence needed to explain failures.
That gives product and engineering teams a much stronger basis for release decisions.
They can evaluate whether the system stayed within its intended knowledge boundary, whether retrieval supplied the evidence the task required, whether known degradations were detected, and whether the important findings survive evaluator variation.
The practical implication becomes more important as foundation models improve:
The better the model becomes, the less a plausible answer tells you about where that answer came from.
The full Plumloom RAG Evaluation Benchmark, including methodology and results, is available at plumloom.ai/research/rag-evaluation-benchmark.
Plumloom Trueline is building evaluation infrastructure for teams that need reliable evidence before shipping RAG and agent products.