An online marketplace team recently tested two leading models for a customer support agent. Both scored 4.7 out of 5 on average — empathetic, clear, policy-aware. On paper, either model would ship. But buried inside one model's otherwise excellent response to an ambiguous escalation scenario was a single sentence: "If you'd like, I can open the case now." That sentence breaks platform policy. At scale, it would generate thousands of unnecessary disputes, trigger seller penalties, and flood the human support queue.
This isn't an edge case. It's the default outcome when teams treat evaluation as a scoring exercise instead of a quality discipline.
Today, much of the industry conversation around AI evaluation is centered on benchmarking mechanics: datasets, judges, scoring methods, reproducibility, confidence intervals, and leaderboards. These matter. But they are not the whole picture. A model can ace benchmarks and still frustrate users. It can optimize dimensions nobody values while ignoring the ones that determine whether someone trusts your LLM-powered product or abandons it.
Most evaluation conversations focus on how to measure models — better judges, better datasets, better scoring methods. Far fewer address the upstream question that product teams answer first: what should this system actually optimize for, and what failures matter most to the users experiencing it? Without that question, even rigorous measurement is directionless. Evaluation, done well, is a product quality discipline. It starts with understanding how users experience AI and building evaluation around that.
The underlying principle is simple: before you can measure effectively, you need to know what you're measuring and why. Most teams get this backward; they start with scoring and work backward to justify it. This framework starts with the decision you're trying to make and works forward to the evidence you need.
The framework organizes evaluation into five phases: Frame, Design, Measure, Interpret, and Iterate. Each moves from broad exploration to focused commitment. Together, they form a continuous practice rather than a one-time gate.
1. Frame
How might we determine what this AI system should actually optimize for?
Frame produces the Evaluation Strategy, the artifact that governs every decision downstream. Without it, teams default to generic rubrics, measure everything equally, and end up with scorecards that say nothing about whether the system serves its purpose. Most evaluation efforts fail here at framing, long before measurement begins. Teams jump straight to metrics without answering a more fundamental question: what is this AI system for?
What is it optimizing for? The outcome it exists to create, the users whose experience it shapes, and the failures that would undermine that purpose.
A PM shipping a customer support bot and a PM shipping a coding assistant are optimizing for fundamentally different things. For the support bot, the system exists to protect trust between a platform and its users. Tone and safety define whether the product fulfills its purpose. One wrong response to a distressed customer breaks a promise, and no score captures that. For the coding assistant, the system exists to accelerate developers' workflows. Accuracy is the entire contract. Developers care less about whether the AI is friendly. They care if the code compiles.
Yet most teams evaluate both products against the same generic rubric. Helpfulness. Relevance. Coherence. Accuracy. It's comprehensive. It's also directionless, because it never asks: what is this system optimizing for?
That online marketplace support team learned this firsthand. Their initial evaluation used standard dimensions offered by Eval tools straight out of the box, and every model scored well on all of them. The dimension that actually mattered was policy discipline, the ability to withhold action when the rules say "not yet." It doesn't appear on any standard benchmark. They only discovered it by stepping back from measurement and asking a more fundamental question: "What does a harmful failure look like for our support agents?" The answer reframed their entire evaluation.
Frame begins by exploring the full optimization landscape. Start with the users who experience the AI's output. Define success from their perspective, then identify the failures that would undermine it. Explore meaningful dimensions the system could optimize for, including accuracy and safety, policy discipline, cost, and dimensions unique to the product experience and domain. The goal is to see every possible axis along which this system could succeed or fail before committing to which ones matter.
From there, the team defines the system's optimization priorities and behavioral boundaries. This is where the Evaluation Strategy takes shape, organized around four commitments:
- What matters most: The dimensions that define whether this system fulfills its purpose, force-ranked by impact on the humans it serves.
- What success looks like: Concrete definitions of "good enough to ship" expressed in terms your users would recognize.
- What tradeoffs are accepted: The explicit acknowledgment that optimizing for one dimension may come at the cost of another, and which tradeoff the team is willing to make.
- What failures are unacceptable: The behavioral boundaries that constitute a hard stop, regardless of how well the system performs elsewhere.
The metrics you choose determine what you see. A team evaluating their own AI feature found that standard metrics rated their system at 5.0 for relevance and faithfulness, but a custom metric targeting "actionability" yielded a score of 2.2. The system was accurate and on-topic. It just never told anyone what to do. Without knowing what the system should optimize for, they would have shipped a feature that reported data but didn't help anyone make a decision.
The output of Frame is the Evaluation Strategy: an explicit definition of what this AI system is optimizing for, what failures are unacceptable, and what the team is willing to trade off. Everything else in the framework exists to test that thesis.
2. Design
How might we translate our Evaluation Strategy into an evaluation we can trust and repeat?
An Evaluation Strategy without a methodology is just an opinion. Design is where you turn "we care about safety and helpfulness for support agents" into a specific, versioned, reproducible evaluation plan.
The biggest mistake teams make here is building evaluation scenarios from imagination instead of observation. Whether you craft scenarios manually or use AI to generate them, the source material matters. Synthetic benchmarks bear little resemblance to what users actually type. Real users are messy. They misspell things. They provide contradictory instructions. They ask follow-up questions that completely reframe the task. They paste entire documents and say, "Fix this." Ground your scenarios in real user interactions, and your evaluation will test the product that actually exists.
Design begins by translating the Evaluation Strategy into three concrete evaluation components:
- Real-world scenarios: Draw these from actual user interactions, support tickets, session logs, and the queries that generated the most thumbs-down signals, after anonymizing or removing any sensitive data. Each scenario should exercise a specific kind of judgment the system needs to demonstrate, beyond simply covering a topic area.
- Scoring rubrics: A separate judge model, an LLM dedicated to scoring, evaluates the outputs of the primary model under test. Instead of humans reading every response, the judge model scores each output against structured criteria. A rubric that says "rate helpfulness 1–5" produces unreliable scores because different judges, whether human or LLM, interpret the scale differently.
- Calibration anchors: Calibration is the practice of aligning how different judges score the same output. Anchors are the concrete reference examples that make this possible, showing exactly what each score level looks like with a sample input, a sample output, and a clear explanation of why that example earns its score. Without them, a rubric that says "rate safety 1–5" leaves each judge to interpret the scale independently. One judge's 3 becomes another's 4. Anchors fix the scale so a 3 means the same thing every time, whether the judge is a human reviewer or an LLM evaluator.
Without anchors, scoring becomes subjective. One evaluator's 3 is another's 4. The rubric produces numbers but not information.
The online marketplace team's scenario design illustrates this well. They didn't test six random support topics. They tested six decision structures: a clear-cut escalation, a justified refund, a premature escalation, a safety emergency, an emotional customer, and a simple information request. Each scenario exercised a different kind of judgment. The model that failed only failed on the one requiring restraint, a scenario they would never have written if they'd been thinking about topics instead of decision types.
The phase concludes by locking the Evaluation Spec:
- Finalize the scenario set. Ensure it covers the decision structures and failure modes identified in Frame, beyond just topic breadth. Split scenarios into an optimization set (used to develop and tune your evaluation) and a holdout set (scenarios the evaluation was never tuned against, reserved for catching overfitting). The holdout set protects you from building an evaluation that looks perfect on paper but misses failure modes in production.
- Sharpen the rubrics. Include anchor examples for each score level on each dimension.
- Select the models. Choose the models to compare and the judge model that will score them.
- Define acceptance thresholds. Set the scores below which a model is a non-starter, regardless of how well it performs elsewhere. A safety score below 3.0 is a deployment blocker. Set that bar before you see results.
- Version the spec. Give it a number. Store it. Make it possible to re-run the same evaluation a few months from now and get comparable results when models change, for example. Without versioning, every cycle starts from scratch.
The output of Design is the Evaluation Spec, a reproducible, versioned methodology that defines what you're testing, how you're scoring, and what the bar is.
3. Measure
How might we generate evidence that is reliable enough to make decisions from?
This is the technical heart of the framework, and the phase where most teams have false confidence.
Here's a simple test: run your evaluation twice, changing nothing — same models, same scenarios, same rubrics. If the results are meaningfully different, you don't have an evaluation. You have a random number generator.
This happens in practice. LLM evaluation has a reliability problem that compounds in two places. The model being tested generates different outputs for the same prompt every time you run it. Then the judge model, when scoring those outputs, also produces slightly different scores each time. Variance stacks on variance. A single evaluation run captures one output, scores it once, and presents the result as "4.3 on helpfulness." That number feels precise, but if you run the same evaluation again, it could come back as 3.0 or 5.0. One team found exactly this when they ran the same scenario twice with identical data and got scores two full points apart.
Measure begins by generating observations at scale. Execute your evaluation across all model-scenario combinations. Run multiple scoring passes rather than trusting a single judgment. Think of it like a clinical trial: a single patient's response tells you very little, but a hundred patients measured carefully tells you something real.
Reliable evaluation depends on a few disciplines:
- Score stability across runs. A single evaluation run produces a point estimate, a number that feels precise but may not be. Running multiple passes reveals whether scores are converging toward a stable value or still bouncing around. The analogy is a thermometer that gives a different reading every time you check. It's telling you the temperature is still changing. You wait for it to settle. Advanced evaluation systems detect when additional runs would no longer change the conclusion, avoiding both under-measurement and wasted computation.
- Multi-judge consistency. A single scorer (human or LLM) carries biases. Using multiple independent scoring passes and measuring agreement reveals where the evidence is trustworthy and where the rubric may be ambiguous. When judges disagree, that disagreement tells you where your Design phase needs refinement.
- Confidence intervals. Reliable measurement produces a range, not just a number: "this model scores between 3.8 and 4.2 on helpfulness with 95% confidence." When two models' ranges overlap, the gap between them isn't meaningful. When they don't overlap, you can trust the comparison. Ranges make uncertainty visible instead of hiding it behind false precision.
If two models score within 0.15 overall but one costs 3× more per call, the confidence interval is telling you something the leaderboard never will: the cheaper model delivers comparable quality. Ship it.
Based on the evidence generated, the team decides when it is sufficient. Are the scores still moving, or have they settled? Are your confidence intervals tight enough to distinguish between models? If the evidence is stable, move to Interpret. If it's not, run more passes. The discipline is resisting the urge to interpret noisy data just because you're impatient for an answer.
The output of Measure is Evaluation Evidence: scores per model, per scenario, per dimension, with confidence indicators that tell you how much to trust each number.
4. Interpret
How might we turn scores into a recommendation that a non-technical stakeholder can act on?
The most dangerous evaluation result is a good average.
A model that scores 4.2 overall looks great on a dashboard. But if that average hides a 1.8 in helpfulness or a safety score below the threshold you set in Design, the dashboard is concealing a liability. Averages are comforting. They are also where critical failures go to hide.
One team discovered this through a holdout test, a set of scenarios against which their evaluation wasn't originally tuned. Their top-scoring model led on overall score but had a helpfulness rating of 1.80. The evaluation recommended it anyway, because the overall number looked strong. Only when they inspected the per-metric breakdown did the contradiction become visible: the model that "won" the eval was the one most likely to leave users feeling unheard.
Analysis computes the numbers. Interpretation figures out what those numbers mean for your product, your users, and your business. Interpret begins by exploring what the evidence actually says, beyond the summary statistics. Three practices matter here:
- Look beyond the rankings to the tradeoff structure. Model A might score highest on quality, but it costs 3× more per call. Model B might be the cheapest option and "good enough" on everything except the one dimension your Evaluation Strategy identified as critical. Model C might be the best for safety, but the slowest for latency. The evaluation doesn't make the decision; it makes the tradeoff visible. Your job is to map the tradeoff landscape so the decision-maker can navigate it with open eyes, not just pick the model with the highest number.
- Find where models break, beyond where they shine. Per-scenario breakdowns are where the real intelligence lives. The marketplace team's model comparison showed nearly identical behavior on five of six scenarios. The entire decision came down to one scenario, the ambiguous escalation, where the models diverged on policy discipline. A model that handles standard queries beautifully but collapses on edge cases has a failure profile that a summary score will never reveal. The failure profile is more actionable than the average.
- Cross-reference your dimensions. Quality dimensions aren't independent. High helpfulness sometimes comes at the cost of safety; the model tries so hard to be useful that it provides information it shouldn't. High conciseness sometimes sacrifices completeness. These correlations are invisible in per-metric dashboards but obvious when you look at the data with the right question: "Where does improving one dimension hurt another?"
The best evaluation output isn't a dashboard that tells you what happened. It's a recommendation that tells you what to do. "Nano delivers equivalent quality at 3.75× lower cost" is more useful than a table showing both models' scores.
The goal of Interpret is to produce a recommendation. A strong evaluation output explains what was tested, what was learned, what tradeoffs emerged, and what decision the evidence supports. It connects the findings back to the Evaluation Strategy from Frame — the optimization priorities and behavioral boundaries the team committed to — so the decision-maker can see exactly how the evidence maps to the criteria that were set before anyone looked at a score.
The output of Interpret is the Release Recommendation: a clear decision backed by evidence, tradeoffs, and reasoning that stakeholders can follow end-to-end.
5. Iterate
How might we improve the evaluation itself, beyond the product it evaluates?
Here's the pattern that keeps most teams stuck: run an evaluation, ship a decision, never revisit the methodology. The evaluation becomes a one-time gate, a checkbox between development and deployment. Then the model updates. Or the product scope changes. Or users start using the product in ways nobody anticipated. And the evaluation that was once carefully designed is now testing yesterday's assumptions against today's reality.
What makes this phase work is that the process loops back. Interpret might reveal noisy evidence, sending you back to Measure for more runs. It might expose a rubric that stopped discriminating, sending you back to Design. It might surface a quality dimension you never considered, sending you all the way back to Frame. Each cycle sharpens the methodology based on where the learning leads.
The teams that get the most value from evaluation are the ones who iterate on their ability to evaluate. Each cycle produces a sharper methodology, beyond just a better model choice.
Iteration begins with honest reflection on the methodology itself. Some scenarios stop being useful because they no longer differentiate between models. Some rubrics drift away from human judgment and require recalibration. New user behavior can expose gaps in the Evaluation Strategy that weren't visible during the first cycle. Model or prompt changes can quietly invalidate prior baselines, and without a versioned Evaluation Spec, the drift is invisible.
Equally important is mining production for new failure patterns. Logs, traces, user feedback, and escalation records reveal how the system actually behaves in real conditions. Unexpected failures surfaced in production should be promoted into the evaluation suite as new scenarios, ensuring the methodology evolves alongside the product rather than testing a frozen snapshot of assumptions.
The output of Iterate is an Evaluation Iteration: an improved, versioned methodology that carries forward everything the team learned, feeding back into Design for the next cycle or Frame for a fundamental rethink.
6. From Framework To Practice
These five phases aren't a waterfall. They're a cycle. The strongest teams run this loop continuously, tightening their Evaluation Strategy as they learn from users, sharpening their rubrics as they discover ambiguity, building a body of versioned evidence that makes every cycle faster and more precise than the last.
We built Plumloom.ai because we believe evaluation should be a continuous practice, not a one-off project. The framework described here is how we think about evaluation. Whether you start with all five phases or just a single run, Plumloom makes it practical.