You built an agent and demoed it. It handled every question you threw at it, the room nodded, and you shipped. That is exactly how it should go. A demo is the right first proof: it shows the agent can produce a great answer.
Reliability is the next question, and the one your customers actually feel. Does it produce a great answer every time, or just in the room?
1. A vending machine vs. a slot machine
Traditional software is like a vending machine. Press the same button, get the same result. That predictability is why a passing test means something: green today, green tomorrow, until the code changes.
An AI agent is closer to a slot machine. Today it behaves correctly. Tomorrow, with the same input, it may take a different path. It may even reach the right outcome while calling the wrong tools along the way. Would you call that a pass or a fail?
This is the nature of the technology, not a bug you fix once. An agent runs on a large language model, which is stochastic: identical input can produce a different answer, a different route, a different tool call from one run to the next. Which means one successful demo tells you very little about reliability. You saw one pull of the lever. Your customers will pull it thousands of times.
You cannot check those by hand. You cannot read thousands of conversations, and a gut read stops scaling past a few dozen. The failures you most need to see are the ones nobody happens to look at.
2. So you run an eval, and the score won't sit still
An evaluation, or eval, does the checking for you. Instead of reading each response yourself, you let an eval grade every answer against your rubric, the written standard for what a good response looks like. Run it across your test set and you get a number.
Say the agent scores 82 out of 100. Something to track, finally, something to compare against.
Then you run the same eval again, and it scores 74. Same agent, same test set, same rubric.
Did the agent regress, or is the evaluation itself noisy?
Right now you cannot tell, and that is the problem. A drop from 82 to 74 is easy to read as "the agent got worse." It does not mean that. It means something moved, and you do not yet know whether it moved in the agent or in the measurement. Until you can separate those two, the number cannot make any decision for you.
3. Stable first, then stable at what
Before you can trust a score, you need to know whether the measurement itself holds still. That fix is direct: run the evaluation more than once and watch where the score settles. An LLM-based judge varies run to run, the same way the agent does, so a single pass is a single pull of the lever. Several passes tell you the real number, the way five thermometer readings tell you the real temperature. When the score stabilizes, say 82 give or take a couple of points, you finally have something you can trust to mean the same thing twice.
But a stable number is only useful if it is measuring the right thing. That is the fuller job, and it is the part that is unmistakably a PM's to own. It comes down to a practice any product manager can run, no code required.
Define the correct outcome, before you build. Write down what a good result actually is, in the outcome you care about: the refund resolves correctly, the booking completes without a human stepping in. That definition becomes your rubric, the standard the judge scores against, so the model applies your bar the way you would. This is your evaluation spec, the agent's version of a PRD, and the team aligns on it first, the same way it aligns on a feature.
Inspect the full trace, not just the answer. An agent can reach the right answer through the wrong path, and the most dangerous failures are the silent ones: a final response that looks fine while the steps behind it went wrong. You only catch those by reading the full trace, what the agent understood, which tools it called, what it did with the results. This is error analysis, and it is a natural fit for a PM. You already read the hard support tickets before you redesign a flow.
Run the evaluation more than once. This is the reliability step from above, and it belongs in the practice, not off to the side. One run is a caveat. A number earns trust when it holds across repeats.
Connect the score back to customer impact. A score means nothing on its own. It matters when it ties to something real: fewer escalations, cleaner resolutions, a customer who got what they came for. "Refund-handling holds 84, give or take 2, and escalations drop as volume grows" is a sentence your product leadership can act on. A raw pass rate is not.
4. The payoff
You started with demo-day confidence, which was real but small: the agent can work. You end somewhere sturdier. You know how reliably it works, with a number that means the same thing twice and connects directly to what customers feel.
That is the difference the practice makes. Not a better demo, but an answer that survives a second look, a real failure, and a skeptical question from your team.
So for a PM, the real question was never simply "did the agent pass?" It is this: did it reach the right outcome, in an acceptable way, consistently enough that you would trust it with a customer?
Building the evidence to answer that takes tooling that runs evals reliably and reads back both the single response and the full path behind it. That is the problem Plumloom is built to solve.
Does your agent actually work? Now that is a question you can answer.