Most LLM evaluation still treats the model like a vending machine: one prompt in, one answer out, score the answer. That made sense when the thing you shipped was a single completion. It makes far less sense now, when the thing you ship is a support agent that runs six tool calls before it says a word to the customer, or an assistant that holds a ten-turn conversation and only goes wrong on turn eight.
The gap is simple to state. You evaluate single prompts, but your product behaves as a trajectory. The score you trust in testing describes an interaction your users never actually have.
Closing that gap means letting the evaluation match the shape of the real work. Three shapes cover almost everything teams build today.
1. Single prompts: the classic case, still essential
The single-prompt evaluation has not gone anywhere, and it should not. When you want to know how a model answers one well-defined input, this is the cleanest signal you can get. You write the inputs by hand, define what a good answer looks like, and score across your test cases.
This is where you catch the fundamentals: does the model get the facts right, does it follow the format, does it stay on policy. It is fast to set up and fast to read, and for a large class of features it is all you need. The mistake is stopping here when your product has already moved past it.
A single-prompt score answers "can the model respond well?" It does not answer "can the model handle the whole exchange?"
2. Conversations: scoring the whole exchange
A multi-turn assistant can look perfect turn by turn and still fail the conversation. It forgets a constraint the user gave three turns ago. It contradicts something it said earlier. It handles every message competently and still leaves the user without the thing they came for.
None of that shows up when you score turns in isolation. It only shows up when you score the exchange end to end, against what the conversation was supposed to accomplish.
A conversation evaluation takes a complete transcript, the full back-and-forth including any system prompt, and grades it as a single unit against an expected outcome. You describe what a successful resolution looks like, and the judge reads the whole thing the way a reviewer would: not "was each reply fine" but "did this conversation get the user where they needed to go."
That reframing is the point. The unit of quality for a chat product is the chat, not the message.
3. Agent traces: scoring how it got there
Agent products raise the stakes again, because with an agent the final answer can be right for the wrong reasons, and that is a problem waiting to happen.
Consider a cancellation agent that tells a customer their order is cancelled and a refund is on the way. The reply reads perfectly. But look at the trajectory: the agent called the order-lookup tool, saw the order had already shipped, and processed the refund anyway, outside policy. Score only the final message and it passes. Score the trajectory and you catch a costly failure before it reaches a real customer.
An agent-trace evaluation scores the full run: the sequence of model calls, tool calls, and retrieval steps the agent took to reach its result. You upload the trace, describe the end state the agent was supposed to reach, and the judge evaluates the trajectory step by step against it, grounded in the tool outputs the agent actually saw. Standard OpenTelemetry traces work, so a trace you are already emitting for observability doubles as an evaluation input.
The result the user sees is one slice of agent quality. The path it took to get there is the rest, and it is usually where the real risks live.
4. New models to judge and be judged
Evaluation is only as good as the models doing the work, on both sides: the model you are testing, and the judge scoring it. A judge that runs out of budget mid-reasoning or misreads a long trace gives you a score you cannot act on. So the model roster matters, and it just grew by five, all open-weight, all served through Together AI.
Inkling, Thinking Machines' first open model, leads the group. It is a large mixture-of-experts reasoner with a one-million-token context window that was deliberately trained to flag its own uncertainty rather than bluff, and it reaches comparable coding performance to Nemotron-3-Ultra on roughly a third of the tokens. Calibrated, honest reasoning is exactly what you want in a judge.
DeepSeek-V4-Pro is a 1.6-trillion-parameter MoE (with about 49 billion active per token) and ranks among the strongest open-weight reasoning models for agentic work. Its one-million-token context and large output ceiling make it well suited to long, complex agent traces that would overflow a smaller judge.
GLM-5.2 from Zhipu is a coding-and-agent-first model, also MIT-licensed, with a genuine one-million-token window and selectable reasoning effort. It is a strong, efficient default for everyday evaluation work.
MiniMax-M3 pairs frontier-level reasoning with a sparse-attention design that keeps long-context runs fast and economical, a good fit when you are running at volume.
Nemotron-3-Ultra rounds out the set as NVIDIA's latest open-weight flagship, a capable general-purpose reasoner for both sides of the bench.
More open-weight choice means more control: over cost, over which model judges your work, and over matching a model's strengths to the job in front of it.
5. Match the eval to the work
The through-line across all three shapes is one idea. The evaluation should look like the thing you are actually shipping. If you ship single answers, score answers. If you ship conversations, score conversations. If you ship agents, score the trajectory, not just the last line of it.
Do that, and the number you trust in testing finally describes the interaction your users will have. That is the whole job of an evaluation: to tell you the truth about your product before your product tells it to your customers.
Plumloom evaluates all three — single prompts, conversations, and agent traces — with the reliability to know when a score is one you can act on.