The Plumloom Blog
Practical notes on LLM evaluation, shipping AI features safely, and building products your users can trust.
Autoeval for Coding Agents: Improve AI Quality, Verify Reliability, Gate Releases
Developers are increasingly using coding agents to write code, fix failures, and iterate on AI applications. These same tools can now help create evaluations, run them, inspect results, and guide the next improvement.
Read articleWhy Plumloom Exists: Reliable Evaluation for the Replaceable Model Era
As foundation models become more capable and more interchangeable, differentiation shifts to the systems built around them: proprietary context, workflow integration, evaluation, and learning.
Read articleFrom RAG Failure to Release Evidence
A failed answer tells you that the task failed. A useful evaluation tells you where the failure occurred, whether the evaluator can detect it, and whether the resulting signal is reliable enough to support a release decision.
Read articleYour RAG Agent Can Be Right for the Wrong Reason
A document assistant can give a correct-looking answer while relying on information outside the document it was asked to use. Here is how to test whether your RAG system stayed inside its knowledge boundary.
Read articleDoes Your AI Agent Actually Work?
A demo proves your agent can produce a great answer. Reliability proves it does so every time. Here is how a PM closes the gap between "can work" and "reliably works."
Read articleYour Product Is a Conversation. Your Evals Should Be Too.
Single-prompt scoring made sense when you shipped single completions. Today you ship conversations and agent trajectories — your evals should match the shape of the real work.
Read articleWhy One Evaluation Run Is Often the Wrong Basis for Model Selection
Across 45 controlled scenarios, multi-run evaluation reduced hard winner instability by 72.73%. A single run is one thermometer reading — repeated evaluation checks whether the reading holds.
Read articleA Reliable Evaluation Must Respond Correctly to the Evidence It Sees
What 38 evaluation lifecycles taught us about measurement correctness — zero false convergence decisions, zero false continuations, zero artifact mismatches.
Read articleThe Cheapest Model Won We Ran the Viral Anti-Sycophancy Prompt Through 4 LLMs
Ben Horowitz's viral anti-sycophancy system prompt promises to stop models from agreeing with you. We built a structured eval across 4 models to see whether it actually works — and the smallest, cheapest model came out on top.
Read articleThe Evaluation Number You're Trusting Might Be Wrong
Why single-judge LLM scoring is structurally unreliable, and what to do about it.
Read articleWhy Your AI Aces Benchmarks and Fails Users
A framework for treating LLM evaluation as a product quality discipline
Read articleWhy LLM Evals Matter More Than You Think
Vibes-based testing breaks the moment your product reaches real users. Here's why evaluation is the unglamorous work that decides whether your LLM feature ships or stalls.
Read articleWelcome to the Plumloom Blog
A new place for practical notes on LLM evaluation, shipping AI features safely, and the lessons we learn building Plumloom.
Read article