The Plumloom Blog
Practical notes on LLM evaluation, shipping AI features safely, and building products your users can trust.
Does Your AI Agent Actually Work?
A demo proves your agent can produce a great answer. Reliability proves it does so every time. Here is how a PM closes the gap between "can work" and "reliably works."
Read articleYour Product Is a Conversation. Your Evals Should Be Too.
Single-prompt scoring made sense when you shipped single completions. Today you ship conversations and agent trajectories — your evals should match the shape of the real work.
Read articleWhy One Evaluation Run Is Often the Wrong Basis for Model Selection
Across 45 controlled scenarios, multi-run evaluation reduced hard winner instability by 72.73%. A single run is one thermometer reading — repeated evaluation checks whether the reading holds.
Read articleA Reliable Evaluation Must Respond Correctly to the Evidence It Sees
What 38 evaluation lifecycles taught us about measurement correctness — zero false convergence decisions, zero false continuations, zero artifact mismatches.
Read articleThe Cheapest Model Won We Ran the Viral Anti-Sycophancy Prompt Through 4 LLMs
Ben Horowitz's viral anti-sycophancy system prompt promises to stop models from agreeing with you. We built a structured eval across 4 models to see whether it actually works — and the smallest, cheapest model came out on top.
Read articleThe Evaluation Number You're Trusting Might Be Wrong
Why single-judge LLM scoring is structurally unreliable, and what to do about it.
Read articleWhy Your AI Aces Benchmarks and Fails Users
A framework for treating LLM evaluation as a product quality discipline
Read articleWhy LLM Evals Matter More Than You Think
Vibes-based testing breaks the moment your product reaches real users. Here's why evaluation is the unglamorous work that decides whether your LLM feature ships or stalls.
Read articleWelcome to the Plumloom Blog
A new place for practical notes on LLM evaluation, shipping AI features safely, and the lessons we learn building Plumloom.
Read article