ONE LINE · FOUR DAYS · 10,010 VERDICTS
One engineer edits one line of a support agent’s prompt on a Thursday morning; Monday 09:41 it reaches 100% of traffic. In between: golden sets, robot judges with thumbs on the scale, three humans and a κ of 0.67, two hundred agent episodes, error bars, an $11 gate, and a 48-hour canary. The scenes play themselves — watch, or just scroll. are tappable. You are the diff.
Thursday, 09:41:12. An engineer adds one line to the system prompt of a working support agent and the diff looks so obviously safe it barely earns a review. In 96 hours the change will serve 100% of traffic, and this tutorial will have a running meter: Σ verdicts — recorded pass/fail judgments about this specific change. A verdict only counts if it is written down and re-runnable; watch how long the meter stays at zero.
The edit adds Keep replies under 120 words; always state the refund timeline. to the system prompt of the refund agent from the agent-patterns tutorial. Blast radius: ~13,000 conversations a day, 7 intents, 2 tools downstream — and a prompt is a code change with no compiler, because nothing type-checks English. The reviewer reads it, nods, writes LGTM. Recorded verdicts about what this line actually does to those 13,000 conversations: zero. This tutorial is about the machinery that turns that zero into a number you can stand on — the .
“I tried it and it looked fine” tested one draw from a distribution. Thinking Machines sent 1,000 identical requests to Qwen3-235B at temperature 0 — the setting everyone calls deterministic — and got 80 distinct completions. The cause: dynamic batching changes the order of floating-point reductions inside GPU kernels, so the same prompt takes a slightly different numerical path depending on who else is on the server. The playground answer you read was one of eighty. An is how you grade the distribution instead of the anecdote.
The model under the prompt is not frozen either. Stanford and Berkeley researchers measured the same GPT-4 API endpoint in March and June 2023: asked is 17077 prime? think step by step, accuracy fell from 97.6% to 2.4% — the June model had largely stopped following the chain-of-thought instruction. Same API name, silently different behavior, no changelog you could have read. A prompt tuned by feel is calibrated to a target that moves — which is why serious teams treat every provider model bump as a release and re-run the whole suite.
The vibe check happens anyway: three playground prompts, three good answers, ship it. Here is the arithmetic of small samples: a version that fails 30% of the time passes a 3-prompt check with probability 0.7³ = 34% — one broken build in three walks straight through. And vibes don’t version: you cannot diff a feeling against last Tuesday, or bisect it when something breaks. The rest of this tutorial replaces “seems fine” with recorded, re-runnable verdicts — starting with why the everyone already ran can’t help.
Why “deterministic” isn’t. Temperature and top-p shape the sampling distribution; a seed pins the sampler. None of them pin the arithmetic: batch-size-dependent reduction order inside GPU kernels means the logits themselves wobble. Batch-invariant kernels exist (SGLang, vLLM) and produce bit-identical outputs — at roughly 62% throughput cost in the first implementation.
Pin your snapshots. A floating alias like “-latest” means your system silently changes underneath you; dated snapshots mean upgrades are deliberate events with an eval run attached. The March/June GPT-4 study is the canonical argument.
Prompts are code. Version control, review, changelog, rollback — the whole discipline applies. The one-line diff of this tutorial lives in a repo, and the eval suite is its test folder.
The vibe check isn’t worthless — it’s just not evidence. Trying a few prompts catches catastrophic breakage in seconds and builds intuition for where to point real evals. The failure mode is stopping there.
MMLU, HumanEval, the Arena — thousands of public measurements already exist, so why not just check the leaderboard? Look at what those numbers actually measure: saturated exams, leaked answer keys, miscounted questions, and one spectacular case of teaching to the test. The Σ meter stays at zero the whole way through, and that silence is the lesson.
Public have lifespans. MMLU — 57 subjects, 14,042 test questions — went from GPT-3’s 43.9% in 2020 to GPT-4’s 86.4% in 2023 to routinely above 90%, at estimated-human-expert level: . HumanEval’s 164 coding problems went from 28.8% (Codex, 2021) to ~95%+ in three years. A maxed-out ruler can’t rank what it can’t separate — which is why the field keeps minting harder exams, and why the number on a model card tells you which model to pick, not whether your prompt edit broke your agent.
Benchmarks live on the same internet models train on. Scale AI rebuilt GSM8k from scratch — same difficulty, same style, brand-new questions (GSM1k) — and several model families dropped up to 13%, with a measurable correlation (r² = 0.32) between how readily a model recites GSM8k text verbatim and how far it falls on the fresh set. means a public score can measure memory instead of ability — frontier models showed minimal gaps, but you can’t tell which case you’re looking at from the leaderboard alone.
MMLU-Redux hand-re-annotated 5,700 MMLU questions and found 6.49% contain errors — wrong answer keys, no correct option, unparseable questions; in the Virology subset, 57%. Fixing the errors visibly reshuffled reported model rankings. A is a dataset built by humans on a deadline, and nobody audits it until the numbers start mattering — remember that when a single decimal point separates “SOTA” from second place.
Even a clean question isn’t measured cleanly. Across 20 LLMs, models systematically prefer specific option letters — token bias: extra probability mass parked on A before the choices are even read — so shuffling answer order moves benchmark scores. Multiple-choice accuracy partly measures the alphabet. Debiasing methods (permute the options, estimate the prior, subtract it) are a real research subfield, which tells you how deep the crack runs.
, it ceases to be a good measure. April 2025: Meta submitted an unreleased, chat-optimized “experimental” Llama 4 Maverick to LMArena — verbose, emoji-warm — and it ranked №2. The public model everyone could actually download ranked №32. LMArena rewrote its leaderboard policies within the week. And the deeper reason this chapter’s meter never moved: not one of these thousands of public verdicts is about your agent, your users, or your one-line diff. For that you need a ruler nobody else can buy.
The bestiary. Knowledge: MMLU → GPQA. Code: HumanEval → SWE-bench. Math: GSM8K → AIME. Agents: GAIA, τ-bench, WebArena, OSWorld. Each generation exists because the previous one saturated; treat every score with a date attached.
The audit pattern. SWE-bench Verified is what auditing looks like: OpenAI put 93 developers on 1,699 sampled tasks and flagged 68.3% — 38.3% underspecified, 61.1% with unit tests that could fail correct solutions — keeping 500 clean ones. Benchmarks are datasets; datasets need QA.
Anti-contamination moves. Private held-out sets (ARC-AGI’s hidden test, SEAL leaderboards), canary strings baked into eval files so their presence in training data is detectable, n-gram overlap scans, and fresh-question rebuilds like GSM1k.
What leaderboards are still good for. Choosing a base model, tracking the frontier’s slope, and calibrating expectations — real jobs, just different from “did my Thursday diff break refund denials.” Use both; confuse neither for the other.
If no public ruler measures your agent, you build a private one out of the only material that is truly yours: what your users actually said. Sixty conversations, three deterministic checks each, and Thursday afternoon delivers the tutorial’s plot twist — the “obviously safe” one-liner broke something a regex can prove.
The : 60 real, redacted pulled from the agent’s own history and curated for coverage — 18 refund requests including January’s duplicate-charge ticket, 9 ALL-CAPS angry ones, 6 prompt-injection attempts, 8 non-English, 19 known past failures and edge cases. The pattern is the same one SWE-bench used at scale: it didn’t invent programming puzzles, it harvested 2,294 real GitHub issues. Reality writes better test cases than imagination — your production queue is the goldmine.
The cheapest graders are string operations. Each trace re-runs under the new prompt and faces 3 programmatic : must cite the invoice ID (an INV-… regex), must cite policy §4.2 on any refund denial, and must pass the structural checks — valid tool-call JSON, reply under the length cap, lookup_invoices before issue_refund. 60 × 3 = 180 verdicts in 41 seconds for $0.00 — deterministic, diffable, re-runnable forever. The meter finally moves.
The grid comes back 177 green, 3 flagged. Two are the real thing: on refund denials, the new 120-word cap squeezed out the required policy citation — the exact a 3-prompt vibe check had a ~0% chance of meeting, caught by a regex. The third is the test’s own fault, a brittle exact-match. Telling those apart is eval maintenance, and it never ends: OpenAI’s auditors flagged 61.1% of original SWE-bench tasks for unit tests that could unfairly fail correct solutions. Your suite has those too.
The fix is four words — …under 120 words unless citing policy — and then the point of the whole apparatus: rerun the identical 180 checks and read a diff, 177/180 → 180/180, instead of a feeling. An you can rerun turns “it seems better now” into version control for behavior. Σ 360, the suite now guards every future diff for free — and everything it cannot see (tone, empathy, whether an apology apologized) goes to the next chapter’s grader.
Assertions that age well. Property checks over snapshot checks: assert invariants (“cites some invoice ID”, “no promise of instant refunds”) rather than exact strings that break on every rewording. Negative assertions matter as much as positive ones.
Fixtures, not production. Golden traces re-run against mocked tools and frozen fixtures, so a database migration doesn’t masquerade as a prompt regression.
Don’t Goodhart your own ruler. Tune the prompt against the golden set daily and you will overfit it — the private version of the leaderboard disease. Keep a small holdout, rotate cases in from production, retire stale ones.
How big is big enough? Dozens with teeth beat thousands of mush: coverage of intents and known failure modes matters more than raw count. Sixty curated traces caught what three enthusiastic prompts missed; a thousand uncurated ones might not have.
“Contains policy §4.2” can’t grade tone, empathy, or whether an apology actually apologized. The fix the whole industry converged on: make another model read every trace against a written rubric. This chapter puts the robot judge on the bench, measures its thumbs on the scale, and then does the only thing that makes its 2,000 verdicts admissible — grades the grader.
got its charter in one NeurIPS 2023 paper (Zheng et al.): a strong judge model agrees with human preferences over 80% of the time — the same rate humans agree with each other — making the judge as reliable as one more human, at machine speed. Friday morning it reads 500 re-run traces against a written 4-dimension — accuracy, tone, grounding, completeness, each a binary pass/fail with a required written reason: 2,000 verdicts, 11 minutes, ≈$4. The same 500 traces would cost a human rater roughly three working days.
The same paper measured the judge’s failure modes. : show a judge two answers, get a verdict, swap their order — the verdict can flip. : longer answers win points for length alone (remember which Llama 4 charmed the Arena). Self-enhancement: models favor prose from their own family. The mitigations are mechanical, not moral: judge every pair twice with sides swapped and keep only agreements, length-normalize, and never let a system grade its own model family.
A judge is only as sharp as its . Rate helpfulness 1–10 produces mush — scores pile into the indistinct middle. What works reads like law: binary criteria (FAIL if the refund timeline is absent), a written justification required before the verdict, and a few anchored examples of each grade. MT-Bench — the benchmark that launched LLM-as-judge — is only 80 questions: rubric precision, not question volume, is where judging quality lives.
2,000 verdicts mean nothing until the grader itself is graded. Standing infrastructure: 100 traces carry human labels (the next chapter’s court), and the judge is scored against them like any classifier — this rubric-judge pair agrees with humans 87%, above the ~80% , with per-dimension true-positive and true-negative rates on a dashboard. When agreement dips, you fix the rubric, not the model. An uncalibrated judge is just a vibe check with an invoice.
Judge ≠ defendant. Use a different model family for judging than the one under test — self-enhancement bias is real, and cross-family judging is the cheap fix.
Mechanics that raise agreement. Swap-and-average for pairs; chain-of-thought-then-verdict over verdict-first; few-shot anchor examples of each grade; reference-guided judging (hand the judge a gold answer) as the highest-agreement mode.
Small judges, juries. Fine-tuned small judges and reward models grade one narrow thing cheaply at scale; juries — three diverse judges plus majority vote — trade cost for robustness on high-stakes dimensions.
Cost is a design input. Judge tokens are a real line item: cache verdicts for unchanged traces, sample instead of exhausting, reserve the expensive judge for the dimensions assertions can’t reach.
The judge’s 87% was measured against human labels — so where do those come from, what do they cost, and how often do the humans even agree with each other? This chapter runs the diff through its human court: three reviewers, one afternoon, one κ statistic — and a cautionary tale about what happens when green dashboards outvote human unease.
The diff gets its own trial: 100 traces, old-prompt reply vs new-prompt reply, sides shuffled per card (position bias applies to humans too), three reviewers voting blind — 300 verdicts, about 3.5 hours of total human time. Result: the new prompt preferred 61:32, with 7 ties. Pairwise is the trick: people are noisy at “rate this 1–10” and sharp at “which of these two is better,” which is why every serious preference pipeline — RLHF included — is built on comparisons, not scores.
The three reviewers split on 14 of the 100 cards — mostly tone calls. (Cohen’s κ = 0.67 here, “substantial”) is the honest number hiding behind every “ground truth”: on chat quality, humans agree with each other only ~80% of the time, and that is the ceiling any judge — silicon or human — can be measured against. Disagreement isn’t noise to average away: the 14 contested cards go to adjudication, and most of them expose a gap in the rubric, not a careless rater.
Scale the mini-arena up 10,000× and you get Chatbot Arena (LMSYS, 2023): anyone prompts, two anonymous models answer, you vote, then the names reveal — 3.5M+ votes across 400+ models by late 2025, ranked with the , chess Elo’s statistical cousin. It is the closest thing to un-gameable ground truth in public AI… which is exactly why gaming it pays (chapter 01’s Maverick), and why LMArena now publishes style-controlled rankings that statistically strip out the win-rate boost of sheer length and formatting.
April 2025. OpenAI updates GPT-4o: offline evals pass, A/B numbers look positive — but expert testers report the model “feels off.” The metrics win the argument; it ships. Within days ChatGPT is agreeing with everything — praising a “shit on a stick” business plan, endorsing users quitting medication — and the update is rolled back four days after launch. The postmortem’s core admission: the vibe was the signal, and the dashboards had no metric for it. Expert unease is data; it just arrives unquantified.
Annotation ops. A rater guideline doc (the human rubric), calibration rounds before real labeling, adjudication for splits, and κ tracked per dimension — 0.4 is a coin with opinions, 0.67 is workable, 0.8 is expensive.
Experts vs crowds. Crowd raters are fine for fluency and preference; domain calls (was §4.2 applied correctly?) need experts at 10–50× the per-label price. Route each dimension to the cheapest rater who can actually judge it.
Arenas have their own threat model. Published research shows arena rankings can be nudged by coordinated voting; style control exists because verbosity and markdown inflate win rates. Public arenas rank models; they cannot review your diff.
Why 300 private votes beat 3.5M public ones. They are about your users, your policies, your failure modes. The big arena answers “which model”; your mini-arena answers “which prompt” — and only one of those was Thursday’s question.
Every eval so far read the agent’s replies. But this is an agent — it changes real systems, and the reply can be flawless prose about having refunded the wrong invoice. So grade what it did, not what it wrote: tool trajectories, database end-states, and the honest test of running the same scenario eight times in a row.
Output evals see the final text; a replays the whole episode and grades the route: right tools, right order, right arguments — issue_refund’s amount must equal the computed over-charge to the cent — and, just as important, no extra destructive calls along the way. Each scripted scenario carries 5 checkpoint assertions on the tool sequence. A wrong-invoice refund followed by a beautiful apology passes every text check ever written and fails checkpoint 4.
The sharpest agent evals don’t read text at all — they read the world afterwards. τ-bench (Sierra, 2024) runs agents against mock APIs with an LLM-simulated customer, then grades by comparing the final database state to the correct one. If the refund row is wrong, no amount of good writing rescues the score. Our suite copies the pattern — sandboxed billing and mail fixtures, and the pass condition is the ledger, not the prose. is the closest an eval gets to caring about what actually happened.
Agents compound randomness across every step, so one clean run is nearly meaningless: τ-bench measured a GPT-4o agent passing retail tasks ~61% of the time once, but succeeding on all 8 of 8 repeats only ~25% of the time. — pass every one of k tries — is the honest metric, because your customer doesn’t re-roll until you get lucky. Friday night: 25 scenarios × 8 runs = 200 episodes × 6 verdicts each = 1,200 verdicts. The new prompt holds pass^8 = 84%; one flaky scenario is quarantined for investigation.
While the knowledge saturated (chapter 01), agent benchmarks opened a canyon in the other direction. GAIA (2023) built questions “conceptually simple for humans yet challenging for AIs”: humans scored 92%, GPT-4 with plugins 15%. WebArena’s realistic websites: the best 2023 agent finished 14.41% of tasks against humans’ 78.24%. Those gaps have narrowed dramatically since — which is the point: agent evals are where a ruler still has room to measure, and they age like all the others.
Sandboxes, not prod. Episodes run against mock tool servers with seeded fixtures — a refund eval that touches real billing is an incident, not a test. Deterministic seeds make failures replayable.
The simulated user is a flaky dependency. τ-bench’s own authors note the LLM-played customer misbehaves — going off-script, leaking info early. When an episode fails, first ask which side of the conversation broke.
Partial credit. All-or-nothing outcomes hide progress; checkpoint-level scoring (5 of 6 verdicts passed) tells you where routes break. Both numbers matter: checkpoints for debugging, outcomes for shipping.
The canyon after GAIA. OSWorld (2024) put agents in front of real desktop apps: humans 72.4%, the best agent 12.2%. Computer-use evals are the current frontier ruler — expect the same saturation story, on the same schedule.
What 200 episodes cost. Each episode is a full agent run — dozens of model calls; the suite is an overnight CI job, not a pre-commit hook. Budget it like one: nightly full runs, per-PR samples.
The new prompt scores 92.0 against the old prompt’s 90.5 on the judged suite. Before anything merges: is that a real difference, or is it noise? Ten minutes of statistics separate eval theater from eval evidence — and they end with the only button that matters.
An eval score is a sample statistic, not a constant — rerun the same suite and it wobbles. Anthropic’s “Adding Error Bars to Evals” (November 2024) opens with the field’s embarrassing habit: SOTA-in-bold, no , no significance test — “highest number is best” science. First discipline of the gate: every reported score wears a confidence interval, and two overlapping intervals mean you have learned approximately nothing yet. The judge says 92.0 vs 90.5; the whiskers say ±2.1. Not a fact. Not yet.
The rescue is that both prompts answered the same 500 traces, so you can analyze per-question instead of comparing two independent averages — each trace’s difficulty cancels against itself, variance collapses, and the same data suddenly separates: mean Δ = +1.5 points, 95% CI [+0.4, +2.6], cleanly excluding zero. Paired analysis and power analysis (how many questions you need, computed before running) are two of the paper’s five recommendations, and they are free — same data, better question.
One more trap: our 60 golden traces come from 9 intent families, and traces within a family rise and fall together — they are not 60 independent samples. Anthropic’s paper measured the damage of pretending otherwise: clustered standard errors run up to 3× the naive ones. Ignore clustering and your tight little interval is a comforting fiction; the honest whisker is wider, and the gate’s regression thresholds are set against that one.
Friday 17:50, the release candidate triggers the gate: the full suite re-runs from scratch — 180 assertions, 2,000 judge verdicts, 1,200 agent-episode verdicts — 3,380 verdicts in 22 minutes for ≈$11, compared against with paired stats and clustered error bars. Gate law: no tracked metric regresses beyond its noise band, and the 6 injection cases are a hard-fail set — 5/6 is not 83%, it is a blocked merge. All green. MERGED, Friday 18:12. Σ 7,240.
The toolkit. Standard error via CLT or bootstrap; resample the model k times per question when nondeterminism (chapter 00’s ghost) dominates; paired t-tests on per-question deltas; power analysis before, not after.
A worked power example. Distinguishing 90% from 93% at 95% confidence needs on the order of a thousand questions. That is why tiny-suite “wins” of a point or two are usually noise, and why the judge set has 500 traces while the golden set needs only 60 (its checks are near-deterministic).
Multiple comparisons. Track 12 metrics and one will “regress” by chance regularly; Bonferroni corrections or an expected-false-positive budget keep the gate from crying wolf until nobody believes it.
Gate mechanics. Compare against a frozen baseline run, not last week’s cached number; thresholds as noise-band multiples, not vibes; hard-fail sets (safety, injection) exempt from statistics entirely — some cases are laws, not samples.
Offline, the change is clean, judged, human-preferred, statistically real. One eval remains — the only one with real customers in it. This chapter takes the diff into production on a leash: a 10% canary, monitors that never sleep, a cautionary tale about dashboards that lie politely, and the flywheel that turns this weekend’s flagged traces into next month’s golden cases.
Saturday 09:41, a : the new prompt serves 10% of traffic — 2,600 conversations over 48 hours — while monitors grade every one of them live. Resolution rate, escalation rate, retry-and-rephrase rate (a user restating their question is a verdict, whatever the thumbs say), p95 latency, cost per ticket, plus a cheap online judge sampling tone. The guardrails were the always-on all along — same checks, no dataset, running forever.
The other half of chapter 04’s story: the sycophantic GPT-4o passed its A/B tests — users click 👍 at a model that flatters them, and OpenAI’s postmortem admits the aggregate reward signal leaned into exactly that. Thumbs measure this-moment satisfaction, not truth or long-term trust; optimize them naively and you build a yes-bot with excellent dashboards. Online metrics deserve the same skepticism as offline ones — always ask what, precisely, this number rewards.
Sunday’s harvest: 170 explicit ratings (Σ 10,010) and — more valuable — 4 conversations the monitors flagged: a new invoice-format edge case, a Portuguese refund idiom the tone judge misread. All four are drafted into the : 60 → 64. This is the flywheel that makes eval suites fossil records of every past failure — OpenAI’s own fix list after the sycophancy incident included making sycophancy an explicit launch-blocking eval. The suite you’ll have in a year is being written by the incidents you have now.
Canary clean, deltas inside their noise bands, no human unease on the sampled traces. Monday 09:41 the rollout dial sweeps 10 → 50 → 100%, exactly 96 hours after the edit — the same four days it took OpenAI to un-ship its update. Final ledger: one line changed, 10,010 recorded verdicts, ~$31 of judge tokens, ~3.5 hours of human time. And the only verdicts that ever existed for the old way of shipping prompts were three green cards in a playground.
The plumbing. Feature flags and percent rollouts; A/B for causal reads, interleaving for cheap preference signal; auto-rollback triggers on hard thresholds vs paged humans for soft ones.
The implicit-signal catalog. Rephrase, abandon, escalate-to-human, copy-the-answer, return-next-day — behavioral verdicts that outnumber explicit thumbs enormously and are harder to charm. Instrument them before you need them.
Judging production traffic. Sampled judge audits over live traces, with redaction before grading — the privacy line matters; store verdicts, not transcripts, where you can.
Drift, both ways. The model under you changes (chapter 00), and your traffic changes — new products, new languages, new user habits. A golden set needs a freshness SLA, or it becomes a museum without anyone noticing.
The quiet endgame. The eval suite becomes the spec: prompt engineering becomes making the suite pass, and arguing about “better” becomes editing a rubric everyone can read. That’s the whole trade — vibes for verdicts.
The payoff the hero promised: every gate the one-line diff passed, and what each contributed. The first two rows bought nothing — that was the point of chapters 00 and 01 — and the three playground cards from Thursday morning appear nowhere below, because a verdict that isn’t recorded isn’t evidence.