Five of five is not a statistic

What repeated eval runs can tell you, what they cannot, and why the two get confused.

A scenario goes from zero of five passing to five of five after a prompt change. Someone writes “100% success” in the summary. Both numbers are true and neither is a success rate.

Five runs of an agent tell you almost nothing about how often it will succeed in production. The confidence interval on 5/5 runs from roughly 50% to 100%. On 0/5 it runs from 0% to about 50%. Anyone quoting the point estimate is quoting noise.

What five runs do tell you is whether the behavior moved. A change from 0/5 to 5/5 is very unlikely under “nothing changed”; the two distributions barely overlap. That is strong evidence that the intervention did something. It is weak evidence about what the system will do on the sixth attempt, and no evidence at all about the population of tasks the scenario didn’t sample.

So the repeated run answers a debugging question, not a measurement question. Did the fix touch the failure? Was the original failure a bad draw or a contract bug? Deterministic contract bugs fail every time; model variance fails some of the time. Five runs separate those two cleanly and cheaply. Estimating a rate to within five points takes hundreds, and by then the model has probably changed.

The confusion happens because both come out as a fraction. Reporting the debugging result as a percentage invites it to be read as the measurement result. I’ve started writing “0/5 → 5/5” in the summary and refusing to convert it, so the reader sees the sample size before the ratio.