What an eval suite lets you believe
Five clean runs, one scenario, a suite, a model upgrade, a condition nobody tested. Following one pass rate through each shows what the evidence establishes, what it assumes, and how quickly it expires.
A pull request changes a prompt. A scenario that asks the agent to update a customer record passes five times out of five, and the summary says so. Everyone reading it knows five runs is not many. Nobody can say, on the spot, how much five runs should change what they believe about the agent, so the number stands and gets repeated as if it meant what it says.
I want to work that question out carefully, because answering it is the whole job an eval suite does. A suite is not a test the agent passes. It is the instrument by which a team updates its beliefs about what the agent can do. Each run moves a distribution. The distribution after today’s runs is where tomorrow’s begin. New scenarios widen what the distribution is about. And everything the distribution says is conditional on a system that will not stay fixed.
That last clause is the one to keep in view. A pass rate belongs to a particular agent under particular conditions. Written out, it is θ(model, harness, tools, world, grader, date): the probability that this model, inside this harness, with these tools, in this simulated world, judged by this grader, on this date, completes the task. Each section of this essay changes one part of that expression while holding the rest still. First everything is fixed and we run once, then again. Then we vary the scenario and look at the suite. Then we vary the system and ask what survives. Finally we vary the conditions inside the world and ask where the behavior holds. At each step something enters the argument that the runs alone did not supply, and the point of writing the argument in Bayesian form is to keep those additions visible.
One successful run
Hold the whole expression fixed and let only the randomness of execution vary. There is some unknown probability θ that the agent passes this scenario. Each attempt is a draw from it.
A pass is nine times as likely under a 90% pass rate as under a 10% one. Observing a pass therefore multiplies the odds between those two possibilities by nine. But a pass is only 1.125 times as likely under 90% as under 80%. One success separates a very unreliable agent from a reliable one far better than it separates a reliable agent from an excellent one. That asymmetry is most of what follows.
Before the run, we hold a distribution over possible rates. The outcome gives more weight to the rates that would have produced it more often. The result is the posterior, and it describes our uncertainty about θ; it is not a spread of scores from individual runs, which remain plain passes and failures. Start, for this worked example, with a uniform prior: every rate from zero to one equally plausible. After one pass the posterior leans toward high rates but remains broad. Its mean is two-thirds, which under this model is also the probability that the next attempt passes. The middle 90% of its probability stretches from about 22% to 97%.
Keep going and watch two different quantities move. After three clean runs, the next-pass prediction is 80%; after five, 86%; after ten, 92%. Meanwhile the probability that θ exceeds 90% rises from 19% after one pass to 34%, 47%, and 69%. A third quantity moves slowest of all. The probability that the next eight attempts all pass, which is closer to what a user experiences than any single-run number, is 43% after five clean runs.1
| After observing 5/5, under a uniform prior | |
|---|---|
| Probability the next attempt succeeds | 86% |
| Probability the underlying rate exceeds 90% | 47% |
| Probability the next eight attempts all succeed | 43% |
These are three answers to “what do we now believe,” and they are not interchangeable. A report that says “100%” has chosen not to say which one it means. Readers will hear the first and act on the third.
| Evidence | Next pass | 90% credible interval | P(rate > 90%) |
|---|---|---|---|
| 1/1 | 66.7% | 22.4%–97.5% | 19.0% |
| 3/3 | 80.0% | 47.3%–98.7% | 34.4% |
| 5/5 | 85.7% | 60.7%–99.1% | 46.9% |
| 10/10 | 91.7% | 76.2%–99.5% | 68.6% |
Switch the figure to ten attempts with failures. At nine of ten, the probability of a rate above 90% is about 30% under the uniform prior. At eight of ten it is 9%. A single failure moves the upper tail a long way, because near-perfect rates had assigned that failure very little probability. Mixed results are evidence about the shape of reliability, not a degraded version of a green one.
The arithmetic behind all of this is compact. A Beta(α, β) prior and k passes in n attempts give a Beta(α + k, β + n − k) posterior, with mean (α + k)/(α + β + n). The uniform prior is Beta(1, 1).2 Its virtue is that the observations and the starting assumptions stay separately visible in the result.
One assumption sits under every number in this section: that five runs are five independent draws at one rate. That is a claim about the harness, and it can be false. Eval-Driven Development records a case where it was: five parallel attempts shared one sandbox and one worker log, and what looked like variation in the model was partly a concurrency bug. Runs that share a sandbox, a seed, a cached response, or a world’s leftover state are correlated, and five correlated passes are fewer than five observations. The posterior is then narrower than the evidence warrants, and the count alone does not say by how much.
What we believed before
The uniform prior is a convenient place to start a worked example. It is not a description of what anyone believes about a real scenario. A scenario written this morning may exercise a mechanism the team has tested for months; another may depend on a tool the agent has never used successfully. They are both new files, and they do not begin with the same evidence.
Change the prior in the figure. Beta(2, 2) puts more weight near the middle. Beta(8, 2) is centered at 80% and more concentrated. After five clean runs, their next-pass predictions are 78% and 87%, against 86% under the uniform prior. The more useful difference is in the tail. Beta(8, 2) gives a higher posterior mean than the uniform prior and a lower probability of exceeding 90%. A belief concentrated around 80% is confident the agent is good and skeptical that it is nearly perfect, and no single mean can express both at once.
In clinical-trial planning this was the sparse-site problem: a site with two patients had little evidence of its own, but it was not the first site anyone had opened. A scenario with five runs is in the same position, and related scenarios can inform its prior, provided we can say why they are related.
The account across time
The prior that matters most in practice is not a family’s. It is the scenario’s own history. The posterior after today’s runs is, in the plainest reading, the prior for tomorrow’s: what we believe about a scenario is a running account, and the suite is how the account gets updated. Whether that reading is right depends entirely on what happens between today and tomorrow.
If nothing in θ(model, harness, tools, world, grader) changed, carrying the posterior forward is exactly correct. It is only batching. Fifty runs spread across ten weeks give the same posterior as fifty runs on one afternoon, and there is no bias in treating them so.
If something changed, the runs on either side of the change are observations of different quantities, and carrying introduces a specific distortion: inertia, growing with the length of the history. Suppose a scenario has accumulated 50 clean runs, Beta(51, 1). A model update lands, and this week’s batch is three of five. Under the carried prior, a batch that poor was a 0.6% event. Yet the updated posterior is Beta(54, 3), with mean 95%. The surprise is enormous and the update is tiny, because 50 pseudo-counts outweigh 5 observations. The longer the suite has been running, the more a plain carry-forward makes it blind to the regressions it exists to catch, and the same mechanism hides improvements after a fix.
The way out is to let the data decide whether the carry was warranted. Hold two hypotheses about the scenario: it is unchanged, in which case its prior is the old posterior; or it changed, in which case its prior is uniform or a family’s. The batch updates both the rate and the probability of each hypothesis. In the example, starting with four-to-one odds that nothing changed, three of five gives an 87% posterior probability that the scenario did change, and the rate estimate moves with it: the changed branch alone has a mean of 57%, and the mixture’s mean is about 62%, against 95% under the plain carry.3 A surprising batch is evidence that the identity moved. A plain carry absorbs the surprise; the mixture acts on it. Operationally this is “carry forward, watch how surprised the carried belief is by each new batch, and reset when it is surprised,” which is a rule a team can run without fitting anything.
The bookkeeping that makes any of this possible is recording the identity of every run. Without it, the account cannot say which runs belong to the same θ, and the question “can we carry this forward” has no answer.
A suite has more than one answer
Now hold the system fixed and vary the scenario. Give the agent ten of them. Each has its own rate and its own posterior, and the suite is a collection of uncertain behaviors rather than one number.
Consider two suites, each with 80 passes in 100 attempts. In the first, every scenario passes eight of ten. In the second, eight scenarios pass ten of ten and two fail every time. The totals match. The engineering work they call for does not. The first describes unreliability spread across everything; the second describes two specific mechanisms the agent cannot do.
| Quantity | Posterior mean | Posterior SD | Expected count |
|---|---|---|---|
| Suite average | 75.0% | 3.8% | — |
| Expected scenarios above 90% | — | — | 0.90 of 10 |
Under independent uniform priors both suites have a posterior mean of 75% for the equally weighted average of the ten rates, and even that average carries different uncertainty in the two cases. The mean does not say where the failures live.
A suite can answer several questions, and they are different calculations from the same set of posteriors. The average rate. The number of scenarios below some requirement. Whether every scenario clears it. How many failures the next full execution will contain. The last two are the claims a green board seems to make, and they are the most expensive to support. If 50 scenarios each pass five of five, independent uniform priors give each a 53% probability of sitting below 90%, so the expected number below the bar is about 27, and the probability that all 50 clear it is effectively zero. Reaching 95% probability on “every scenario is above 90%” takes 65 clean runs per scenario. Those figures hold only if the scenarios have nothing in common. Assume the opposite, that all 50 share a single rate, and the same 250 passes put the chance that it sits below 90% at about three in a trillion; one clean run per scenario would already have been enough. The same board supports almost nothing or almost everything depending on how alike the scenarios are, and that is an assumption nobody wrote down. The board is green at five. Nothing on it says which claim it is entitled to.4
More scenarios make the average precise while leaving each member uncertain. Fifty scenarios at ten of ten give the average a posterior mean of 92% with a standard deviation near one point; each scenario’s own 90% interval still runs from 76% to nearly 100%. Treating the 500 passes as evidence about one shared rate would answer a different question under a stronger assumption, that all 50 scenarios have the same reliability. Treating them as unrelated refuses to learn across them. A hierarchical model sits between, estimating how much they can share.5 That sharing should follow a mechanism the team can defend: the same tool, the same kind of ambiguous identity, the same recovery from a timed-out write. Pooling by folder name is not a mechanism.
The system changes underneath the suite
So far every θ has belonged to one fixed system. Now vary that instead. A prompt changes. A tool gains a field. The harness is refactored. The model is replaced. Each is a change to one argument of θ(model, harness, tools, world, grader, date), and each produces a new quantity. The runs we accumulated describe the old one.
There are two ways to read a suite across such changes, and a team needs both. They answer different questions, they cost different amounts, and the order in which to adopt them is not a matter of taste.
The first instrument: this week against last week
The first is to run the suite fresh each time and compare this run with the previous one. This is a difference question, and it needs no history beyond the two runs being compared. Did any regression scenario drop? Did any frontier scenario rise? Has a frontier scenario passed a declared clean batch, so that it graduates into the regression set and becomes a behavior we preserve rather than one we are reaching for?
This instrument is where a suite should start, and it is more powerful than it looks when the comparison is paired: the same scenarios, the same worlds, the same seeds where possible, under both versions. Paired, a large move shows in the five or ten runs that could never support a level claim: a scenario that passed five of five last week and two of five this week has, under uniform priors, a 97% probability of having dropped. A small move does not show. If the true rate slides from 95% to 85%, ten runs a side show fewer passes about 64% of the time, and they do so 27% of the time when nothing changed at all.6 It teaches three things cleanly: what regressed, what progressed, and what is ready to be promised. It is honest in a specific way, since each week’s conclusion stands on that week’s evidence and nothing is carried that could go stale.
What it cannot teach is anything about levels. Five runs against five runs says whether the scenario moved. It does not say whether the scenario is above 90%, and no week-over-week comparison ever will, because that claim has a price: under a uniform prior it takes 28 clean runs to be 95% sure that one scenario is above 90%, and, as the suite section showed, 65 per scenario for a suite of 50 unrelated ones. A team that only compares week to week will never know its agent’s reliability. It will know its agent’s direction.
The second instrument: the running account
The second is the running account introduced earlier: a posterior per scenario, carried forward and discounted, that accumulates what the suite has learned. It exists because learning the agent’s true levels and true boundaries takes far more evidence than any single week produces. Many runs per scenario, many scenarios per mechanism, and the same scenarios revisited as the system changes. Starting from scratch every Monday means never arriving.
The account teaches four things the first instrument cannot. Levels: whether a scenario, or a family, is above a bar, which is the claim a reliability dashboard pretends to make and only accumulation can support. Slow drift: this week’s five runs against last week’s five cannot see 95% sliding to 85%, since the difference sits inside the noise; this week’s five against a tight carried belief can, because the comparison is against a running estimate rather than another small sample. Allocation: the width of each posterior says where next week’s runs should go, which a table of deltas does not. And a cheaper baseline for the first instrument itself: the old version usually has a long history and the new one has none, so the baseline side can borrow its carried posterior and only the candidate side needs fresh runs. That halves the cost of a comparison and gives up the pairing, since the borrowed runs did not share worlds or seeds with the new ones. It is enough for a routine check; a close call still deserves both versions run side by side.
Its price is the bookkeeping from the account section: an identity on every run, and a rule for how much to carry across a change. Without the rule, the account is ruled by inertia. The account section gave the rule a shape; the discount gives it a number.
How much to carry
Write the discount as δ between zero and one. The old posterior Beta(α, β) becomes the prior Beta(1 + δ(α − 1), 1 + δ(β − 1)); the old runs are worth δ of what they were.7 Teams choose δ on every change, usually without noticing, and the choice takes one of three values. Carrying everything, δ = 1, assumes the change did nothing, which is the hypothesis the new runs are supposed to test. Starting over, δ = 0, is honest and ruinous: it is the first instrument with extra steps, and no scenario ever reaches 28 clean runs because the system moves before it gets there. Something in between is what anyone sensible does, and the number is rarely written down.
It should be, because the new runs read differently under each choice. Take a scenario at ten of ten under the old model, then upgrade the model and run it five times clean. Carrying everything, the posterior is Beta(16, 1) and the probability of being above 90% is 81%. Starting over, Beta(6, 1), 47%. Keeping three of the ten, Beta(9, 1), 61%. Same scenario, same five new passes, and a conclusion that runs from a coin flip to a confident claim depending on a parameter nobody mentioned.
| Stage | Distribution | Next pass | P(rate > 90%) |
|---|---|---|---|
| Old version | Beta(11, 1) | 92% | 69% |
| Prior for new version | Beta(4, 1) | 80% | 34% |
| New version | Beta(9, 1) | 90% | 61% |
Two things make δ less arbitrary than it looks. Changes have footprints: a schema change in one tool touches the scenarios that use that tool and leaves the rest alone, and a model swap touches everything but not equally. If scenarios are tagged by mechanism, as the suite section already wanted, the discount can be applied by family: near zero where the change landed, near one where it did not, with the surprise rule from the account section catching the cases where the footprint was guessed wrong. And the team has history: across the last several model upgrades, how far did pass rates move, and in which families? That record is an empirical prior on the size of a change. In the clinical portfolio, 2020 broke every enrollment parameter and left the structure of the model intact. A model upgrade is a smaller version of the same event; the scenarios and the mechanism taxonomy survive, and the rates are what has to be re-learned.
Some changes are not deliberate. The provider updates the model under the same name. A fixture’s upstream data shifts. The judge is replaced. For these there is no diff to pair against, and the useful instrument is a control chart rather than a gate: a small set of canary scenarios run on a schedule, with their posteriors tracked over time. The best canaries are the boring ones. A scenario that has passed ten of ten for months is the one whose first failure is most informative, because under a rate near 99% a failure in ten attempts is a 10% event and two are a 0.4% one. For finding frontier capability, the uncertain scenarios carry the information. For detecting drift, the stable ones do.
Put together, this says that reliability claims have a half-life. “This scenario is above 90%” is never true unconditionally. It is true as of a version, and its probability decays with every change that was not re-tested. The record of a run therefore needs an identity, which is the full expression θ was written with, and the evidence carried from a previous identity needs its discount stated. “Five of five on harness 3.2 with model M; prior carried from 3.1 at δ = 0.5” is a sentence a colleague can argue with. “Five of five” is not.
From tested cases to a boundary
Return to the customer record and vary the last thing: the conditions inside the world. The name was unique, and the agent updated the right record. Now give two customers that name. Remove the identifier. Let the write commit before the tool reports a timeout. Take away the user’s permission.
These are different conditions, not additional samples of the original scenario, and θ depends on them. Repeated attempts at each condition estimate a local reliability. Comparisons across conditions show which differences in the world matter. A boundary starts to appear: ordinary updates work, ambiguous identities need a clarifying question, recovery after an uncertain write is still unsafe.
The purpose of all this is worth stating plainly. The true boundary of an agent’s competence is unknown. The suite is the charted part of it. Production is where users discover the uncharted part, one incident at a time, and the job of the suite is to find the edges before they do. The measure of a suite is not its pass rate. It is how much of what production later finds was already on the chart.
Seen that way, the conditions an agent might meet fall into four regions. Conditions tested and found to work. Conditions tested and found to fail, which are often the most valuable scenarios a team owns, because each is a known limit with a name. Conditions named but not yet run: the backlog, the known unknowns. And conditions nobody has named. Repetition narrows uncertainty in the first two regions. New scenarios move conditions from the third into the first two. Nothing in the suite touches the fourth. It shrinks only by production incidents, by deliberately adversarial exploration, and by varying conditions systematically until something breaks. A suite can only ever chart the conditions someone thought to name.
The figure makes the four regions visible in a space with two conditions, which is already a simplification: real condition spaces have many axes, some of them categorical, and the assumption that nearby conditions behave alike is itself a claim about similarity. Even so, the figure shows the three things the prose cannot. A new scenario can land inside the charted region and fail, because precision at the neighboring points said nothing about the gap between them. A week of production drops incidents mostly in the fog, at conditions nobody named, and each one is a scenario waiting to be written, which is the loop described in Eval-Driven Development and the only mechanism by which the fourth region becomes the third. And the button that reveals the true boundary exists only because the data is invented. No such button exists for a real agent.
Two confusions are easy on a dashboard where everything is a high number. A narrow posterior is evidence of precision, not of coverage: a thousand clean runs with a unique name establish nothing about what happens when the name is ambiguous. Carrying evidence from one condition to another is the same act as carrying it across a model upgrade, with a δ between conditions instead of between versions, and with no stated reason to think the two behave alike that δ is zero. And two scenarios near 50% can mean opposite things. One has barely been run and sits there because the prior is doing the work; it needs evidence. The other has been run extensively and fails about half the time; it needs a change. A scenario that fails about half the time sometimes hides a boundary rather than a coin: an agent that succeeds on every unambiguous request and fails on every ambiguous one is 50% reliable on a mixed sample. More runs will narrow that number without explaining it. A scenario that records the condition would.
The boundary is also behavioral. At some conditions the right outcome is to ask which customer, to obtain authorization, or to stop and verify what a timed-out write did, and an agent that proceeds confidently where it should have asked is as far outside its boundary as one that fails outright. Counting both as one kind of failure hides the difference that matters most for deciding how much autonomy to grant.
How cases are chosen decides what the pass rate estimates. Regression cases, frontier cases, adversarial cases, and a sample of production tasks each measure something different, so a suite can become more informative while its aggregate score falls, because it has begun asking questions it did not know to ask. Learning where the boundary runs, and how to spend the next scenario on moving it, is the subject of the next essay.8
What to put in the report
The point estimate is not the problem. Five of five belongs in the report. The problem is a report that offers the number and lets the reader supply the interpretation, because the reader will supply the strongest one available.
A report that keeps the argument visible has five parts. What happened: five of five. What we infer, under a stated model: an 86% chance the next attempt passes and a 47% probability that the underlying rate exceeds 90%. What that is conditional on: model, harness version, world version, grader version, date. What was carried from before: the previous version’s evidence at δ = 0.5, or nothing. And what it licenses: the action that follows, against a threshold set before the batch ran. Each part is a claim someone can check, and together they say how much the number is worth and for how long.
The fifth part is the one that changes what the team does. A posterior supports three actions: promise the behavior, change the agent, or buy more runs. Say a scenario graduates into the regression set once the probability that its rate exceeds 90% passes 80%. The upgraded scenario from earlier, ten of ten on the old model and five of five on the new, graduates today if everything is carried. Keeping three of the ten, it needs seven more clean runs; starting over, it needs ten. The discount nobody wrote down decides whether the behavior is promised today or two batches from now.
The last obligation is to find out whether the inferences deserve to be believed. Record the predictions before the next batch runs, then compare. Of the scenarios given an 80% chance of passing, did about 80% pass? Do the misses cluster around a tool, a mechanism, a model version? Check within families as well as overall, since good aggregate calibration can hide a systematic error in one corner, and remember that a model which always predicts the average is calibrated and useless.9 A posterior is a model’s answer. Future runs are how we learn whether the model has earned the trust the report asks for.
An eval suite, read this way, is a running account of what we believe about an agent: updated by every run, widened by every new scenario, and dated by every change to the system it describes. The useful output is not a green board. It is an increasingly specific statement of where the agent can be trusted, under which conditions, as of which version, and which observation would change our mind.
Footnotes
-
Under a uniform prior after n clean runs, the probability that the next m attempts all pass is (n + 1)/(n + m + 1); for n = 5 and m = 8 that is 6/14. The τ-bench authors introduced pass^k for this reason: an agent that completes a task once is not deployable if it fails the same task on the next call. Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, 2024. The prediction concerns repeated attempts under the conditions the scenario models. ↩
-
The Beta density is proportional to θ to the power α − 1 times (1 − θ) to the power β − 1; multiplying by the binomial likelihood gives the update in the text. α + β is the prior’s effective sample size, so Beta(8, 2) carries the weight of ten prior runs at an 80% rate. All numerical examples assume conditionally independent outcomes with a fixed rate, and intervals are central 90% credible intervals. The height of a density curve at a point is not a probability; probability belongs to an interval and is the area beneath the curve over it. ↩
-
Under the carried prior Beta(51, 1), the predictive probability of exactly three passes in five is 0.6%; under a uniform prior every count from zero to five is equally likely, one in six. With prior odds of four to one for “unchanged,” the posterior odds for “changed” are (1/5 × 1/6) / (4/5 × 0.0061) ≈ 6.8, or 87%. This is the simplest change-point model; the fully dynamic version lets θ drift between versions and updates it as a time series. The practical rule, reset when the carried belief is surprised, is the same test run by eye. ↩
-
With a uniform prior, n clean runs give P(θ < θ*) = θⁿ⁺¹ per scenario, so the count below the bar across S independent scenarios is Binomial(S, θⁿ⁺¹). The probability that all S clear the bar is (1 − θ*ⁿ⁺¹)ˢ; for 50 scenarios at 90% this first exceeds 0.95 at n = 65. If the scenarios instead share one rate, 250 clean runs give Beta(251, 1) and P(θ < 0.9) = 0.9²⁵¹ ≈ 3 × 10⁻¹². The Stan guide to posterior predictive sampling covers the general method of drawing rates from the joint posterior and computing the quantity of interest on each draw. ↩
-
Bob Carpenter’s Hierarchical Partial Pooling for Repeated Binary Trials compares complete pooling, no pooling, and the hierarchical model, including prediction on held-out trials. Learning the family parameters jointly propagates their uncertainty; fitting them once and treating them as fixed understates it. A family made only of scenarios admitted after clean batches has been selected for observed success, and fitting it as if unselected exaggerates what a new scenario should inherit. ↩
-
With both versions run on the same worlds, the posterior on the difference in rates is what the decision needs, and the branch note works through why a ranking cannot carry it. A change can improve a weak scenario substantially while leaving it unreliable, or leave a strong scenario above an absolute bar while degrading it meaningfully; the paired comparison sees both where a level claim sees neither. ↩
-
This is the power prior of Ibrahim and Chen, “Power Prior Distributions for Regression Models”, Statistical Science, 2000: the historical likelihood raised to a power δ before being combined with new data. For a Beta posterior it reduces to the shrinkage of pseudo-counts in the text. A per-week δ applied to time rather than to deliberate changes is exponential forgetting, and a value between 0.5 and 0.8 gives the account a memory of a few weeks, which is a reasonable default for drift nobody announced. ↩
-
Learning where a success function crosses a threshold, rather than its value everywhere, is active level-set estimation; Gotovos et al., “Active Learning for Level Set Estimation”, 2013. The decision-theoretic version is expected value of sample information: the next observation is worth most where it is most likely to change a consequential decision, which is not the same as where uncertainty is largest. Applying either to agents requires task features that mean something and a check on the similarity assumptions. ↩
-
Posterior and prior predictive checks compare a model’s implications with observed patterns. Prospective calibration on new runs complements them. Neither establishes coverage of conditions absent from the evaluation process. ↩
- Essay
- Essay
- Field Note
- Field Note
- Jun 2026Previous essay