What an eval suite lets you believe

A pull request changes a prompt. A scenario that asks the agent to update a customer record passes five times out of five, and the summary says so. Everyone reading it knows five runs is not many. Nobody can say, on the spot, how much five runs should change what they believe about the agent, so the number stands and gets repeated as if it meant what it says.

I want to work that question out carefully, because answering it is the whole job an eval suite does. A suite is not a test the agent passes. It is the instrument by which a team updates its beliefs about what the agent can do. Each run moves a distribution. The distribution after today’s runs is where tomorrow’s begin. New scenarios widen what the distribution is about. And everything the distribution says is conditional on a system that will not stay fixed.

That last clause is the one to keep in view. A pass rate belongs to a particular agent under particular conditions. Written out, it is θ(model, harness, tools, world, grader, date): the probability that this model, inside this harness, with these tools, in this simulated world, judged by this grader, on this date, completes the task. Each section of this essay changes one part of that expression while holding the rest still. First everything is fixed and we run once, then again. Then we vary the scenario and look at the suite. Then we vary the system and ask what survives. Finally we vary the conditions inside the world and ask where the behavior holds. At each step something enters the argument that the runs alone did not supply, and the point of writing the argument in Bayesian form is to keep those additions visible.

One successful run

Hold the whole expression fixed and let only the randomness of execution vary. There is some unknown probability θ that the agent passes this scenario. Each attempt is a draw from it.

A pass is nine times as likely under a ninety-percent pass rate as under a ten-percent one. Observing a pass therefore multiplies the odds between those two possibilities by nine. But a pass is only 1.125 times as likely under ninety percent as under eighty. One success separates a very unreliable agent from a reliable one far better than it separates a reliable agent from an excellent one. That asymmetry is most of what follows.

Before the run, we hold a distribution over possible rates. The outcome gives more weight to the rates that would have produced it more often. The result is the posterior, and it describes our uncertainty about θ; it is not a spread of scores from individual runs, which remain plain passes and failures. Start, for this worked example, with a uniform prior: every rate from zero to one equally plausible. After one pass the posterior leans toward high rates but remains broad. Its mean is two-thirds, which under this model is also the probability that the next attempt passes. The middle ninety percent of its probability stretches from about twenty-two to ninety-seven percent.

Keep going and watch two different quantities move. After three clean runs, the next-pass prediction is eighty percent; after five, eighty-six; after ten, ninety-two. Meanwhile the probability that θ exceeds ninety percent rises from nineteen percent after one pass to thirty-four, forty-seven, and sixty-nine. A third quantity moves slowest of all. The probability that the next eight attempts all pass, which is closer to what a user experiences than any single-run number, is forty-three percent after five clean runs.¹

These are three answers to “what do we now believe,” and they are not interchangeable. A report that says “100%” has chosen not to say which one it means. Readers will hear the first and act on the third.

Switch the figure to ten attempts with failures. At nine of ten, the probability of a rate above ninety percent is about thirty percent under the uniform prior. At eight of ten it is nine. A single failure moves the upper tail a long way, because near-perfect rates had assigned that failure very little probability. Mixed results are evidence about the shape of reliability, not a degraded version of a green one.

The arithmetic behind all of this is compact. A Beta(α, β) prior and k passes in n attempts give a Beta(α + k, β + n − k) posterior, with mean (α + k)/(α + β + n). The uniform prior is Beta(1, 1).² Its virtue is that the observations and the starting assumptions stay separately visible in the result.

What we believed before

The uniform prior is a convenient place to start a worked example. It is not a description of what anyone believes about a real scenario. A scenario written this morning may exercise a mechanism the team has tested for months; another may depend on a tool the agent has never used successfully. They are both new files, and they do not begin with the same evidence.

Change the prior in the figure. Beta(2, 2) puts more weight near the middle. Beta(8, 2) is centered at eighty percent and more concentrated. After five clean runs, their next-pass predictions are seventy-eight and eighty-seven percent, against eighty-six under the uniform prior. The more useful difference is in the tail. Beta(8, 2) gives a higher posterior mean than the uniform prior and a lower probability of exceeding ninety. A belief concentrated around eighty percent is confident the agent is good and skeptical that it is nearly perfect, and no single mean can express both at once.

In clinical-trial planning this was the sparse-site problem: a site with two patients had little evidence of its own, but it was not the first site anyone had opened. A scenario with five runs is in the same position, and related scenarios can inform its prior, provided we can say why they are related.

The account across time

The prior that matters most in practice is not a family’s. It is the scenario’s own history. The posterior after today’s runs is, in the plainest reading, the prior for tomorrow’s: what we believe about a scenario is a running account, and the suite is how the account gets updated. Whether that reading is right depends entirely on what happens between today and tomorrow.

If nothing in θ(model, harness, tools, world, grader) changed, carrying the posterior forward is exactly correct. It is only batching. Fifty runs spread across ten weeks give the same posterior as fifty runs on one afternoon, and there is no bias in treating them so.

If something changed, the runs on either side of the change are observations of different quantities, and carrying introduces a specific distortion: inertia, growing with the length of the history. Suppose a scenario has accumulated fifty clean runs, Beta(51, 1). A model update lands, and this week’s batch is three of five. Under the carried prior, a batch that poor was a 0.6 percent event. Yet the updated posterior is Beta(54, 3), with mean ninety-five percent. The surprise is enormous and the update is tiny, because fifty pseudo-counts outweigh five observations. The longer the suite has been running, the more a plain carry-forward makes it blind to the regressions it exists to catch, and the same mechanism hides improvements after a fix.

The way out is to let the data decide whether the carry was warranted. Hold two hypotheses about the scenario: it is unchanged, in which case its prior is the old posterior; or it changed, in which case its prior is uniform or a family’s. The batch updates both the rate and the probability of each hypothesis. In the example, starting with four-to-one odds that nothing changed, three of five gives an eighty-seven percent posterior probability that the scenario did change, and the rate estimate follows that branch to a mean near fifty-seven percent.³ A surprising batch is evidence that the identity moved. A plain carry absorbs the surprise; the mixture acts on it. Operationally this is “carry forward, watch how surprised the carried belief is by each new batch, and reset when it is surprised,” which is a rule a team can run without fitting anything.

The bookkeeping that makes any of this possible is recording the identity of every run. Without it, the account cannot say which runs belong to the same θ, and the question “can we carry this forward” has no answer.

A suite has more than one answer

Now hold the system fixed and vary the scenario. Give the agent ten of them. Each has its own rate and its own posterior, and the suite is a collection of uncertain behaviors rather than one number.

Consider two suites, each with eighty passes in a hundred attempts. In the first, every scenario passes eight of ten. In the second, eight scenarios pass ten of ten and two fail every time. The totals match. The engineering work they call for does not. The first describes unreliability spread across everything; the second describes two specific mechanisms the agent cannot do.

Under independent uniform priors both suites have a posterior mean of seventy-five percent for the equally weighted average of the ten rates, and even that average carries different uncertainty in the two cases. The mean does not say where the failures live.

A suite can answer several questions, and they are different calculations from the same set of posteriors. The average rate. The number of scenarios below some requirement. Whether every scenario clears it. How many failures the next full execution will contain. The last two are the claims a green board seems to make, and they are the most expensive to support. If fifty scenarios each pass five of five, independent uniform priors give each a fifty-three percent probability of sitting below ninety, so the expected number below the bar is about twenty-seven, and the probability that all fifty clear it is effectively zero. Reaching ninety-five percent probability on “every scenario is above ninety” takes sixty-five clean runs per scenario. The board is green at five. Nothing on it says which claim it is entitled to.⁴

More scenarios make the average precise while leaving each member uncertain. Fifty scenarios at ten of ten give the average a posterior mean of ninety-two percent with a standard deviation near one point; each scenario’s own ninety-percent interval still runs from seventy-six to nearly one hundred. Treating the five hundred passes as evidence about one shared rate would answer a different question under a stronger assumption, that all fifty scenarios have the same reliability. Treating them as unrelated refuses to learn across them. A hierarchical model sits between, estimating how much they can share.⁵ That sharing should follow a mechanism the team can defend: the same tool, the same kind of ambiguous identity, the same recovery from a timed-out write. Pooling by folder name is not a mechanism.

The system changes underneath the suite

So far every θ has belonged to one fixed system. Now vary that instead. A prompt changes. A tool gains a field. The harness is refactored. The model is replaced. Each is a change to one argument of θ(model, harness, tools, world, grader, date), and each produces a new quantity. The runs we accumulated describe the old one.

There are two ways to read a suite across such changes, and a team needs both. They answer different questions, they cost different amounts, and the order in which to adopt them is not a matter of taste.

The first instrument: this week against last week

The first is to run the suite fresh each time and compare this run with the previous one. This is a difference question, and it needs no history beyond the two runs being compared. Did any regression scenario drop? Did any frontier scenario rise? Has a frontier scenario passed a declared clean batch, so that it graduates into the regression set and becomes a behavior we preserve rather than one we are reaching for?

This instrument is where a suite should start, and it is more powerful than it looks when the comparison is paired: the same scenarios, the same worlds, the same seeds where possible, under both versions. Paired, the question “is the new rate lower than the old by an amount that matters” can often be settled in the five or ten runs that a level claim never could.⁶ It teaches three things cleanly: what regressed, what progressed, and what is ready to be promised. It is honest in a specific way, since each week’s conclusion stands on that week’s evidence and nothing is carried that could go stale.

What it cannot teach is anything about levels. Five runs against five runs says whether the scenario moved. It does not say whether the scenario is above ninety percent, and no week-over-week comparison ever will, because the first section already established the price of that claim: twenty-eight clean runs for one scenario at ninety, sixty-five per scenario for a suite of fifty. A team that only compares week to week will never know its agent’s reliability. It will know its agent’s direction.

The second instrument: the running account

The second is the account from the previous section: a posterior per scenario, carried forward and discounted, that accumulates what the suite has learned. It exists because learning the agent’s true levels and true boundaries takes far more evidence than any single week produces. Many runs per scenario, many scenarios per mechanism, and the same scenarios revisited as the system changes. Starting from scratch every Monday means never arriving.

The account teaches four things the first instrument cannot. Levels: whether a scenario, or a family, is above a bar, which is the claim a reliability dashboard pretends to make and only accumulation can support. Slow drift: this week’s five runs against last week’s five cannot see ninety-five percent sliding to eighty-five, since the difference sits inside the noise; this week’s five against a tight carried belief can, because the comparison is against a running estimate rather than another small sample. Allocation: the width of each posterior says where next week’s runs should go, which a table of deltas does not. And a sharper baseline for the first instrument itself: in a paired comparison, the old version usually has a long history and the new one has none, so the baseline side can borrow its carried posterior and only the candidate side needs fresh runs, which halves the cost of every comparison.

Its price is the bookkeeping from the previous section: an identity on every run, and a rule for how much to carry across a change. Without the rule, the account is ruled by inertia. The previous section gave the rule a shape; the discount gives it a number.

How much to carry

Write the discount as δ between zero and one. The old posterior Beta(α, β) becomes the prior Beta(1 + δ(α − 1), 1 + δ(β − 1)); the old runs are worth δ of what they were.⁷ Teams choose δ on every change, usually without noticing, and the choice takes one of three values. Carrying everything, δ = 1, assumes the change did nothing, which is the hypothesis the new runs are supposed to test. Starting over, δ = 0, is honest and ruinous: it is the first instrument with extra steps, and no scenario ever reaches twenty-eight clean runs because the system moves before it gets there. Something in between is what anyone sensible does, and the number is rarely written down.

It should be, because the new runs read differently under each choice. Take a scenario at ten of ten under the old model, then upgrade the model and run it five times clean. Carrying everything, the posterior is Beta(16, 1) and the probability of being above ninety is eighty-one percent. Starting over, Beta(6, 1), forty-seven percent. Keeping a third, Beta(9, 1), sixty-one percent. Same scenario, same five new passes, and a conclusion that runs from a coin flip to a confident claim depending on a parameter nobody mentioned.

Two things make δ less arbitrary than it looks. Changes have footprints: a schema change in one tool touches the scenarios that use that tool and leaves the rest alone, and a model swap touches everything but not equally. If scenarios are tagged by mechanism, as the suite section already wanted, the discount can be applied by family: near zero where the change landed, near one where it did not, with the surprise rule from the account section catching the cases where the footprint was guessed wrong. And the team has history: across the last several model upgrades, how far did pass rates move, and in which families? That record is an empirical prior on the size of a change. In the clinical portfolio, 2020 broke every enrollment parameter and left the structure of the model intact. A model upgrade is a smaller version of the same event; the scenarios and the mechanism taxonomy survive, and the rates are what has to be re-learned.

Some changes are not deliberate. The provider updates the model under the same name. A fixture’s upstream data shifts. The judge is replaced. For these there is no diff to pair against, and the useful instrument is a control chart rather than a gate: a small set of canary scenarios run on a schedule, with their posteriors tracked over time. The best canaries are the boring ones. A scenario that has passed ten of ten for months is the one whose first failure is most informative, because under a rate near ninety-nine percent a failure in ten attempts is a ten-percent event and two are a half-percent one. For finding frontier capability, the uncertain scenarios carry the information. For detecting drift, the stable ones do.

Put together, this says that reliability claims have a half-life. “This scenario is above ninety percent” is never true unconditionally. It is true as of a version, and its probability decays with every change that was not re-tested. The record of a run therefore needs an identity, which is the full expression θ was written with, and the evidence carried from a previous identity needs its discount stated. “Five of five on harness 3.2 with model M; prior carried from 3.1 at δ = 0.5” is a sentence a colleague can argue with. “Five of five” is not.

From tested cases to a boundary

Return to the customer record and vary the last thing: the conditions inside the world. The name was unique, and the agent updated the right record. Now give two customers that name. Remove the identifier. Let the write commit before the tool reports a timeout. Take away the user’s permission.

These are different conditions, not additional samples of the original scenario, and θ depends on them. Repeated attempts at each condition estimate a local reliability. Comparisons across conditions show which differences in the world matter. A boundary starts to appear: ordinary updates work, ambiguous identities need a clarifying question, recovery after an uncertain write is still unsafe.

The purpose of all this is worth stating plainly. The true boundary of an agent’s competence is unknown. The suite is the charted part of it. Production is where users discover the uncharted part, one incident at a time, and the job of the suite is to find the edges before they do. The measure of a suite is not its pass rate. It is how much of what production later finds was already on the chart.

Seen that way, the conditions an agent might meet fall into four regions. Conditions tested and found to work. Conditions tested and found to fail, which are often the most valuable scenarios a team owns, because each is a known limit with a name. Conditions named but not yet run: the backlog, the known unknowns. And conditions nobody has named. Repetition narrows uncertainty in the first two regions. New scenarios move conditions from the third into the first two. Nothing in the suite touches the fourth. It shrinks only by production incidents, by deliberately adversarial exploration, and by varying conditions systematically until something breaks. A suite can only ever chart the conditions someone thought to name.

The figure makes the four regions visible in a space with two conditions, which is already a simplification: real condition spaces have many axes, some of them categorical, and the assumption that nearby conditions behave alike is itself a claim about similarity that the follow-up to this essay takes up. Even so, the figure shows the three things the prose cannot. A new scenario can land inside the charted region and fail, because precision at the neighboring points said nothing about the gap between them. A week of production drops incidents mostly in the fog, at conditions nobody named, and each one is a scenario waiting to be written, which is the loop described in Eval-Driven Development and the only mechanism by which the fourth region becomes the third. And the button that reveals the true boundary exists only because the data is invented. No such button exists for a real agent.

Two confusions are easy on a dashboard where everything is a high number. A narrow posterior is evidence of precision, not of coverage: a thousand clean runs with a unique name establish nothing about what happens when the name is ambiguous. And two scenarios near fifty percent can mean opposite things. One has barely been run and sits there because the prior is doing the work; it needs evidence. The other has been run extensively and fails about half the time; it needs a change. A scenario that fails about half the time sometimes hides a boundary rather than a coin: an agent that succeeds on every unambiguous request and fails on every ambiguous one is fifty percent reliable on a mixed sample. More runs will narrow that number without explaining it. A scenario that records the condition would.

The boundary is also behavioral, not only a pass rate. At some conditions the right outcome is not to complete the task but to ask which customer, to obtain authorization, or to stop and verify what a timed-out write actually did. The chart has at least three colors, and an agent that proceeds confidently where it should have asked is as far outside its boundary as one that fails outright. Scoring both as a single failure count hides the difference that matters most for deciding how much autonomy to grant.

How cases are chosen decides what the pass rate estimates. Regression cases preserve known behavior. Frontier cases probe uncertain capability. Adversarial cases hunt for failures. A sample of production tasks measures something else again, and the four need not move together. A suite can become more informative while its aggregate score falls, because it has begun asking the questions it previously did not know to ask. Learning where the boundary runs, and how to spend the next scenario on moving it, is the subject of the next essay.⁸

What to put in the report

The point estimate is not the problem. Five of five belongs in the report. The problem is a report that offers the number and lets the reader supply the interpretation, because the reader will supply the strongest one available.

A report that keeps the argument visible has four parts. What happened: five of five. What we infer, under a stated model: an eighty-six percent chance the next attempt passes and a forty-seven percent probability that the underlying rate exceeds ninety. What that is conditional on: model, harness version, world version, grader version, date. And what was carried from before: the previous version’s evidence at δ = 0.5, or nothing. Each part is a claim someone can check, and together they say how much the number is worth and for how long.

The last obligation is to find out whether the inferences deserve to be believed. Record the predictions before the next batch runs, then compare. Of the scenarios given an eighty-percent chance of passing, did about eighty percent pass? Do the misses cluster around a tool, a mechanism, a model version? Check within families as well as overall, since good aggregate calibration can hide a systematic error in one corner, and remember that a model which always predicts the average is calibrated and useless.⁹ A posterior is a model’s answer. Future runs are how we learn whether the model has earned the trust the report asks for.

An eval suite, read this way, is a running account of what we believe about an agent: updated by every run, widened by every new scenario, and dated by every change to the system it describes. The useful output is not a green board. It is an increasingly specific statement of where the agent can be trusted, under which conditions, as of which version, and which observation would change our mind.