Which branch would you bet on

Four branches, one eval suite, one to merge. All four pass the regression gate; that part is not a decision, it is a precondition. The decision is on the frontier scenarios, the ones that fail today and mark where the product’s capability ends. Run each branch against them, rank by pass rate, pick the top.

I’ve stopped trusting that ranking, for three reasons that are specific to frontier scenarios.

They are the noisiest scenarios you own. They fail today, so any movement is small-sample. B takes a scenario from 1/10 to 4/10; C takes it to 3/10. The posterior on that difference is wide enough that “B beat C” is close to a coin flip presented as a ranking. Regression scenarios don’t have this problem, because 10/10 staying 10/10 is not a measurement. On the frontier, the small-sample problem is the whole comparison.

They are valued unequally, and the leaderboard hides that it weighted them at all. Closing “the agent handles a multi-record CRM workflow” is worth more than closing “the agent picks the nicer date format.” Any ranking by pass rate is a weighted sum with every weight set to one, which is a product-priority statement nobody made on purpose. Writing the weights down is uncomfortable exactly because it forces the statement.

Movement has a price. A scenario goes from 2/10 to 8/10 because the branch added retries or a larger model. That is capability bought with tokens and latency. The rate alone rewards spending; the honest unit is the pair, movement and what it cost.

Put those together and the question changes. It is not “which branch scored highest.” It is: for the gaps we actually care about, what is the probability that B moved them more than C, at what cost, and is that probability far enough from one half to act on?

Often it isn’t. Five frontier scenarios at ten runs each cannot separate two branches whose true rates differ by ten points. When the probability sits near one half, the leaderboard still produces a winner, because ranking always produces a winner. The decision-theoretic answer is different: don’t pick, spend the next thirty runs on the two scenarios that account for most of the uncertainty, and decide tomorrow. That fourth option, buy information instead of choosing, is the thing a ranking structurally cannot express, and it is the one an experienced engineer reaches for anyway, usually in a meeting after the number has already anchored the room.¹

The underlying mistake is treating an evaluation result as a score when it is an input to a bet. A score is a property of the branch. A bet needs two more things the score does not carry: what you would rather have, and how sure you are. Frontier scenarios are precisely the place where both are missing by default, because they are new, sparse, and unequally important. That makes them the place where writing both down changes the decision, rather than decorating it.

Footnotes

1. The width shrinks faster by pooling than by rerunning. Frontier scenarios that share a mechanism, the same tool, the same failure class, carry information about each other, and a hierarchical estimate borrows it. That is a separate note. ↩