Known-bad candidates

Before I trust a scenario, I want to see it fail on purpose.

The check is simple. Hand the grader a candidate that is wrong in the way the scenario exists to catch: a workflow that writes before its prerequisite search resolves, an email that uses the account’s field where the lead’s was required, a repair that silently switches to a different connection. If the scenario passes it, the scenario is not measuring the thing its title says it measures. It is measuring something adjacent, usually the shape of the output rather than its consequence.

Then the reverse. Hand it a candidate that is correct but takes a route the author didn’t anticipate: fewer steps, a different tool, no intermediate lookup. If that fails, the grader has encoded one implementation rather than the requirement, and every future improvement that arrives by a different path will register as a regression.

This is mutation testing, applied to the test instead of the code, and it is rare in agent evals because green results are comfortable and the machinery is expensive to run. But an eval suite that has never seen a known-bad candidate is a suite whose false-negative rate is unknown. You have a classifier with no measured recall.

The cheapest version costs one extra fixture per scenario: the intended solution with its critical step removed or inverted. If that still passes, stop and fix the grader before reading anything else the scenario says.