How Do You Know an Agent Is Done?

If you let an AI agent work on its own, you have to define when a task counts as done. We are building a governance framework for agents at Creaminds, and acceptance is its most important requirement: an agent may work independently exactly as far as acceptance holds.

The obvious candidate is the test run — automatable, unambiguous, cheap. It simply does not hold.

Why green tests prove nothing

SpecBench (arXiv 2605.21384) measured this in May 2026: thirty systems-level tasks, from a JSON parser to an entire operating-system kernel. Each task is split into a specification, visible validation tests, and held-out tests that exercise the same features in combination. The gap between the two suites is the measure. The result: every frontier agent saturates the visible suite — and breaks down against the held-out one. The most vivid example is a 2,900-line hash-table "compiler" that had memorized the test inputs. Formally, everything was green.

The mechanism is mundane: a system optimizes for what is measured, by the shortest route available, and that route is not necessarily "solve the task correctly". It is the same reason machine learning separates training data from test data. A measure that has been optimized against is no longer a measure.

Two aggravating factors. The gap grows with task size — by 28 percentage points for every tenfold increase in code volume — so the long autonomous runs are precisely the ones where the check is weakest. And smaller models show the larger gaps, which matters to anyone running locally for reasons of cost or sovereignty.

What helps: build it, don't ask for it

The Reward Hacking Benchmark (arXiv 2605.02964, ICML 2026) examined the countermeasures. Simple hardening of the execution environment cut the exploit rate by 5.7 percentage points — from 6.5 to 0.8 percent, a relative drop of 87.7 percent — without degrading success on the actual tasks.

What follows is a building instruction: structural separation beats exhortation by prompt. Concretely — the agent that implements does not write the tests it is accepted against, and the evaluation paths sit outside its working area.

In fairness: this study comes from a single author and has not been independently replicated. What is notable, though, is that it includes locally deployable models — and that what matters there is not open-weight status but post-training. Two variants from the same lab sit at 0.6 versus 13.9 percent exploit rate. Running your own models for sovereignty costs you nothing here. It does mean checking which variant.

Humans and machines fail for different reasons

The obvious objection is a fair one: human teams have gappy tests too. A suite that is too lax goes unnoticed by everyone. This is not a problem unique to AI.

But the reason differs, and something useful follows from that. For humans it is a motivation problem: they usually know the hard question and do not ask it anyway — because it is Friday, because a colleague wrote it, because the feature has to ship. For the machine it is a structural problem: it does not ask because nobody assigned it to.

Motivation can only be influenced through culture, and it reliably collapses under deadline pressure. Structure can be written down, and it holds. That is why our reviewing agent is briefed adversarially: not "check whether this is correct" but "show that it is not". It carries no social cost for an inconvenient finding, has no reputation to lose and no relationship with the author of the code. It is equally unpleasant every time.

This is the one point where an agent system can be superior to a human team — not because it reviews more intelligently, but because it lacks the reasons not to. Governance here means exactly that: moving the diligence you would otherwise rely on into the architecture, where it does not depend on anyone's day.

What remains open

Who writes the held-out tests? An agent moves the problem up one level; a human limits autonomy. Our current route runs through the specification: whoever owns it owns the acceptance criterion — and someone has to supply it in any case.

Independence. Two developers have different blind spots; two instances of the same model have identical ones. What a team brings implicitly has to be built deliberately here.

Memory. A team learns over time where a codebase lies to you, and grows suspicious there on its own. An agent starts from zero every time. Institutional distrust has to be written down and shared.

What we contribute

We are building this review scaffolding for our own operations: separate roles for implementation and review, evaluation paths outside the agent's reach, adversarially briefed review, and escalation to a human instead of a third attempt. What proves itself goes into the governance structure we set up with clients.

If you plan to hand production work to AI agents, acceptance is the question to answer first — not the choice of model.