A paper circulating this week from a joint Stanford–MIT group makes an argument that will be uncomfortable for anyone whose roadmap is "bigger model, longer chain of thought". Across 1,100 tasks and twelve agent scaffolds, the strongest predictor of task success was not model size and not reasoning-token count, but whether the scaffold contained an external verification step. The correlation was 0.71. Model size, measured within the same family, correlated at 0.28.
The experiment
The design is simple enough to reproduce. Take one frontier model, hold it fixed, and run it in three scaffolds on the same tasks: no check, self-check (the model reviews its own output before submitting), and external check (a test suite, a compiler, a schema validator, or a retrieval lookup that can contradict the claim).
On software tasks, success went from 46% with no check to 61% with self-check to 79% with external check. On data extraction the pattern was steeper: 52%, 58%, 84%. On tasks with no mechanical verifier available — open-ended writing, summarisation — the gap collapsed to three points, which is the paper's own honest limit.
Why self-checking barely helps
The explanation is correlated error. When a model produces a wrong answer and then reviews it, it applies the same weights and the same priors that produced the mistake. It approves the error for the same reason it made it. The paper measures this directly: on the subset where the first attempt was wrong, self-check caught 19% of errors. External check caught 71%.
That should reframe how you read vendor claims about self-critique and reflection loops. They are not worthless — 19% is not zero — but they are an order of magnitude weaker than a test that can actually fail.
The cost side
External checks are not free. The software tasks ran 34% more tokens and 22% more wall-clock time with a test-suite check, because failing runs get retried. But total cost per successful task fell 18%, because retries replaced human cleanup. That inversion — more tokens, less money — is the part most teams miss when they budget for agents.
The paper also finds a threshold effect. One external check is worth a lot. Two independent checks are worth somewhat more. Three or more produce diminishing returns and, in 9% of runs, contradictory signals that the scaffold handles badly by looping. Design for one good checker, not a panel.
What it means for how you build
Stop tuning prompts to make a model more careful and start writing checkers. The cheapest checker is usually already in your stack: the test suite you have, the JSON schema your API expects, the database constraint, the second service that can confirm a fact. If a task has no mechanical verifier, that is a signal about task selection, not about model quality.
The paper's closing line is blunter than its evidence: before scaling the model, scale the ability to detect failure. For teams shipping agents into production this quarter, that is the more actionable half of the finding.