The verification tax: AI speed becomes trust only where the output is checkable
An AI that answers fast saves nothing if you must re-check it. We measured the residual error after self-verification (qwen3-coder:30b + glm-5.2, n=25–120): on hard reasoning self-checking catches only ~1/3 of errors (residual ~30%), a stronger model is no better and an independent one doesn't rescue it (errors correlate across models) — yet a cheaply-checkable task gets caught ~100%. The tax tracks checkability, not difficulty: the generation–verification (NP) asymmetry, measured in LLMs.
A faster worker is only a more productive worker if you can trust the work without redoing it. For AI that is the whole game: a model that answers in a second saves you nothing if you have to spend a minute checking whether the answer is right. So the number that actually matters is not speed and not first-pass accuracy — it is the residual error after verification: of the answers the AI gets wrong, how many survive a check and reach you still wrong. We measured it, on local models plus a frontier model, with runnable tests.
Part 1 — self-checking catches only about a third of hard errors
The popular hope is that a model can "check its own work." It can — but only on the right kind of task. We had a model answer, then verify its own answer, across tasks from cheaply-checkable to open-ended:
| task | first-pass error | self-verify catch | residual error |
|---|---|---|---|
| arithmetic (output cheaply checkable) | 0.08 | 1.00 | 0.00 |
| MMLU-Pro (hard knowledge, not checkable) | 0.30–0.44 | 0.17–0.36 | ~0.28 |
| multi-hop QA (hard reasoning, not checkable) | 0.42–0.48 | 0.33–0.36 | ~0.30 |
On a checkable task self-verification catches 100% of errors and the residual is zero — fast output becomes trustworthy output. On hard reasoning it catches only about a third (firmed at n=120: mean catch 0.26). After the model has "checked its work," ~30% of its hard-reasoning answers are still wrong — and it believes they are right.
Part 2 — a stronger model is no better at this
Surely a more capable model checks itself better? We ran the same self-verification on a strong frontier model (glm-5.2). It caught no more of its own hard-reasoning errors than the small local model — fewer in our run (~0.19 vs ~0.34), but at n=25 per task the two sit within overlapping confidence intervals, and glm-5.2's higher measured error is partly strict grading of its formatting, so we do not claim it is genuinely worse. What is clear is the thing that didn't happen: capability bought no extra self-knowledge (it flagged none of its own answers wrong across n=25 — a zero count too small to call a property). This is the published result, not a new one: LLMs cannot reliably self-correct reasoning without external feedback (Huang et al., ICLR 2024). The hope that "better models will self-correct into trustworthiness" is not supported on exactly the work where it would matter.
Part 3 — a second, independent model doesn't rescue it either
If a model can't check itself, pay for an independent one? We had the strong frontier model check the small model's answers. The independent checker caught no more errors than the model's own self-check (~0.23 vs ~0.34, again within noise at n=25). The reason is the engine under all of this: LLM errors are correlated across model families — a different model is blind to many of the same hard questions, so it nods along to plausible-looking wrong answers. This is now documented: models agree ~60% of the time when both are wrong, and more accurate models have more correlated errors (Kim et al., ICML 2025). No stack of models — self, stronger-self, or independent-and-stronger — pushed hard-reasoning error-catching clearly above ~1/3.
| checker on hard reasoning | catch rate |
|---|---|
| the model checks itself | ~0.34 |
| a stronger model checks itself | 0.19 (and never doubts itself) |
| an independent stronger model checks it | 0.23 |
Part 4 — the keystone: checkability, not difficulty
So far "hard" and "uncheckable" travelled together — arithmetic was both easy and checkable, reasoning both hard and uncheckable. Which one drives the tax? We built the decisive control: a task that is hard to solve but easy to check — a constraint search ("find a 3-digit number whose digits sum to S, divisible by D, of given parity"). The model has to search to solve it, and it failed first-pass 35% of the time — genuinely hard. But every constraint is mechanically checkable, and self-verification caught 100% of the errors. Residual: zero.
Read honestly, this confirms the framing rather than discovering it. A "mechanically checkable" task is one that has a sound, cheap verifier — which is exactly what NP formalizes (Cook–Levin, 1971): solutions can be hard to produce yet cheap to check. A task with such a verifier has zero residual by construction, so the control is close to definitional. Still, it isolates the variable cleanly: what governs the tax is whether a cheap ground-truth check exists — not how hard the answer was to produce.
The pattern
AI speed becomes trust only where the output is cheaply checkable — and, in our tests, capability doesn't pay the tax down, checkability does.
Wherever an answer can be cheaply verified (arithmetic, code that runs, a constraint that can be re-checked, a fact with a lookup), AI speedups convert into trustworthy output no matter how hard the answer was to produce. Wherever the output cannot be cheaply checked — open-ended reasoning, judgement, synthesis — there is a persistent verification tax of ~30% residual error (n = 25–120, two models) that our stronger model did not reduce and a second model did not remove — consistent with the published finding that intrinsic self-correction doesn't close the reasoning gap (Huang et al. 2024). Two models can't make that a scaling law, so we don't call it irreducible; but on this work it has to be paid by a human, or avoided by restricting the work to checkable form.
What to actually do
- For checkable work, trust the speedup. Pair the model with cheap automated verification (run the code, re-check the constraint, look up the fact) and ship — the residual really is near zero.
- For open-ended hard work, budget the human check as a fixed cost. It was persistent across our tests; don't expect a better model or a second model to remove it. The productivity gain is real but capped by verification, not by the model.
- Don't buy a "second opinion" from another model on shared-blind-spot tasks. It catches fewer errors than the model's own check, because the blind spots are shared.
- Where you can, convert un-checkable work into checkable form — force intermediate steps, demand a re-checkable artifact, ground the answer in a source that can be verified. This is process supervision (Lightman et al., 2024), and it works for a specific reason: an external channel — a code run, a retrieved source, a machine-checkable sub-step — is not drawn from the model's training distribution, so it doesn't share the blind spot a second model does. That is what moves the residual, not a bigger model.
The open question. "Un-checkable" here really means "no cheap ground-truth channel at inference time" — the MMLU-Pro and multi-hop items do have gold answers, which is how we graded them. So the residual may be a property of the verification channel, not of hard reasoning itself: does the ~30% survive once the verifier can act — run code, retrieve a source, decompose into machine-checkable claims — rather than just re-read its own answer? That is the test we'd run next, and it is where the tax is likely paid down.
Caveats and prior art: measured on a local model (qwen3-coder:30b) and one frontier model (glm-5.2); n = 25–60 per task (firmed cells n = 60; constraint search n = 40); strict automated grading. At these n the cross-model catch rates (~0.34 / ~0.19 / ~0.23) sit within overlapping 95% confidence intervals, so the model ranking is not statistically established — we report it but do not lean on "stronger is worse", and the frontier model's higher error is partly format/grading strictness. "Catch" means the model flags its own answer as wrong; "residual = first-pass-error × (1 − catch)" is an identity, not an independently-measured constant. This is a measured LLM instance of known results, not a new law: the generation–verification asymmetry is the definition of NP (Cook–Levin 1971); intrinsic self-correction failing without external feedback is Huang et al. (ICLR 2024); correlated errors across families is Kim et al. (ICML 2025); "verification tax" is the economists' costly state verification (Townsend 1979). Every number comes from a runnable experiment — the probe (protocol, result files, confidence intervals) is at research/probes/verification_tax.
FAQ
What is the verification tax in AI? A faster AI worker only helps if you can trust its output without redoing it. When you must re-verify the work, the speed gain is taxed away — and for hard, non-checkable tasks the tax is steep.
Can an AI reliably verify its own answers? Only when the output is cheaply checkable. On arithmetic, self-verification catches 1.00 of errors (residual 0.00). On hard knowledge (MMLU-Pro) it catches only 0.17–0.36 (residual ~0.28); on multi-hop QA ~0.33–0.36 (residual ~0.3).
Does a stronger model verify better? No better. No stack — self, a stronger self, or an independent-and-stronger model — pushed hard-reasoning error-catching clearly above ~1/3. Our single frontier run (glm-5.2) scored lower, but at n=25 the model ranking is within noise and its error is partly strict grading, so we don't claim "stronger is worse". LLM errors are correlated across families (Kim et al. 2025), so an independent checker shares the same blind spots.
What is the implication for using AI? AI speed pays off where verification is cheap. For hard reasoning, plan for a persistent verification cost rather than assuming self-checking closes the gap.