Why a more capable AI can be more confidently wrongWhy a more capable AI can be more confidently wrong

Give a reasoner more evidence and it should get both more accurate and more sure. On independent evidence, it does. But real evidence is rarely independent — sources copy each other, datasets overlap, models train on the same web. And on correlated evidence, something unsettling happens: scaling up capability makes a system more confident without making it more right.

We built the smallest model that shows it. A reasoner estimates an unknown quantity from K sources whose errors are partly shared (correlation 0.4 — a realistic echo-chamber level). As we raise K — its "capability," the amount of evidence it can bring to bear — we track two things: accuracy (how close its estimate lands) and calibration (does its stated confidence match reality? — measured as how often its 95% confidence interval actually contains the truth).

capability K | accuracy (error) | its "95%" interval actually covers the truth 2 | 0.84 | 58% 10 | 0.68 | 50% 100 | 0.64 | 18%

Accuracy barely improves — it hits a floor set by the shared error that more correlated evidence can't remove. But calibration collapses: at K=100 the reasoner's "95% sure" interval is right only 18% of the time. More evidence didn't make it more right. It made it more confidently wrong.

The mechanism is simple. A naive reasoner treats 100 correlated sources as 100 independent votes, so its confidence interval shrinks toward zero — while its actual error plateaus. Confidence and correctness come apart, and the gap widens with scale. The fix is equally simple and is the whole point: count the number of effectively independent sources, not the raw count. Do that, and calibration holds steady (≈87% coverage) at any capability — the scissors closes.

This is the uncomfortable shape of the AI-scaling era. Adding parameters, data, and tools to a system trained on a correlated world does not automatically make it know what it doesn't know — and a capable system that is confidently wrong is more dangerous than a weak one that is uncertain. The scarce, valuable resource is not capability. It is calibration: honest uncertainty, bought by maximizing effective-independent evidence and anchoring on the outside world — not by scale.

What would change our mind: if a reasoner's calibration held up (or improved) as capability rose on correlated evidence, the scissors would be an artifact. It does not — the collapse is robust, and the effective-independence correction reliably reverses it. (One honest limit: that correction restores most of the calibration, not all of it — it fixes the count of evidence, and residual overconfidence remains. Calibration is hard; that is rather the point.)