Why a more capable AI can be more confidently wrong

Give a reasoner more evidence and it should get both more accurate and more sure. On independent evidence, it does. But real evidence is rarely independent — sources copy each other, datasets overlap, models train on the same web, and frontier models make correlated errors even across different providers. On correlated evidence, pooling more of it makes a system more confident without making it more right.

A note on the word "capability." In the smallest model below, "capability" means the amount of evidence a system can bring to bear — the number of sources K it aggregates — not raw model ability. That distinction is the whole point, and we return to it.

We built the smallest runnable model that shows it — and the mechanism is a classic. A reasoner estimates an unknown quantity from K sources whose errors are partly shared (correlation ρ = 0.4, a realistic echo-chamber level — though, as we'll see, the size of the effect depends heavily on ρ). As we raise K, we track two things: accuracy (how close its estimate lands) and calibration (does its stated confidence match reality? — how often its 95% confidence interval actually contains the truth).

Accuracy barely improves — it hits a floor set by the shared error that more correlated evidence can't remove (the error can't drop below √ρ ≈ 0.63, straight from the algebra of averaging correlated numbers). But calibration collapses: at K=100 the reasoner's "95% sure" interval is right only 18% of the time. More evidence didn't make it more right. It made it more confidently wrong. (That 18% is the worst case for ρ=0.4; at ρ=0.1 the same interval still covers ~43%, at ρ=0.05 ~57% — the collapse is real, but its severity is an assumption about how correlated your world is.)

The mechanism is textbook: the survey design effect (Kish, Survey Sampling, 1965). Correlated observations carry less information than their headcount implies, so the effective number of sources is K / (1 + (K−1)ρ), not K. Naive iid aggregation — treating 100 correlated sources as 100 independent votes, the failure mode when you don't discount overlap, as when many agents share one base model — shrinks the interval toward zero while the actual error plateaus. Xiao-Li Meng's "big-data paradox" (2018) is the same law: a tiny correlation shrank a 2.3-million-person survey to the information of about 400 people. So does the dependent-case Condorcet jury theorem (Ladha 1992) for votes. We didn't discover the effect; we made it concrete for AI aggregation.

The fix is to count effectively-independent sources, not raw ones — but the naive version has a catch. Divide by the design effect and calibration recovers only partway (to ~87% coverage at large K), because the spread you measure within a correlated sample understates the true spread by √(1−ρ): the shared component cancels when the sources look at each other, so the correction still uses too small a yardstick. Supply the true variability from outside — equivalently, an estimate of ρ — and the interval closes fully: 95% coverage at every K. So the residual over-confidence is not a deep limit. It is that you cannot self-calibrate against a shared error from the correlated data alone. You have to know how correlated your sources are — and a pool of agreeing sources is exactly what can't tell you.

This is the uncomfortable shape of the scaling era. Adding parameters, data, and tools to a system trained on a correlated world gives it more evidence — not the knowledge that its evidence isn't independent. You might object that a truly capable reasoner would discount the correlation. Exactly: that discounting is the capability that scale doesn't automatically buy, and worse, it is the kind of calibration that RLHF actively erodes. The scarce, valuable resource is not capability. It is calibration — bought by engineering independence (different base models, disjoint sources, deduplication) and anchoring on the outside world, not by scale.

What would change our mind: if a reasoner's calibration held up (or improved) as it pooled more correlated evidence, the effect would be an artifact. It doesn't — the collapse is robust and the effective-independence correction reliably reverses it. The one honest limit is practical, not fundamental: you can apply the full correction only if you can estimate the cross-source correlation from outside the pool — and in a live system that number is unobservable and content-dependent, rising exactly when every source is echoing the same mistake.

FAQ

Does more evidence make an AI both more accurate and better calibrated? Only when the evidence is independent. With correlated sources — a realistic echo chamber, and frontier models do make correlated errors even across providers (Kim et al. 2026) — accuracy plateaus while calibration breaks: pooling more of it makes the reasoner more confidently wrong.

Why does correlated evidence break calibration? This is the survey design effect (Kish 1965): correlated observations carry less information than their headcount implies, so the effective sample size is K/(1+(K−1)ρ), not K. A reasoner that counts them as independent shrinks its interval toward zero while its actual error plateaus. Meng's big-data paradox is the same law.

Is this a model flaw or a mathematical fact? A mathematical fact, and a textbook one — the design effect, six decades old. Treating correlated sources as independent understates uncertainty regardless of the reasoner; the severity depends on the correlation ρ (at ρ=0.4 the 95% interval covers only 18% of the time by K=100; at ρ=0.1 it is ~43%).

What is the fix, and does it fully work? Discount for source dependence: estimate the inter-source correlation and count effectively-independent evidence. But you cannot read that correlation off the correlated data alone — the sample spread understates the true spread — so the correction only closes fully to 95% when the correlation is supplied from outside. The practical lever is engineering independence (different base models, disjoint sources, deduplication), not computing a number you cannot observe.

Related research

• The most confident systems are the least grounded
• A meter for when an AI is confidently wrong
• The same hidden law in four AI-memory mechanisms