The most confident systems are the least grounded

Three failures look unrelated. An AI model trained on its own output degrades into nonsense ("model collapse"). Seventy expert teams handed the same brain-imaging dataset reach different conclusions; twenty-nine teams analyzing the same soccer dataset report effects ranging from "no effect" to "strong effect." Markets and technical standards lock onto an inferior option and stay there. We found a shared pattern across these — and built the smallest models that could test it.

The pattern

A system builds confidence from internal consistency — agreement accumulated over time (consensus), or a narrow interval bought with more data (precision). That confidence tracks the truth only in proportion to how much external information the system is coupled to. Call that coupling g. As g falls, confidence and accuracy can decouple — sometimes the system grows more certain via its own internal dynamics while staying wrong; sometimes its confidence simply stops moving at all while its accuracy collapses underneath it. Either way: high confidence is not evidence of grounding. This is not a new mechanism from scratch — it bridges several established results (credited below) that, to our knowledge, hadn't been put on one page together.

What we measured

We built the smallest models that could show this and ran them.

• One shared model, two famous failures. A single opinion-formation model — K options, one truly best, an external-truth-signal fraction φ mixed against self-reinforcement strength α — reproduces both a self-feeding learner losing the true signal below a critical external-information threshold, and quality-blind winner-take-all lock-in (the model becomes indifferent to which option is actually better as self-reinforcement α grows) — the mechanism is exactly Brian Arthur's increasing-returns lock-in (1989), run at low external signal. Both sit on the same rising critical curve φ_c(α): the more aggressively the system reinforces its own signal, the more external grounding it needs to still track truth (φ_c rose from ~0.01 at α=1 to ~0.27 at α=3 in our runs).
• Curing is dearer than preventing. The same model is bistable. Once it locks onto a wrong consensus, escaping costs far more external grounding than preventing the lock would have (in our runs, recovering from a false lock at α=2 needed φ≈0.28, while merely maintaining a correct one needed φ≈0). This hysteresis is the same asymmetry Scheffer's critical-transitions work documents in ecological and social regime shifts.
• A second, complementary illustration — a different mechanism, not the same equations. We ran a second, structurally unrelated toy model: an omitted-confounder regression where g is how well a control proxies the true confound. It shows a starker version of the same THEME (confidence uninformative about grounding) via a different SIGNATURE: precision-as-confidence stayed essentially flat regardless of g at every sample size we tried (N=50 to N=20,000) — data volume, not identification, sets it — while accuracy collapsed sharply as g fell. So this mechanism doesn't show confidence rising as grounding falls, the way the opinion-formation model does; it shows confidence staying put while only accuracy moves — arguably a more dangerous signature, since nothing about the confidence score itself would tip you off. This is the textbook identification-vs-precision point in econometrics (more data narrows a confidence interval without fixing bias from an under-identified estimate).

It matches real data

The "many-analysts" studies are a direct test: give many expert teams the same dataset and the same question, and watch how much the answer moves. It moves a lot. Silberzahn et al. (2018): 29 teams, odds ratios from 0.89 to 2.93 on identical data. Breznau et al. (2022): 73 teams; identifiable coded methodological decisions explained only 2.6% of the variance in numerical results (95.2% remained unexplained even adding researcher characteristics) — a purely theoretical simulated ceiling put the absolute most any coding scheme could explain at just over 16%, and real analysts fell well short of even that. Most of the spread has no identifiable cause at all, under any coding tried — a starker finding than "structural, not sampling noise," and one we understated in an earlier version of this post. Botvinik-Nezer et al. (2020): 70 neuroimaging teams, no two analysis pipelines identical. A newer crowd-reanalysis (Aczel et al., Nature 2026, 100 studies) found only 34% of results replicated within a tight tolerance. This is consistent with the pattern's prediction: when a question is under-identified, the answer is set by which defensible specification you pick, not by how much data you have. A narrow confidence interval is no evidence that you are right. This is a qualitative match, not a fitted point estimate — and an honest limit worth stating plainly: these studies measure cross-team specification variance, not a system's own self-reported confidence the way our toy models define it. We're using them as a real-world parallel to the precision-≠-truth mechanism, not as a direct measurement of either toy model's specific confidence dynamics.

The one practical rule

Across all of these — AI training, scientific analysis, markets, and any system that learns from itself — the rule is close to identical: do not read internal consistency as evidence of truth. Consensus among the parts of a system, and a narrow interval from abundant data, are both cheap and internal. Truth-tracking requires an external anchor, and you have to keep paying for it. The single most dangerous regime is high confidence with low grounding — maximal certainty exactly where it is least earned, and (per the structural illustration) that confidence may never visibly move to warn you. Practically: keep an external-information stream above a floor; the more aggressively a system reinforces its own outputs, the larger that floor must be; and read cross-specification stability, never interval width or consensus alone, as your evidence of being identified. One more caveat worth stating plainly: a grounding score you start acting on becomes a target — a system (or a person) that knows it's being scored on "external grounding" can learn to look grounded (cite sources selectively, pass a robustness checklist) without becoming more accurate. Measuring grounding doesn't retire Goodhart's law; it just moves the goalpost one level up. One detectable early-warning signal, suggested but not tested by this post: watch the co-movement of confidence and diversity, not confidence alone — if an ensemble's or a team's stated confidence is rising while the spread of independent answers it's drawn from is collapsing, that pairing is visible from logs/outputs alone, without ever needing ground truth, and arrives earlier than a visible drop in accuracy.

What would change our mind

A self-referential system that stays well-calibrated while starved of external information — its confidence-accuracy gap staying near zero as grounding falls — would break the pattern. So would a large many-analysts study in which between-team disagreement is no larger than ordinary sampling error. And if an adversarial system can drive our grounding measure to look high while accuracy stays low, that would show the practical rule doesn't survive contact with an incentive to fake it — a test we haven't run yet.

Prior art we're building on

None of the individual pieces here are new; the cross-domain bridge is what we're contributing. Winner-take-all lock-in from self-reinforcement is Brian Arthur's increasing-returns / path-dependence program (1989). Identification vs. precision — more data narrows an interval without fixing bias from an under-identified estimate — is textbook econometrics, formalized furthest in Charles Manski's partial-identification work. Confidence staying high (or rising) while accuracy falls under distribution shift is the mainstream ML calibration/overconfidence literature (Guo et al. 2017 showed modern neural networks are systematically miscalibrated toward overconfidence). Hysteresis in escaping a locked-in regime is Marten Scheffer's critical-transitions program. Self-reinforcing belief lock-in from insufficient external coupling is studied directly in opinion-dynamics models (Deffuant, Hegselmann–Krause) and echo-chamber/epistemic-bubble theory. The general shape of "internal coherence substituting for external checking" is also independently named in older literatures we hadn't credited: Kuhn's paradigm entrenchment (The Structure of Scientific Revolutions, 1962 — confidence in a paradigm persists while anomalies accumulate, until crisis forces revision), Minsky's Financial Instability Hypothesis (1975/1986 — stability itself breeds the leverage and overconfidence that cause the next crash, largely ignored until the 2008 "Minsky moment"), and Janis's groupthink (1972 — cohesive, insulated groups manufacture consensus-confidence while losing contact with disconfirming evidence). None of these three formalize the mechanism the way Arthur, Manski, or Scheffer do; they're cited here as convergent naming, not shared mathematics — three fields independently noticed the same shape of failure without a common formal model, which is itself informative about how real the pattern is versus how rigorously anyone has pinned it down. What we add: a runnable model showing model-collapse-style decay and winner-take-all lock-in emerge from one shared critical curve φ_c(α), the quantified hysteresis gap on that same model, and a second illustration making the identification-vs-precision point concrete with a specific, reproducible signature.

Honest caveats

The thresholds come from minimal simulations, so the exact numbers are model-specific, not universal constants. The two mechanisms above are unified by a shared THEME (an external-grounding fraction that confidence should but doesn't reliably track), not by shared mathematics — the opinion-formation model (collapse + lock-in + hysteresis) is genuinely one system; the identification/precision illustration is a separate, structurally different model that shows the same theme with a different signature (flat confidence, not rising). The real-data comparison is a direction-and-order-of-magnitude match — the multi-analyst studies confirm that specification dispersion dwarfs sampling error, which is the pattern's core — not a fitted point estimate. What we stand behind is the structure: confidence can decouple from truth as external grounding falls, via more than one mechanism, across domains that don't otherwise resemble each other. We call this a pattern, not a law: the evidentiary bar for "law" is quantitative, repeated, independent testing across systems, which this doesn't clear yet.

FAQ

Is there a law linking confidence and grounding? We'd call it a candidate pattern, not a law yet: across model collapse, winner-take-all lock-in, and many-analyst disagreement, confident systems tend to be poorly grounded — but confidence decouples from accuracy via at least two different signatures (rising confidence, or flat confidence with collapsing accuracy), not one universal curve.

What real-data evidence supports it? Many-analyst studies: 73 teams where identifiable coded methodological decisions explained only 2.6% of the variance in numerical results (a theoretical ceiling was ~16%, real analysts fell short of even that); 70 neuroimaging teams with no two identical pipelines; 29 teams reporting odds ratios from 0.89 to 2.93; a newer 100-study crowd-reanalysis found 34% replicated within a tight tolerance. This is a qualitative, direction-and-order-of-magnitude match against cross-team specification variance, not a direct measurement of confidence and not a fitted point estimate.

What is the practical rule? Treat high confidence with low grounding as a red flag, not reassurance. Measure grounding separately instead of reading a confidence score as if it were correctness — and remember a grounding score you act on can itself be gamed.

What would change your mind? A domain where confidence reliably rises with grounding rather than against it, or a large many-analysts study with disagreement no bigger than sampling noise — or a system that learns to fake a high grounding score while staying inaccurate.

Is this a new discovery? No — increasing-returns lock-in (Arthur 1989), identification-vs-precision (Manski), calibration/overconfidence under distribution shift, and hysteresis in regime shifts (Scheffer) are each established separately. What's new here is a runnable model showing two of these mechanisms (collapse, lock-in) share one critical curve and a quantified hysteresis gap, plus a second model making the identification/precision point concrete with a specific, reproducible signature.

Related research

• A meter for when an AI is confidently wrong
• Why a more capable AI can be more confidently wrong
• We hunted for the tipping point in 8 systems — only one is real