More data, more confidently wrong: a 95% interval (Bayesian or frequentist) loses coverage under a hidden confounder
A 95% interval (Bayesian or frequentist) feels like a guarantee — but under a hidden confounder it measures sampling noise, not model error: coverage of the truth collapsed from 1.4% at n=50 to 0% by n=200, and more data only buys more false confidence.
A 95% Bayesian credible interval feels like a guarantee: "there's a 95% chance the true value lies in here." That reading is only valid when the model is correctly specified. Under the kind of misspecification that pervades real data — most commonly an omitted confounder — the credible interval measures your sampling noise, not your model error, and its actual coverage of the truth can fall far below 95%. Worse, it degrades as you collect more data. This is a well-established result — that a misspecified posterior concentrates on the wrong (KL-pseudo-true) value and its credible sets stop being valid confidence sets is the misspecified Bernstein–von Mises theorem (Kleijn & van der Vaart 2012; Müller 2013), and the omitted-confounder bias driving it is textbook (it does not vanish as n grows). What we add is a runnable receipt for exactly how fast coverage collapses.
One honest clarification (the word "Bayesian" in the title could mislead): this is not a Bayesian-specific quirk — a frequentist confidence interval fails identically here. It is omitted-variable bias, i.e. non-identification: with the confounder dropped, the estimator is consistent for the best-linear-predictor coefficient (≈1.6 in this setup), and — because the Gaussian, homoskedastic structure here makes the model-based and sandwich variances coincide — the same interval does cover that pseudo-true 1.6 at ~95% throughout (measured at every n); what collapses is its coverage of the true causal 1.0. (That free calibration is a Gaussian accident: under generic misspecification even the pseudo-true value needs a sandwich correction to stay valid — Müller 2013.) And the point estimate isn't getting "more wrong" with n; its bias is fixed at +0.60. What degrades is the interval's confidence around the wrong center — more data buys more false confidence, not more error.
Method. We simulated y = x + z + noise with x and z correlated (rho = 0.6), then fit a Bayesian model that omits z and read off the 95% credible interval for x's coefficient (true value 1.0). We measured how often that interval actually contained the truth, across sample sizes, over 3,000 datasets each.
What we found.
| sample size | credible-interval coverage | interval width | bias |
|---|---|---|---|
| 50 | 1.4% | 0.53 | +0.60 |
| 200 | 0.0% | 0.26 | +0.60 |
| 1,000 | 0.0% | 0.12 | +0.60 |
| 20,000 | 0.0% | 0.03 | +0.60 |
Coverage collapses to zero. The reason is structural: the omitted-confounder bias is fixed (~0.60) while the credible interval shrinks like 1/sqrt(n). More data buys more precision around the wrong answer. The posterior becomes more confident and less correct at the same time.
The practical rule. A credible interval's width quantifies sampling uncertainty, not model error — and only the first of those shrinks with n. When misspecification is plausible (and with observational data it usually is), do not read coverage off the posterior. Bound the effect under the structure you might be omitting (sensitivity analysis), or use a design that identifies the effect rather than a model that assumes it away. Calibration on your assumed model is not coverage of reality.
What would change our mind. If a misspecified model's 95% credible interval retained near-nominal coverage as n grew — the bias washing out on its own — the warning would be overstated. It does the opposite: coverage went to zero by n = 200 and stayed there.
(All figures from simulation.)
FAQ
Does a 95% Bayesian credible interval contain the truth 95% of the time? Only if the model is correctly specified. Under a common misspecification — an omitted confounder — the interval measures sampling noise, not error, so its real coverage of the true value collapses: 1.4% at n=50 and 0.0% by n=200 — and a frequentist confidence interval fails identically (it is omitted-variable bias, not a Bayesian quirk).
Why does more data make it worse? Because more data shrinks the interval (width 0.53 → 0.03) around a biased point (+0.60 throughout). The estimate gets more precise about the wrong value, so coverage falls to zero as n grows — “more data, more confidently wrong.”
What setup produced this? We simulated y = x + z + noise with x and z correlated (ρ = 0.6), fit a Bayesian model that omits z, and read the 95% credible interval for x's coefficient over 3,000 datasets per sample size.
What's the takeaway? A tight credible interval is not evidence of correctness — it quantifies sampling noise under your assumptions. Check for omitted confounders and model misspecification before trusting any interval, especially with large n.