Robustness checks aren't ritual - but they filter only if the tests are independent
You can't prove a causal effect is real - you only watch it survive severe tests. Surviving several independent ones should sharply cut the chance it is a false positive; this is textbook (Ioannidis's positive predictive value). The catch: real robustness checks share the same data and identifying assumption, so they are not independent - and a shared bias can make five checks worse than one honest test.
In causal inference you can never prove an effect is real. You can only subject it to severe tests - placebo-in-time, placebo-in-space, leave-one-out, pre-trend checks - and trust an estimate a little more each time it survives one. This is Karl Popper's corroboration and Deborah Mayo's severe testing: a claim earns credibility not by proof but by surviving honest attempts to kill it. The question practitioners rarely quantify: how much does surviving one more test actually buy you?
The arithmetic is textbook. Take a population of candidate causal claims in which 70% are spurious - a deliberately harsh base rate, though not an outlandish one: replication audits reproduce only about 36% of psychology results and 61% of economics results. Put each claim through five independent falsification tests, where a genuinely real effect survives any one test about 85% of the time and a spurious one about 45%. Each test is then a likelihood ratio of 0.85/0.45 ≈ 1.9 for the claim, and five of them chain by Bayes' rule. Among the claims that survive exactly k of the five, the share still false - the posterior false-positive risk, one minus the positive predictive value - falls like this:
| Independent tests survived (of 5) | Posterior false-positive risk |
|---|---|
| 0 | ~100% |
| 1 | ~99.6% |
| 2 | ~97% |
| 3 | ~82% |
| 4 | ~40% |
| 5 | ~9% |
Surviving all five drops the false-positive risk from 70% to under 10%. This is simply John Ioannidis's “why most published findings are false” calculation run forwards: each independent confirmation multiplies the odds. So the craft habit of running robustness checks is not, in principle, ritual - it can earn trust. (The runnable model reproduces every number here.)
But there is a load-bearing word, and it carries the whole result: independent. Five variants of the same placebo check share a blind spot - if a hidden bias lets a spurious effect past one, it slips past all five together, and the filter collapses toward the power of a single test: a false-positive risk of about 55%, only modestly better than the 70% you started with. This is the survey design effect in another costume - correlated tests count as fewer independent tests (the “effective number of tests,” Cheverud 2001; Nyholt 2004), exactly as correlated observations shrink an effective sample size.
How fast does it collapse? We modeled it two ways. First, let the five outcomes share a common component with correlation ρ (a Gaussian copula):
| Test correlation ρ | False-positive risk after surviving all 5 |
|---|---|
| 0 | ~9% |
| 0.2 | ~20% |
| 0.4 | ~30% |
| 0.6 | ~38% |
| 0.8 | ~45% |
| 0.95 | ~51% |
And that understates the damage, because a Gaussian copula has no tail dependence - it never lets tests fail in perfect lockstep. Real robustness checks share one identifying assumption, so a cleaner model is a deterministic shared confound: a fraction φ of spurious claims are fooled by it and pass every test together. That puts an irreducible floor under the false-positive risk, and the floor climbs fast - φ = 0.05 gives ~26%, and φ = 0.3 gives ~62%, worse than a single honest test. A robustness appendix built on one shared assumption does not multiply your evidence; it photocopies it.
And this is still the optimistic case, because it assumes you see every test that was run. In practice the author both picks the checks and knows the answer, so failing specifications quietly vanish from the appendix. Under that censoring, “it survived all our robustness checks” can be nearly uninformative no matter how independent the checks are - which is why placebo tests are reported as supporting the design about 99% of the time (Dreber, Johannesson & Yang, 2024).
What this is, and isn't
This is a deliberately minimal model, not field data, and the mechanism is textbook: Bayesian positive predictive value under (in)dependent tests, the same design-effect logic behind our calibration piece. We did not discover it. The honest, published form of “many severe tests, jointly and transparently” already exists - specification-curve analysis (Simonsohn, Simmons & Nelson 2020) and multiverse analysis (Steegen et al. 2016), which report the whole distribution of defensible specifications instead of a hand-picked table. What is worth keeping is the shape of the trade-off: credibility comes from the diversity of your tests, not their count - five correlated robustness checks are worth about one. It is the lesson William Whewell called consilience in 1840: evidence is strong when it converges from genuinely independent directions.
What would change our mind - and what we will measure next: add a reporting fraction f (the share of run checks actually shown) and map the false-positive risk over both correlation ρ and f. The open question is whether pre-registering the set of checks, rather than diversifying them, is what actually restores the filter.
(The two tables' numbers come from assumed inputs and a toy model, not field data - they illustrate the logic of corroboration; the replication rates (36%, 61%) and the placebo-reporting rate (99%) are real and cited. “False-positive risk” here is the Bayesian posterior P(false | survived) = 1 − positive predictive value, not the Benjamini-Hochberg multiple-testing FDR.)
FAQ
Do robustness checks actually filter out false findings? Only if the tests are independent. In the idealized case - five independent severe tests - the posterior false-positive risk falls from a 70% base rate to under 10%. But real robustness checks share data and identifying assumptions, so the realistic filter is far weaker, and a shared bias can push it past the ~55% you would get from a single honest test.
How much does each severe test help? An independent test with ~85% real-survival and ~45% spurious-survival is a likelihood ratio of about 1.9; chaining five gives the sharp fall above. Dependent tests add almost nothing beyond the first.
Why must the tests be independent? Correlated checks re-test the same weakness - placebo-in-time and leave-one-out on the same regression fail together if the identifying assumption is wrong. This is the design effect: correlated tests count as fewer effective independent tests (Cheverud 2001; Nyholt 2004).
Is “false-positive risk” the same as the false discovery rate? Not exactly. Here it is the Bayesian posterior P(false | survived) = 1 − positive predictive value (Ioannidis 2005), not the Benjamini-Hochberg multiple-testing FDR. We use “false-positive risk” (Colquhoun 2017) to keep them distinct.
What does this mean for practice? Report which independent tests a claim survived; prefer a different dataset or design (a genuine replication) over another variant of the same regression; and pre-register the check set so failed tests cannot vanish. Or use specification-curve / multiverse analysis, which do this transparently.