A 95% CI that covers 31% of the time: diff-in-diff, 1 treated unit
Replication: with one treated unit and serially correlated errors, difference-in-differences' nominal 95% confidence interval covered the true effect only 31% of the time — synthetic control restored ~89% coverage, at about 4x wider intervals.
The claim. When you run difference-in-differences (DiD) with a single treated unit and errors that are correlated over time, the "95%" confidence interval it reports is badly overconfident. In a clean replication, that nominal 95% interval contained the true effect only 31% of the time. The point estimate is rarely the main problem — the inference is.
What we measured. We ran the classic single-treated-unit, serially-correlated design (Bertrand, Duflo & Mullainathan 2004; Conley & Taber 2011): 30 units, 12 time periods (8 pre-treatment), exactly one treated unit, a true treatment effect of zero, AR(1)-correlated errors (rho = 0.7), over 800 simulated experiments. For each, one question: did the method's 95% confidence interval actually contain the true (zero) effect?
| method | 95% CI coverage (nominal 0.95) | mean abs error | RMSE | mean CI width |
|---|---|---|---|---|
| Difference-in-differences | 0.305 | 0.95 | 1.27 | 0.90 |
| Synthetic control | 0.891 | 0.78 | 1.02 | 3.49 |
DiD's intervals are narrow (width 0.90) — which is exactly why they fail: they are confidently wrong. Synthetic control nearly restored nominal coverage (0.89, still a touch under 0.95), but at the cost of intervals about 4x wider. Two things changed at once in that row, though — the estimator and the inference: synthetic control buys its coverage mainly from its placebo/permutation inference (Abadie, Diamond & Hainmueller 2010), not from the SC estimator as such.
Why it happens. With one treated unit there is effectively a single cluster of correlated residuals, so the usual standard errors have almost nothing to average over and badly understate the true uncertainty. The estimate can be roughly fine while the error bars are fiction. This is a well-established result — the serial-correlation under-coverage of DiD was shown by Bertrand, Duflo & Mullainathan (2004, QJE), and inference with few treated units by Conley & Taber (2011, REStat). The single-treated-unit, serially-correlated design we run is theirs; Alvarez & Ferman (2020, working paper) study the related few-treated-units / spatial-correlation case. The honest fix is not "abandon DiD" — it is to use valid inference (Conley–Taber, or randomization/permutation) on the DiD estimate; synthetic control is one route that also changes the estimand and the intervals' width.
The falsifierGive DiD many treated units (so the cluster-robust variance has enough independent clusters), or truly independent errors, and coverage should climb back toward 95%. If it does not, this explanation is wrong. And synthetic control's fix is not free: its intervals here were ~4x wider, so if that width is uninformative for your decision, "just use SC" is not automatically the answer.
The practical takeaway. If a DiD result rests on one treated unit — one state, one market, one product — with serially correlated outcomes, treat its p-value and confidence interval with deep suspicion. The headline estimate may be the more trustworthy part — it is roughly centered, even if high-variance — while the significance stars are the fiction.
Method: simulation (single-treated-unit DiD with serial correlation, in the spirit of Bertrand-Duflo-Mullainathan / Conley-Taber), 800 reps, true effect = 0; reproducible in our lab ledger.
FAQ
Can a 95% confidence interval really cover the truth only 31% of the time? Yes. With difference-in-differences on a single treated unit and serially-correlated errors, the nominal 95% interval covered the truth just 0.305 of the time in our replication — it is badly over-confident.
What design exposes this? The classic single-treated-unit, serially-correlated DiD design (Bertrand-Duflo-Mullainathan 2004; Conley-Taber 2011): 30 units, 12 periods (8 pre-treatment), exactly one treated unit, true effect zero. With one treated unit the standard errors have too few independent clusters, so the interval is far too narrow.
Does synthetic control fix it? Partly — and not uniquely. Synthetic control nearly restored coverage (0.891) at about 4× wider intervals, but it buys that mostly from its placebo/permutation inference (Abadie, Diamond & Hainmueller 2010), not from the estimator. Valid inference on the DiD estimate itself — Conley–Taber, or randomization/permutation — also restores coverage. The real fix is the inference, not abandoning DiD.
When is DiD coverage trustworthy? When you have many treated units (enough independent clusters for the cluster-robust variance) or genuinely independent errors. With one treated unit and correlated errors, report a method with correct coverage instead.