A 95% confidence interval that covers 31% of the time: difference-in-differences with one treated unit95% interval spolahlivosti, ktory pokryva len 31 % pripadov: difference-in-differences s jednou ostatkovanou jednotkou
The claim. When you run difference-in-differences (DiD) with a single treated unit and errors that are correlated over time, the "95%" confidence interval it reports is badly overconfident. In a cleanReplikacia: s jednou ostatkovanou jednotkou a korelovanymi chybami pokryva 95% CI metody DiD len ~31 %; synthetic control obnovi ~89 %, ale za cenu ~4x sirsich intervalov.
The claim. When you run difference-in-differences (DiD) with a single treated unit and errors that are correlated over time, the "95%" confidence interval it reports is badly overconfident. In a clean replication, that nominal 95% interval contained the true effect only 31% of the time. The point estimate is rarely the main problem — the inference is.
What we measured. We replicated the Alvarez & Ferman (2020) design: 30 units, 12 time periods (8 pre-treatment), exactly one treated unit, a true treatment effect of zero, AR(1)-correlated errors (rho = 0.7), over 800 simulated experiments. For each, one question: did the method's 95% confidence interval actually contain the true (zero) effect?
| method | 95% CI coverage (nominal 0.95) | mean abs(bias) | RMSE | mean CI width |
|---|---|---|---|---|
| Difference-in-differences | 0.305 | 0.95 | 1.27 | 0.90 |
| Synthetic control | 0.891 | 0.78 | 1.02 | 3.49 |
DiD's intervals are narrow (width 0.90) — which is exactly why they fail: they are confidently wrong. Synthetic control nearly restored nominal coverage (0.89), but at the cost of intervals about 4x wider.
Why it happens. With one treated unit there is effectively a single cluster of correlated residuals, so the usual standard errors have almost nothing to average over and badly understate the true uncertainty. The estimate can be roughly fine while the error bars are fiction.
The falsifierGive DiD many treated units (so the cluster-robust variance has enough independent clusters), or truly independent errors, and coverage should climb back toward 95%. If it does not, this explanation is wrong. And synthetic control's fix is not free: its intervals here were ~4x wider, so if that width is uninformative for your decision, "just use SC" is not automatically the answer.
The practical takeaway. If a DiD result rests on one treated unit — one state, one market, one product — with serially correlated outcomes, treat its p-value and confidence interval with deep suspicion. The headline estimate may be the trustworthy part and the significance stars the fiction.
Method: simulation (Alvarez-Ferman 2020 replication), 800 reps, true effect = 0; reproducible in our lab ledger.
Tvrdenie. Ak robis difference-in-differences (DiD) s jednou ostatkovanou jednotkou a chybami korelovanymi v case, "95%" interval spolahlivosti, ktory ti vyjde, je vyrazne prehnane sebavedomy. V cistej replikacii ten nominalne 95% interval obsahoval skutocny efekt iba v 31 % pripadov. Problemom nie je bodovy odhad — problemom je inferencia.
Co sme zmerali. Replikovali sme dizajn Alvarez & Ferman (2020): 30 jednotiek, 12 obdobi (8 pred zasahom), presne jedna ostatkovana jednotka, skutocny efekt nula, AR(1)-korelovane chyby (rho = 0,7), cez 800 simulovanych experimentov. Pri kazdom jedna otazka: obsahoval 95% interval danej metody skutocny (nulovy) efekt?
| metoda | pokrytie 95% CI (nominal 0,95) | priemer abs(bias) | RMSE | priemerna sirka CI |
|---|---|---|---|---|
| Difference-in-differences | 0,305 | 0,95 | 1,27 | 0,90 |
| Synthetic control | 0,891 | 0,78 | 1,02 | 3,49 |
Intervaly DiD su uzke (sirka 0,90) — a presne preto zlyhavaju: su sebavedomo nespravne. Synthetic control takmer obnovil nominalne pokrytie (0,89), ale za cenu zhruba 4x sirsich intervalov.
Preco sa to deje. Pri jednej ostatkovanej jednotke je v podstate jediny zhluk korelovanych rezidui, takze bezne standardne chyby nemaju z coho priemerovat a hrubo podhodnotia skutocnu neistotu. Odhad moze byt zhruba v poriadku, no chybove usecky su fikcia.
Falsifier — co by nas vyvratilo. Daj DiD vela ostatkovanych jednotiek (aby mala cluster-robustna variancia dost nezavislych zhlukov), alebo naozaj nezavisle chyby, a pokrytie by malo stupnut spat k 95 %. Ak nie, toto vysvetlenie je nespravne. A oprava cez synthetic control nie je zadarmo: jej intervaly tu boli ~4x sirsie.
Prakticky zaver. Ak DiD vysledok stoji na jednej ostatkovanej jednotke — jeden stat, jeden trh, jeden produkt — s casovo korelovanymi vystupmi, ber jeho p-hodnotu a interval spolahlivosti s velkou nedoverou. Bodovy odhad moze byt ta doveryhodna cast a hviezdicky vyznamnosti fikcia.
Metoda: simulacia (replikacia Alvarez-Ferman 2020), 800 opakovani, skutocny efekt = 0; reprodukovatelne v nasom lab ledgeri.