Spillovers Don't Bias Your Experiment — They Change the Estimand
When units interfere, a randomized difference-in-means doesn't break — it consistently estimates the TOTAL effect, not the direct one, and the gap grows with coupling (to ~96% of the direct effect near criticality). The fix: choose your estimand and a design that targets it. Corrected re-publication.
The claim (corrected). When units interfere — your treatment of one unit spills over to others — a randomized experiment does not break. What changes is the estimand: a naive difference-in-means stops measuring the direct effect and instead consistently measures the total (equilibrium) effect, and the two pull apart as coupling between units grows. On a lattice simulation that gap rises from under 1% at zero coupling to nearly 100% of the direct effect near the stability boundary. The headline lesson is the opposite of "the design doesn't matter": under strong interference, design and estimand choice matter more, not less — you have to decide which effect you want and pick a design that targets it. (An earlier version of this post framed the gap as RCT "bias" and called the curve a "phase diagram." Both were wrong; this is the corrected, honest version — see the note at the end.)
The mechanism — it's the social multiplier, not a failure of randomization. Model the outcomes as linear-in-means: each unit's outcome is its own treatment plus a weighted average of its neighbours' outcomes, y = τD + ρ·W·y + e. Solve it and y = (I − ρW)⁻¹(τD + e). The propagator (I − ρW)⁻¹ is the matrix form of the linear-in-means reduced-form multiplier 1/(1 − ρ) (Manski's 1993 reflection-problem model; the "social multiplier" framing is Glaeser, Sacerdote & Scheinkman 2003), and it diverges as ρ approaches its critical value (the largest eigenvalue's reciprocal). Difference-in-means on this y is a consistent, well-defined estimator — of the total effect (direct treatment + all the spillovers that propagate through the network). It is not "contaminated"; it is answering a different, larger question than the structural coefficient τ = 2.0, which is only the direct effect.
The measurement. A linear-in-means process on a 20×20 lattice (400 units, randomized treatment of half, direct effect τ = 2.0), difference-in-means as coupling ρ approaches criticality:
| ρ / ρ_crit | difference-in-means (direct τ = 2.0) | total − direct gap |
|---|---|---|
| 0.00 | 2.01 | 0.6% |
| 0.50 | 2.14 | 6.9% |
| 0.80 | 2.49 | 24.6% |
| 0.90 | 2.91 | 45.4% |
| 0.95 | 3.15 | 57.4% |
| 0.99 | 3.93 | 96.4% |
Read the last column correctly: it is not estimation error. It is how far the total effect (what difference-in-means consistently targets here) sits above the direct coefficient — a gap that diverges because the social multiplier 1/(1−ρ) diverges, by construction, near the stability boundary. The shape (a smooth, accelerating departure) is the point; the 96.4% peak is just the value at ρ = 0.99 of critical and grows without bound as ρ→ρ_crit.
Why the gap grows near the stability boundary
Below the boundary, units are nearly independent: spillovers are small, so the total effect ≈ the direct effect and difference-in-means ≈ τ. As coupling rises, each unit's outcome increasingly reflects its neighbours': a treated unit lifts its neighbours, who lift their neighbours, and the equilibrium response is amplified by the social multiplier. Difference-in-means faithfully captures that amplified, system-wide total — so it departs further and further from the direct coefficient. Near the critical coupling the multiplier (and the variance) diverge, which is the same algebra behind percolation/epidemic thresholds and critical slowing-down (a useful parallel, not the same mechanism: those are driven by network heterogeneity, this by a tuned scalar coupling on a lattice).
What to do about it
The correct response is the opposite of "stop worrying about design":
- 1. Decide which estimand you want. Direct effect (the unit-level treatment response) and total/overall effect (including spillovers) are different quantities — Hudgens & Halloran (2008) name direct / indirect / total / overall. Neither is "the" effect; choose deliberately.
- 2. Pick a design that targets it. To estimate the total/overall effect, cluster-randomize at a scale larger than the spillover range. To recover the direct effect under interference, use exposure-mapping / ego-cluster designs (Aronow & Samii 2017; Forastiere & Sävje). Under strong interference the design is the lever — it matters more, not less.
- 3. Report the interference regime. A line on how coupled the units are (or how fast effects propagate) tells a reader which effect a number is even measuring. Its absence is the real gap.
Why it matters. A confident "significant" difference-in-means in a highly-coupled system (viral markets, contagious finance, social platforms) may be a perfectly valid estimate of the total effect while a reader assumes it is the direct one. The error isn't in the randomization; it's in the silent mismatch between the estimand you reported and the estimand you implied.
The falsifierThis is one linear-in-means model on one topology (a lattice) — a 1-D curve, not a phase diagram, and we did not actually compare designs (A/B vs DiD vs synthetic control vs cluster-randomization) under matched coupling, so the "design matters" claim is argued, not yet measured here. Open tests: (a) does the gap's shape survive on a scale-free network; (b) does a cluster-randomized estimator of the total effect stay unbiased with finite variance as ρ→ρ_crit, or does critical slowing-down make it unidentifiable? If a cluster design recovers a stable total effect near criticality, that confirms "design matters more"; if nothing is estimable, the claim weakens to "near criticality the effect is barely defined."