Food Nudges Aren't 2.5× Better — Food Is the Small-Study Domain

The short answer. A famous 2021 PNAS meta-analysis found food-choice nudges the most responsive behavioral domain — food d ≈ 0.65 versus the least responsive domain, finance, at d ≈ 0.24, a ≈2.7× gap that travelled as "food is ~2.5× more nudgeable." We checked the one thing that ranking hangs on: study size. In the authors' own data, food is by far the smallest-study domain — about 113 participants per effect versus ~861 for finance and ~1,400–16,000 for every other domain. A domain built from tiny studies is exactly where a significance filter inflates effects most, and a runnable model reproduces the whole ≈2.6× gap from zero true difference once you feed it that size asymmetry. **The honest twist (below): this is small-study fragility, not a proven publication-bias artifact** — the real-data bias test found food's bias signal the weakest of any domain.

The claim, stated precisely. Mertens, Herberz, Hahnel & Brosch (2021) pooled ~450 nudge effects and found food the most responsive domain (d = 0.72 as first published; 0.65 after a formal correction), with finance the least (d ≈ 0.24). The headline "food is ~2.5× more responsive" is a derived food-vs-finance ratio (really ~2.7× corrected, ~3.0× as first published) — the paper never states "2.5×," and food is only ~1.5× the pooled mean (d ≈ 0.43). It nonetheless travelled into talks and policy decks as if it were an intrinsic fact about eating behavior.

The catch — and now we can check it. Food-choice nudges are typically small point-of-choice field and cafeteria trials with dozens to low-hundreds of participants. Nudges in other domains (default enrollment, tax letters, organ donation) pull in large administrative and transaction datasets. When a literature is filtered by statistical significance, small studies survive only when their estimate is large — so a domain made of small studies is systematically inflated relative to a domain made of large ones, even if the true effect is identical everywhere. That premise is checkable, and in Mertens's own Table 1 it holds decisively:

(Cohen's d as first published; food's corrected value is 0.65 — see the correction.) Food is the lowest-precision domain by a wide margin — ~7.6× smaller per study than finance, ~13× smaller than health/environment, ~100–140× smaller than prosocial/other. This is exactly the size asymmetry that manufactures a domain ranking out of measurement noise.

The mechanism is textbook

That small studies are inflated more by a significance filter — and that a subgroup difference can therefore be manufactured by differential small-study effects — is not new. It is the small-study effect (Egger et al. 1997), stated as an explicit warning for meta-analysts (Sterne et al. 2011: funnel asymmetry across subgroups "should not be equated with publication bias"), and it is Type-M (magnitude) error (Gelman & Carlin 2014): a low-powered study that clears significance overstates the effect it detects. Our contribution is only a small runnable receipt that the specific ~2.5× domain ratio falls out of exactly the size gap we just verified.

The runnable demonstration

The smallest model that isolates the mechanism: give every domain the same true effect (Cohen's d = 0.20). Make "food" out of small studies (per-group n ≈ 30) and "other" out of large ones (n ≈ 300) — a 10× asymmetry, conservative against the real 7.6–100×. Publish a study only if it reaches p < .05 in the expected direction (the file-drawer filter). Then read the observed food/other ratio.

The control is the load-bearing line: with equal sample sizes the artifact vanishes and the ratio returns to ~1.0, so the inflation is driven by the size asymmetry, not by anything intrinsic to food or a bug in the simulation. It is a dose-response — the ratio grows monotonically as food studies shrink, and the famous ~2.5× appears right around a ~10× gap:

Every number here is re-runnable: research/probes/nudge_pubbias_artifact.py (MIT, zero external data).

The honest complication (the part that keeps us honest)

This is a plausibility demonstration, not proof that food's ranking is publication bias — and the direct real-data test cuts against the simplest version of that story. Maier, Bartoš, Stanley, Shanks, Harris & Wagenmakers (2022, PNAS) re-analyzed Mertens's own corrected data and found the whole pooled nudge effect collapses to d ≈ 0.04 (95% CrI [0.00, 0.14], including zero) after correcting for publication bias — "no evidence remains that nudges are effective." But of all domains, food showed the weakest direct publication-bias evidence (Bayes factor BF ≈ 2.49, only "moderate"), while the others showed strong bias (BF > 10). If food's lead were a pure funnel-asymmetry artifact, food should show the strongest bias signal — it shows the weakest.

So the honest reading is narrower than "it's publication bias": food is the smallest-n, lowest-precision domain, so its effect is the most fragile and Type-M-inflated — and its uniformly tiny studies give the funnel-based bias tests low power, so BF ≈ 2.49 may mean "can't detect bias here," not "no bias here." Small-n is the substrate of both the inflation and the inability to prove it. The ranking is not robust evidence of intrinsic food nudgeability; pinning the cause specifically on demonstrated publication bias is more than the data supports.

What the ranking actually tracks

Ranking domains by raw Cohen's d compares non-comparable outcomes. Food nudges are measured on near-zero-cost, proximal choices (take the apple at the tray line); finance and retirement nudges on high-cost, distal behavior (save for 30 years). The "2.5×" largely reflects the cost and immediacy of the measured behavior, not domain "nudgeability" — and field evidence agrees: across 126 real nudge-unit trials, DellaVigna & Linos (2022, Econometrica) find effects average 1.4pp versus 8.7pp in academic journals, with selective publication and low power explaining most of the gap. Practitioners rank by lever (a default beats an informational nudge in any domain), not by domain.

What this does and doesn't say

• It does not claim nudges have zero effect — that is a separate question (and Maier's real-data answer is "the corrected average is indistinguishable from zero").
• It does show the food-is-2.5×-more-responsive ranking is not robust evidence of an intrinsic property: food is the small-study domain, the ratio reproduces from a verified size asymmetry with no true difference, and the ranking confounds measured-outcome cost.
• It corrects our own earlier framing: we do not claim the ranking is a proven publication-bias artifact — Maier's per-domain test found food's bias signal weakest, so the defensible cause is small-n fragility, not demonstrated bias.

The falsifierWe pre-registered: get the per-domain study sizes; if food is not systematically smaller, the size story fails. We checked Mertens's data — food is by far the smallest (~113 vs ≥861). What would still overturn the honest claim: a within-domain small-study correction (PET-PEESE / RoBMA / selection model) run at adequate power that leaves food's ranking intact after the size gap is accounted for. Maier's food BF ≈ 2.49 is too underpowered to settle it either way — which is itself the point.

FAQ

Do food nudges actually work? This doesn't answer that — it tests the between-domain ranking. Nudges may have a real (if modest) average effect; Maier's bias-corrected estimate is near zero. What we show is that "food is 2.5× more responsive than other domains" isn't good evidence of an intrinsic food advantage.

Is food's ranking publication bias? Not demonstrably. Food is the smallest-n domain, so its effect is the most fragile and inflated by small-study/Type-M error — but the direct bias test (Maier 2022) found food's funnel-asymmetry signal the weakest of any domain (partly because tiny studies give the test low power). So: untrustworthy ranking, but small-n fragility rather than proven bias.

Why does sample size matter so much? Measurement noise scales as ~1/√n. Small studies need a bigger observed effect to clear significance, so the filter selects their largest, most inflated estimates. The smaller the studies in a domain, the bigger this selection inflation.

Does this contradict Mertens et al. (2021)? It challenges one derived quantitative reading — the food-vs-finance ratio as evidence of intrinsic responsiveness — using their own study sizes. It agrees with the real-data re-analysis by Maier et al. (2022) that bias correction sharply shrinks nudge effects.

Is this just a simulation? The ratio-reproduction is — deliberately the smallest one that isolates the mechanism. But the size asymmetry it assumes is now checked against Mertens's real Table 1 (food ~113 vs finance ~861), and the falsifier says exactly what real-data evidence would overturn it.