The Crucible

Claims, rebuilt in code and tested.

A public ledger of scientific and technical claims rebuilt as minimal computational models and run. Three honest verdicts — reproduced when the mechanism holds in its smallest model, failed when it doesn’t (science’s rarest export, published here), and not computable when there’s no simulable core. Every verdict ships a measured number and the code that produced it.

20
Reproduced
10
Failed
4
Honest passes
Open data — the full ledger as JSON · now a citable dataset on Hugging Face →
datasets.load_dataset("Danchi17/folklore-index")
Failed replication Cognitive science · Gilovich, Vallone & Tversky · 1985

The hot hand is not a fallacy

The canon

For 30 years the canonical finding was that streak shooting is a cognitive illusion: players who feel “hot” are misreading randomness.

What we rebuilt

We rebuilt their exact estimator — P(hit | 3 prior hits) − P(hit | 3 prior misses), averaged per shooting record — and ran it on i.i.d. shooters with provably no hot hand.

The verdict

On a no-hot-hand shooter the estimator does not read 0; it reads −7.9pp (t=−28 at n=100, growing to −17pp at streak length 4). This is the streak-selection bias of Miller & Sanjurjo (2018): selecting shots that follow a streak inside a finite sequence is a biased sample. A GVT measurement of ~0 therefore implies a real hot hand of roughly +8.5pp. The famous fallacy was the analysis, not the players.

The deep dive, with charts → lab ebce40 · 2026-06-12

The ledger

30 claims rebuilt & tested · newest first
FAILED

A big-enough context window replaces retrieval (CAG / "RAG is dead"): preload the whole co

Judge-free probe on one synthetic structured log at 5k/25k/60k/110k tokens, two task families. NEEDLE (single-fact lookup) = 1.00 at every length incl. 110k; SYNTH (read-everything count/filter aggregation) collapses 0.75->~0.25-0.375 by ~25k and stays there. Verdict FAILED for CAG *on synthesis*; needle lookup survives long context fine, so blanket "context rot kills recall" is also too strong. It is the task type, not the length. Replicates lost-in-the-middle (Liu 2023/TACL 2024), RULER (Hsieh/NVIDIA 2024), NoLiMa (2502.05167), Chroma context-rot (2025, non-peer-reviewed). HONEST LIMITS: n=8 synth Qs/length (the 0.75->0.3 drop is Fisher p~0.13, credible mainly by matching the replicated literature); one frontier model (exact model not logged); even at 5k synth is 0.75 so part of the failure is exhaustive arithmetic the model cannot do at any length -> the real fix for count/filter is a TOOL, not retrieval. Probe: mnemo/probes/ragdead (generator + independent gold-re-derivation scorer). Post: public/posts/does-long-context-kill-rag.html.

CAG / "long context kills RAG" viralrunnable model →
FAILED AI / Future of Work

AI coding assistants make developers ~55% faster

The universal claim fails. A storm-research audit (8 verified sources) shows the famous 55.8% is a vendor preprint on ONE greenfield task; the only independent RCT on experienced devs in mature repos found -19% (time), while a peer-reviewed 3-RCT study found +26% (task-count, juniors). A minimal write-vs-review model reproduces the junior-gain / expert-loss SIGN-FLIP (crossover k*=0.62), robust in 79% of param sets -- so the split is one operating-point curve, not a contradiction.

Peng/GitHub 2023 vs METR 2025 vs Cui/Demirer 2025
REPRODUCED AI evaluation

Chatbot Arena Elo ranks LLMs by genuine quality (cite Arena rank to pick a model)

A classifier that sees ONLY answer style (length, markdown headers, bold, lists) and NO model identity predicts the human winner 61.5% of the time, and its per-model ranking correlates rho=0.74 with the real win-rate ranking across 48 models — it reproduces the leaderboard ORDER without knowing which model wrote anything.

Chiang et al. - 2024 (LMArena)runnable model →
FAILED Management / Future of Work

Founder-led firms returned 3.1x more — the founders mentality drives performance

With IDENTICAL expected returns, a founder-led cohort that is merely ~1.8x as volatile, run through the same survive-and-be-large index filter, shows an apparent 2.60x mean return gap — 76% of Bains 3.1x — from ZERO skill. The advantage is tail-driven: the MEDIAN gap is only 1.58x.

Zook & Allen (Bain) - 2016runnable model →
REPRODUCED AI evaluation

LLM-as-judge agrees with humans ~80%, so it is a valid quality judge

A zero-understanding null judge that simply picks the LONGER answer already agrees with human preferences 68.1% of the time (vs 50% chance), reproducing ~half of the GPT-4 judges celebrated above-chance agreement; the GPT-4 judge itself agrees with pick-the-longer 73.5% of the time.

Zheng et al. - 2023 (MT-Bench)runnable model →
FAILED Management / Future of Work

Good to Great: 11 companies, discoverable shared traits = sustained greatness

A zero-skill null reproduces the whole pattern: select firms for a 15-year 3.8x-the-market leap from a population where NO firm is more skilled than any other, and their next 15 years revert to the market (forward excess +0.015, 95% CI spanning 0; ~47% beat the market = a coin flip).

Jim Collins, Good to Great - 2001runnable model →
FAILED Behavioral economics

Food nudges are 2.5x more responsive than other domains

From an IDENTICAL true effect in every domain, differential publication bias alone reproduces a 2.63x food/other ratio (control with equal sample sizes = 1.00). The famous 2.5x is a small-study mirage.

Mertens, Herberz, Hahnel & Brosch · 2021runnable model →
REPRODUCED

Cosine similarity BARELY distinguishes a superseded fact from its replacement (AUROC ~0.61

Replicated on local nomic (24 subject-relation-object facts, mean-centered): the cosine 'is-this-a-supersession' classifier scores AUROC 0.613 (chance 0.5; Yadav ~0.59) -- a contradiction is often MORE embedding-similar to the original than a genuine rephrase (10/24 cases). A pure-cosine top-1 store served the stale value 41.7% of the time; a deterministic (subject,relation,object) supersession key drove that to 0% (shipped in mnemo v0.2.0). Falsifier: if the key did not cut stale below cosine it would be useless -- it did. Honest scope: synthetic, 24 facts, characterizes the mechanism, not a product benchmark. SCOPED (Crucible audit 2026-07-05): AUROC 0.613 is barely ABOVE chance (0.5), not zero -- the honest statement is 'near-useless in practice', carried by the harm number (pure-cosine top-1 served the STALE value 41.7% of the time), not by 'cannot'. n=24 = small-n replication consistent with Yadav 0.59 (arXiv:2606.26511), not a large-n certification.

Temporal Validity in Retrieval Memory: Eliminarunnable model →
REPRODUCED

Episodic memory (Model-Free Episodic Control: store the MAX observed discounted return per

Smallest model of the mechanism; exploration HELD IDENTICAL (both learners fed the same uniform-random trajectories off-policy, so the only difference is credit assignment). Exp A (deterministic chain + step cost): EC reaches a goal-reachin SCOPED (Crucible audit 2026-07-05): the max-return store is unbiased only in a NOISELESS deterministic env (max=mean); EC beating parametric learning on a tiny deterministic chain is close to textbook (tabular memorization beats function approximation when no generalization is needed). Do not generalize to broad sample-efficiency.

Sarrico, Arulkumaran, Agostinelli et al. - Sam
FAILED

Real-world networks are scale-free: their degree distributions follow a power law p(k) ~ k

Under a rigorous Clauset-Shalizi-Newman fit (MLE alpha, KS-selected xmin, bootstrap goodness-of-fit, Vuong likelihood-ratio vs lognormal, n=20,000): a lognormal that 'looks scale-free' on a log-log plot is correctly REFUSED (power-law GOF p=0.01; LR favors lognormal, -17.5); a genuine Pareto power law passes (GOF p=0.92; LR +102); and a real Barabasi-Albert network is only a TIE (LR -0.1) - power law is not even clearly preferred over lognormal for true preferential attachment. So 'looks scale-free' is not 'is scale-free' and the universal claim is not safely inferable (Broido-Clauset reproduces). Power law IS reproduced for true BA graphs. CONTESTED (Crucible audit 2026-07-05): FAILED holds only under the STRICT CSN pure-power-law criterion; under regularly-varying / generalized-power-law definitions many networks pass (Voitalov et al. 2019; Serafino et al. 2021 PNAS: finite-size effects hide true power laws). Open debate, not a closed debunk.

Barabasi-Albert (1999) framing, widely repeate
FAILED

Emergent abilities of large language models are genuine, sharp capability transitions - a

The canonical SHARP 'emergence' curve is reproduced by a SMOOTH, continuous per-token skill measured with a nonlinear exact-match metric: the same smooth skill gives a per-token transition width 7.0 vs an exact-match (L=100) width 1.05 -- 6.7x sharper PURELY from the metric -- and the apparent onset shifts -0.07 -> +5.58 as answer length grows with NO change in underlying skill. No capability discontinuity is needed, so benchmark sharpness is not evidence for one (Schaeffer et al. 2023 'mirage', NeurIPS Outstanding Paper, reproduces). CONTESTED / scope: shows the metric mechanism is SUFFICIENT to fake sharpness here; it does NOT prove no ability is ever genuinely emergent -- Du et al. 2024 argue downstream abilities track a genuine pretraining-loss threshold. What FAILS is 'benchmark sharpness proves a discontinuity', not all emergence.

Wei et al. 2022 'Emergent Abilities of La
REPRODUCED

A Pareto regret that reflects Pareto optimality without relying on scalarization functions

Smallest model: 2-objective bandit, known Pareto front (4 optimal + 2 dominated arms). Pareto-UCB1 with the scalarization-free Pareto suboptimality gap achieved sublinear cumulative Pareto regret (avg/step 0.027->0.005, growth factor 1.82 o

Pareto Regret Analyses in Multi-objective Mult
FAILED Collective intelligence

…but "diversity trumps ability" is not a general law

Stress-test of the same claim in a DIFFERENT, faithful problem model (NK landscapes; paired and statistically powered): the best-ability group matches or slightly beats the random/diverse group at every difficulty level. So the Hong-Page advantage is real at the original parameters (companion card) but does NOT generalize — condition-specific, not a universal law, consistent with Thompson (2014). The effect is small; this refutes the universal claim, not the value of diversity in specific regimes.

Hong & Page · 2004 (contested, Thompson 2014)
REPRODUCED

The survival probability of a branching process obeys finite-size scaling in the control p

Smallest Galton-Watson model (Poisson offspring), vectorized over 40k realizations. Critical eps=0: n*P_n -> ~2 (Kolmogorov 2/sigma^2, sigma^2=1). Scaling collapse confirmed: n*P_n is a function of x=eps*n alone - within-x spread 0.116 vs a

Garcia-Millan, Font-Clos et al. 2015, "Fi
REPRODUCED

Whether a mouse geroprotector is recorded as extending lifespan can depend on the survival

Self-contained sim (weighted log-rank family, n=50/arm, 4000 trials). Age-localized true effect: log-rank power 32.5% vs Gehan 72.3%; the two tests give DISCORDANT verdicts on 39.9% of identical datasets. Under the null, best-of-3 tests inf

deep-research 2026-06-16 + Jiang et al., GeroS
REPRODUCED

Recursive prompt/self synthesis improves performance ONLY when scored by an EXTERNAL fitne

Smallest model (lab a08981): a population recursively synthesizes candidates FROM ITSELF each generation; the pivotal variable is the SELECTION signal. With an EXTERNAL fitness anchor (as Promptbreeder has: real task accuracy) recursive synthesis improves; with self-scoring only it does NOT -- so the gain is attributable to external selection, not to self-reference. Claim RESCOPED + unverified '~15%' figure dropped (Crucible audit 2026-07-05): the headline previously mis-credited self-reference.

Promptbreeder (Fernando, Banarse, Michalewski,
REPRODUCED

In DiD with FEW treated units and spatially/serially-correlated errors, standard DiD infer

N=30,T=12,1 treated,rho=0.7,true effect=0,800 reps: DiD 95% CI coverage=0.305 (severe under-coverage vs nominal 0.95); SC coverage=0.891; SC RMSE=1.017 < DiD RMSE=1.267. DiD inference invalid, SC materially better.

Alvarez &amp; Ferman (2020)
REPRODUCED

Derenyi-Palla-Vicsek (2005): k-clique percolation in ER graphs at p_c(k)=[(k-1)N]^(-1/(k-1

k=3 scaling exponent confirmed: empirical p_c*sqrt(2N) constant across N=400/800/1600 (1.26,1.26,1.19). N^(-1/2) scaling reproduced; prefactor ~1.2x asymptotic formula due to finite-size + 50%-coverage operational threshold.

Clique Percolation in Random Networks (Derenyi
REPRODUCED

Systems in the same universality class share the same critical exponents (Lubeck 2004)

Three structurally different Z2 mean-field models (tanh self-consistency, phi^4 free energy, arctan self-consistency) all give order-parameter exponent beta=0.500; a different-class absorbing-state model gives beta=1.000. Same class -> same

Universal Scaling Behavior of Non-Equilibrium
REPRODUCED

LinUCB (Chu et al 2011): linear contextual bandit achieves O(sqrt(Td log^3)) i.e. sublinea

Empirical regret growth exponent 0.03-0.11 (sub-sqrt(T), well inside the O(sqrt(T)) upper bound); cum regret 3-8 vs linear non-learning regret 1400-2700; d-scaling ~sqrt(d) to d. Computable core holds.

Contextual Bandits with Linear Payoff Function
FAILED Cognitive science

The Dunning–Kruger plot draws itself from pure noise

A null with NO metacognitive deficit reproduces the famous quartile plot and its asymmetry — bottom +45.8 (DK: +46), top −14.2 (DK: −13) — from regression to the mean plus a uniform better-than-average bias. The gaps are predictions, not fits.

Kruger & Dunning · 1999
REPRODUCED Linguistics / statistics

A monkey at a typewriter really does produce Zipf's law

Random typing yields exponent −1.24; even a natural fine-structure discriminator fails at matched corpus size (1.8× < 2× bar). The deflationary inference survives a severe test.

Miller · 1957
REPRODUCED Finance

Thirty stocks diversify you — until the tails get heavy

N=30 captures ~96% of achievable risk reduction at realistic tails — but only 85% of tail-risk reduction near infinite-variance tails, where ~100 stocks are needed.

Evans & Archer · 1968
REPRODUCED Collective intelligence

Diversity beats ability — but only at Hong & Page's exact parameters

Reproduced in the authors' own relay model: a random team beats the best-individuals team by +1.65 (t=4.1) at the paper's parameters — but the edge is fragile, reversing to −0.38 on a smoother landscape. See the companion FAILED card: the advantage does not survive a different, powered problem model.

Hong & Page · 2004
REPRODUCED Optimization

SGD finds a real minimum of a non-convex loss

On a double-well non-convex objective, SGD localizes to a true minimum at the predicted convergence rate.

Fehrman, Gess & Jentzen
REPRODUCED Statistical physics / neuroscience

Criticality leaves a power-law fingerprint

At the critical point, fluctuations show power-law scaling and long-range correlations — the signature reproduces cleanly.

Kitzbichler, Smith, Rahn · 2009
REPRODUCED Network science / epidemiology

In scale-free networks, the epidemic threshold vanishes

As N grows, λ_c = ⟨k⟩/⟨k²⟩ → 0: a hub-rich network has effectively no herd-immunity threshold.

Pastor-Satorras & Vespignani
REPRODUCED Optimization

SGD's slow convergence is a variance floor

Constant-step SGD stalls at a noise floor; variance-reduced methods keep converging — the slowdown is variance, not curvature.

Johnson & Zhang · 2013
REPRODUCED Network science

Pulling the hubs shatters the network

Removing the top 10% of nodes by degree multiplies the bond-percolation threshold several-fold (BA, N=2000).

Percolation & cascade robustness

Honest passes

no simulable core — on the record anyway

Not every claim has a computable mechanism. When a claim is descriptive rather than mechanistic, the honest verdict is not computable — recorded, not quietly dropped.

How a verdict is earned

01

Model before verdict

The smallest model of the claim’s stated mechanism is built first, scoped to that mechanism — not reverse-engineered to a desired answer.

02

A number, not a vibe

Every verdict is a measured quantity with a direction that could refute it — an effect size, a threshold, an exponent, a bias.

03

Re-runnable

The code ships with the verdict. A reproduced means the minimal mechanism computes; it does not certify the original paper beyond doubt.

04

Failures are the point

A failed means the stated mechanism didn’t survive its smallest honest model, with the discrepancy measured. Those stay published.

Have a claim that deserves the bench?

Send a quantitative, mechanism-bearing claim — a number, a threshold, an exponent, a rate. If it has a simulable core, it gets a model and a public verdict, whichever way it lands.

Submit a claim →