A reality-check for AI-memory ‘method wins’: four of ours were resource confounds

The trap. In RAG and agent memory, a new trick almost always "beats the baseline" on the first run — a smarter re-ranker, a corroboration gate, a decomposed judge. Ship it and half the time the gain wasn't the method: it was a longer prompt, more tokens, a proxy feature, or a known impossibility wearing a new name. This is not a new observation. It is the oldest lesson in empirical ML — and it kept catching us this month, so we wrote down how we now try to kill our own results before we believe them.

Not our idea. "Find the source of a claimed gain before crediting the method; wins vanish under fair comparison" is a whole genre. Lipton & Steinhardt named the exact failure mode — "failure to identify the sources of empirical gains." Ferrari Dacrema et al. beat 6 of 7 neural recommenders with properly-tuned simple baselines. Musgrave et al. found a decade of metric-learning "progress" marginal under equal comparison. Henderson et al. traced deep-RL gains to seeds and implementation. Bouthillier et al. showed single-run deltas evaporate once you model variance. We are standing on that genre; what's ours is four fresh receipts in AI memory, two of which we nearly shipped. (We run our retriever comparisons the same way.)

Five checks — read them as diagnostics, not verdicts. A check firing does not mean "bad method." It means the win isn't coming from where you named it, so you don't generalize a confound. (1) Variance first — report a bootstrap CI on per-item deltas; a delta inside the noise band is not a result. (2) Compute- and plumbing-match — if the method makes K calls or K× the tokens, give the baseline the same budget and the same judge/generation prompt; a compute win can still be the right production call, but you should know that's what you bought. (3) Proxy-vs-target — add a cheap-proxy arm (length, recency, "just spend more tokens"); if the proxy ties the method, the method is the proxy. (4) Ablation-to-localize — turn the claimed mechanism off; if the number barely moves, that mechanism isn't the source. (5) Impossibility / prior-art — does it reduce to a known impossibility (Sybil, Douceur 2002) or a named principle (Biba integrity, risk-based access control)? Scope the claim and cite the rest.

Receipt A — the proxy check kills a norm re-ranker. Our own idea: nomic keeps raw vector norms, so re-rank by norm to "recover recall cosine throws away." It worked — norm-aware re-rank beat cosine by +0.043 recall@10 on LoCoMo (bootstrap CI excludes 0). Then the proxy arm: the raw norm correlates −0.71 with token count, and a pure length prior ties it exactly — norm minus length = −0.0001, CI [−0.026, +0.026], crossing zero. There is no specificity signal beyond length. The cross-cutting gotcha: embedding magnitude tracks length, quietly corrupting recency/importance heuristics and any cosine-vs-dot choice — the same length confound that inflates LLM-as-judge scores. (This says nothing about the norm-specificity paper's own setup, which we did not reproduce.)

Receipt B — the compute check kills our own "decomposition law" (scoped, honestly). We claimed a decomposed judge (check each sub-claim) beats a holistic one, and the gap grows with complexity. Capped at one tight call it looked huge — Δ up to +1.00 at 8 sub-claims. But decomposition makes K calls = K× tokens. Give the holistic judge equal tokens and Δ = 0 at every complexity, across deepseek-v4-flash, deepseek-v4-pro, kimi-k2.6 (two model families) and Claude — and Δ = 0 on chained logical fallacies too. The honest scope, stated loudly: this dissolves our law only in the regimes where decomposition already showed no edge. It does not dissolve decomposition in general. The published decomposition-judging result lives in a regime we did not token-match and that stands — Theoria reports 90.6% vs 62.5% on hidden-premise detection. And "extra tokens supply the reasoning" is itself contested: Pfau et al. show filler tokens can substitute for chain-of-thought, but only on specific synthetic tasks and only with dense supervision to learn it; Lanham et al. found filler gives no gain on natural benchmarks. Matched-compute is a necessary control, not a universal explanation.

Receipt C — the discrimination check kills a poison-monitor we proposed. Our corroboration gate trusts a value on ≥2 distinct sources; two fresh domains forge that, so we proposed a stateful monitor that flags all-fresh-source bursts. It can't tell a coordinated Sybil burst from two genuine new sources reporting at once — true-positive rate = false-positive rate = 1 in the fresh-burst regime — and a dripped or pre-aged cluster walks past it. Not a detector; a false-positive surface.

Receipt D — the impossibility check on a reader's idea (with credit). In an r/LangChain thread, jacksonxly proposed scaling a memory's authority by an action's blast radius. His instinct is sound, and we built and stress-tested it with him: it buys back the recall tail. But the high-stakes tier still rests on an unforgeable-independence test, so it maps onto the Sybil-cost frontier Douceur formalized in 2002, and mechanically it is standard risk-based / Biba-integrity authorization (CaMeL is the current agent-side statement). A good idea, correctly scoped — not a debunk.

The tool (and exactly what it does not do). reality_check.py is ~120 lines, zero dependencies. It runs two of the five checks in code — the variance CI and the proxy tie-test — on paired score lists you supply. It does not run your method, does not do the compute-matching for you, and does not correct for multiple comparisons; compute-match, ablation, and the impossibility check are discipline you apply. Its self-demo returns PROXY-SUSPECTED on the norm=length case and LOCATED on a genuinely-real synthetic method, so it is not just a confound-detector. The bootstrap CI captures item-level sampling variance, not full run-to-run seed variance — pool several seeds if you want that.

Honest scope. We did not show decomposition is useless (its hidden-premise regime stands), that the norm paper is wrong (its own setup, untested), or that compute-match is a production verdict (in production, latency and token budget are the constraint). A related confound we didn't chase: parametric leakage — a "retrieval win" that's really the base model answering from memory. The checks catch cheap confounds; a method can pass all five and still be real, or fail them and still help in a regime you didn't test.

Takeaway. Before you believe your memory/RAG method beat its baseline: match the compute, add a proxy arm, put a CI on the delta. Two of ours survived that; two dissolved into length and tokens; one was a 24-year-old impossibility. Run it on your own candidate win — the receipts are git clone && python probe.py and print every number.

The falsifierIf, at matched tokens and against a length proxy, our two dissolved wins had kept their edge, the checks would be theatre. They didn't — across four models and two families, on data where the ground truth is independent of the judge.

FAQ

Is "control for compute before crediting a method" a new idea? No — it is the reality-check / strong-baseline genre (Lipton & Steinhardt 2018; Ferrari Dacrema 2019; Musgrave 2020; Henderson 2018; Bouthillier 2021). Our contribution is four measured AI-memory instances plus a small runnable helper, not the principle.

Does this prove decomposed judging doesn't work? No. It dissolves our own decomposition claim only in regimes where it had no edge at matched compute. The published hidden-premise result (Theoria, 90.6% vs 62.5%) is in a regime we did not token-match and does not touch — it stands.

What does reality_check.py actually do? Two checks on paired score lists you provide: a bootstrap confidence interval on method−baseline (variance), and a tie-test of method−proxy (is the gain distinguishable from a cheap proxy). It does not run your experiment or match compute for you; those are discipline.

Is a compute-matched loss the same as a useless method? No — it is a diagnostic. A method that keeps its win only by spending more tokens can still be the right call if you can afford the tokens; the point is to know that's what you're buying.