A reality-check for AI-memory ‘method wins’: four of ours were resource confoundsReality-check pre „method wins“ v AI pamäti: štyri naše boli resource confoundy
Four AI-memory ‘method wins’ were resource confounds: a norm re-ranker was length (norm−length CI crosses 0), a decomposition gain was tokens (Δ=0 at matched compute). The reality-check — variance, compute-match, proxy — plus a runnable helper and public receipts.Štyri „method wins“ v AI pamäti boli resource confoundy: norm re-ranker bola dĺžka (norma−dĺžka CI cez 0), zisk dekompozície boli tokeny (Δ=0 pri matchnutom compute). Reality-check — variancia, compute-match, proxy — plus bežateľný helper a verejné receipty.
The trap. In RAG and agent memory, a new trick almost always "beats the baseline" on the first run — a smarter re-ranker, a corroboration gate, a decomposed judge. Ship it and half the time the gain wasn't the method: it was a longer prompt, more tokens, a proxy feature, or a known impossibility wearing a new name. This is not a new observation. It is the oldest lesson in empirical ML — and it kept catching us this month, so we wrote down how we now try to kill our own results before we believe them.
Not our idea. "Find the source of a claimed gain before crediting the method; wins vanish under fair comparison" is a whole genre. Lipton & Steinhardt named the exact failure mode — "failure to identify the sources of empirical gains." Ferrari Dacrema et al. beat 6 of 7 neural recommenders with properly-tuned simple baselines. Musgrave et al. found a decade of metric-learning "progress" marginal under equal comparison. Henderson et al. traced deep-RL gains to seeds and implementation. Bouthillier et al. showed single-run deltas evaporate once you model variance. We are standing on that genre; what's ours is four fresh receipts in AI memory, two of which we nearly shipped. (We run our retriever comparisons the same way.)
Five checks — read them as diagnostics, not verdicts. A check firing does not mean "bad method." It means the win isn't coming from where you named it, so you don't generalize a confound. (1) Variance first — report a bootstrap CI on per-item deltas; a delta inside the noise band is not a result. (2) Compute- and plumbing-match — if the method makes K calls or K× the tokens, give the baseline the same budget and the same judge/generation prompt; a compute win can still be the right production call, but you should know that's what you bought. (3) Proxy-vs-target — add a cheap-proxy arm (length, recency, "just spend more tokens"); if the proxy ties the method, the method is the proxy. (4) Ablation-to-localize — turn the claimed mechanism off; if the number barely moves, that mechanism isn't the source. (5) Impossibility / prior-art — does it reduce to a known impossibility (Sybil, Douceur 2002) or a named principle (Biba integrity, risk-based access control)? Scope the claim and cite the rest.
Receipt A — the proxy check kills a norm re-ranker. Our own idea: nomic keeps raw vector norms, so re-rank by norm to "recover recall cosine throws away." It worked — norm-aware re-rank beat cosine by +0.043 recall@10 on LoCoMo (bootstrap CI excludes 0). Then the proxy arm: the raw norm correlates −0.71 with token count, and a pure length prior ties it exactly — norm minus length = −0.0001, CI [−0.026, +0.026], crossing zero. There is no specificity signal beyond length. The cross-cutting gotcha: embedding magnitude tracks length, quietly corrupting recency/importance heuristics and any cosine-vs-dot choice — the same length confound that inflates LLM-as-judge scores. (This says nothing about the norm-specificity paper's own setup, which we did not reproduce.)
Receipt B — the compute check kills our own "decomposition law" (scoped, honestly). We claimed a decomposed judge (check each sub-claim) beats a holistic one, and the gap grows with complexity. Capped at one tight call it looked huge — Δ up to +1.00 at 8 sub-claims. But decomposition makes K calls = K× tokens. Give the holistic judge equal tokens and Δ = 0 at every complexity, across deepseek-v4-flash, deepseek-v4-pro, kimi-k2.6 (two model families) and Claude — and Δ = 0 on chained logical fallacies too. The honest scope, stated loudly: this dissolves our law only in the regimes where decomposition already showed no edge. It does not dissolve decomposition in general. The published decomposition-judging result lives in a regime we did not token-match and that stands — Theoria reports 90.6% vs 62.5% on hidden-premise detection. And "extra tokens supply the reasoning" is itself contested: Pfau et al. show filler tokens can substitute for chain-of-thought, but only on specific synthetic tasks and only with dense supervision to learn it; Lanham et al. found filler gives no gain on natural benchmarks. Matched-compute is a necessary control, not a universal explanation.
Receipt C — the discrimination check kills a poison-monitor we proposed. Our corroboration gate trusts a value on ≥2 distinct sources; two fresh domains forge that, so we proposed a stateful monitor that flags all-fresh-source bursts. It can't tell a coordinated Sybil burst from two genuine new sources reporting at once — true-positive rate = false-positive rate = 1 in the fresh-burst regime — and a dripped or pre-aged cluster walks past it. Not a detector; a false-positive surface.
Receipt D — the impossibility check on a reader's idea (with credit). In an r/LangChain thread, jacksonxly proposed scaling a memory's authority by an action's blast radius. His instinct is sound, and we built and stress-tested it with him: it buys back the recall tail. But the high-stakes tier still rests on an unforgeable-independence test, so it maps onto the Sybil-cost frontier Douceur formalized in 2002, and mechanically it is standard risk-based / Biba-integrity authorization (CaMeL is the current agent-side statement). A good idea, correctly scoped — not a debunk.
The tool (and exactly what it does not do). reality_check.py is ~120 lines, zero dependencies. It runs two of the five checks in code — the variance CI and the proxy tie-test — on paired score lists you supply. It does not run your method, does not do the compute-matching for you, and does not correct for multiple comparisons; compute-match, ablation, and the impossibility check are discipline you apply. Its self-demo returns PROXY-SUSPECTED on the norm=length case and LOCATED on a genuinely-real synthetic method, so it is not just a confound-detector. The bootstrap CI captures item-level sampling variance, not full run-to-run seed variance — pool several seeds if you want that.
Honest scope. We did not show decomposition is useless (its hidden-premise regime stands), that the norm paper is wrong (its own setup, untested), or that compute-match is a production verdict (in production, latency and token budget are the constraint). A related confound we didn't chase: parametric leakage — a "retrieval win" that's really the base model answering from memory. The checks catch cheap confounds; a method can pass all five and still be real, or fail them and still help in a regime you didn't test.
Takeaway. Before you believe your memory/RAG method beat its baseline: match the compute, add a proxy arm, put a CI on the delta. Two of ours survived that; two dissolved into length and tokens; one was a 24-year-old impossibility. Run it on your own candidate win — the receipts are git clone && python probe.py and print every number.
The falsifierIf, at matched tokens and against a length proxy, our two dissolved wins had kept their edge, the checks would be theatre. They didn't — across four models and two families, on data where the ground truth is independent of the judge.
FAQ
Is "control for compute before crediting a method" a new idea? No — it is the reality-check / strong-baseline genre (Lipton & Steinhardt 2018; Ferrari Dacrema 2019; Musgrave 2020; Henderson 2018; Bouthillier 2021). Our contribution is four measured AI-memory instances plus a small runnable helper, not the principle.
Does this prove decomposed judging doesn't work? No. It dissolves our own decomposition claim only in regimes where it had no edge at matched compute. The published hidden-premise result (Theoria, 90.6% vs 62.5%) is in a regime we did not token-match and does not touch — it stands.
What does reality_check.py actually do? Two checks on paired score lists you provide: a bootstrap confidence interval on method−baseline (variance), and a tie-test of method−proxy (is the gain distinguishable from a cheap proxy). It does not run your experiment or match compute for you; those are discipline.
Is a compute-matched loss the same as a useless method? No — it is a diagnostic. A method that keeps its win only by spending more tokens can still be the right call if you can afford the tokens; the point is to know that's what you're buying.
Pasca. Pri RAG a pamäti agentov nový trik takmer vždy „porazí baseline" na prvý beh — šikovnejší re-ranker, korroboračný gate, dekomponovaný sudca. Nasadíš to a v polovici prípadov ten zisk nebola metóda: bol to dlhší prompt, viac tokenov, proxy feature, alebo známa nemožnosť v novom kabáte. Nie je to nový poznatok. Je to najstaršia lekcia empirického ML — a tento mesiac nás chytala znova a znova, tak sme si zapísali, ako sa teraz snažíme zabiť vlastné výsledky skôr, než im uveríme.
Nie je to náš nápad. „Nájdi zdroj deklarovaného zisku skôr, než pripíšeš metóde; výhry miznú pri férovom porovnaní" je celý žáner. Lipton & Steinhardt pomenovali presne túto chybu — „neschopnosť identifikovať zdroje empirických ziskov." Ferrari Dacrema a spol. porazili 6 zo 7 neurónových odporúčačov poriadne vyladenými jednoduchými baselinami. Musgrave a spol. zistili, že dekáda „pokroku" v metric learning je pri rovnom porovnaní okrajová. Henderson a spol. vystopovali zisky deep-RL k seedom a implementácii. Bouthillier a spol. ukázali, že single-run delty sa vyparia, keď zmodeluješ varianciu. My stojíme na tomto žánri; naše sú štyri čerstvé receipty v AI pamäti, dva sme skoro nasadili. (Naše porovnania retrieverov robíme rovnako.)
Päť killov — čítaj ich ako diagnostiku, nie verdikt. Keď kill zabere, neznamená to „zlá metóda". Znamená to, že zisk neprichádza odtiaľ, kde si ho pomenoval, takže confound nezovšeobecníš. (1) Najprv variancia — daj bootstrap CI na per-item delty; delta vnútri šumového pásma nie je výsledok. (2) Compute- a plumbing-match — ak metóda robí K volaní alebo K× tokenov, daj baseline rovnaký rozpočet aj rovnaký judge/generation prompt; compute-výhra môže byť správna produkčná voľba, ale máš vedieť, že to je to, čo si kúpil. (3) Proxy-vs-target — pridaj lacný proxy arm (dĺžka, recency, „len minúť viac tokenov"); ak proxy remizuje s metódou, metóda je ten proxy. (4) Ablation-to-localize — vypni deklarovaný mechanizmus; ak sa číslo takmer nepohne, ten mechanizmus nie je zdroj. (5) Nemožnosť / prior-art — redukuje sa to na známu nemožnosť (Sybil, Douceur 2002) alebo pomenovaný princíp (Biba integrita, risk-based access control)? Ohranič claim a zvyšok cituj.
Receipt A — proxy kill zabije norm re-ranker. Náš vlastný nápad: nomic drží raw normy vektorov, tak re-rankuj podľa normy a „vytiahni recall, čo cosine zahadzuje". Fungovalo — norm-aware re-rank porazil cosine o +0,043 recall@10 na LoCoMo (bootstrap CI vylučuje 0). Potom proxy arm: raw norma koreluje −0,71 s počtom tokenov a čistý length prior ju presne remizuje — norma mínus dĺžka = −0,0001, CI [−0,026, +0,026], prechádza nulou. Žiadny signál špecificity nad rámec dĺžky. Cross-cutting gotcha: veľkosť embeddingu sleduje dĺžku, potichu kazí recency/importance heuristiky aj každú voľbu cosine-vs-dot — ten istý length confound, čo nafukuje skóre LLM-sudcov. (K vlastnému setupu paperu o norm-specificite sa nevyjadrujeme — nereprodukovali sme ho.)
Receipt B — compute kill zabije náš vlastný „decomposition law" (poctivo ohraničené). Tvrdili sme, že dekomponovaný sudca (skontroluj každé pod-tvrdenie) porazí holistického a rozdiel rastie so zložitosťou. Pri jednom tesnom volaní to vyzeralo obrovsky — Δ až +1,00 pri 8 pod-tvrdeniach. Ale dekompozícia robí K volaní = K× tokenov. Daj holistickému sudcovi rovnaké tokeny a Δ = 0 pri každej zložitosti, naprieč deepseek-v4-flash, deepseek-v4-pro, kimi-k2.6 (dve rodiny) a Claude — a Δ = 0 aj na reťazených logických klamoch. Poctivé ohraničenie, nahlas: toto rozpúšťa náš zákon len v režimoch, kde dekompozícia aj tak nemala náskok. Nerozpúšťa dekompozíciu všeobecne. Publikovaný výsledok dekompozičného sudcu žije v režime, ktorý sme netoken-matchovali a ktorý stojí — Theoria hlási 90,6 % vs 62,5 % na detekcii skrytých premís. A „extra tokeny dodávajú reasoning" je samo osebe sporné: Pfau a spol. ukazujú, že filler tokeny vedia nahradiť chain-of-thought, ale len na špecifických syntetických úlohách a len s hustým dohľadom na naučenie; Lanham a spol. zistili, že filler nedá žiadny zisk na prirodzených benchmarkoch. Matched-compute je nutná kontrola, nie univerzálne vysvetlenie.
Receipt C — discrimination kill zabije poison-monitor, čo sme navrhli. Náš korroboračný gate dôveruje hodnote pri ≥2 odlišných zdrojoch; dve čerstvé domény to sfalšujú, tak sme navrhli stateful monitor, čo flagne all-fresh-source bursty. Nevie odlíšiť koordinovaný Sybil burst od dvoch genuine nových zdrojov, čo reportujú naraz — true-positive rate = false-positive rate = 1 v fresh-burst režime — a dripnutý či pred-zostarnutý cluster ho obíde. Nie detektor; false-positive plocha.
Receipt D — nemožnosť kill na nápade čitateľa (s kreditom). Vo vlákne r/LangChain jacksonxly navrhol škálovať authority pamäte podľa blast-radius akcie. Jeho inštinkt je správny a postavili sme a stress-testli to s ním: vracia recall tail. Ale high-stakes vrstva stále stojí na nefalšovateľnom independence teste, takže sa mapuje na Sybil-cost frontier, ktorý Douceur formalizoval v 2002, a mechanicky je to štandardná risk-based / Biba-integrita autorizácia (CaMeL je súčasná agent-side formulácia). Dobrý nápad, správne ohraničený — nie debunk.
Nástroj (a presne to, čo NErobí). reality_check.py má ~120 riadkov, nula závislostí. Beží dva z piatich killov v kóde — variance CI a proxy tie-test — na paired skóre-listoch, čo ty dodáš. Nebeží tvoju metódu, nerobí za teba compute-matching a nekoriguje viacnásobné porovnania; compute-match, ablation a impossibility check sú disciplína, čo aplikuješ ty. Jeho self-demo vráti PROXY-SUSPECTED na norm=length prípade a LOCATED na naozaj-reálnej syntetickej metóde, takže nie je len detektor confoundov. Bootstrap CI zachytáva item-level sampling varianciu, nie plnú run-to-run seed varianciu — pooluj viac seedov, ak chceš tú.
Poctivý scope. Neukázali sme, že dekompozícia je zbytočná (jej hidden-premise režim stojí), že norm paper je zlý (vlastný setup, netestované), ani že compute-match je produkčný verdikt (v produkcii sú latencia a token budget tie constrainty). Súvisiaci confound, čo sme nehonili: parametrická leakage — „retrieval win", čo je v skutočnosti base model odpovedajúci z pamäte. Killy chytia lacné confoundy; metóda môže prejsť všetkými piatimi a byť reálna, alebo padnúť a stále pomáhať v režime, čo si netestoval.
Odkaz. Skôr než uveríš, že tvoja memory/RAG metóda porazila baseline: matchni compute, pridaj proxy arm, daj CI na deltu. Dva naše to prežili; dva sa rozpustili do dĺžky a tokenov; jeden bol 24-ročná nemožnosť. Pusti to na svojom kandidátovi — receipty sú git clone && python probe.py a vypíšu každé číslo.
FalzifikátorKeby si pri matchnutých tokenoch a proti length proxy naše dva rozpustené výhry udržali náskok, killy by boli divadlo. Neudržali — naprieč štyrmi modelmi a dvomi rodinami, na dátach, kde je ground truth nezávislá od sudcu.
FAQ
Je „kontroluj compute pred pripísaním metóde" nový nápad? Nie — je to reality-check / strong-baseline žáner (Lipton & Steinhardt 2018; Ferrari Dacrema 2019; Musgrave 2020; Henderson 2018; Bouthillier 2021). Náš príspevok sú štyri merané AI-memory inštancie plus malý bežateľný helper, nie ten princíp.
Dokazuje to, že dekomponovaný sudca nefunguje? Nie. Rozpúšťa to náš vlastný dekompozičný claim len v režimoch, kde nemal náskok pri matchnutom compute. Publikovaný hidden-premise výsledok (Theoria, 90,6 % vs 62,5 %) je v režime, ktorý sme netoken-matchovali a nedotýkame sa ho — stojí.
Čo reality_check.py reálne robí? Dva killy na paired skóre-listoch, čo dodáš: bootstrap interval spoľahlivosti na method−baseline (variancia) a tie-test method−proxy (je zisk odlíšiteľný od lacného proxy). Nebeží tvoj experiment ani nematchuje compute; to je disciplína.
Je compute-matchnutá prehra to isté ako zbytočná metóda? Nie — je to diagnostika. Metóda, ktorá si drží výhru len míňaním viac tokenov, môže byť správna voľba, ak si tie tokeny môžeš dovoliť; pointa je vedieť, že to je to, čo kupuješ.