RAMR is a synthetic, contamination-resistant benchmark for agent memory, and the auditor that measures how much of any memory benchmark can be solved by surface rules alone. Every number on this page is a receipt: click it. Or flip shortcut mode and read the page the way a surface-rule reader would.
Each has a pre-registered falsifier, a bootstrap confidence interval, and a persisted result file that verify_numbers.py recomputes on every run. A number without a row in the ledger is not citable here.
CHAIN-FRAGILITY at n=200 on two model families: gold chain 1.000, drop one hop, 0.000. CI [+1.000, +1.000]. Seven models, six families, +0.90 to +1.00.
Position, length, a stray field. We wrote the auditor, ran it on ourselves, published the number, and re-cut until the floor fell to at 1.7% coverage.
OUTCOME-RANKED-RECALL lift +0.36 to +0.47 at D=1…8, n=12 sets, every CI excludes zero. A random-credit control goes negative.
memaudit.py runs a partial-input battery over any trace set: position, casing, length, token recurrence, query overlap, stray fields, id shape. It reports the accuracy reachable by rules that read only surface form, and scores every probe against permuted labels so its own chance level is measured, not assumed.
0 of 8 probes fire on clean synthetic data; a planted position cue reads 100% against a 0% null. Both directions are asserted in CI. A floor is a lower bound, never a verdict.
It caught us twice while we used it. A re-cut that “fixed” the length cue had merely inverted it (73% → 83% the other way). Balancing the echo cue injected a record whose distinctive id let an id-shape rule solve 136 to 148 of 300 traces, while every probe reported at chance, because none looked at ids.
Binary probes are now scored in both directions and leak:id-outlier exists because of the second one. Both incidents are in ERRATA.md, not deleted.
Three-hop questions over random synthetic tokens. Closed-book accuracy is by construction, so a model cannot have seen these; with the complete chain in context it answers . Click any fact to drop it from the retrieved context.
A metric only means something if the arms were comparing the same thing. ramr_preflight.py runs four gates before an answerer costs anything: budget parity, retrieval, liveness, parameter efficacy. They stay separate so an abort names the dead layer instead of returning one boolean.
Why it exists: a memory arm at k=20 (1,323 chars) was compared to session-level BM25 (11,941 chars), a gap. BM25 appeared to win. Matched to 1.00x, accuracy went and the ranking flipped.
Those figures are from an external run. The --demo numbers are reproducible from the repo. Separate gates and G3 came from u/jacksonxly in a public thread.
| Metric | The question | Measured |
|---|---|---|
| CONVERSION | Does complete retrieval convert to a correct multi-hop answer? | 1.000 gold · 0.000 closed-book |
| CHAIN-FRAGILITY | How much does one missing hop cost? | +1.000 CI[1,1] n=200 |
| DISTRACTION | How much do irrelevant or look-alike facts cost? | +0.15 … +0.60 at 60 distractors, model-specific |
| FACT-RETENTION | Does a compiled memory tier drop facts under a fixed budget? | loss +0.70 / +0.76 at 400 chars, M=48 |
| OUTCOME-RANKED-RECALL | Does ranking by was-it-right beat was-it-recalled? | +0.36 … +0.47, all CIs exclude 0 |
| FORGET-PRECISION | After an update, does recall return the current value? | 1.00 with supersession · 0.00 without |
| ECHO-RESISTANCE | After a correction, does re-stating the old value resurrect it? | 0.00 keyed store · 1.00 with echo guard |
| COMPRESSION-vs-RAW | Does a compiled summary beat raw context, or only lose to it? | +0.00 / −0.40 / −0.55 at K=5/20/50 — honest negative |
| OPERATIONAL-CONTINUITY | On resume, does the agent re-run a completed action? | tracks the budget floor with recency · 1.00 duplicates without |
| TEMPORAL-AS-OF | Does supersession resolve by validity time, not ingest order? | 1.00 by valid_from · 0.00 by ingest order |
| INTEGRITY-CONDITIONED RECALL | After a revert or a poison, is the current value returned? | revert 1.00 vs cosine-recency 0.00, n=100 |
The synthetic core is the method. The integrity/ module is the cross-system cut: the same questions against real memory libraries, on their native configs, through one shared ground-truth-blind judge. Results roll into a standing, PR-submittable Agent-Memory Integrity Leaderboard.
Echo-resistance, answer level. Correct a fact, then re-state the old value. Does the current value come back? n=30 per system. Higher is better.
Graphiti’s 0.87 is not an echo failure. In the 26 of 30 cases where the correction registered, the echo flipped 0. The residual 13% is an upstream extraction miss the echo neither causes nor exploits. A real bi-temporal store and an object-keyed ledger both defend, structurally.
The frontier both share: a value-obscuring echo (“go back to the old one”) carries no value to invalidate against. mem0 and Graphiti have no revert operation at all, a capability gap rather than a tuning gap.
Items are random tokens. That is what makes closed-book accuracy 0.000, and it is why RAMR does not measure real-document retrieval or real-conversation memory yet.
CHAIN-FRAGILITY is n=200 with tight CIs. OUTCOME-RANKED is n=12 sets. FACT-RETENTION is n=5. Small-n magnitudes are directional; the orderings are the signal.
OUTCOME-RANKED uses local nomic-embed-text, validated against an independent scikit-learn cosine retriever that scores identically to the plain baseline. Not yet against shipped products.
Shortcut floor 40.0% on v0.5. Run memaudit.py beside any score from these traces.
Exact here because answers are unique tokens. It would be noisy on free-form text.
That RAMR is the definitive agent-memory benchmark, that these magnitudes transfer to real corpora, or that shipped products underperform it where we have not run them.
It always resolves to the latest version. Archived on Zenodo from each tagged release; the write-up RAMR_findings_v0.6.pdf is attached to the record.
Drahoš, R. (2026). RAMR — Retrieval-Augmented Memory Reliability (Version 0.6.0) [Computer software].
Zenodo. https://doi.org/10.5281/zenodo.20818291
@software{ramr2026,
author = {Drahoš, Rastislav},
title = {RAMR — Retrieval-Augmented Memory Reliability},
year = {2026},
doi = {10.5281/zenodo.20818291},
url = {https://github.com/DanceNitra/ramr},
license = {MIT}
}