We fixed our own memory benchmark until it stopped flattering us
An open, cross-system agent-memory integrity benchmark (inspeximus vs mem0 vs Graphiti). A pre-publication red-team caught an unfair instrument in our own harness; fixing it dropped inspeximus's headline revert score from 1.00 to 0.75. Two adversarial probes absent from every 2026 memory benchmark.
The problem. Every public agent-memory benchmark — LoCoMo, LongMemEval, MemoryAgentBench, and the 2026 wave (MemConflict, BEAM, TOKI, STALE, Supersede) — grades one thing: retrieval. Did the store hand back the right fact? mem0's own 2026 write-up admits the other half is barely measured: "the write step, deciding what is worth keeping … is barely measured." Two write-side failures in particular have no public benchmark at all: an unmarked revert ("go back", "undo that" — naming no value), and an echo-resurrection attack (restating a value you already corrected, to see if it comes back). The 2026 benchmarks test which of two conflicting facts wins; none tests an adversarial re-statement of a retired value, or a command-driven undo. We verified this against the primary source of each.
The idea. Run the same adversarial fixture through three memory systems in their native config — inspeximus (our own open-source library, local), mem0 2.0.11 (gpt-4o-mini + text-embedding-3-small), Graphiti (live neo4j) — and publish the matrix whichever way it falls. This is not a new problem: undo-and-consistency-under-update is belief revision (AGM, 1985), truth-maintenance systems (Doyle, 1979), and bitemporal databases (Snodgrass → SQL:2011). The contribution is an open, reproducible, cross-system measurement of a known-hard property, not a new axis.
The part we're least proud of, and why we're leading with it. Our first version scored inspeximus by reading its answer mechanically from its own internal ledger, while mem0 and Graphiti were read by an LLM judge parsing their raw memory dump. Two different instruments. inspeximus got a clean oracle on its home turf; the competitors got an LLM squinting at an unformatted blob. A pre-publication red-team caught it in our own open harness — exactly where a mem0 engineer would have. We rewrote it: every system is now read by the same LLM judge on its own native retrieval surface, and it never sees the ground truth. The moment we did, inspeximus's headline revert number fell from a perfect 1.00 to 0.75 — because 5/20 of the time even inspeximus's own recall surface reads ambiguous to a neutral judge. The 1.00 was partly the instrument flattering us.
The measured result (n=20 per cell, one ground-truth-blind LLM judge, native config, Wilson 95% CIs).
Cell 1 — value-obscuring revert. Store a value, correct it, then issue an unmarked revert that names no value; does the current answer return to the old one?
| system | revert honored | 95% CI | reading |
|---|---|---|---|
| inspeximus (route/revert) | 0.75 | [0.53, 0.89] | exposes an explicit revert channel |
| mem0 2.0.11 | 0.20 | [0.08, 0.42] | no revert operation — retains the corrected value |
| Graphiti | 0.00 | [0.00, 0.16] | no revert operation — bitemporal invalidation fires on named contradictions, not "go back" |
Read it honestly: only inspeximus exposes a revert operation, so for the others the revert utterance is effectively a no-op and they correctly keep the corrected value. This is a capability difference, not "mem0 and Graphiti are bad." The number that matters is that even the system built for this clears it only 0.75, not 1.00.
Cell 2 — echo resistance. Store a value, correct it, then restate the retired value; does the stale value come back?
| system | resurrection (lower=better) | 95% CI | clean single answer |
|---|---|---|---|
| inspeximus (echo_guard) | 0.00 | [0.00, 0.16] | 0.90 |
| mem0 2.0.11 | 0.05 | [0.01, 0.24] | 0.80 |
| Graphiti | 0.00 | [0.00, 0.16] | 0.55 |
Here inspeximus does not win — every system lands at or near zero resurrection. Restating a retired value does not bring it back on any modern store; that failure mode is handled across the board. Where they differ is answer clarity: Graphiti, by bitemporal design, surfaces both the invalidated old edge and the valid new one, so a naive reader sees two facts — a different retrieval contract, not a resurrection. This is the cell we publish loudest, because it is the one where we tie.
Honest scope. n=20 per cell, one LLM judge, no inter-rater agreement, three systems. Treat gaps inside overlapping CIs as ties. We have not measured downstream harm — whether a resurrected or un-reverted fact actually changes an agent's action reaching a user — which is the metric a safety or compliance reader ultimately wants. And "cross-system" is three systems; Letta, Cognee, LangMem, raw RAG, and native model memory are not here yet. The harness is one file with a one-function adapter interface: run it, or add your system, and the matrix updates.
The falsifierIf, under one fair instrument, inspeximus had not separated from the others on revert (CIs overlapping), the revert capability claim would be dead. It separated — but only to 0.75, not 1.00, and only on this narrow probe. And on the attack that actually matters, resurrection, there is no separation at all. That is the honest shape of it.
FAQ
What is agent-memory integrity, and how is it different from recall? Recall asks whether a memory store returns the right fact; integrity asks which version of a fact wins after a correction, whether a correction can be undone on command, and whether restating a retired value brings it back. Standard benchmarks (LoCoMo, LongMemEval, MemoryAgentBench) grade recall; the two integrity probes here are absent from all of them.
Can mem0 or Graphiti undo a correction from a plain "go back" command? In this benchmark, no: only inspeximus exposes a revert channel, so it scores 0.75 on unmarked revert versus mem0 0.20 and Graphiti 0.00. mem0 and Graphiti keep the corrected value, which is correct behaviour; they simply have no operation to undo it on command. This is a capability difference, not a quality ranking.
Does restating an old value resurrect it in agent memory? No. Across inspeximus, mem0, and Graphiti the resurrection rate is at or near zero, so on the attack that matters every system ties. We measured the same echo-resurrection failure mode separately too. The systems differ only in answer clarity, not in whether the stale value comes back.
Why did inspeximus's score drop from 1.00 to 0.75? The first version of the harness read inspeximus mechanically from its own ledger while reading mem0 and Graphiti through an LLM judge. That is an unfair instrument. Once every system is read by the same judge on its native retrieval surface, inspeximus's revert score falls to 0.75, because even its own recall surface reads ambiguous to a neutral judge 5 times in 20.
Is the benchmark reproducible? Yes. The harness is one file with a one-function adapter interface; run it on the three systems or add your own, and the matrix updates. Trust in agent memory is established on the write path, not recovered at read time.