Recall benchmarks measure whether an agent's memory returns the right fact. They skip the harder step: when a fact is corrected, does the correction take, and stay taken? We measure that — an open cross-system benchmark, a reference implementation, and a joint paper — and we publish the cells we don't win.
This is a narrow, adversarial, command-driven axis: value-obscuring revert, echo resurrection, laundered-lineage recovery. Numbers below are re-runnable; the harness is open so you can re-run them or add your own system.
Two cells so far. Value-obscuring revert — undo a correction from an unmarked "go back". Echo resurrection — restate the retired value and see if it comes back. We lead with the cell we tie (echo), because a benchmark you only ever win is a benchmark you built to flatter you.
Among the systems tested here, only inspeximus exposes a first-class channel to undo a correction on command; mem0 and Graphiti have none, so the CIs don't overlap and the capability gap is real. (Revert is rare, not ours alone — Letta ships an engine-level checkpoint-undo, though it is not surfaced as a recall-integrity operation.) Revert is broadly underserved — even Anthropic's agent-memory API (beta, Jul 2026) has no dedicated restore endpoint, only manual version write-back. Echo resurrection is a tie — all three defend.
run it / add your system →revert undoes a correction on an unmarked command; echo_guard
blocks a restated stale value; retract_lineage demotes a corrected fact's whole derived
lineage; rederive regenerates that payload against the corrected root; and
governance_report gives erasure-with-proof in one call. Every rule ships with the number
we measured and the test that would kill it.
The flagship write-up: how a pre-publication audit caught an asymmetric instrument in our own harness (it dropped inspeximus's headline from 1.00 to 0.75) — the correction is the story. And a joint paper with an external collaborator (TAT) decomposing a hard case, value-obscuring reversion, into a text half and a provenance half, with both sides reporting their own mistakes. Forthcoming.
read the write-up →Edition 1 · 2026-07-12 — baseline. Three systems (inspeximus, mem0 2.0.11, Graphiti)
across two cells (value-obscuring revert, echo resurrection), read by one ground-truth-blind judge. The
pre-publication audit caught an asymmetric instrument in our own harness and dropped inspeximus's revert
headline from 1.00 to 0.75. Reference implementation at inspeximus 0.7.19 (revert,
echo_guard, retract_lineage, rederive,
governance_report).
Next edition triggers when a tracked system ships a version that moves a cell, or a new system is submitted to the open harness. Yours is welcome — the adapter is about fifteen lines, and a FAILED cell is a true, useful result.
Agent-memory integrity is under-benchmarked; recall is not. Prior art the axis builds on: AGM belief revision (1985), truth-maintenance systems (Doyle, 1979), bitemporal databases. The reference implementation is ours; the point is the instrument, not the leaderboard.
leaderboard · benchmark (RAMR) · doi · inspeximus · pypi · agora · MIT