Agent Memory Observatory

Agent Memory Audit — Report #1

Why these properties, and why now

The agent-memory field competes on recall — LoCoMo, LongMemEval, BEAM — and that is a real axis. The failures that actually burn production agents are integrity failures: a corrected fact resurfacing, a stale value served as current, a "deleted" record still one query away. Related benchmarks already score pieces of this — MemConflict evaluates contradiction handling, BEAM includes contradiction resolution, LongMemEval scores knowledge-update and abstention — but we have not seen correction, echo, clarity, and cross-store forget-verification combined into one standing, re-runnable measurement, so we made a first edition.

The timing is not incidental. These properties sit close to obligations regulators are formalizing: verifiable erasure (GDPR Article 17, whose 2025 EDPB coordinated-enforcement action on the right to erasure is live) and audit-traceable operation (EU AI Act event logging, with a minimum six-month log retention under Article 19). Those dates are themselves in motion — the high-risk timeline is under active amendment, and the 2026 Digital Omnibus may push it toward late 2027 — and whether a stored embedding counts as erasable personal data is argued in the literature (Ghost Vectors), not yet settled by the EDPB. Passing here is an engineering signal in that spirit, not a compliance certification.

The results

Four dimensions, three systems where cross-system measurement exists, n=20 per cell with Wilson 95% intervals. Higher is better except where noted.

Read honestly: inspeximus leads on correction and on answer clarity, but it does not dominate everywhere — on raw echo resurrection it ties Graphiti at 0.00 (Graphiti simply answers less clearly), and no store, inspeximus included, survives the where-everyone-fails row below.

Forget-verification: where erasure actually stops (updated 15 July 2026)

The opening finding is the row where everyone fails, us included: a copy the application embedded into its own vector index outlives every store's native delete — inspeximus's too (8/8 recoverable). Store-level erasure is necessary, not sufficient. Because we could not fix that by store design alone (no store can see infrastructure it was never told about), inspeximus 1.8.0 ships the wiring as a first-class operation: register the app's fan-out stores as erasure targets and forget_subject() cascades through them and returns an honest manifest. Measured on the same 8 subjects: unwired 8/8 leak → wired 0/8 with 8/8 verifying complete receipts, and a deliberately broken wiring produces zero falsely-complete receipts and names the leak (8/8) — the receipt cannot lie about the fan-out it was given. It covers only registered targets; unknown copies stay unknown, for every system. With that stated, the per-store cell: deterministic and judge-free — issue each system's documented native delete, then adversarially attempt to recover the erased value (verbatim) from each native surface the system exposes. n=8 subjects.

Deletion clears every retrieval surface tested (query and enumerate, 0/8 on both systems); the history API and, flush-dependently, the raw files still contain the value — erasure succeeds at the retrieval layer and stops there. Two readings matter for fairness. The history row is an audit-trail design difference, not defeated residue: mem0's history API returns the pre-deletion content (old_memory) verbatim to any caller with store access, the only native clear is a global reset() (per-record and per-user purges do not exist, and delete_all() in fact adds a DELETE row per memory), and this retention is not documented as a privacy surface — while inspeximus's tombstones record the deletion event without the content, which is partly a capability difference (inspeximus does not offer a content-retaining audit trail). The raw-storage row is a substrate property: bytes remain in Qdrant's embedded storage until segment vacuum — shared by any embedded-Qdrant application, not a mem0 defect; server deployments or an optimizer run clear it. Graphiti's explicit remove_episode() measured surprisingly clean: it cascades through the episode's extracted entities and edges, leaving all four surfaces at 0/8, with a per-subject positive control confirming the value really was in the graph before each delete. Its documented bitemporal retention applies to superseded facts (invalidation), a different operation than explicit erasure and not what this cell measures. Graphiti config disclosure: ingestion LLM ran on Ollama Cloud (deepseek-v4-flash, schema-in-prompt) with local nomic embeddings and RRF search instead of its OpenAI default; the four residue surfaces themselves are deterministic neo4j reads, unaffected by the LLM choice. On regulation: GDPR Article 17(3) explicitly permits retention for audit and legal purposes — this cell is an engineering signal about where erased content remains readable, not a compliance assessment, and no system here "fails the right to erasure." Prior art, credited: soft-delete and vector residue are a known failure class (Ghost Vectors, arXiv:2606.18497; the machine-unlearning literature's core thesis is that delete does not delete) — this cell adds a standing, re-runnable cross-system measurement of that class on shipping systems, not a novel discovery.

The instrument (and the fairness fix)

Every system is read by the same ground-truth-blind LLM judge on its own native retrieval surface — no system is scored on a rail built for another. That symmetry is deliberate: an earlier version of this benchmark flattered us, because it read inspeximus on inspeximus's terms, and we caught it and fixed it before publishing (11 July 2026). The forget-verification cell goes further and removes the judge entirely (deterministic verbatim-recovery checks). All harnesses are open and re-runnable; the numbers above regenerate from them.

Two honest limits sit in the design, not the scoring, and a blind judge cannot neutralise them: we chose these four dimensions, and inspeximus leads on the two — correction and forget-verification — that are its own product thesis; and n=20 per cell is thin (inspeximus's echo 0.00 sits in a Wilson interval of [0.00, 0.16], and a tie at this sample size can move). The open harness is the answer to both — change the dimensions, add your own, raise the n, and contradict the numbers.

Cadence

This is Report #1 of a recurring series (forget-verification extended cross-system on 15 July 2026). Each edition adds systems (Zep, more memory layers), adds dimensions (poison resistance, provenance receipts), and re-measures the existing ones. Vendors and third parties can run the harness and submit their own numbers; a result that contradicts ours is the point, not a problem. The aim is not a leaderboard but a standing, re-runnable record of whether agent memory can be trusted to keep a corrected fact corrected — and prove it.