Does agent memory keep a corrected fact? We measured it

Restate a fact an agent already corrected, and many memory systems quietly bring the old value back — we measured it across backends. Here is why, and the one-line check that catches it.

Embedding similarity is famously bad at one specific thing: telling a contradiction apart from a duplicate. There is a number for it — cosine separates the two at roughly AUROC 0.59, barely above a coin flip, and a contradicted value often lands more similar to the original than a genuine rephrase does. (Yadav, Temporal Validity in Retrieval Memory, arXiv:2606.26511.)

That sounds academic until you meet the failure it causes in an agent's memory.

The failure

You tell an agent "the region is Frankfurt." Later you correct it: "actually, it's Ohio." A good memory layer now answers Ohio. Fine.

Then the old value gets said again. Not even maliciously — a user repeating a preference they forgot they changed, or a stray line in a long transcript: "the region is Frankfurt." Does the corrected fact stay corrected, or does Frankfurt come back from the dead?

We call this the echo: a restatement of a value that was already corrected away. On any store whose supersession is last-writer-wins by recency, the echo is the newest assertion of the old value, so it wins. And because similarity can't flag it — the echo is nearly identical to the original it's reviving — a similarity-based store has no signal to catch it either.

Measuring it

We built a small synthetic probe: assert a fact, correct it, restate the old value in different words, then score at the answer level — recall the top-k memories, hand them to an LLM, ask "what is the current value?" That protocol is fair to add-based stores that keep both values and reconcile at read time, not just to stores that overwrite. n=30; a demonstration, not a definitive benchmark.

Echo-resistance = fraction where the answer is still the corrected value. 1.0 is good.

Every backend keeps the correction itself when read carefully. But under a reworded echo:

• A naive keyed store (our own default, guard off) resurrects the stale value essentially every time — the restatement is the newest write.
• mem0, run in its own recommended config (gpt-4o-mini + text-embedding-3-small), returns the retired value roughly 30-63% of the time (95% CI on n=30; point estimate ~47%). This reproduced across an all-local Ollama run too, so it is config-robust, not an artifact of one judge. This is not a "mem0 bug" — it is the honest tradeoff of an add-based design that keeps both values and lets the reader reconcile; sometimes the reader picks the retired one, and mem0's extractor may even write a "reverted back to Frankfurt" record that the reader then trusts.
• A superseded-value guard holds the corrected value.

The fix is old and boring

The guard is not clever. It is essentially AGM belief revision (Alchourrón-Gärdenfors-Makinson, 1985) and bitemporal databases (Snodgrass, ~1993) re-applied: once a value is corrected away, a bare restatement must not revive it. Key on the value, not on similarity — a re-assertion of an already-superseded value is a restatement-of-superseded, and it does not win. A genuine change of mind ("actually, switch it back") is allowed, but only through an explicit reaffirm signal, never a silent echo.

We shipped this as an opt-in guard in our small memory library. The interesting result is not that it works — it is textbook — but that the adversarial echo was unmeasured: benchmarks like STALE and LongMemEval run a single correction, never a re-injection of the old value. (Related: the RAG supersession blind spot, and our work on memory-poisoning defenses and provenance, not truth.)

The part nobody solves

Our guard has a hard boundary. It keys on the value token, so it only catches echoes that name the old value. The genuinely open case is the value-obscuring restatement — "let's go back to what we had before", "undo that last change" — where the old value is never spoken. On a fixture of those, both of our methods are blind: value-match F1 0.03, cosine at chance. The discriminating signal is not in the content at all; it is the discourse relation — is this utterance re-opening a settled decision? That is a structural problem, and it is where we think a structural (not similarity) approach could finally earn its place.

Takeaway

Mostly we are publishing the check, not the scores: does your correction survive the old value being restated? It is cheap to test, the failure is silent, and almost no memory eval runs it. The harness is runnable against any store through a small adapter: github.com/DanceNitra/ramr.

Honest limits, stated plainly: small synthetic n=30, a single judge model, answer-level only, value-preserving echoes; the mem0 number is a point on a wide interval, not a verdict. Treat it as a demonstration of a real failure mode — and a reason to add one cheap regression test to your memory layer.

FAQ

Q: Is this saying mem0 is broken? No. mem0's add-based design keeps both values and reconciles at read time, which is a reasonable choice. The echo failure is a consequence of that tradeoff, measured in its own recommended config, reported with a confidence interval — not a bug claim.

Q: Why does embedding similarity fail here? Because a contradiction and a duplicate look almost identical to cosine (AUROC ~0.59). The restated old value is nearly identical to the original it revives, so a similarity threshold cannot separate "stale echo" from "same fact again."

Q: What is the fix? Deterministic supersession keyed on the value: once a value is corrected away, a restatement of it cannot revive it without an explicit reaffirm. This is AGM belief revision / bitemporal validity, not a new idea — but the adversarial restatement it defends against was unmeasured.

Q: What still fails? Value-obscuring restatements ("go back to what we had before") that never name the old value. There both value-match and cosine fail; the signal is the discourse relation, which needs a structural method.