Why similarity-only RAG serves stale facts: the supersession blind spot, reproduced

The short version. A memory that ranks by embedding similarity alone has no model of time. When a fact is updated (a function renamed, a price changed, an API key rotated), the old value and the new one sit at nearly the same point in embedding space. Cosine search cannot tell which is current, so it silently returns the stale one. The counterintuitive part, and the reason a better embedder won't save you: a contradiction — the new value — is often more embedding-similar to the original than a faithful rephrase is. We independently reproduced this and re-measured the fix.

This is a credited replication, not a discovery. The supersession blind spot was named and measured by Neeraj Yadav (MemStrata, arXiv:2606.26511, a June 2026 preprint), who reports a cosine classifier separating current from superseded facts at AUROC ~0.59 — near chance — and proposes the deterministic-key fix that takes stale-serving to ~0%. Both halves are his. We re-ran it on a different stack and shipped a zero-dependency open-source implementation of his fix in inspeximus — a runnable receipt, not a new idea. And the underlying facts are textbook: embeddings are trained for topical similarity and are famously negation-insensitive (the STS-vs-NLI gap), so a one-token value flip barely moves cosine; the fix is the classic bi-temporal / slowly-changing-dimension pattern (Snodgrass; Kimball SCD Type 2), which production agent-memory like Zep/Graphiti already ships.

Scope, so nobody has to say it in the comments. This is the failure mode of a similarity-only store. Real systems already sidestep it — upsert-by-id, metadata/recency filters, bi-temporal validity windows. The narrow, quantified point is that the cosine signal itself carries no supersession information, so anything relying on similarity alone to pick "the current fact" is structurally blind — you have to add a key or a time layer, which is exactly what those systems do.

The claim, replicated

On 24 (subject, relation, object) facts embedded with local nomic-embed-text (mean-centered, because nomic is anisotropic):

The AUROC is the whole story. A contradiction — the new value — is often more embedding-similar to the original than a genuine rephrase is, so no similarity threshold can reliably flag "this record supersedes that one." What reproduces here is the direction, not a precise value: at n=24 the AUROC's 95% CI is roughly [0.42, 0.81] — our 0.613 and Yadav's 0.59 are both statistically indistinguishable from chance (0.5), which is the point. Verdict: REPRODUCED (the mechanism, on synthetic data — not a benchmark rate).

What it costs you

Store the original, then the update, then ask for the current value:

A similarity-only store serves the superseded value a large fraction of the time (here ~40%, with a wide small-n band). That is not a tuning problem — the similarity signal is structurally blind to it. The 0% is not a measured surprise: a key that retires the old value can never serve it, by construction.

The fix: a key, not a threshold

Similarity fails because it answers the wrong question. Supersession is not "are these two texts similar?" — it is "do these two records describe the same (subject, relation)?" That is a deterministic key, not a distance. When a new value arrives for an existing (subject, relation) key, retire the old one — no embedding, no LLM, no threshold.

This is exactly the fix Yadav's paper proposes (a deterministic (subject, relation, object) rule that retires the stale value in a bi-temporal ledger). It is also, older still, the classic slowly-changing-dimension / bi-temporal pattern from data warehousing (Kimball SCD Type 2; Snodgrass). We ship a zero-dependency implementation of it in inspeximus v0.2.0:

remember("Billing API auth method: API keys", key="billing-api::auth")

retires every active record with that key, so recall never returns the stale value. It is bi-temporal — a back-filled earlier value cannot overwrite the current one — and append-only: the old value is demoted, not deleted (still there with include_superseded=True). Our contribution here is the runnable open-source receipt, not the mechanism — the mechanism is Yadav's, and the pattern is decades old.

Honest limits

• Synthetic, 24 facts, one embedder, no significance. This characterizes the mechanism, not a product benchmark. AUROC ~0.61 is a point estimate whose CI straddles chance; the stale-fact rate is 10/24 with a wide band. Both move with your data and embedder — the probe is one file, re-run it on yours.
• The near-chance AUROC is partly by design. The contradictions are minimal one-value edits (high overlap) and the rephrases are surface reworks, which is why cosine can't separate them — it's a faithful demonstration of the STS-vs-NLI gap, not a discovered empirical rate.
• The fix is a key, and 0% is definitional. Upsert-by-key can never serve the old value by construction — the interesting number is the 40%-ish that similarity-only pays, not the 0%. It only applies to facts you can assign a (subject, relation) to (config, prices, versions, status, identities). For free-text memory with no natural key you are back to the hard retrieval-side problem Yadav's paper attacks.

FAQ

Does a better embedding model fix stale-fact retrieval? No. The problem is structural: a contradicted fact is often more embedding-similar to the original than a rephrase is (we measured AUROC 0.61, near chance). A stronger embedder shifts the similarity band, not the blind spot.

What is supersession in AI memory? It is when a stored fact is replaced by a newer value — a renamed function, a changed price, a rotated key. A memory with no model of time keeps both and cannot tell which one is current.

How do you fix it without an LLM? Assign facts a deterministic (subject, relation) key and retire the old value when a new one arrives — no similarity threshold, no model call. This is the fix Yadav's MemStrata paper proposes, and it is the classic bi-temporal / slowly-changing-dimension pattern; a key that retires the old value serves 0% stale by construction.

Is this your discovery? No. Neeraj Yadav's MemStrata preprint (arXiv:2606.26511) both names the blind spot and proposes the deterministic-key fix. We independently reproduced the direction (our AUROC ~0.61 and his ~0.59 are both indistinguishable from chance at small n) and shipped an open-source implementation of his fix in inspeximus. Our part is a replication receipt and runnable code, not the idea.

Related research

• Your RAG store is rotting — the fix is what you keep
• Does long context kill RAG? We measured it
• What should an AI agent forget? Two tiers beat every single rule