1. After a correction, does a restatement bring the old value back?
Correct a fact, then have someone assert the retired value again in different words — not a byte-for-byte replay, a paraphrase, which is what actually happens in a conversation. The number is how often the store then serves the stale value. Lower is better. Three independent models generated the paraphrases; n=39 per arm.
| resolution policy | deepseek-v4-flash | kimi-k2.7-code | glm-5.2 | mean |
|---|---|---|---|---|
| inspeximus | 0.128 | 0.154 | 0.162 | 0.148 |
| mem0 (faithful policy) | 1.000 | 1.000 | 1.000 | 1.000 |
| Graphiti (faithful policy) | 1.000 | 1.000 | 1.000 | 1.000 |
| MemStrata (faithful policy) | 1.000 | 1.000 | 1.000 | 1.000 |
| inspeximus 0.6.8 — our own older version | 0.436 | 0.436 | 0.486 | 0.453 |
| similarity only | 0.359 | 0.333 | 0.378 | 0.357 |
| last-mention-wins | 1.000 | 1.000 | 1.000 | 1.000 |
Last-writer-wins and bitemporal resolution both return the retired value every single time. That is not a bug in either — it is what those designs say to do: the newest assertion wins, or the edge with the later validity time wins, and a paraphrased echo IS the newest assertion. inspeximus records the superseded object against its (subject, relation) key, so a later assertion of an already-retired value is recognised whatever words carry it.
The italic row is our own version 0.6.8, which fails a third of the time: this is the same panel we use to attack ourselves, which is why we trust it pointing the other way. The panel measures each system's documented resolution logic, which isolates the supersession rule and grants every policy oracle value-extraction so it is not measuring extractor quality — see the live run below for the same question put to the real software.
1b. The same attack, against the live products
The section above models each system's documented logic. The obvious objection is "that is not our product, that is your model of our product", so here is the same procedure run against the real software: write a fact, correct it, restate the retired value in different words, then ask each system at its own retrieval surface and read rank 1 — what an agent would actually act on. Each system in its shipped product configuration. n=8.
| system | serves the retired value | n |
|---|---|---|
| inspeximus (product surface) | 0.000 | 8 |
| mem0 2.0.14 | 1.000 | 8 |
| Graphiti | NOT RUN — no NEO4J_PASSWORD configured; a graph database is required | |
mem0 does not overwrite the corrected fact — it keeps both and ranks the stale one first. Searching the corrected fact returns "…is db-old-07" at 0.872 and its own "changed from db-old-07 to db-new-12" at 0.828, so rank 1 is the retired value. Graphiti is reported as NOT RUN, not estimated: it needs a graph database we could not reach, and carrying the modelled number over and relabelling it "live" would be the exact overclaim this run exists to remove.
And the configuration matters, so here it is. Measured through the product
surface — the MCP server, the CLI, the editor plugin — inspeximus scores 0.000. Constructed as a
bare library object before version 1.87.0 it scored 1.000, because the echo guard shipped off for
byte-identical legacy compatibility. That is not a footnote we polished: it is why the guard is now on by
default in the library too. Set echo_guard = False and you get the old number back.
2. Erasure that reaches the stores we do not own
Every memory library shares a blind spot: delete a subject from the store and the copy your application embedded into its own vector index is untouched. Measured on ourselves:
| configuration | residue in the app's index | receipt |
|---|---|---|
| store-native delete only (the category default) | 8/8 | — |
| wired to a registered target | 0/8 | manifest complete 8/8, chain verifies |
| wired, but the integration is broken | 8/8 | falsely-complete receipts 0/8, leak named 8/8 |
The third row is the point. A broken integration still leaks — no library can reach into
a store it was never given — but it cannot produce a clean receipt while leaking. The claim is not
"we erase everywhere"; it is that we will not tell you an erasure was complete when it was not. Since
this release forget_subject() returns a coverage field stating exactly that,
and with nothing registered it says so outright rather than returning a bare success.
3. Where we do not win
On each system's own retrieval surface, after its own native delete, the subject's value is equally unrecoverable. Same procedure, same data, no rubric of ours, no LLM judge.
| system | residual recoverability | rate |
|---|---|---|
| inspeximus 0.7.21 | 0/8 | 0.00 |
| mem0 2.0.11 | 0/8 | 0.00 |
A tie, and it belongs on this page: a comparison that reported a win on every axis would tell you more about the author than about the systems. Graphiti is bitemporal by design and retains invalidated facts as history, which is a different trade-off on this axis rather than a worse score.
4. Verifiable without trusting anyone
Read live from PyPI when this page was generated. Hard dependencies exclude optional extras.
| package | version | hard deps | licence | python |
|---|---|---|---|---|
| inspeximus | 1.86.0 | 0 | MIT | >=3.8 |
| mem0ai | 2.0.14 | 8 | Apache-2.0 | <4.0,>=3.10 |
| graphiti-core | 0.29.3 | 7 | Apache-2.0 | <4,>=3.10 |
A deployment property, not a quality claim: fewer dependencies does not mean better recall, and all three are permissively licensed.
What is not measured here
- Retrieval accuracy on a shared corpus. No symmetric cross-system measurement exists yet, so treat any accuracy ranking between these three — including one in our favour — as unsupported.
- Graphiti on the erasure cells. It needs a graph database we did not stand up for this run, so it is absent rather than estimated.
- Cost and latency. Not measured at all.
- Graphiti live. Sections 1b and 3 could not reach a graph database, so it is absent from both rather than estimated. Configure one and the harness fills the cell in.
Check it yourself
Each cell is one file: echo_attack_probe_v2.py, erasure_manifest_wired_cell.py, erasure_completeness_xsystem.py. Each states its own scope limits in its header, above any result. The replication ledger behind this project's other claims is the Crucible (DOI), and what we have retracted is on the track record.