A reality-check on agent-memory poisoning defenses: you price the residual, you don't close it

The trap. An AI agent that learns from its own memory can be poisoned: feed it a fact often enough, from enough angles, and it starts acting on that fact. The instinct is to stack defenses — trust the value only if several sources corroborate it, weight a memory by the agent's reputation, cap what an unproven memory is allowed to touch, add an anomaly monitor — and assume the layers multiply into a wall. We spent a week composing exactly that stack against a patient poisoner, then red-teamed every layer to destruction. The layers are real and each one helps. They do not multiply into a wall. Against an attacker who is willing to wait, they compose into a priced, appealable residual — and every claim we made about why they compose is textbook. That is the honest result, and it is more useful than a wall we can't build.

Not our idea — and it's important to say so loudly. "Redundant defenses don't fail independently, so you can't multiply their pass-rates" is the oldest lesson in dependability. Eckhardt & Lee (1985) and Littlewood & Miller (1989) formalized coincident failure: independently-built redundant channels fail dependently, so you cannot assume independence (only deliberately forced diversity can ever beat it — and it usually isn't there). The reputation half is settled too — Cheng & Friedman (2005) proved no non-trivial symmetric reputation function is Sybil-proof, Douceur (2002) is the Sybil impossibility itself, and Friedman & Resnick (2001) priced the "cheap pseudonyms" whitewashing tax. Capability ceilings are Saltzer & Schroeder (1975) least-privilege; the detection floor is Page (1954) / Lorden (1971) / Moustakides (1986) CUSUM; the deterrence math is Becker (1968); and "a detector keyed on a signal the attacker controls falls to an attacker who supplies that signal" is Carlini & Wagner (2017). We are standing on all of it. What's ours is only this: four composition claims checked against that literature, a small harness that shows the checks bite on real agent memory, and five runnable primitives built once we knew their limits.

We put the four load-bearing claims through a five-lens verification (Storm) before writing a word. All four came back correct — and textbook. That pairing is the point: a claim being textbook is not a reason to drop it, it is a reason to cite the prior art and never dress it up as new.

Claim 1 — "layering is superadditive" is backwards; the multiplicative product is the NULL. If two defenses failed independently, attacker success would be the product of the two pass-rates. That product is the independence null (Bliss 1939), not a synergy — "superadditive protection" would mean beating it, i.e. a residual below the product. Real defenses share failure modes, so coincident failure pushes you the wrong way, above the product. Calling a stack "superadditive" without a null is the single most common overclaim in defense-in-depth, and we caught ourselves about to make it. (Which null — Bliss-multiplicative or Loewe-additive — is itself unsettled in the combination-effect literature, unreconciled since the 1992 Saariselkä agreement; that is exactly why you must state one.)

Claim 2 — earned-reputation and authenticated-identity fail TOGETHER against a patient sleeper. One authenticated identity that quietly earns real standing satisfies both the identity gate and the reputation gate at once — so they are not two independent taxes, they are one common-cause axis. Our harness makes this concrete rather than proving it: with poison blended into LoCoMo memories (one poison model, one retriever) and forging its own corroborating source-count, most of the poison earns standing on its own genuine usefulness, and an attacker who pre-banks standing before turning erases the gate's benefit entirely. Neither is a surprising measurement — both are local confirmations of the adaptive-attacker canon: a gate keyed on usefulness cannot separate a useful poison from a useful-but-unproven legitimate memory, so it blocks them at roughly the same rate. There is no free separation, because standing tracks usefulness, not verified correctness. The only layer genuinely independent of a Sybil identity is the standing-independent capability ceiling — what an unproven-domain memory is structurally forbidden to touch, no matter how useful or well-reputed it looks.

Claim 3 — a per-action blast ceiling doesn't stop a slow multi-shot; the ceiling has to be cumulative. Cap each action and the patient attacker simply salami-slices: many individually in-bounds, in-domain actions that aggregate past an irreversibility threshold. Sub-threshold aggregation defeating per-event limits is a named pattern (salami-slicing; the same shape as low-and-slow fraud, and recently demonstrated against multi-turn LLM systems). So the ceiling must be enforced on cumulative per-source, per-domain influence — and the moment you watch a cumulative statistic you inherit the change-point detector's iron law: the detection-latency floor. CUSUM's average-run-length-vs-delay duality (Lorden 1971; exact under Moustakides 1986) says you trade false alarms against how long the slow attacker runs before you catch him. You cannot set that floor to zero.

Claim 4 — standing is a hot-path phenomenon, so most memory can never earn it. Only repeatedly-reused memory accrues reputation. Memory reuse is heavy-tailed — one-hit-wonders (objects requested exactly once) dominate a cache, and preferential attachment (Barabási–Albert 1999) concentrates reuse on a head — so a large, permanent tail of legitimate memory is structurally sub-standing and can only ever be governed by the capability ceiling, never by earned trust. A reputation-only defense silently abandons the cold tail, which is exactly where a fresh poison lives.

So what's actually left: a priced residual, not a wall. Put the four together and the honest picture is a detection-latency floor you can move but not close. A CUSUM-type monitor on cumulative per-source influence catches the careless poisoner, but its reference rate k is a tolerated-rate price, not a threshold you can wish to zero: an attacker who holds his per-source bad-rate at or below k gives the statistic no expected drift to accumulate, so its detection latency is unbounded — you only catch him by accepting unbounded false alarms. He simply nets a bounded k × exposure. Lowering k to chase him just buys those false alarms on your honest high-variance sources; it is Goodhart's law, and the threshold tells a salami attacker exactly how thin to slice. The lever that remains is Becker (1968) deterrence: make the expected penalty exceed the gain. That means the accrued standing itself has to be the bond — caught once, you forfeit all of it — so a patient sleeper can't out-earn a single bad act. You move the residual to a priced, tolerated rate behind a human reviewer. You don't close it.

The load-bearing substrate is provenance that survives transformation — old idea, still under-built. Every layer above assumes you can attribute a bad outcome back to the memory that caused it. But agents summarize, merge, and rewrite memories constantly, and naive provenance dies at the first summary. If a derived record doesn't inherit its parents' origins as a transitive taint, the patient attacker just launders his poison through one summarization step and the whole deterrence stack loses the defendant. Information-flow tracking and dynamic taint analysis are fifty years old (Denning 1976); what we hadn't seen shipped is transitive taint driving retroactive forfeiture inside an agent-memory core — not source-tagging you can read, but lineage that survives summarization and can revoke standing after the fact. A sharp reader (jacksonxly, in the r/RAG thread that started this) correctly named this as the part that mattered.

What we shipped — five primitives, each with its limit stated in the code. Textbook theory doesn't mean the plumbing exists; mostly it doesn't. Into our open-source memory core inspeximus we added: remember(derived_from=…) — transitive taint so a summarized slice still attributes to its origin (the substrate); influence_gate_report() — the standing gate, honest that its cost is density-dependent (it blocks a large fraction of legitimate memory in a sparse store and far less in a dense one) and oracle-dependent; slash(scope='source') — retroactive forfeiture of a caught source's entire accrued standing, making reputation the bond; restore() — the reversible appeal, because slashing on an imperfect detector is a weapon; and monitor(ids, outcome, k, h) — the per-source CUSUM whose cumulative breach is the detection event, with auto_slash defaulting off. That default is the whole lesson: a drifting base rate guarantees false alarms, taint means one false positive can nuke a downstream tree, and if the outcome signal is attacker-influenceable — a memory-injection attack like MINJA induces poisoned records through ordinary interaction, no backend access — an auto-trigger becomes a framing weapon (feed bad outcomes attributed via taint to a rival and you've auto-slashed the rival, which is RepTrap bad-mouthing with no human in the loop). Seventy years of automated-penalty systems converged on the same shape: automatic detection, human-reviewable reversible penalty, never auto-fired forfeiture.

The biggest caveat, named plainly: the whole deterrence stack is mortgaged on an outcome oracle most deployments don't have. Standing, CUSUM, slash, and the "pricing" that makes the residual appealable all need a labeled harm signal to meter. Strip that oracle — and in production, outcomes are usually sparse, delayed, or never attributed — and four of the five layers go inert, collapsing back onto bare provenance. A patient attacker doesn't beat the pricing; he chooses long-fuse, diffuse-harm payloads whose damage never resolves to an attributable event, and the meter never moves. Where there is no oracle, the defense that still works is a different kind: read-time certified robustness (RobustRAG-style isolate-then-aggregate; Xiang et al. 2024) bounds the influence of any k poisoned memories structurally, at retrieval, with no outcome signal and no pricing at all — it sits entirely outside the residual logic. The viable-defense space bifurcates on whether an outcome oracle exists.

Honest scope. We did not find a Sybil-proof blast cap — Cheng & Friedman say there isn't one, and our harness is one demonstration of why, not a proof. We did not benchmark the standing-independent domain ceiling; we argued it's the only Sybil-independent axis and shipped the gate, but a full ceiling that stays useful while blocking cold-tail poison is proposed, not measured — so we are not presenting it as the fix. Our numbers are one poison model on one benchmark with one retriever: illustrative existence-proofs, not rates you should carry to another setting. And the Matthew-effect framing is Merton (1968) sociology, an analogy, not a mechanism we measured.

Takeaway. Before you claim a layered defense "multiplies" against memory poisoning: name your null (it's the product, and coincident failure means you're below it), check whether two of your layers are really one Sybil identity, put the ceiling on cumulative influence not per action, remember the cold tail your reputation never reaches — and ask first whether you even have the outcome oracle the whole scheme needs. What you can build where you do is a priced, appealable residual on top of provenance that survives transformation; where you don't, reach for read-time certified robustness instead. We shipped the first set as runnable primitives in inspeximus, each with its limit written next to the code.

The falsifierIf, against a patient attacker who pre-earns standing and then holds his bad-rate at or below the monitor's k, any layer had produced a blast-reduction without a matching utility cost and at a forgeable identity, the "priced residual" thesis would be wrong — that would be the free separation Cheng & Friedman say can't exist. It didn't: the reduction tracked the utility cost, the pre-earner erased it, and the below-k attacker left the statistic with no drift to accumulate.

FAQ

Is "layered defenses multiply" ever right? Only if the layers fail independently — and redundant defenses almost never do (Eckhardt & Lee 1985; Littlewood & Miller 1989). The multiplicative product is the null you have to beat to claim synergy; coincident failure puts real stacks below it. State the null before you claim superadditivity.

Doesn't a corroboration or reputation gate stop poisoning? It raises the cost, it doesn't close it. Corroboration is forgeable by fresh sources; reputation is Sybil-earnable by being genuinely useful first, and a patient pre-earner defeats it entirely — the adaptive-attacker result (Carlini & Wagner 2017) and Cheng & Friedman (2005): no non-trivial symmetric reputation is Sybil-proof.

What is k and why can't you just lower it? k is the per-source bad-rate the CUSUM monitor tolerates before it starts accumulating. An attacker at or below k gives the statistic no expected drift, so detection latency is unbounded — k is a price, not a wall. Lowering it to chase him raises false alarms on honest high-variance sources — Goodhart, and it tells a salami attacker exactly how thin to slice.

Why does monitor() not auto-punish on a breach? Because the outcome signal can be attacker-influenceable (MINJA), and with transitive taint one false positive can nuke a whole downstream tree. Auto-firing forfeiture turns the detector into a framing weapon (RepTrap bad-mouthing). Detection is automatic; the reversible penalty needs a human, and restore() is one call.

What if my agent has no labeled outcomes? Then most of this doesn't apply to you — standing, CUSUM, slash and pricing all need a harm signal to meter. Reach instead for read-time certified robustness (RobustRAG-style isolate-then-aggregate; Xiang et al. 2024), which bounds a poisoned memory's influence at retrieval with no oracle. The defense space bifurcates on whether that oracle exists.

What's actually novel here? Not the theory — it's textbook composition, Sybil, change-point and adaptive-attacker results, all cited. What's ours is the demonstration that the checks bite on real agent memory, the "priced residual, not a wall" framing with the outcome-oracle caveat named, and five runnable primitives built on transitive taint that drives forfeiture — the plumbing, not the principle.