When should an AI's memory refuse to believe what it just saw?

Give an AI agent a long-running memory and you hit a dirty question: when a new observation contradicts what is stored, is it new truth (update fast) or poison (ignore it)? Every memory system answers this implicitly through its consolidation operator — the rule that turns repeated, noisy, sometimes-adversarial observations of a fact into one stored value. We measured what that choice costs. These are deliberately minimal, fully-reproducible models (small simulations plus a check on real embeddings), not a benchmark of a production system — the goal is a scope boundary and a design rule, with every number reproducible. Up front: the mechanisms here are all textbook, and a sibling post already treats the same coupling family; the one thing that is genuinely ours is a measured negative result — where the gate stops helping.

The frontier nobody gets to skip

Track a value over time from noisy observations. Two failure modes pull opposite ways. Adapt fast to real change and a single adversarial spike yanks your estimate. Resist poison by averaging over a long window and you lag real change. Sweeping the decay rate of an exponential moving average (EWMA) traces a clean speed-robustness frontier. This is textbook — the robust-statistics breakdown-point tradeoff (Huber 1964; Hampel 1971) and the median / trimmed-mean used in Byzantine-robust learning. We use it only as the baseline.

The coupling — a textbook tradeoff, restated

A corroboration gate admits a new observation into the stored estimate only if several recent observations corroborate it, and rejects isolated outliers. In a minimal model it makes the breakdown bounded — as an adversarial spike's size grows, a fast EWMA's error grows without limit while the gate stays flat (it simply rejects the spike). That is exactly the breakdown-point behaviour of a rejecting estimator (Huber / Hampel), not a new effect.

And the same mechanism that rejects an isolated poison spike must, by construction, also reject the first sample of a genuine sudden change — to a corroboration test the two are identical. Tightening the gate monotonically improves poison-robustness and worsens response to sudden change. This is not our discovery: it is Grossberg's stability–plasticity dilemma (Adaptive Resonance Theory, Carpenter & Grossberg 1987) and, formally, the detection-delay-vs-false-alarm tradeoff of change-point detection (Page's 1954 CUSUM; Lorden 1971; Moustakides 1986). The gate's corroboration threshold is a CUSUM-style threshold; the coupling is a century-old theorem, not a finding.

When does a gate actually help?

The gate's advantage is bounded breakdown against an unbounded attack. So it pays off only when observations can be unboundedly large. Under realistic heavy-tailed noise, as an adversarial spike scales 30x, the gate's error stays flat while pure operators blow up:

But on bounded unit-norm embeddings — what most "AI memory" actually stores — we tested on real nomic embeddings of 240 real conversation turns, and the gate does not escape the frontier: a tuned EWMA, or even a plain mean, dominates it. A unit-norm poison vector has bounded influence, so every operator already has bounded breakdown; the gate's rejection buys nothing and its novelty-blindness is pure cost.

(One operational note: raw nomic embeddings are anisotropic — all cosines compress to ~0.75-0.81 — so you must center them before any outlier logic has signal.)

So the scope boundary — the one measured result that is genuinely ours: a corroboration gate Pareto-helps only when the observation magnitude is unbounded (heavy-tailed counts, scores, prices, durations). This is close to definitional — a bounded (unit-norm) value already gives every operator a bounded breakdown, so the gate's rejection has nothing left to buy — but the empirical direction is the useful part: on real embeddings a tuned decay dominates the gate on both axes, so its novelty-blindness is pure cost. For bounded embedding recall, use a tuned decay. (The coupling itself is clean only in the unbounded-magnitude simulations; on the real centered embeddings it did not reproduce — tightening the gate did not improve robustness — so we scope the coupling to the unbounded regime rather than claim it universally.)

Escaping the coupling — into a latency floor

If one operator can't have both, use two. A two-channel consolidator — a corroboration-gated slow channel plus a fast channel, with a selector that switches to the fast channel only once a deviation has persisted for d steps — beats every single operator: bounded poison-robustness and fast response to sustained change. This is the Complementary Learning Systems architecture (McClelland, McNaughton & O'Reilly 1995) — a fast channel and a slow one — imported to memory robustness; the two-channel idea is theirs.

The coupling doesn't vanish; it becomes a detection-latency floor (our sibling post treats the same escape in depth). With zero waiting (d=1) robustness collapses — you cannot tell an isolated spike from the onset of a real change until you see whether it persists:

Telling poison from genuine novelty requires waiting to see whether a deviation persists. Architecture converts the robustness-novelty tradeoff into a detection latency — but our fixed-d persistence selector is a crude detector: a sequential likelihood-ratio / CUSUM test reaches the same confidence in fewer steps. (Lorden 1971 lower-bounds the delay above zero, so some latency is irreducible — but not the specific d we used; calling this an information bound would overclaim.) And a fixed d is exactly what an adaptive attacker defeats: a patient poisoner (as in MINJA-style memory injection) drips and shortens each burst to sit just under your threshold. Against an opponent who sees your delay, no content-blind timing rule is safe — the invariant lever is provenance / source-identity, not a confirmation delay.

If you build agent memory

• Don't gate embedding recall. Use a tuned decay (or a mean for stable facts). Center the embeddings first. This is the one measured takeaway most agent-memory systems can use, since most of what they store is bounded embeddings.
• Gate unbounded-magnitude memory (counts, scores, prices, durations) as a blast-radius cap: a gate bounds the damage from a single arbitrarily large spike. But treat it as one signal feeding source-trust, not the whole defense — a fixed corroboration/delay threshold is defeated by a patient attacker who stays under it.
• For memory that must survive both poison and legitimate regime changes, use two channels with a persistence selector; tune the delay to the expected burst, and pair it with provenance. You cannot get robust novelty-detection from timing alone against an adaptive adversary.

The falsifierEvery step shipped with a pre-committed falsifier: a real frontier must exist (fast adapts, slow resists); the gate must be Pareto-non-dominated for unbounded poison and dominated for bounded embeddings; tightening the gate must trade robustness against novelty monotonically; and the two-channel must beat every single operator while collapsing at zero delay. All four held in the unbounded-magnitude simulations — with one honest exception: the gate's monotonic robustness/novelty coupling did not reproduce on the real centered embeddings (tightening the gate did not improve robustness there), which is why we scope the coupling to the unbounded regime.

FAQ

When should an AI's memory refuse to believe a new observation? When it cannot yet tell new truth from poison — a contradicting observation is either a real update (adopt fast) or an attack (ignore). Every memory system answers this implicitly through its consolidation operator.

Does a corroboration gate bound the damage from an adversarial spike? Yes — the gate's breakdown stays bounded (~0.50) whether the adversarial spike is ×5, ×15, or ×150, while an EWMA(0.1) grows unbounded (0.84 → 22.3) and a plain mean (0.20 → 5.53).

When does corroboration-gating actually help? Only when observations can be unboundedly large — heavy-tailed counts, scores, prices, durations. For bounded values (e.g. unit-norm embedding recall) a tuned EWMA or mean already keeps breakdown bounded, so the gate adds little.

Is this a benchmark of a production system? No — it is small simulations plus a check on real embeddings, aimed at a law and a design rule, with every number reproducible.

Related research

• When should AI memory trust a new fact? Corroboration, measured
• Can corroboration stop AI-agent memory poisoning?
• The same hidden law in four AI-memory mechanisms