Our grounding 'firewall' for confidently-wrong AI turned out to be mostly an artifact
The problem is real and worse than we thought: given a document that asserts a false answer, LLMs follow it - and being a newer or stronger model does not protect you: two frontier reasoners follow it far more than a weak 7B (glm-5.2 22/24, deepseek-v4-pro 20/24, versus 5/24) - though a code-tuned model (kimi, 4/24) resisted, so susceptibility tracks context-faithfulness, not raw capability. But our 'grounding firewall' - abstain when the answer is sensitive to deleting the document - was measured in a regime that flattered it: the model already knew every fact, so the sensitivity signal and the 'wrong' label are the same number. The idea is also textbook, and it flags grounding, not lying. The real defense is provenance and corroboration.
The problem is real - and frontier models are not spared. Give a model a retrieved document that asserts a false answer - a poisoned context: stale data, an injected line, a wrong source - and it can follow it at full confidence. We measured this on 24 basic facts. With no context, every model we tried answers all 24 correctly. Add a document that says “the answer is <the false option>” and the most context-faithful models flip on most of them:
| Model | Followed the poisoned document (of 24) |
|---|---|
| qwen2.5-7B (the post's model) | 5 / 24 |
| kimi-k2.7-code | 4 / 24 |
| deepseek-v4-pro | 20 / 24 |
| glm-5.2 | 22 / 24 |
Every model had 0 errors with no context, so this is not ignorance - it is deference. Two frontier reasoners (glm-5.2, deepseek-v4-pro) followed a false document - in both answer-order presentations - on 20-22 of 24 facts, far more than the weak 7B (5/24); a code-tuned model (kimi) resisted (4/24), so the effect is not simply “bigger is worse.” Following a trusted-looking retrieved document is exactly what context-faithful models are trained to do, which is why context poisoning works so well (PoisonedRAG: ~90% attack success from five injected texts). And confidence does not warn you - the model is most sure precisely when it has adopted the poison. (Runnable probe.)
What we tried - and why it does not hold up. Our idea was a “grounding firewall”: measure how much the answer depends on the document rather than how sure the model sounds. For an answer, sensitivity = | p(answer | context) - p(answer | context removed) |; abstain when sensitivity is high. On the 24-fact poisoned set (open model qwen2.5-7B, where token probabilities are available) the grounding signal predicted correctness far better than the model's own confidence (risk-coverage AUC 0.028 vs 0.095, where lower is better). We now think that number is mostly an artifact, for four reasons.
1. Saturated prior. The model already knew every fact (p(answer | no context) = 1.00 on all 24; and every model above scored 0 prior errors). When the prior is 1, sensitivity = 1 - p(answer | context) - which is the same quantity as “the model followed the poison.” So the firewall beats confidence by algebra, not by insight. With n=24, 5 wrong answers, and a single “confidently-wrong” case, there was never a real test. (Probe: the saturated-prior confound and a signed-margin baseline.)
2. It flags grounding, not lying. A high-sensitivity answer is one that depends on its context - which is exactly what you want in correct RAG. It looks like a firewall here only because every context is poison. On a realistic corpus, where most retrieved documents are correct, abstaining on high sensitivity means abstaining on your best grounded-correct answers.
3. The comparison was folded. Confidence = max(p, 1-p) throws away the direction; sensitivity keeps it. In this regime the honest signal is just the signed margin toward the poisoned option - one model call, no context-removal call needed.
4. It cannot run where the problem is worst. The method needs token logprobs. The strong cloud models (glm-5.2, deepseek-v4-pro, kimi) do not expose them, and neither do closed APIs like Anthropic - so our earlier claim that it “works on closed APIs where you only see inputs and outputs” was wrong. The firewall runs only on the weak open model, where it is an artifact.
This is textbook
Ablating the context to see how much an answer depends on it is already published: ContextCite (NeurIPS 2024) attributes a generation to context by ablation and explicitly demonstrates detecting poisoning; ReDeEP/SEReDeEP score external-context versus parametric-knowledge reliance. Abstaining below a confidence threshold is Chow's reject option (1970); “delete a data point and watch the answer move” is Cook's distance (1977) and influence functions. The logprob-free way to do black-box checking is SelfCheckGPT. We did not invent this.
What actually helps
Against a poisoned document the deployable defenses are provenance and corroboration - do not let a single unverified source drive an answer - plus consistency/NLI checks that need no logprobs. And if the sensitivity idea is to carry real information, it must be stratified by prior entropy: it can only add to confidence when the model is genuinely unsure, which is exactly the case this experiment never tested. That stratified test - sensitivity versus a signed-margin baseline, on a mixed clean+poison corpus, in the uncertain-prior cell - is the honest next experiment.
The falsifierOn the all-poison, known-answer set the “firewall beats confidence” result is not informative, because sensitivity and the wrong-label are the same quantity there. The claim survives only if sensitivity beats a signed-margin baseline on a mixed clean+poison corpus with uncertain priors - which we have not shown.
Honest scope. The original version of this post framed the grounding signal as a working “firewall” that “catches what confidence cannot.” A full audit found that headline was mostly a measurement artifact and prior art. What stands: LLMs - especially the strongest, most context-faithful ones - over-trust poisoned retrieved documents, and confidence does not warn you. All numbers here are from runnable probes (poison-following, the confound), not field data.
FAQ
Do LLMs really follow a poisoned document? Yes, and it does not spare frontier models. With no context every model we tried answered all 24 facts correctly, but a document asserting the false answer flipped glm-5.2 on 22/24 and deepseek-v4-pro on 20/24 - far more than a weak 7B (5/24), though a code-tuned model (kimi) resisted (4/24). Context-faithful models defer to retrieved text - the PoisonedRAG threat.
Does the “grounding firewall” catch it? Not really. In our test the model already knew every fact, so the sensitivity signal equals the “followed the poison” label by construction; the apparent win over confidence is an artifact of that saturated-prior setup, not a working detector.
Is the idea new? No. Ablating context to attribute an answer is ContextCite (NeurIPS 2024); the reject option is Chow (1970); leave-one-out influence is Cook's distance (1977).
Does it need model internals? It needs token logprobs, which closed APIs (Anthropic) and strong cloud models (glm-5.2, kimi) do not expose - so it cannot run on the models where poisoning is worst. A logprob-free consistency check (SelfCheckGPT) is the black-box option.
What actually reduces confidently-wrong answers under poison? Provenance and corroboration - do not trust a single unverified document - not a confidence score; and stratifying any sensitivity signal by whether the model was already sure.