Does long context kill RAG? We ran it — and both viral camps are wrong
A loud fight keeps recurring: one camp says "with a big enough context window you don't need retrieval — dump everything in, the fact must be somewhere"; the other says "long context rots — quality collapses past ~100k tokens." We ran a judge-free probe to settle it. The answer is neither: it's the task type, not the length. Single-fact lookup is perfect even at 110k tokens; read-everything aggregation collapses by ~25k and stays collapsed. So "RAG is dead" is false for anything that combines facts, and "context rot kills recall" is too strong for simple lookup. This replicates a known effect (lost-in-the-middle, context rot, RULER, NoLiMa) — the point is a clean, reproducible, neutral adjudication of the viral fight.
The fight
Three mutually-contradictory positions go viral, none of them neutrally benchmarked:
- "Don't do RAG / CAG" (Chan et al. 2024): preload the whole corpus into a long context; "it must be somewhere", so retrieval is obsolete. (The CAG paper itself scopes this to small, constrained knowledge bases — the viral version drops that caveat.)
- "Context rot": effectiveness drops sharply past ~100k tokens, regardless of the advertised window.
- "RAG still wins": brute-force context-stuffing loses to a well-built retriever on accuracy, cost, and speed.
They can't all be right. The disagreement is testable — so we tested it.
The test (smallest runnable model, judge-free)
A synthetic structured log of N records (region, shipments, status). Two task families on the same log at four growing lengths (~5k → ~110k tokens):
- NEEDLE — single-record lookup ("how many shipments did Record 04210 report?"). What CAG predicts stays easy.
- SYNTH — read-everything aggregation (counts, max-by-region, filters). What context-rot predicts collapses.
Exact gold, no LLM judge (set-containment / integer match). The reader is a frontier long-context model (the exact model + date for this run were not logged — a reproducibility gap; the probe regenerates so you run your own), and it answered from reading the log into its context — not by grepping or running code (that's enforced, and the result self-validates: a script would make aggregation perfect too).
The result
| context length | NEEDLE (lookup) | SYNTH (aggregation) |
|---|---|---|
| ~5k tokens | 1.00 | 0.75 |
| ~25k | 1.00 | 0.375 |
| ~60k | 1.00 | 0.25 |
| ~110k | 1.00 | 0.375 |
Needle recall is perfect at every length, including 110k. Synthesis collapses to ~0.25–0.38 by 25k and stays in that band (no clean length trend within it — the 0.25↔0.38 wiggle, including 110k ticking up, is within the n=8 noise). Tellingly, at 110k the model itself reported it "could not reliably aggregate by mental tally over a log this size" — which is exactly the failure, and confirms it answered from the context rather than a script. Two nuances the per-question data shows: (1) it is exhaustive aggregation — counting and filtering across all records — that collapses; finding a single extreme (the maximum) often survives even at 110k, because that is effectively one pass, not a full tally; and (2) even at 5k synthesis is only 0.75, not 1.00, so part of the failure is exhaustive mental-arithmetic the model can't do at any length — long context makes it worse, it doesn't cause it. So the failure is specifically combine-everything, not scan-for-one — and it is as much an aggregation-arithmetic limit as a "context rot" one.
The verdict
FAILED — for CAG's "it must be somewhere ⇒ correct answer", on synthesis. Long context does not replace retrieval for anything that must combine information across the window; that breaks early and stays broken. But needle lookup survives long context fine, so the blanket "context rot kills recall past 100k" is also too strong. The honest rule: it's the task, not the length. Use long context for find-the-fact; use retrieval (or structured decomposition) for combine-the-facts. And for exhaustive count/filter aggregation the real fix is neither long context nor retrieval but a tool — a code interpreter or a query over the data — because no model tallies thousands of records in its head reliably at any length. Both viral headlines conflate all of this.
Why this is a fair, valid probe
- Judge-free — exact gold, no LLM grading.
- Self-validating — needle staying flat at 1.0 while synthesis collapses proves the model answered from its context; a grep/script would make synthesis perfect too.
- Same model, same data, only length varies — so the effect is the length×task interaction, not a model or prompt artifact.
Honest limitationsWe replicate a known phenomenon (lost-in-the-middle, Liu et al. 2023/TACL 2024; RULER, Hsieh et al./NVIDIA 2024; NoLiMa, Modarressi et al. 2025; and Chroma's "context rot", 2025 — an industry report, not peer-reviewed) — no new discovery. Scope: one frontier model, synthetic structured haystacks, lengths up to ~110k (not the 1M the loudest claims invoke), and only n=8 synthesis questions per length — so at this n even the 0.75→~0.3 drop is suggestive on its own (Fisher's exact ≈ 0.13, not significant); it is credible mainly because it reproduces the heavily-replicated lost-in-the-middle / RULER / NoLiMa effect. The contribution is the neutral, runnable, judge-free adjudication of the viral fight, not the underlying effect.
The whole probe is one generator + a length-swept reader + a scorer; the haystacks, questions, and gold regenerate from one command, and the generator + independent gold-re-derivation scorer are public at research/probes/ragdead. If you want to know where your model's aggregation breaks, run it on your model and your data — that's the point.
FAQ
Does a bigger context window replace RAG? No. In our test, single-fact lookup (the needle) stays perfect at 1.00 at every length, but synthesis/aggregation accuracy collapses from 0.75 to ~0.3 as context grows 5k→60k tokens. “Dump everything in the prompt” fails for any question that aggregates across the context.
What is context rot? The measured degradation of a model’s ability to use information as the input grows — even when the needed fact is present. We reproduce it: retrieving one fact survives long context, but reasoning that combines many facts degrades steeply.
Is RAG still needed with long-context models? Yes, for aggregation and synthesis workloads. Long context handles single-fact lookup fine; RAG protects multi-fact reasoning by keeping the working set small instead of forcing the model to aggregate over a huge window.
At what context length does aggregation break? In our runnable, judge-free probe the collapse is already steep by ~25k tokens — earlier than the “~100k” folklore. The exact breakpoint depends on your model and data; the probe regenerates from one command so you can measure your own.