Multi-hop recall on LoCoMo: put the model in the retrieval loop
This applies a known agentic-retrieval recipe (the IRCoT / Self-RAG / PRISM family) to a memory benchmark. It is not a new method and not a SOTA claim. On LoCoMo multi-hop, naive flat dense retrieval gets the complete evidence chain 14.5% of the time at a 50-passage budget; a model-in-the-loop pipeline gets 56.5% at the same 50-passage budget. That 3.9× looks large mainly because the baseline is naive — a strong agentic baseline would start far higher. The real contributions are modest and honest: (1) a standard supporting-fact recall reported cleanly — full-evidence recall@50 on LoCoMo multi-hop, rarely broken out there — and (2) a cloud-free, reproducible harness.
Where this sits — prior art & an honest baseline
The mechanism is mainstream, not ours. "Let an LLM read its first results, name the missing fact, write a follow-up query, retrieve again, then rerank" is the well-established iterative / agentic retrieval line: IRCoT (Trivedi et al. 2022, ACL 2023), Self-RAG (Asai et al. 2023, ICLR 2024), and 2025 successors like PRISM, FAIR-RAG, FrugalRAG; the fusion step is RRF (Cormack et al. 2009) and the reranker is LLM-listwise (RankGPT-style). Those methods already report large multi-hop recall gains over strong iterative baselines (e.g. PRISM lifts MuSiQue passage recall from IRCoT's ~57% to ~83%). We claim no methodological novelty against that work.
Our baseline is deliberately naive, and that matters. The 14.5% starting point is single-shot dense top-50 by question similarity — not a strong agentic baseline like IRCoT. So the 3.9× should be read as "a known recipe fixes a weak baseline", not "we beat the state of the art." Whenever you see a big recall multiple, ask how strong the baseline was — ours was weak on purpose, to isolate the lever, and we say so.
What is genuinely useful here is narrow — and it is not the metric. Full-evidence recall (all the gold supporting turns retrieved) is the standard joint / supporting-fact recall from multi-hop QA (HotpotQA, Yang et al. 2018), just applied to LoCoMo, which is normally scored by end-task QA F1/judge. So the value is not a "new metric": it is (1) reporting that standard recall cleanly on LoCoMo multi-hop where it is rarely broken out, and (2) a cloud-free, reproducible harness (local embeddings) plus the honest 0.145 → 0.565 delta. That's the entire claim.
The numbers
Long-conversation multi-hop questions need to chain 2+ facts stated in different turns. We measured how often retrieval pulls all the gold evidence turns into a fixed 50-passage context, on LoCoMo (n=276 multi-hop, conversations ~597 turns). A diagnostic first: the gold turns are mostly present but rank-buried (recall@100 = 0.514, recall@150 = 0.641; mean gold size 3.17), so this is a ranking problem. Four compounding stages, all at the same 50-passage budget:
| stage | full-recall@50 (n=276) | × naive baseline |
|---|---|---|
| naive flat top-50 (dense, question only) | 0.145 | 1.0× |
| + LLM-in-the-loop follow-up queries | 0.297 | 2.0× |
| + RRF fusion over the follow-up queries | 0.326 | 2.25× |
| + LLM reranker (pick the chain from a top-100 pool) | 0.482 | 3.3× |
| + adaptive multi-round + rerank a deeper pool | 0.565 | 3.9× |
Reading the stages: round-1 retrieves top-15 by question similarity; the model names the missing bridge entity and emits 1–2 follow-up queries; RRF over those queries (dropping the original question ranking) fuses best; an LLM reranker selects the evidence chain from a top-100 pool; a second round names what's still missing and reranks a deeper pool. (Aside we'll keep: asking for more queries hurt — it dilutes the fusion; 1–2 sharp bridge queries were optimal, observed across the fusion runs, not tabulated here.)
Why the within-result comparison is fair
- Equal final-context budget. Every row returns exactly 50 passages, same metric — so the lift is not a bigger returned context. It is not free either: stages 1/3/4 spend extra per-query LLM calls and the rerankers inspect a top-100 / deeper pool, so the gain buys "look at a deeper pool + spend more compute, then compress to 50" — a real cost/recall knob, not equal compute.
- No leakage. The in-loop reader sees only the question and the already-retrieved turns — never the gold. The follow-ups are inferred.
- Judge-free. Full-evidence recall is exact set-containment against LoCoMo's annotated evidence.
The honest caveat that matters mostThe headline multiple is only as impressive as the baseline is weak — and ours is weak by design. The fair next test, which we have not run, is the same pipeline against a strong agentic baseline (IRCoT / PRISM-class) on this exact metric, ideally at a fixed token budget. Until then, treat this as "a known recipe + a standard recall reported cleanly on a real, unsaturated frontier (LoCoMo multi-hop)," not as a method that beats the field.
Other caveats: absolute recall is ~57%, not solved; stages 1, 3, 4 are per-query model calls — a real cost/recall knob, not free; one dataset (two-person conversations, where the bridge entity is a name and relatively easy to surface). And the caveat that matters most: recall is a proxy, and we did not measure whether it moves the end task. We never checked whether lifting full-evidence recall 0.145 → 0.565 improves answer accuracy — multi-hop questions are often answerable from partial evidence or parametric knowledge, and supplying the complete gold chain caps multi-hop accuracy near ~0.66, so the recall→accuracy transfer function is unmeasured and may be well below the recall gain. Every number comes from a runnable experiment — the harness + result files are at research/probes/locomo_multihop_recall.
FAQ
Does putting the model in the retrieval loop improve multi-hop recall? Yes. On LoCoMo, full-evidence recall@50 rises from 0.145 (naive single-shot dense top-50) to 0.565 via a model-in-the-loop pipeline — all at the same 50-passage budget. That 3.9× looks large mainly because the baseline is deliberately naive.
Is this a new method or a SOTA result? No. Iterative, model-driven retrieval is mainstream — IRCoT (Trivedi et al. 2022, ACL 2023) and successors like PRISM (which lifts MuSiQue passage recall from IRCoT's ~57%→83%). The contribution is a standard supporting-fact recall reported cleanly on LoCoMo (usually scored by QA F1) plus a cloud-free reproducible harness, not a SOTA claim.
What is full-evidence recall@50? Whether all the gold supporting turns for a multi-hop question land in the top-50 retrieved passages (mean gold size 3.17, n=276). LoCoMo is normally scored as end-task QA F1/judge, so this exact retrieval recall is rarely broken out.
What actually drove the gains? Four compounding stages at a fixed 50-passage budget: LLM-in-the-loop follow-up queries (0.297), RRF fusion over those queries (0.326), an LLM reranker picking the chain from a top-100 pool (0.482), and adaptive multi-round rerank of a deeper pool (0.565). One or two sharp bridge queries are optimal — more dilutes the fusion.