Agent-memory retrieval, measured: recency 0.024, a vector DB ties BM25, the cheap hybrid wins

What we measured. "Agent memory" tooling has quietly inherited the web-RAG default: embed everything, put it in a vector database, retrieve by cosine. We ran the cheap, self-hostable options head-to-head on a real multi-session memory benchmark and mapped where each one fails. Three things held up under cluster-aware statistics: the recency / "last-N" window that many agent frameworks actually ship is catastrophic (recall@20 of 0.024); a single vector index — even with a strong embedder — does not beat a zero-dependency BM25 (it ties); and the cheap BM25 + embedder hybrid robustly beats every standalone retriever. None of this is new information retrieval — it reproduces BEIR's "BM25 is a strong baseline" lesson on agent-memory data — but the runnable receipt, and the recency number, are worth having.

The setup. LoCoMo is a very-long-term multi-session conversational-memory benchmark. The publicly released 10-conversation dataset has 5,882 dialogue turns and ~1,986 questions (841 single-hop, 282 multi-hop, 321 temporal, 96 open-domain, plus ~446 adversarial); we score the 1,531 answerable questions whose gold-evidence turns are present in the transcript. Each conversation is one user-pair's full multi-session history (~590 turns); we index and retrieve within a conversation — i.e. one user's memory store, recalling across that user's own past sessions. For each question we retrieve turns and measure recall@20 — the fraction of that question's gold-evidence turns that land in the top 20 — and full-evidence recall (all gold turns in the top 20). Six retrievers, all self-hostable, all vanilla:

• recency — the 20 most-recent turns, query-blind (the "just keep the last N" default)
• BM25 — zero-dependency lexical ranking (k1=1.5, b=0.75)
• nomic — nomic-embed-text cosine, run correctly with its required search_query: / search_document: prefixes
• mxbai — mxbai-embed-large cosine, a strong open embedder, with its retrieval query prompt
• hybrid_nomic / hybrid_mxbai — Reciprocal Rank Fusion of BM25 with each embedder

(An earlier draft of this run embedded nomic without its task prefixes — a configuration bug an adversarial re-audit caught before publication. Running it correctly closed most of the gap, which is exactly why the corrected story below is "tie," not "BM25 wins.")

The measurement (recall@20).

Read these as relative comparisons on a hard task, not a solved one: in absolute terms even the winning hybrid recovers only ~61% of evidence turns at k=20, and the complete evidence set for just ~55% of questions (full-evidence recall@20 ≈ 0.549). Turn-level retrieval on multi-session memory is far from solved — the question here is which cheap option is least bad, and why.

Three results stand out:

1. 1. Recency is a cliff, not a baseline. At 0.024 recall@20 it is ~23× worse than BM25 and loses in all 10 conversations. The "remember the last N messages" pattern that ships in a lot of agent scaffolding is, for multi-session recall, close to retrieving nothing — the evidence you need is scattered across old sessions, exactly where a recency window cannot see it. This is the least surprising result in principle and the most ignored in practice.
2. 2. A vector DB does not beat BM25 here — it ties. With the strong embedder, mxbai (0.526) versus BM25 (0.552) is not a significant difference (paired Wilcoxon p = 0.36; conversation-level 95% bootstrap CI on the gap includes zero). "You need a vector database for agent memory" is not supported as a standalone claim on this benchmark. Where embeddings do look worth their cost is multi-hop questions (mxbai 0.313 vs BM25 0.241 — a directional per-category gain we did not separately significance-test) — the semantic-matching regime — while lexical wins on entity/temporal recall.
3. 3. The cheap hybrid wins, robustly. BM25 + mxbai (0.609) beats BM25 alone by +0.057, with a conversation-level bootstrap CI of [+0.039, +0.076] (excludes zero) and a win in 9 of 10 conversations. Fusing a lexical and a semantic channel recovers what each misses. Notably this needs only a small local embedder, not a bigger one: hybrid_nomic (0.604) ≈ hybrid_mxbai (0.609).

At the budget you actually have (k=3–5)

recall@20 is a fair retrieval ceiling, but an agent rarely spends 20 chunks of context per turn — in practice the budget is k≈3–5. So we report the smaller cutoffs too, and the picture sharpens:

The hybrid's edge moves in opposite directions against the two baselines as k shrinks. Against the single vector index it widens (+0.083 → +0.109 → +0.118 at k=5 — missing the one exact-token hit hurts most when you can only keep five chunks). Against BM25 alone it shrinks (+0.057 → +0.040 → +0.012 at k=5): at the realistic budget, BM25 by itself is essentially the hybrid. So shrinking k makes the conclusion more BM25-first, not less — the embedder's marginal value drops as the budget tightens. Recency stays ~0 throughout (0.002 at k=5).

Why the lexical channel is so strong here

LoCoMo is conversational self-narrative: people reuse the same names, dates, and event words across sessions, so a question and its gold-evidence turn usually share surface vocabulary. That is the best case for lexical search and a demanding test for pure semantics — which is precisely why BM25 is hard to beat and why the hybrid's gain comes from the minority of questions (multi-hop, paraphrase) where lexical overlap breaks down. It also reconciles a result we previously measured — that naive RRF does not help when one channel already dominates with a good embedder: fusion pays off only when the two channels are complementary and comparably strong, which is the regime LoCoMo sits in and a single-embedder web-RAG corpus often is not.

The statistics (because 1,531 questions live in only 10 conversations)

Point estimates to three decimals would overstate the certainty: the 1,531 questions are nested in 10 conversations, so they are not independent. We therefore report, against BM25: a paired Wilcoxon signed-rank test per question; a per-conversation win-rate over the 10 clusters; and a 95% bootstrap CI on the per-conversation mean delta. The honest summary: recency loses (0/10, CI far below 0); nomic and mxbai are statistically indistinguishable from BM25 at the conversation level (CIs include 0); both hybrids beat BM25 (9–10/10, CIs exclude 0). The strong claims are the recency cliff and the hybrid win; "vector beats lexical" is not a claim this data supports.

What to do instead

For self-hosted agent memory, the cheap layered stack beats reaching for a bigger model:

1. 1. Don't make recency your retrieval. A last-N window is fine as a recency bias on top of retrieval, never as the retriever — on multi-session memory it recalls almost nothing.
2. 2. Start with BM25, add the embedder as a hybrid. Lexical-first costs a text index (no model, no GPU, no stored vectors); the embedder then buys a robust +0.057 in fusion, with a small local model. A bigger embedder was not the lever here; the second channel was.
3. 3. Add a freshness layer separately. Retrieval recall is not the whole story for memory: similarity cannot tell a superseded fact from its replacement (measured separately — see the supersession post below — a vector store served the stale value about 42% of the time, AUROC ≈ 0.6 for the stale-vs-fresh decision). Currency is a deterministic (subject, relation) supersession problem, not a retrieval one — keep it out of the embedder.

Why it matters. The reflexive "spin up a vector DB" answer for agent memory is, on this benchmark, neither the cheapest nor the most accurate option — and the recency default that many frameworks ship is far worse than the lexical index they skipped. The win is boring and cheap: lexical-first retrieval, a small embedder fused on top, and a separate freshness ledger.

Honest scope

This is a replication with a receipt, not a new law. The direction (lexical ≈/≥ zero-shot dense; fusion helps when channels are complementary) is textbook — BEIR established BM25 as a strong zero-shot baseline years ago. Specific caveats: (a) recall@gold-evidence-turn slightly under-credits embeddings, since a semantically-equivalent but non-annotated turn scores zero; (b) LoCoMo is high-lexical-overlap and monolingual English — a paraphrase-heavy or cross-lingual workload (e.g. retrieving across languages, where BM25 scores zero) would move the gap toward embeddings; (c) one benchmark, vanilla retrievers, no reranker. The numbers are means over 1,531 questions and reproduce on re-run from the cached embeddings.

Related research

• Why RAG serves stale facts: the supersession blind spot, reproduced — the freshness problem retrieval alone cannot solve.
• Does long context kill RAG? — when "just retrieve more / dump it all in" stops working.
• Can corroboration stop AI-agent memory poisoning? — trust, not just recall, in agent memory.

FAQ

Do you need a vector database for AI agent memory? Not as your only retrieval layer, on this evidence. On LoCoMo a single vector index — even with a strong embedder (mxbai-embed-large) — did not beat a zero-dependency BM25 (recall@20 0.526 vs 0.552, statistically a tie). Vectors earned their cost only inside a hybrid (BM25 + embedder = 0.609) and on multi-hop/semantic questions. Start with BM25; add embeddings as a fused second channel.

Why is recency-based memory so bad? Recency (keep the last N turns) is query-blind, so on multi-session memory where the relevant fact is in an old session it recalls almost nothing — recall@20 of 0.024, ~23× worse than BM25, losing in all 10 conversations. Use recency as a tie-breaker on top of retrieval, never as the retriever.

Does a bigger embedder fix it? No. The strong embedder (mxbai-embed-large) was statistically indistinguishable from BM25 and from the small local nomic-embed-text inside the hybrid (hybrid 0.604 vs 0.609). The lever was adding a lexical channel, not scaling the model.

Is "BM25 beats vectors" a new finding? No — this reproduces BEIR's well-known result that BM25 is a strong zero-shot baseline, here on agent-memory data with a runnable script. The "you probably don't need a vector DB" angle is also already well-trodden; our contribution is the measured receipt and the recency and hybrid numbers, not the opinion.

The falsifierIf, on this same LoCoMo set, a single vector index with a strong embedder (run with correct prefixes, no reranker) beats BM25 at recall@20 with a conversation-level CI that excludes zero — or if a recency window reaches BM25-level recall — the core claims break. The script and the raw per-method results are public, and the embeddings regenerate deterministically, so anyone can reproduce or refute this.