Pure vector search got my agent memory stack to 63% on LoCoMo.
Adding a sparse retriever and a reranker that runs on a card I already owned pushed it to 80%.
The accuracy came from a stage that adds maybe 40ms per query, and the queries it fixed were exactly the ones I cared about: specific dates, error codes, and "who said what in which session" needles buried in months of conversation history.
If you're running a local agent that recalls facts across long conversations, this is the retrieval layer under everything else.
A bad memory stack doesn't crash.
It quietly hands the model the wrong three chunks and lets it confabulate a confident answer.
That failure mode is worse than an outage because nothing tells you it happened.
The setup and why I benchmarked at all My agents run on a memory stack I've written about before: a six-layer architecture for Claude Code with a wiki layer, a vector store, and an activation-based cognitive layer.
The vector store is the workhorse.
When an agent needs to recall a fact from a past session, it embeds the query, pulls the top-k nearest chunks, stuffs them into context, and answers.
That worked well enough that I never questioned it.
Then I ran LoCoMo against it.
LoCoMo is a long-term conversational memory benchmark.
It gives you multi-session dialogues that span hundreds of turns, then asks questions whose answers are scattered across those sessions.
Single-hop lookups, multi-hop reasoning, temporal ordering, the works.
It's a good proxy for what an agent memory system actually has to do, because the answer is never in the most recent turn.
It's three sessions back, phrased differently than the question.
My vector-only stack scored 63%.
Not terrible.
Not good enough to trust an agent to act on.
The interesting part wasn't the number, it was the shape of the failures.
What I tried first (and why it was the wrong lever) My first instinct was the obvious one: the embeddings must be too weak.
Swap the model, get better vectors, problem solved.
So I did the thing everyone does.
I moved from a general-purpose embedding model to a larger, higher-ranked one on the MTEB leaderboard.
Re-embedded the whole corpus.
Re-ran LoCoMo. 63% went to 65%.
Two points.
Hours of re-embedding for two points.
That's when I actually looked at the failures instead of the aggregate score, and the pattern was obvious in hindsight.
The questions I was getting wrong weren't semantically hard.
They were lexically specific: "What was the ticket number the user mentioned?" — the chunk with in it wasn't in the top-k, because "ticket number" as a query embeds close to a hundred chunks that talk about tickets in general. "Which date did they say the migration finished?" — the model retrieved chunks about the migration, just not the one sentence with the actual date. "What did Maria say about the vendor?" — proper nouns get averaged into oblivion by dense embeddings. "Maria" and "the vendor" are needles, and cosine similarity is bad at needles.
This is the well-documented weakness of dense retrieval.
Embeddings capture meaning, and they're great at "find me things about database migrations." They're bad at "find me the exact string TICKET-4471," because that string's meaning is thin.
There's nothing semantic about an identifier.
A better embedding model doesn't fix a problem that isn't about semantics.
The second thing I tried was cranking k.
If the right chunk isn't in the top 5, pull the top
20.
That helps recall, and it did nudge the score.
It also blows up the context window with noise and triggers the "lost in the middle" problem, where the model ignores relevant chunks buried between irrelevant ones.
I was trading a retrieval problem for an attention problem.
Not a win.
The actual fix: sparse recall, then rerank for precision The move that mattered was splitting retrieval into two jobs it was badly trying to do at once.
Recall is "get the right chunk into the candidate set somehow." Precision is "put the right chunk at th