In the first post I described how I organized my local multi-agent setup, : Codex executes, Claude reviews, other agents enter in bounded ways, and the human keeps closure authority.
I also argued that operational memory should not depend on a single conversation or be confused with the vector index.
By the time I closed that first post, I already had continuity mechanisms: handoffs, routing rules, append-only logs, experiential memory in JSONL/YAML, a rebuildable vector collection, and the skill.
My problem was not absolute amnesia.
It was that I still could not prove what the system retrieved, when it confused a coincidence with evidence, and when it should admit that it did not have an answer.
This second part, then, is not about inventing memory from scratch.
It is about turning still-fragile operational continuity into a traceable, testable, and rebuildable mechanism.
The idea of an external working memory is not new.
It echoes Bush's old ambition of augmenting recall through a personal archive and the extended-mind intuition that notes and tools can become part of cognition.12 My claim here is narrower: local traces are useful only if I can retrieve them with provenance and audit how they were used.
Spoonful 1: the problem was not storing, but retrieving well Storing information is easy.
The difficult part, I think, is retrieving the right piece when there are successive decisions, similar names, contradictory versions, and explanations spread across several files.
To organize that "memory" in my own setup, I separated its operational layers: Table 1 - System memory layers Layer Question it answers Effective implementation Canonical record What happened, and what was decided? , , , and curated events.
Retrieval index Where is the relevant evidence?
Qdrant with for curated context and for operational memory; both are derived.
Episodic history How did a session unfold?
Selected Markdown handoffs, compact context events, and operational logs; full sessions are not indexed raw.
Working context What context do I need to provide now?
MCP (, ) consumed by the skill.
The distinction matters.
A retrieved result is not yet a verified decision.
It is a clue that must preserve provenance, date, and a link to its source.
In the current contract, local JSONL/YAML files are the source of truth; Qdrant is rebuilt from them.
If the index contradicts a current file, the file wins, and the right fix is to reindex or correct ingestion, not to publish vector proximity as if it were final evidence.
The actual flow ended up like this: Figure 1 - Operational flow of auditable memory in my multi-agent setup.
Note: local JSONL/YAML files are the source of truth; Qdrant and BM25 are derived retrieval indexes, not final evidence.
RAG, without turning it into magic A language model stores part of what it learned during training in its parameters.
That memory does not necessarily include what I decided yesterday in a local repository.
Retrieval-Augmented Generation, or RAG, adds an external memory: before answering, a retriever searches for relevant passages and gives them to the generative model as context.3 In my implementation, acts as the retriever: it queries Qdrant, fuses signals with the canonical records, and returns candidates through MCP.
Codex, Claude, or Gemini then use that context according to the workflow.
That is why I prefer to describe this as retrieval-augmented agents rather than as an "autonomous RAG": storing and ordering fragments is not the same as reasoning over them.
Searching by meaning and by words Semantic search transforms each fragment into a vector.
The query is represented with the same model and then compared by orientation through cosine similarity: sim(q,d)=∥q∥∥d∥q⋅d If the query and the document point in similar directions, their similarity increases.
This makes it possible to find paraphrases even when they do not share the exact same words.
Lexical search covers the complementary problem: identifiers, acronyms, paths, proper names, and exact terms.
BM25 does more than count matches; it weighs term rarity and frequency, saturates repetitions, and partially adjusts for document length.4 Today I combine five signals: dense search in Qdrant with embeddings; lexical fallback over canonical events; BM25 sparse vector with ; event recency; lifecycle type: , , , or .
I still do not use RRF in the evaluated version.
The active fusion is a weighted formula in : when there is a semantic signal, it assigns 0.60 to dense similarity, 0.28 to lexical evidence, 0.07 to recency, and 0.05 to lifecycle type.
When there is no semantic signal, the fallback mostly weights lexical evidence.
BM25 contributes as an additional reordering signal, but it is not enough by itself to rescue a result.
One less flashy virtue is still missing: teaching the system to stay quiet.
A query with no answer in the corpus should not receive a fragment only because it looks nearby.
In this version I measure retriever rejection: a candidate qualifies if it passes any of these criteria in the gate run, , , or .
I still do not have an automated abstention metric for the final generator.
If a model receives context and invents anyway, this evaluation will not catch it.
Spoonful 2: from a useful index to evaluated retrieval What I left working uses these pieces, verified against the local repo and active services: Table 2 - Active components Function Component Verified configuration Canonical source and Rebuildable local records; at review time, had 1,829 lines and had
904.
Dense vectorization Ollama + model, 566.70M parameters, F16, 8192 context, 1024-dimensional embeddings.
Vector database Local Qdrant Collections and ; the MCP reported status, 14 canonical contexts, and 1,769 operational points.
Lexical retrieval Canonical JSONL + FastEmbed BM25 computes lexical matching; feeds the sparse vector.
Fusion Fuses dense, lexical, BM25, recency, and lifecycle signals; deduplicates by or .
Reranking Not active in the published run An opt-in reranker exists with , but the validated gate does not use it.
Agent interface MCP and ; the skill uses it as its primary path.
Test automation and The is enabled weekly; the gate records history in .
I run this on a local Arch Linux/KDE workstation.
Ollama and Qdrant are queried over loopback; the timers are timers.
That detail is not cosmetic: if I isolate the canonical fallback or the gate environment is missing, retrieval changes materially.
That is why the gate service explicitly sets , , and .
The embedding model I actually used BGE-M3, in its original implementation, supports dense, sparse, and multivector representations, works with more than one hundred languages, and accepts long sequences.5 But those model capabilities do not prove that all of them are exposed in my stack.
I use , served by Ollama through .6 In practice I use only the dense representation returned by Ollama: 1024-dimensional vectors in the local installation.
The sparse branch of the system does not come from BGE-M3; it comes from BM25 with FastEmbed and Qdrant.78 Pooling, tokenization, and normalization are encapsulated in the Ollama runtime, not in custom repo code.
I also did not chunk the whole workspace indiscriminately.
The ingestion works with selected documents: sanitized Markdown handoffs and compact events.
When importing handoffs, the extractor takes sections such as goal/context, decision, and outcome; compacts them; preserves , , , , , , branch, and date; and limits text to short fields.
I do not dump full conversations or raw private files into the index.
A question set that makes the system uncomfortable I evaluated with a local golden set of 40 questions: 32 positives and 8 negatives.
Positive cases point to expected documents through suffixes relative to the workspace; the evaluator does not open or emit the source content.
Difficulty labels are not mutually exclusive: there are 10 exact-keyword cases, 9 paraphrase cases, 7 cross-language cases, 3 collo