Integrations should not own document text: shrink the side-car to ids + metadata

Summary

The four integration wrappers persist a JSON side-car holding the full document/node text alongside the .tvim index. turbovec should own indexing and search; the frameworks should own documents. The side-car should shrink to ids + metadata only, and the text should go.

This is the structural fix behind #425 (side-car 5.5x larger than the index at 100k docs) — text is what makes it large.

Why this is safe: all four frameworks support resolving text separately

Validated empirically, not from documentation. Wheel built with maturin build --release, installed non-editable into a fresh Python 3.11.12 venv; framework versions langchain-core 1.5.3 / langchain-classic 1.0.8, llama-index-core 0.14.23, haystack-ai 3.0.0, agno 2.8.6.

Framework	Resolution step	Status
LangChain	MultiVectorRetriever takes a docstore and calls docstore.mget(ids) on our hits	first-class, works today
LlamaIndex	StorageContext.docstore owns text; SimpleVectorStore itself sets stores_text = False	first-class
Haystack	AutoMergingRetriever / SentenceWindowRetriever accept a different document_store than the one that searched	first-class
Agno	none — user subclasses and overrides search	hand-rolled

Confirmed running: MultiVectorRetriever(vectorstore=TurboQuantVectorStore(...), docstore=InMemoryStore(), id_key="doc_id") retrieves documents that provably came from the docstore; LocalFileStore likewise. AutoMergingRetriever(document_store=parent_store).run(documents=turbovec_hits) returned the parent's full text while every document held in turbovec had content=None.

What has to change

Deleting the JSON writer alone is not enough — three read paths reconstruct results from the side-car, so the in-memory duplication would simply remain undisk'd:

1. LlamaIndex — query() hard-requires its own copy of the node: metadata_dict_to_node(data["node_dict"]) at turbovec-python/python/turbovec/llama_index.py:544, called from :848. Blanking the payload raises ValueError: Node content not found in metadata dict. Needs add() to stop storing node payloads and query() to return VectorStoreQueryResult(nodes=None, ids=..., similarities=...), matching SimpleVectorStore. A ~40-line prototype over turbovec.IdMapIndex was verified end to end: no vector-store JSON written at all, retrieval returns full text with the correct top hit.

   • Accepted cost: VectorStoreIndex.from_vector_store(vs) raises ValueError: Cannot initialize from a vector store that does not store text. This is by design and is what SimpleVectorStore does too.
   • Note the shipped class already accepts stores_text=False as a constructor arg, but still writes default__vector_store.nodes.json (2862 B) containing the text — so text is duplicated on disk today even when the caller opts out.

2. Agno — correctness bug, independent of this work. TurboQuantVectorDb._dedup_by_content hashes md5(doc.content). With blank content a search(limit=3) silently returns 1 result. Should dedup on id, not content.

3. LangChain — no vector-in door. add_texts embeds exactly what it stores, and add_embeddings does not exist (sorted(m for m in dir(vs) if m.startswith("add")) == ['add_documents', 'add_texts']). Passing page_content="" does not raise — it goes semantically dead: all three test documents embedded identically, every score exactly 0.2286243885755539, wrong document ranked first. Adding add_embeddings(embeddings, ids, metadatas) makes text-free a supported path rather than a trick.

4. Haystack — no change required. write_documents accepts content=None, and embedding_retrieval ranked correctly with all content blank (p1 1.004 vs p2 0.1243).

What stays

Metadata stays in the side-car. LangChain's dict and callable filters and Haystack's filter_documents both walk the local document table, so metadata cannot leave without breaking filtering. Metadata is small, and it is arguably index-adjacent anyway.

Also worth capturing when this is done: turbovec derives its own ids (_derive_doc_id = md5(f"{base_id}_{content_hash}")), so a caller-supplied id like p1 becomes 225df170ad3940d474744dbc0de92068. Any external store must key on doc.name or metadata rather than the id the user passed in — that needs documenting. Separately, Agno's content_hash_exists() is in-memory only, so with no durable turbovec state the skip-if-already-ingested check always reports "not present" and re-ingests every run; that state needs a home.

Scope

Breaking only for callers that rely on turbovec handing back text it stored. Not a major version.