`insert_chunks` should use upsert semantics — providers silently duplicate or error on repeated chunk IDs

System Info

• OGX version: current main (commit 57886c8bd)
• Python version: 3.12+
• Affected components: src/ogx_api/vector_io/api.py, all VectorIO provider implementations

Information

• The official example scripts
• My own modified scripts

Describe the bug

The VectorIO protocol defines insert_chunks (src/ogx_api/vector_io/api.py:58) with no documented contract for duplicate chunk_id handling. Five providers already implement upsert semantics (PGVector, Qdrant, Elasticsearch, OCI 26AI, Infinispan), but the remaining five use pure insert — causing silent data duplication (FAISS, Milvus, Weaviate), errors (ChromaDB), or inconsistent internal state (SQLite-vec) when chunks with existing IDs are re-inserted.

This is a portability bug: the VectorIO abstraction should guarantee consistent behavior regardless of the backend. Since generate_chunk_id (src/ogx/providers/utils/vector_io/vector_utils.py:15) produces deterministic UUIDs from sha256(document_id:chunk_text) — clearly designed for idempotent re-ingestion — upsert is the intended and correct semantic.

All five misaligned backends natively support upsert or can trivially emulate it, so alignment is a matter of calling the right API method, not a design limitation.

Current provider behavior

Milvus — easiest and most impactful fix

The Milvus provider (src/ogx/providers/remote/vector_io/milvus/milvus.py:150) calls client.insert() which silently creates duplicate entries on repeated chunk_id values. Milvus natively supports client.upsert() with identical signature and semantics — it checks the primary key and replaces existing entities. The fix is a single method swap:

# Current (line 150):
await asyncio.to_thread(self.client.insert, self.collection_name, data=data)

# Fixed:
await asyncio.to_thread(self.client.upsert, self.collection_name, data=data)

Note: Milvus upsert() requires the collection to be loaded and has higher memory usage than insert() for large-scale ingestion. This is an acceptable trade-off for correctness, and aligns with how PGVector/Qdrant already behave.

Relevant code paths

• Protocol definition: src/ogx_api/vector_io/api.py:58 — insert_chunks, no duplicate-handling contract specified
• Chunk ID generation: src/ogx/providers/utils/vector_io/vector_utils.py:15 — deterministic UUIDs from sha256(document_id:chunk_text), designed for idempotent re-ingestion
• Milvus (pure insert): src/ogx/providers/remote/vector_io/milvus/milvus.py:150 — client.insert() creates duplicates
• ChromaDB (error on dup): src/ogx/providers/remote/vector_io/chroma/chroma.py:86 — collection.add() raises on duplicate IDs
• SQLite-vec (partial upsert): src/ogx/providers/inline/vector_io/sqlite_vec/sqlite_vec.py:250 — vec0 table does plain INSERT, metadata table correctly upserts at line 236
• FAISS (pure insert): src/ogx/providers/inline/vector_io/faiss/faiss.py:182 — index.add() always appends, no ID-based dedup
• Weaviate (pure insert): src/ogx/providers/remote/vector_io/weaviate/weaviate.py:87 — insert_many() with auto-generated UUIDs, ignores chunk_id as object ID

Reproduction

from ogx_client import OgxClient

client = OgxClient(base_url="http://localhost:8321")

chunk = {
    "content": "test content",
    "chunk_id": "same-id-twice",
    "metadata": {},
    "embedding": [0.1] * 384,
    "embedding_model": "all-MiniLM-L6-v2",
    "embedding_dimension": 384,
}

# Insert the same chunk twice
client.vector_io.insert(vector_store_id="my-store", chunks=[chunk])
client.vector_io.insert(vector_store_id="my-store", chunks=[chunk])

# Result depends on backend:
# - PGVector/Qdrant/ES/OCI/Infinispan: 1 chunk (upserted) — correct
# - Milvus/FAISS/Weaviate: 2 chunks (silently duplicated) — bug
# - ChromaDB: DuplicateIDError on second call — bug
# - SQLite-vec: 1 metadata row, 2 vector entries — bug

Error logs

# ChromaDB raises on duplicate:
chromadb.errors.DuplicateIDError: ...

# Milvus/FAISS/Weaviate: no error, but silent data duplication affecting search quality
# SQLite-vec: no error, but inconsistent state between metadata and vector tables

Expected behavior

Expected behavior

insert_chunks should guarantee upsert semantics across all providers: inserting a chunk with an existing chunk_i d replaces the previous entry. Specifically:

1. Document the contract. The insert_chunks protocol docstring should explicitly state that duplicate chunk_id values result in replacement, not duplication or error.

2. Align all providers to upsert. Recommended fix per provider:

   • Milvus: swap client.insert() to client.upsert() (one-line fix)
   • ChromaDB: swap collection.add() to collection.upsert() (one-line fix)
   • SQLite-vec: change INSERT INTO [vec_table] to INSERT OR REPLACE INTO [vec_table] for the vec0 virtual ta ble
   • FAISS: wrap IndexFlatL2 with IndexIDMap2 to enable ID-based operations, implement upsert as remove_ids()

• add_with_ids()
  • Weaviate: use chunk_id as the Weaviate object UUID (deterministic) instead of auto-generating, and use the b atch REST API with replace semantics or delete-then-insert

1. Prioritize Milvus and ChromaDB. These are one-line fixes with no trade-offs that affect common production deploy ments.