Ingestion: UPSERTS_AND_DELETE misses a removed document when hashes collide, and the transformation cache key is not injective

Two defects in llama-index-core's ingestion path, both in how a unit of work is identified. They are separate and can be split if you prefer.

---

1. UPSERTS_AND_DELETE silently fails to delete a removed document when another document shares its content hash

Where

llama-index-core/llama_index/core/ingestion/pipeline.py, _handle_upserts (sync) and _ahandle_upserts (async); root cause in llama-index-core/llama_index/core/storage/docstore/keyval_docstore.py.

The delete pass computes what is currently stored as:

existing_doc_ids_before = set(self.docstore.get_all_document_hashes().values())
doc_ids_to_delete = existing_doc_ids_before - doc_ids_from_nodes

get_all_document_hashes() returns a dict keyed by hash, with the doc_id as the value:

return {doc_hash: doc_id for doc_id, doc in ... if (doc_hash := doc.get("doc_hash"))}

The map from doc_id to hash is many-to-one, so inverting it collapses every set of same-content documents down to a single doc_id — last writer wins. The delete pass then computes a set difference over a partial view, and a document the user removed from their source is silently retained in both the docstore and the vector store, where it keeps being returned by queries.

Reproduction

Ingest {A, B} with identical content, then re-ingest only {B}; A should be deleted.

doc_A.hash == doc_B.hash: True | ids equal: False

after run 1, docstore ids: ['doc_A', 'doc_B']
after run 2, docstore ids: ['doc_A', 'doc_B']
warnings raised: []
RESULT: doc_A LEAKED (not deleted)

The map collapse, shown directly:

docstore ids              : ['doc_A', 'doc_B']
get_all_document_hashes() : {'67cf986e…df54': 'doc_B'}
  -> .values() sees only  : {'doc_B'}

The asymmetry — why this is an accident rather than a rule

Re-ingesting only doc_A correctly deletes doc_B; re-ingesting only doc_B leaves doc_A behind. Same data, same strategy, opposite correctness, decided by which document won the dict slot:

re-ingest only doc_B -> docstore ids: ['doc_A', 'doc_B']   (wrong)
re-ingest only doc_A -> docstore ids: ['doc_A']            (correct)

No intended semantics produce opposite outcomes for symmetric inputs.

Negative control

Identical script, the two documents differing by one word:

after run 2, docstore ids: ['doc_B']
RESULT: doc_A deleted (ok)

The async twin _ahandle_upserts behaves identically, verified by execution, and its control passes too.

The silence

warnings raised: [] on every run, captured with catch_warnings(record=True) and simplefilter("always"). run() returns normally with n nodes = 0, which reads to the caller as "everything is already up to date". The stale document remains retrievable.

Note

The surrounding code is careful about identity: _handle_upserts keys correctly on ref_doc_id in its main loop, and refresh_ref_docs in core/indices/base.py keys on document.id_. Only the delete pass reaches for get_all_document_hashes().values(), evidently as a convenient way to enumerate stored ids, without noticing the enumeration is lossy. docstore.docs.keys() / get_all_ref_doc_info() enumerate correctly and already exist.

Duplicate-content documents under distinct ids are ordinary in RAG corpora: boilerplate pages, licence text repeated across products, empty or placeholder pages, the same record ingested from two feeds. Note that metadata participates in the hash, so readers that inject a distinguishing file_path will not collide; readers pulling from databases or APIs typically will.

---

2. The transformation cache key is not one-to-one, and the cached value carries identity the key does not

Where

llama-index-core/llama_index/core/ingestion/pipeline.py, get_transformation_hash:

nodes_str = "".join([str(node.get_content(metadata_mode=MetadataMode.ALL)) for node in nodes])
transformation_dict = transformation.to_dict()
transform_string = remove_unstable_values(str(transformation_dict))
return sha256((nodes_str + transform_string).encode("utf-8")).hexdigest()

Two independent problems in one expression. The key omits node identity entirely — no id_, no ref_doc_id, no relationships — yet the cached value is a list of nodes that carries exactly that identity. And the join uses no separator, so node boundaries are ambiguous: ["ab", "c"] and ["a", "bc"] produce the same key.

Reproduction — boundary ambiguity

hash(A) = ad6212ff…  hash(B) = ad6212ff…   COLLISION: True

run on A -> [('1', 'ab|len=2'), ('2', 'c|len=1')]
run on B -> [('1', 'ab|len=2'), ('2', 'c|len=1')]   <- should be [('3','a|len=1'),('4','bc|len=2')]
B got A's cached result: True

Reproduction — end to end, default configuration

No docstore, no flags, disable_cache at its default False:

A -> [('bf227cbb', 'contract_A')]
B -> [('bf227cbb', 'contract_A')]   <- expected ref_doc_id contract_B
WRONG PAIRING: True

Ingesting contract_B returns nodes whose ref_doc_id says contract_A. Every downstream consumer — citations, source attribution, delete_ref_doc, get_ref_doc_info — now points at the wrong document.

Negative controls

# same script with disable_cache=True  (also rules out my setup as the cause)
A -> [('7dcb8d8d', 'contract_A')]
B -> [('4a18c698', 'contract_B')]    WRONG PAIRING: False

# a near miss without boundary ambiguity correctly misses the cache
hash(A) == hash(C)? False   -> C is transformed, not served from A's entry

The silence

No exception, no warning, no log line on any run. run() returns a well-formed node list of the expected length. The only way a caller detects this is by reading ref_doc_id and noticing it names a document they did not pass in.

Note

Caching a transformation on content plus transform config is a sound intent: identical input text plus identical transform yields identical output text. The defect is that the cached value is not text — it is a node list carrying identity that is not a function of the key. The missing separator is a second, independent injectivity break in the same expression. Both are fixable without changing the intent, by including node ids in the key or rewriting identity on a cache hit. It is on by default.

---

Explicitly not a report

_handle_duplicates dropping a document because another with the same content hash is already stored is exactly what DocstoreStrategy.DUPLICATES_ONLY is documented to do. Deliberate, and not what either finding above is about.

Not tested

Only SimpleDocumentStore and SimpleVectorStore were exercised; Redis, Mongo, Postgres, DynamoDB, Firestore and Couchbase implement get_all_document_hashes separately, and iteration order there is backend-defined, so which document leaks may differ — the collapse itself is structural. MockEmbedding and trivial transform components were used to keep runs offline; neither finding depends on the model call. The num_workers > 1 multiprocessing path uses the same keys but was not run. Integration packages were out of scope.

I have not checked whether either was raised before; pointers to existing issues are welcome and I will close this in favour of them.

Version

llama-index-core 0.14.24, source at d2ac544a27c73d2a68e9c57efec4b2ac0ef99892, Python 3.12.