file-embeddings stalled-job recovery forks the batch chain and silently writes duplicate vectors

Split out of #1212, where it was reported by @DonnyPro-aka-Ron with Redis-level evidence. The bug in that thread's title is the O(n²) batch re-scan, which is #1185. This is unrelated to it and considerably worse.

The mechanism

ZIM embedding runs as a self-continuing chain: each EmbedFileJob handles one batch and, on completion, enqueues the next one with an incremented batchOffset. If BullMQ's stalled-job detection recovers such a job while the original is still alive, the result is not a resumed chain but two parallel chains over the same file. Both walk the same offsets, both write to Qdrant, and nothing in the UI or the logs flags it. The chunk counter keeps climbing, so it reads as normal progress.

admin/commands/queue/work.ts:

private getStallOptionsForQueue(
  queueName: string
): { lockDuration: number; maxStalledCount?: number } {
  if (
    queueName === DownloadDrugDataJob.queue ||
    queueName === IngestDrugDataJob.queue
  ) {
    return { lockDuration: 1_800_000, maxStalledCount: 3 }
  }
  return { lockDuration: 300000 }
}

The drug queues get 30 minutes and three stalled retries. file-embeddings falls through to 5 minutes and BullMQ's default maxStalledCount of 1.

The docblock above that function already states why the drug queues needed the override: a transient lock-renewal miss otherwise kills the continuation chain with "job stalled more than allowable limit". That reasoning applies at least as strongly to file-embeddings, which holds the lock through a full ZIM extraction batch. A single batch blocks the event loop for minutes on its own, and any Qdrant or Ollama hiccup extends that well past five minutes.

Two details make the outcome worse than a killed chain:

1. EmbedFileJob.queue runs at concurrency 2, so a forked second chain fits inside the concurrency budget and never contends for a slot. Nothing throttles it.
2. rag_service.ts mints point ids with randomUUID() on every write. A duplicated write therefore cannot overwrite the earlier point; it can only add a new one.

Evidence from the report

During a wikipedia_en_all_maxi ingest, Qdrant was unreachable for about 42 minutes. On recovery there were two active jobs with identical payloads, 115 seconds apart:

> LRANGE bull:file-embeddings:active 0 -1
1750
1751

> HGET bull:file-embeddings:1750 data
{"fileName":"wikipedia_en_all_maxi_2026-02.zim","batchOffset":3480000,
 "totalArticles":18982214,"chunksSoFar":8020083,"startedAt":1787166720191}

> HGET bull:file-embeddings:1751 data
{"fileName":"wikipedia_en_all_maxi_2026-02.zim","batchOffset":3480000,
 "totalArticles":18982214,"chunksSoFar":8020083,"startedAt":1787166835649}

SREM bull:file-embeddings:stalled 1751 returned 1, confirming the second job came from stalled recovery. Every offset appears twice in the logs, roughly 115 seconds apart, with byte-identical messages.

Impact

On the run that was caught: Qdrant held 8,095,435 points for that source against a reported chunksSoFar of 8,020,083, about 75,000 duplicate vectors from a couple of hours of overlap.

On an earlier run that was not caught: roughly 10 million duplicate vectors out of 24 million points, about 40% of that file's index, and several weeks of GPU time.

Corroborating detail from the same thread: raising ZIM_BATCH_SIZE from 50 to 5000 produced duplicate indexes. That is what this mechanism predicts, since larger batches hold the lock longer.

Why this is getting more urgent, not less

Today most ZIM ingests stop after a single batch because of #1240, and a chain of length one cannot fork. #1242 fixes that. Measured on a v1.34.0 test box, wwwnc.cdc.gov_en_all goes from 1 batch to 38 and www.ready.gov_en from 1 batch to 42; a full Wikipedia runs thousands. The fork window scales with chain length, so #1242 moves this from a bug that mostly affects people who raised ZIM_BATCH_SIZE by hand to one that every install is exposed to.

That is not an argument against #1242, which is correct and fixes a worse problem. It is an argument for landing this in the same release.

Fix shape

Two parts. The first alone reduces the frequency but does not make duplicates impossible.

1. Give file-embeddings a realistic lockDuration and an explicit maxStalledCount, in the same function that already special-cases the drug queues.
2. Make the chain safe against a fork. Deriving the Qdrant point id deterministically from source, batch offset, and chunk index would turn a duplicated write into an idempotent overwrite instead of a new point. This protects new writes only; it does not clean up existing duplicates.

Related

• #1203 is the same class of defect on the downloads queue: no maxStalledCount or lockDuration override, so BullMQ's default of 1 fails a download outright and bypasses attempts: 10.
• #1170 covers stale vectors surviving ZIM deletion, which is the other way this collection accumulates points nobody wants.