Threaded StandardPdfPipeline: conversion result depends on machine load (timer-based batch assembly)

Bug

The threaded StandardPdfPipeline assembles model batches on a wall-clock timer (batch_polling_interval_seconds, default 0.5 s), so the batch size actually used depends on how busy the machine is. Model output is not bit-identical across batch sizes, so the same PDF with the same options converts to a different DoclingDocument depending on load — text items merge or split, and labels change (e.g. text vs page_header).

No content is lost: the character multiset is identical every time. What changes is the structure, which is enough to move a block into a different chunk downstream.

Converted on its own the result is byte-identical over repeated runs, so this only shows up in a batch/service setting.

Steps to reproduce

https://files.eric.ed.gov/fulltext/ED134923.pdf - a scanned, OCR'd document that ERIC serves publicly. I am linking rather than attaching it: I could not establish that its licence permits redistribution, and the link is enough to fetch it.

1. Convert it once on its own → baseline.
2. Convert it three times inside a mixed batch together with three other PDFs, all submitted at once (I used ED145334, ED160651, ED110432 from the same source).
3. Compare texts + table cell texts against the baseline.

Observed, via docling-serve with DOCLING_SERVE_ENG_LOC_NUM_WORKERS=5:

configuration	result
1 worker, mixed load	12/12 identical
5 workers, mixed load, round 1	2 of 3 differ
5 workers, mixed load, round 2	1 of 3 differ
5 workers, mixed load, layout_batch_size=1 + table_batch_size=1	12/12 identical

Example difference (same document, same options, one run against the other):

- [list_item]  58 chars: 'Sound it out. Get teady. Move to a. aaa. When aaa is firm,'
- [text]       50 chars: 'move quickly to st. The children are to say saammm'
+ [list_item] 109 chars: 'Sound it out. Get teady. Move to a. aaa. When aaa is firm, move quickly to st. The child...'

and

- [text]         1 char: '2'
+ [page_header]  1 char: '2'

Eight identical requests fired at once agreed with each other; it is the mixed batch that does it, because that is what makes batch assembly irregular.

Analysis

ThreadedQueue.get_batch(size, timeout) returns up to size items — "Blocks until ≥1 item present or queue closed/timeout". With batch_polling_interval_seconds=0.5 and a loaded machine, the producer stage falls behind and the consumer flushes partial batches of 1–3 items where an idle machine would consistently assemble 4. Inference is not bit-identical across batch sizes on GPU, so borderline layout decisions flip.

Two observations that support this:

• forcing layout_batch_size=1/table_batch_size=1 removes the variation entirely while keeping 5 concurrent workers (table above);
• converting solo with layout_batch_size=4 and with =1 gives identical output, so the batch size alone is not the problem — it is that the size is decided by timing.

It is not a memory or configuration problem on my side: each worker has its own converter and its own models (eng_loc_share_models=false), the GPU peaked at 18 of 32 GB, and nothing is logged.

Documents matter: three arXiv papers and two cleanly typeset public classification registers stayed identical under the same load. What reproduces are scanned, OCR'd documents.

Possible direction

The deeper property — batch-size-invariant GPU inference — is not realistic to chase. But the batch composition could be made independent of wall-clock timing: only flush a partial batch when the upstream stage is finished or the queue is closed, rather than whenever the poll interval expires. That would keep throughput and make a conversion reproducible.

Happy to prepare a PR along those lines if that direction is acceptable — please say whether you would rather have an opt-in pipeline option than a change of the current behaviour.

Docling version

docling-serve:latest image, digest sha256:52fe3430c2db05bae0d1878222e89bc52613b9c681f6c29812a990fa968d47fe (built 2026-09-15), containing docling-slim 2.127.0, docling-core 2.96.1, docling-parse 7.19.1, docling-ibm-models 4.0.2, docling-jobkit 3.6.0, docling-serve 1.33.0, torch 2.14.0. Pipeline standard, pdf_backend=docling_parse, table_mode=accurate, do_ocr=false, do_formula_enrichment=true, custom layout preset docling-project/docling-layout-egret-medium. NVIDIA RTX 5090, CUDA, rootless podman.

Python version

Python 3.12 (the docling-serve:latest image).