Document if multiple instances can use the same workspace
Describe the bug
So, this might seem like a silly thing, but I find it hard to understand how the workers and the workspaces relate.
For example, I have an instance running on a few hundred images (not PDFs):
$ olmocr ../archives/olmocr_workspace/ --markdown --pdfs ../archives/barch/R_58_5853e/*.jpg --server http://localhost:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8
This first reports converting the jpgs to PDFs:
2026-07-05 15:18:33,101 - olmocr.pipeline - INFO - Loading file at ../archives/barch/R_58_5853e/R_58_5853_e_0001.jpg as image document
2026-07-05 15:18:33,101 - olmocr.pipeline - INFO - Loading file at ../archives/barch/R_58_5853e/R_58_5853_e_0002.jpg as image document
2026-07-05 15:18:33,106 - olmocr.pipeline - INFO - Loading file at ../archives/barch/R_58_5853e/R_58_5853_e_0003.jpg as image document
2026-07-05 15:18:33,110 - olmocr.pipeline - INFO - Loading file at ../archives/barch/R_58_5853e/R_58_5853_e_0004.jpg as image document
...Now, while this is running, is it safe to start another olmocr command using the same workspace with a different set of files?
Or is it possible to have one persistent worker-olmocr and use other olmocr invocations to just add documents to it?
I tried running this while the previous olmocr instance was still running, and it announced it's converting to PDF files from the previous run (incidentally, I'm very confused about what --pdfs does):
$ olmocr ../archives/olmocr_workspace/ --markdown --pdfs ../archives/barch/RM_98_1196__ktb_uboote/* --server http://localhost:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8
...
2026-07-05 16:24:03,945 - olmocr.pipeline - INFO - Got --pdfs argument, going to add to the work queue
2026-07-05 16:24:03,946 - olmocr.pipeline - INFO - Loading file at ../archives/barch/RM_98_1196__ktb_uboote/RM_98_1196_0001.jpg as image document
2026-07-05 16:24:03,946 - olmocr.pipeline - INFO - Loading file at ../archives/barch/RM_98_1196__ktb_uboote/RM_98_1196_0005.jpg as image document
...
2026-07-05 16:24:03,983 - olmocr.pipeline - INFO - Loading file at ../archives/barch/RM_98_1196__ktb_uboote/RM_98_1196_0415.jpg as image document
2026-07-05 16:24:03,983 - olmocr.pipeline - INFO - Loading file at ../archives/barch/RM_98_1196__ktb_uboote/RM_98_1196_0417.jpg as image document
2026-07-05 16:24:03,984 - olmocr.pipeline - INFO - Loading file at ../archives/barch/RM_98_1196__ktb_uboote/RM_98_1196_0418.jpg as image document
2026-07-05 16:24:03,984 - olmocr.pipeline - INFO - Found 381 regular pdf paths and 0 tarballs to add
Sampling PDFs to calculate optimal length: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 186.11it/s]
2026-07-05 16:24:04,526 - olmocr.pipeline - INFO - Calculated items_per_group: 500 based on average pages per PDF: 1.00
2026-07-05 16:24:04,533 - olmocr.pipeline - INFO - Starting pipeline with PID 221876
2026-07-05 16:24:04,533 - olmocr.pipeline - INFO - Using external server at http://localhost:8000/v1
INFO:olmocr.work_queue:Initialized queue with 2 work items
INFO:httpx:HTTP Request: GET http://localhost:8000/v1/models "HTTP/1.1 200 OK"
2026-07-05 16:24:04,554 - olmocr.pipeline - INFO - vllm server is ready.
2026-07-05 16:24:04,554 - olmocr.pipeline - INFO - Queue remaining: 2
2026-07-05 16:24:04,554 - olmocr.pipeline - INFO -
Metric Name Lifetime (tokens/sec) Recently (tokens/sec)
----------------------------------------------------------------------------------
2026-07-05 16:24:04,554 - olmocr.pipeline - INFO -
Worker ID
---------
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 2 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 3 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 4 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 5 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 6 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 7 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 8 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 9 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 10 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 11 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 12 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 13 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 14 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 15 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 16 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 17 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 18 exiting due to empty queue
2026-07-05 16:24:04,555 - olmocr.pipeline - INFO - Worker 19 exiting due to empty queue
2026-07-05 16:24:04,556 - olmocr.pipeline - INFO - Worker 0 processing work item 1bd628c581c8afad525c64d3f20d38d72ee91c21
2026-07-05 16:24:04,557 - olmocr.pipeline - INFO - Created all tasks for 1bd628c581c8afad525c64d3f20d38d72ee91c21
2026-07-05 16:24:05,349 - olmocr.pipeline - INFO - Converting ../archives/barch/R_58_5853e/R_58_5853_e_0255.jpg from image to PDF format...
2026-07-05 16:24:05,642 - olmocr.pipeline - INFO - Converting ../archives/barch/R_58_5853e/R_58_5853_e_0276.jpg from image to PDF format...
2026-07-05 16:24:05,977 - olmocr.pipeline - INFO - Converting ../archives/barch/R_58_5853e/R_58_5853_e_0324.jpg from image to PDF format...
2026-07-05 16:24:06,733 - olmocr.pipeline - INFO - Converting ../archives/barch/R_58_5853e/R_58_5853_e_0003.jpg from image to PDF format...
2026-07-05 16:24:06,956 - olmocr.pipeline - INFO - Converting ../archives/barch/R_58_5853e/R_58_5853_e_0004.jpg from image to PDF format...Here the R_58_* files are ones being processed in the other olmocr invocation. This makes me suspect it's not really designed to allow two olmocrs to use the same workspace?
Versions
$ uv run python --version Python 3.14.3
No pip here, but here's uv tree output:
$ uv tree Resolved 77 packages in 1ms ocr v0.1.0 ├── olmocr v0.4.27 │ ├── bleach v6.4.0 │ │ └── webencodings v0.5.1 │ ├── boto3 v1.43.40 │ │ ├── botocore v1.43.40 │ │ │ ├── jmespath v1.1.0 │ │ │ ├── python-dateutil v2.9.0.post0 │ │ │ │ └── six v1.17.0 │ │ │ └── urllib3 v2.7.0 │ │ ├── jmespath v1.1.0 │ │ └── s3transfer v0.19.0 │ │ └── botocore v1.43.40 () │ ├── cached-path v1.8.10 │ │ ├── boto3 v1.43.40 () │ │ ├── filelock v3.29.5 │ │ ├── google-cloud-storage v3.12.0 │ │ │ ├── google-api-core v2.31.0 │ │ │ │ ├── google-auth v2.55.1 │ │ │ │ │ ├── cryptography v49.0.0 │ │ │ │ │ │ └── cffi v2.0.0 │ │ │ │ │ │ └── pycparser v3.0 │ │ │ │ │ └── pyasn1-modules v0.4.2 │ │ │ │ │ └── pyasn1 v0.6.3 │ │ │ │ ├── googleapis-common-protos v1.75.0 │ │ │ │ │ └── protobuf v7.35.1 │ │ │ │ ├── proto-plus v1.28.0 │ │ │ │ │ └── protobuf v7.35.1 │ │ │ │ ├── protobuf v7.35.1 │ │ │ │ └── requests v2.34.2 │ │ │ │ ├── certifi v2026.6.17 │ │ │ │ ├── charset-normalizer v3.4.7 │ │ │ │ ├── idna v3.18 │ │ │ │ └── urllib3 v2.7.0 │ │ │ ├── google-auth v2.55.1 () │ │ │ ├── google-cloud-core v2.6.0 │ │ │ │ ├── google-api-core v2.31.0 () │ │ │ │ └── google-auth v2.55.1 () │ │ │ ├── google-crc32c v1.8.0 │ │ │ ├── google-resumable-media v2.10.0 │ │ │ │ └── google-crc32c v1.8.0 │ │ │ └── requests v2.34.2 () │ │ ├── huggingface-hub v1.22.0 │ │ │ ├── click v8.4.2 │ │ │ ├── filelock v3.29.5 │ │ │ ├── fsspec v2026.6.0 │ │ │ ├── hf-xet v1.5.1 │ │ │ ├── httpx v0.28.1 │ │ │ │ ├── anyio v4.14.1 │ │ │ │ │ └── idna v3.18 │ │ │ │ ├── certifi v2026.6.17 │ │ │ │ ├── httpcore v1.0.9 │ │ │ │ │ ├── certifi v2026.6.17 │ │ │ │ │ └── h11 v0.16.0 │ │ │ │ └── idna v3.18 │ │ │ ├── packaging v26.2 │ │ │ ├── pyyaml v6.0.3 │ │ │ ├── tqdm v4.68.3 │ │ │ └── typing-extensions v4.16.0 │ │ ├── packaging v26.2 │ │ ├── requests v2.34.2 () │ │ └── rich v13.9.4 │ │ ├── markdown-it-py v4.2.0 │ │ │ └── mdurl v0.1.2 │ │ └── pygments v2.20.0 │ ├── cryptography v49.0.0 () │ ├── filelock v3.29.5 │ ├── ftfy v6.3.1 │ │ └── wcwidth v0.8.2 │ ├── httpx v0.28.1 () │ ├── lingua-language-detector v2.2.0 │ ├── markdown2 v2.5.5 │ ├── markdownify v1.2.3 │ │ ├── beautifulsoup4 v4.15.0 │ │ │ ├── soupsieve v2.8.4 │ │ │ └── typing-extensions v4.16.0 │ │ └── six v1.17.0 │ ├── orjson v3.11.9 │ ├── pillow v12.3.0 │ ├── pypdf v6.14.2 │ ├── pypdfium2 v5.11.0 │ ├── requests v2.34.2 () │ ├── smart-open v8.0.0 │ │ └── wrapt v2.2.2 │ └── zstandard v0.25.0 └── ipython v9.15.0 (group: dev) ├── decorator v5.3.1 ├── ipython-pygments-lexers v1.1.1 │ └── pygments v2.20.0 ├── jedi v0.20.0 │ └── parso v0.8.7 ├── matplotlib-inline v0.2.2 │ └── traitlets v5.15.1 ├── pexpect v4.9.0 │ └── ptyprocess v0.7.0 ├── prompt-toolkit v3.0.52 │ └── wcwidth v0.8.2 ├── psutil v7.2.2 ├── pygments v2.20.0 ├── stack-data v0.6.3 │ ├── asttokens v3.0.1 │ ├── executing v2.2.1 │ └── pure-eval v0.2.3 └── traitlets v5.15.1 (*) Package tree already displayed
Source: allenai/olmocr