How to reproduce the 83.3% olmOCR-bench score? What are the "adjustments applied to account for our output HTML format"?
Question
The README reports 83.3% on olmOCR-bench for Surya OCR 2, and the "Reproducing" section says:
We score Surya 2 on olmOCR-bench by serving the model with
vllm(orllama.cpp) and running the olmOCR-bench harness from allenai/olmocr, with some adjustments applied to account for our output HTML format.
I've been trying to reproduce this and can only reach ~77.6%. I suspect the gap is entirely in those unspecified "adjustments" for converting Surya's block-HTML output into what the olmOCR-bench harness expects.
Could you share (or describe) exactly what those adjustments are? Ideally the HTML→text/markdown conversion code you used for the benchmark.
My setup
surya-ocr==0.22.1, modeldatalab-to/surya-ocr-2, served via vLLM- Full-page OCR:
RecognitionPredictor(...)(images, full_page=True) - I take each block's
html(inreading_order) and convert to Markdown for the olmOCR-bench harness (allenai/olmocr), running the standardolmocr-benchscorer.
What I found (and fixed on my side)
Starting from a naive HTML→Markdown conversion I got 58.6%. Two conversion issues explained most of the gap:
- Math underscores get escaped. A generic HTML→Markdown step turns
<math>J_1</math>into$J\_1$. The harness's math matching then fails. Protecting<math>content fixedmathfrom 33.6% → 83.1%. - Running headers/footers are included. Surya faithfully emits
PageHeader/PageFooterblocks, but the olmOCR-benchabsenttests expect these to be dropped. RemovingPageHeader/PageFooterblocks fixedabsentfrom 27.5% → 91.9%.
After both fixes:
| type | score |
|---|---|
| overall | 77.6% (95% CI 76.5–78.7) |
| baseline | 99.9 |
| math | 83.1 |
| table | 81.4 |
| order | 72.7 |
| present | 56.2 |
| absent | 91.9 |
Where I'm still stuck
The remaining ~5–6 point gap seems concentrated in present (56.2) and order (72.7), largely on old scans / tiny-text pages. I can't tell whether the official run:
- uses a different HTML→text normalization (e.g. how math delimiters, tables, or block ordering are emitted),
- drops/keeps different block labels,
- renders pages at a specific DPI, or
- applies harness-side tweaks.
Is the benchmark conversion/adjustment script available anywhere? Even a short description of the HTML-handling rules would let the community reproduce the 83.3 number. Thanks!
Source: datalab-to/surya