#541·surya

How to reproduce the 83.3% olmOCR-bench score? What are the "adjustments applied to account for our output HTML format"?

Author: zhangxueren9Created Jul 24, 2026Updated Jul 24, 2026
Labelsbug: output

Question

The README reports 83.3% on olmOCR-bench for Surya OCR 2, and the "Reproducing" section says:

We score Surya 2 on olmOCR-bench by serving the model with vllm (or llama.cpp) and running the olmOCR-bench harness from allenai/olmocr, with some adjustments applied to account for our output HTML format.

I've been trying to reproduce this and can only reach ~77.6%. I suspect the gap is entirely in those unspecified "adjustments" for converting Surya's block-HTML output into what the olmOCR-bench harness expects.

Could you share (or describe) exactly what those adjustments are? Ideally the HTML→text/markdown conversion code you used for the benchmark.

My setup

  • surya-ocr==0.22.1, model datalab-to/surya-ocr-2, served via vLLM
  • Full-page OCR: RecognitionPredictor(...)(images, full_page=True)
  • I take each block's html (in reading_order) and convert to Markdown for the olmOCR-bench harness (allenai/olmocr), running the standard olmocr-bench scorer.

What I found (and fixed on my side)

Starting from a naive HTML→Markdown conversion I got 58.6%. Two conversion issues explained most of the gap:

  1. Math underscores get escaped. A generic HTML→Markdown step turns <math>J_1</math> into $J\_1$. The harness's math matching then fails. Protecting <math> content fixed math from 33.6% → 83.1%.
  2. Running headers/footers are included. Surya faithfully emits PageHeader / PageFooter blocks, but the olmOCR-bench absent tests expect these to be dropped. Removing PageHeader/PageFooter blocks fixed absent from 27.5% → 91.9%.

After both fixes:

type score
overall 77.6% (95% CI 76.5–78.7)
baseline 99.9
math 83.1
table 81.4
order 72.7
present 56.2
absent 91.9

Where I'm still stuck

The remaining ~5–6 point gap seems concentrated in present (56.2) and order (72.7), largely on old scans / tiny-text pages. I can't tell whether the official run:

  • uses a different HTML→text normalization (e.g. how math delimiters, tables, or block ordering are emitted),
  • drops/keeps different block labels,
  • renders pages at a specific DPI, or
  • applies harness-side tweaks.

Is the benchmark conversion/adjustment script available anywhere? Even a short description of the HTML-handling rules would let the community reproduce the 83.3 number. Thanks!