OmniDocBench v1.6 text EditDist: we measure 0.087 vs ~0.042 implied by the reported Overall — CDM/TEDS at parity; requesting eval methodology

Summary

We evaluated baidu/Unlimited-OCR with the official HuggingFace model.infer path and the official OmniDocBench scorer, and found that text EditDist accounts for the entire gap between our result and the reported ~93.92 Overall, while Formula CDM and Table TEDS are essentially at parity. The CDM/TEDS parity suggests our setup is fundamentally correct, so the 2× text gap is puzzling. We'd like the exact eval methodology to close it, and we can share our predictions to localize it directly.

Measured results (OmniDocBench v1.6, 1,651 pages, official scorer)

The Overall is ((1−Text)×100 + CDM×100 + TEDS×100)/3. With the reported ~93.92 Overall plus our matching CDM/TEDS, the implied Text EditDist is 3×93.92 − 95.83 − 90.22 = 0.9576 → Text ≈ 0.042. We measure 0.0870 — roughly 2×, and this accounts for the full ~1.5-point Overall gap; CDM and TEDS match the reported values.

Setup (matches the README gundam example)

• Model baidu/Unlimited-OCR, revision 84757cb0, BF16, trust_remote_code=True
• gundam single-image: base_size=1024, image_size=640, crop_mode=True, max_length=32768, no_repeat_ngram_size=35, ngram_window=128, greedy (temp=0), prompt document parsing.
• Scorer: official opendatalab/OmniDocBench main (v1.6), match_method=quick_match, CDM enabled (texlive-lang-chinese + ImageMagick)
• Hardware: AMD ROCm gfx1100 ×4 (not NVIDIA — see "Open question" below)

What we ruled out (with evidence)

• Config. We use the README gundam example verbatim. A/B test (no_repeat_ngram_size=35 vs the model default 0, on a 100-page subset): identical text EditDist (0.01864 vs 0.01853) — ngram=35 has no effect.
• Matching. quick_match is the official default; top models on the leaderboard reach Text 0.035–0.044 with it, so it isn't the bottleneck.
• Degeneration/looping. A per-page analysis on the official scorer's per-block dump shows 99.8% of text-blocks are clean/non-degenerate. The gap is genuine content: ~65% content divergence (extra section headers, CJK variants, reading order, OCR confusions), ~14% inline-math LaTeX-style residual (CDM 0.958 confirms the math is correct — char-EditDist penalizes style differently), ~8% dense pages. Not garbage/looping.

So 0.087 is the real char-level text output of the model + this config, not an artifact.

Questions to drive localization

Given CDM/TEDS match but Text doesn't, the discrepancy is in (a) the text-metric eval methodology, (b) the inference path/post-processing, (c) the checkpoint revision, or (d) the prompt format. To localize:

1. Text metric. Is the reported Text EditDist produced by the official OmniDocBench scorer (clean_string(textblock2unicode(text)) normalization + quick_match + length-weighted per-page average), or a custom text metric?
2. Inference. Single-page model.infer (gundam) or a serving path (vLLM/SGLang)? Any post-processing of the predictions?
3. Checkpoint. Which weight revision? (We used 84757cb0.)
4. Prompt. The README example uses prompt='document parsing.' (no explicit <image> — model.infer then appends the image tokens after the text). We used '<image>document parsing.' (image before text). Is one the intended format for OCR, and could image-token placement affect text generation? (Our CDM/TEDS match, so both produce sane results, but this is one untested variable.)

Fastest path to localize: if you can share predictions for any ~50 pages from the reported run, we'll diff against ours and pin the gap to inference vs scoring in one step. Conversely, we're happy to share our full 1,651 predictions + per-block dump for you to re-score.

Reproduce our 0.087

# 1. Inference (single-page gundam, official config) -> one .md per page:
python -c "
import torch; from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained('baidu/Unlimited-OCR', trust_remote_code=True)
m = AutoModel.from_pretrained('baidu/Unlimited-OCR', trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda()
m.infer(tok, prompt='<image>document parsing.', image_file='page.png', output_path='./out',
        base_size=1024, image_size=640, crop_mode=True, max_length=32768,
        no_repeat_ngram_size=35, ngram_window=128, save_results=True)
"
# 2. Score with the official scorer (v1.6, quick_match, CDM enabled):
#    config: ground_truth=OmniDocBench.json, prediction=./out, metrics=[Edit_dist, CDM, TEDS]
#    -> Text EditDist 0.0870, CDM 0.9583, TEDS 0.9022, Overall 92.45

Full project with reproducible manifests (commit hashes, configs, env) and the per-page text-divergence analysis: AIwork4me/Unlimited-OCR-ROCm — in particular docs/parity/text-editdist-rootcause-2026-07-12.md.

Open question (backend)

We ran on AMD ROCm (gfx1100), not NVIDIA. We've observed forward-pass numerical divergence on the vLLM/ROCm serving path (documented separately), but the batched PyTorch model.infer path is byte-identical to the per-page path (gated on a 30-page subset of the 1,651, Overall Δ = 0.0), and the per-block analysis shows the gap is genuine model-output divergence, not numerical noise. We flag this for completeness; the questions above are the primary ask.

Thanks for the work and any guidance you can provide.