#1101·marker

[Feature Request] Chinese document parsing resilience — post-processing fallback, structured validation, per-page error isolation

Author: 1printfCreated Sep 7, 2026Updated Sep 7, 2026

Hi maintainers, first of all — thank you for building and maintaining Marker. It's become one of the most useful tools in my document-processing workflow, and I really appreciate how clean the provider -> builder -> processor -> renderer pipeline is.

I'm a first-time contributor looking to add a small, non-invasive set of improvements aimed at Chinese-language PDF documents. Before I write any code, I wanted to check whether this direction aligns with the project's roadmap and whether you'd be open to a PR.

Background

When converting Chinese PDFs, I've run into three recurring problems that all live in the post-processing layer (not the recognition engines):

  1. Punctuation and line-break fragility — Chinese full-width punctuation and CJK line wrapping can produce broken markdown, e.g. unwanted spaces between lines, split headings, or table cells that break on mid-sentence newlines.
  2. No structured output validation — The JSON/chunks output is a tree of blocks, but there's no schema-level check that a SectionHeader, Text, or Table block actually contains well-formed content. A malformed block can silently propagate into the final markdown.
  3. Single-page failure aborts the whole document — If one page throws during processing (e.g. an unusual table boundary or a garbled OCR span), the entire conversion crashes, losing all previously processed pages.

Proposed approach (all in post-processing, zero changes to core engines)

# Change Where it lives
1 Add a ChineseTextResilienceProcessor that normalizes CJK punctuation, merges mid-sentence line breaks, and repairs table-cell boundary splits marker/processors/ (new file, opt-in via --processors)
2 Add pydantic validators for SectionHeader, Text, and Table block content, with auto-correction for common format errors marker/schema/ (new validators module, existing block classes unchanged)
3 Wrap per-page processing in a try/except that logs the error, records the page id in metadata, and continues with remaining pages marker/converters/pdf.py (build_document, minimal change)

Design constraints I would commit to

  • No changes to providers, builders, layout/OCR models, or core recognition logic.
  • Backward compatible — all new behavior is opt-in; default pipeline and output formats stay identical.
  • No new hard dependencies — only pydantic (already used) and stdlib.
  • Full logging via the project existing logger, with page-level error metadata.
  • Unit tests for each fallback path and a small benchmark comparing crash rates on Chinese PDFs before/after.

Questions for you

  1. Is this a direction you would welcome a PR for, or is it better handled by a custom processor in user code?
  2. Would you prefer the Chinese-specific logic as a standalone opt-in processor, or folded into the existing TextProcessor / TableProcessor?
  3. For the per-page error isolation — is there an existing pattern I should follow, or would a new config flag (e.g. --skip-bad-pages) be acceptable?
  4. Any naming or style conventions I should be aware of beyond what's in the codebase?

I am happy to start with just one of the three pieces (e.g. the per-page skip) as a smaller first PR if that is easier to review. Thank you for your time!