[Feature Request] Chinese document parsing resilience — post-processing fallback, structured validation, per-page error isolation
Hi maintainers, first of all — thank you for building and maintaining Marker. It's become one of the most useful tools in my document-processing workflow, and I really appreciate how clean the provider -> builder -> processor -> renderer pipeline is.
I'm a first-time contributor looking to add a small, non-invasive set of improvements aimed at Chinese-language PDF documents. Before I write any code, I wanted to check whether this direction aligns with the project's roadmap and whether you'd be open to a PR.
Background
When converting Chinese PDFs, I've run into three recurring problems that all live in the post-processing layer (not the recognition engines):
- Punctuation and line-break fragility — Chinese full-width punctuation and CJK line wrapping can produce broken markdown, e.g. unwanted spaces between lines, split headings, or table cells that break on mid-sentence newlines.
- No structured output validation — The JSON/chunks output is a tree of blocks, but there's no schema-level check that a SectionHeader, Text, or Table block actually contains well-formed content. A malformed block can silently propagate into the final markdown.
- Single-page failure aborts the whole document — If one page throws during processing (e.g. an unusual table boundary or a garbled OCR span), the entire conversion crashes, losing all previously processed pages.
Proposed approach (all in post-processing, zero changes to core engines)
| # | Change | Where it lives |
|---|---|---|
| 1 | Add a ChineseTextResilienceProcessor that normalizes CJK punctuation, merges mid-sentence line breaks, and repairs table-cell boundary splits | marker/processors/ (new file, opt-in via --processors) |
| 2 | Add pydantic validators for SectionHeader, Text, and Table block content, with auto-correction for common format errors | marker/schema/ (new validators module, existing block classes unchanged) |
| 3 | Wrap per-page processing in a try/except that logs the error, records the page id in metadata, and continues with remaining pages | marker/converters/pdf.py (build_document, minimal change) |
Design constraints I would commit to
- No changes to providers, builders, layout/OCR models, or core recognition logic.
- Backward compatible — all new behavior is opt-in; default pipeline and output formats stay identical.
- No new hard dependencies — only pydantic (already used) and stdlib.
- Full logging via the project existing logger, with page-level error metadata.
- Unit tests for each fallback path and a small benchmark comparing crash rates on Chinese PDFs before/after.
Questions for you
- Is this a direction you would welcome a PR for, or is it better handled by a custom processor in user code?
- Would you prefer the Chinese-specific logic as a standalone opt-in processor, or folded into the existing TextProcessor / TableProcessor?
- For the per-page error isolation — is there an existing pattern I should follow, or would a new config flag (e.g. --skip-bad-pages) be acceptable?
- Any naming or style conventions I should be aware of beyond what's in the codebase?
I am happy to start with just one of the three pieces (e.g. the per-page skip) as a smaller first PR if that is easier to review. Thank you for your time!
Source: datalab-to/marker