bug: a text layer that decodes to the wrong characters is not detected
Deployment
Other
Version
Branch fix/1667-glyph-text-quality-gate at 7a50f41370, and main.
Description
The reported document returns readable-looking text that is actually wrong: the font's own mapping resolves each character, but to the wrong one. Forcing OCR returns correct text, so the content is recoverable.
The work on #1667 routes a page to OCR when the character mapping had to be fabricated, meaning no mapping tier resolved it at all. That is a different shape. The reported file resolves through its own embedded mapping and therefore never reaches that signal.
Measured at that branch with OCR enabled: the document still extracts natively, OCR is not used, and the result is the same as on main. A trace confirms the new routing code ran and declined, because the provenance is a resolved tier rather than a fabricated one.
So the reported case remains open. What #1667 does fix, and this issue does not cover, is that the provenance value was empty for virtually every font because of a lookup key mismatch.
Steps to reproduce
- Extract the reported document with OCR enabled.
- Observe the extraction method is native and OCR was not used.
- Compare the returned text against the same document with OCR forced.
Logs and configuration
The detection needs a signal for text that decodes successfully but decodes to something implausible, rather than for text that failed to decode. Character-class statistics do not separate them: on this document the alphanumeric ratio is 0.473 against a 0.3 floor and the undecodable ratio is 1.3% against a 0.5 trigger, so both gates pass.
Possible directions, none yet measured: dictionary or language-model plausibility on the decoded text, a comparison against an OCR sample of one page, or treating a font whose mapping is non-standard as suspect regardless of whether it resolved.
Source: xberg-io/xberg