textord 噪声滤波器会在带点的阿拉伯文语言中静默地丢弃整行文本(已验证:波斯语、阿拉伯语、乌尔都语); textord_noise_rejrows=0 可将其恢复

Current behavior

With default settings, entire text lines are silently missing from the output for dotted Arabic-script languages — no warning, no trace in confidence stats. Adding -c textord_noise_rejrows=0 -c textord_noise_rejwords=0 recovers every dropped line. Measured per-line similarity (best difflib ratio of each ground-truth line against any output line), default vs. flags:

Language	Lines dropped by default	Recovered by the flags
Persian (fas)	3 of 5 pure-script lines (~30% sim → 71–95%)	all 3
Arabic (ara)	1 of 5 (42% → 100%)	yes
Urdu (urd)	1 of 5 (43% → 82%)	yes
Pashto (pus)	0 on our sample page	—
In every language the control line containing Latin words (Robert Brandom, John McDowell) and the digit-heavy line survive. Pashto is listed for completeness: same script family, did not reproduce on our particular sample — susceptibility appears to depend on the exact dot-to-body ratio of the rendered text.
Sample output of the repro script (Persian):

line   default  rejrows=0
   1       61%        71%
   2       29%        71%  <-- DROPPED by default, recovered by the flag
   3       34%        84%  <-- DROPPED by default, recovered by the flag
   4       55%        55%  (control line with Latin words - survives)
   5      100%       100%  (digit-heavy line - survives)
   6       38%        95%  <-- DROPPED by default, recovered by the flag

Expected behavior

All text lines appear in the output (possibly with recognition errors), or at least a diagnostic is emitted when a whole row is rejected as noise.

Analysis

Letters in these scripts carry dots as an integral part of the glyph (پ چ ژ گ ب ت ث ن / ب ت ث ج خ ...). textord's row-level noise rejection (textord_noise_rejrows, around Textord::clean_noise_from_row) counts these dots as noise specks; when the dot-to-body ratio of a row crosses the threshold, the whole row is rejected as noise before the LSTM recognizer ever sees it. That makes the failure invisible: confidence stats look fine, there is simply less text. The Latin-word asymmetry is what makes this dangerous in practice: a line with one embedded Latin token changes the ratio and survives, so on real mixed pages the data loss looks random and users blame scan quality or the model rather than layout analysis.

Suggested direction

Either exempt scripts whose letters are inherently dotted from row-level noise rejection (script info is available at that stage), or