textord noise filter silently drops entire text lines in dotted Arabic-script languages (verified: Persian, Arabic, Urdu); textord_noise_rejrows=0 recovers them
Author: aminbm1919Created Jul 15, 2026Updated Jul 15, 2026
## Environment
- Tesseract 5.5.0 (`v5.5.0.20241111`, Windows x64, UB-Mannheim build), leptonica 1.85.0, LSTM engine (`--oem 1`)
- Reproduced with **both** `tessdata_best` and the default (fast) models for `fas`, `ara`, `urd` — the loss happens in layout analysis, before recognition, so the model choice does not matter
- Clean synthetic input: 300-dpi grayscale renders of A4 pages, Tahoma 15px, ordinary paragraph text (no noise, no skew, no scanning artifacts)
## Current behavior
With default settings, **entire text lines are silently missing from the output** for dotted Arabic-script languages — no warning, no trace in confidence stats. Adding `-c textord_noise_rejrows=0 -c textord_noise_rejwords=0` recovers every dropped line.
Measured per-line similarity (best difflib ratio of each ground-truth line against any output line), default vs. flags:
| Language | Lines dropped by default | Recovered by the flags |
|---|---|---|
| Persian (fas) | 3 of 5 pure-script lines (~30% sim → 71–95%) | all 3 |
| Arabic (ara) | 1 of 5 (42% → 100%) | yes |
| Urdu (urd) | 1 of 5 (43% → 82%) | yes |
| Pashto (pus) | 0 on our sample page | — |
In every language the control line containing Latin words (`Robert Brandom`, `John McDowell`) and the digit-heavy line **survive**. Pashto is listed for completeness: same script family, did not reproduce on our particular sample — susceptibility appears to depend on the exact dot-to-body ratio of the rendered text.
Sample output of the repro script (Persian):
```text
line default rejrows=0
1 61% 71%
2 29% 71% <-- DROPPED by default, recovered by the flag
3 34% 84% <-- DROPPED by default, recovered by the flag
4 55% 55% (control line with Latin words - survives)
5 100% 100% (digit-heavy line - survives)
6 38% 95% <-- DROPPED by default, recovered by the flag
```
## Expected behavior
All text lines appear in the output (possibly with recognition errors), or at least a diagnostic is emitted when a whole row is rejected as noise.
## Analysis
Letters in these scripts carry dots as an integral part of the glyph (پ چ ژ گ ب ت ث ن / ب ت ث ج خ ...). textord's row-level noise rejection (`textord_noise_rejrows`, around `Textord::clean_noise_from_row`) counts these dots as noise specks; when the dot-to-body ratio of a row crosses the threshold, the whole row is rejected as noise **before the LSTM recognizer ever sees it**. That makes the failure invisible: confidence stats look fine, there is simply less text.
The Latin-word asymmetry is what makes this dangerous in practice: a line with one embedded Latin token changes the ratio and survives, so on real mixed pages the data loss looks random and users blame scan quality or the model rather than layout analysis.
## Suggested direction
Either exempt scripts whose letters are inherently dotted from row-level noise rejection (script info is available at that stage), or lower the default aggressiveness for those scripts, or at minimum document `textord_noise_rejrows=0` as the recommended setting for Arabic-script languages — the silent, warning-free data loss is the worst part.
## Workaround for users
```
tesseract page.png out -l fas -c textord_noise_rejrows=0 -c textord_noise_rejwords=0
```
## Reproduction
Self-contained script (Python 3 + `pip install pymupdf`; renders the test pages itself — needs any font with full Arabic-script coverage, defaults to Windows Tahoma, override with `FONT_DIR`/`FONT_FILE`; on Linux e.g. Noto Naskh Arabic):
repro_textord_dropped_lines.py
```python
# -*- coding: utf-8 -*-
"""Repro: Tesseract's textord noise filter silently drops entire text lines
in dotted Arabic-script languages (fas/ara/urd verified).
Requires: Python 3 + pymupdf (pip install pymupdf), a Tesseract 5.x binary,
and fas/ara/urd traineddata (best or fast).
Environment overrides (all optional):
TESSERACT path to the tesseract binary (default: "tesseract" on PATH)
TESSDATA folder holding the traineddata (default: engine's own)
FONT_DIR folder containing FONT_FILE (default: C:/Windows/Fonts)
FONT_FILE a font with full Arabic-script coverage (default: tahoma.ttf)
Usage: python repro_textord_dropped_lines.py [fas ara urd pus]
"""
import difflib
import os
import subprocess
import sys
import tempfile
import fitz # PyMuPDF
sys.stdout.reconfigure(encoding="utf-8")
TESSERACT = os.environ.get("TESSERACT", "tesseract")
TESSDATA = os.environ.get("TESSDATA")
FONT_DIR = os.environ.get("FONT_DIR", "C:/Windows/Fonts")
FONT_FILE = os.environ.get("FONT_FILE", "tahoma.ttf")
# Per language: dot-heavy full-width lines + one Latin-containing control
# line. The control line demonstrates the asymmetry (it always survives).
SAMPLES = {
"fas": [
"فلسفه در یونان باستان با پرسش از هستی آغاز شد و نخستین فیلسوفان میکوشیدند جهان را بدون توسل به اسطوره توضیح دهند.",
"هگل در کتاب پدیدارشناسی روح استدلال میکند که آگاهی در فرایندی دیالکتیکی از یقین حسی به دانش مطلق حرکت میکند.",
"مارکس این روش را از هگل گرفت اما آن را وارونه کرد؛ به گفتهٔ او دیالکتیک نزد هگل روی سر ایستاده است.",
"پژوهشگران معاصر مانند Robert Brandom و John McDowell خوانشهای تازهای از هگل ارائه کردهاند.",
"این کتاب در سال ۱۸۰۷ منتشر شد و شامل ۸ فصل و ۵۹۱ صفحه است.",
"آیا میتوان گفت فلسفه همان تاریخ فلسفه است؟ برخی فیلسوفان چنین باوری دارند.",
],
"ara": [
"بدأت الفلسفة في اليونان القديمة بالسؤال عن الوجود وكان الفلاسفة الأوائل يحاولون تفسير العالم من غير الرجوع إلى الأسطورة.",
"يرى هيغل في كتاب ظاهريات الروح أن الوعي يتحرك في مسار جدلي من اليقين الحسي إلى المعرفة المطلقة.",
"أخذ ماركس هذا المنهج من هيغل لكنه قلبه رأسا على عقب فقال إن الجدل عنده يقف على رأسه.",
"درس باحثون معاصرون مثل Robert Brandom و John McDowell قراءات جديدة لهذا الكتاب.",
"نشر هذا الكتاب سنة ١٨٠٧ ويضم ثمانية فصول وكثيرا من الصفحات.",
"هل يمكن القول إن الفلسفة هي تاريخ الفلسفة؟ بعض الفلاسفة يعتقدون ذلك.",
],
"urd": [
"فلسفے کا آغاز قدیم یونان میں ہستی کے سوال سے ہوا اور پہلے فلسفی دنیا کو اسطورہ کے بغیر سمجھانے کی کوشش کرتے تھے۔",
"ہیگل اپنی کتاب میں کہتا ہے کہ شعور ایک جدلیاتی عمل میں حسی یقین سے مطلق علم کی طرف بڑھتا ہے۔",
"مارکس نے یہ طریقہ ہیگل سے لیا لیکن اسے الٹ دیا اس کے بقول جدلیات ہیگل کے ہاں سر کے بل کھڑی ہے۔",
"جدید محققین مثلاً Robert Brandom اور John McDowell نے اس کتاب کی نئی تعبیریں پیش کی ہیں۔",
"یہ کتاب ۱۸۰۷ میں شائع ہوئی اور آٹھ ابواب پر مشتمل ہے۔",
"کیا کہا جا سکتا ہے کہ فلسفہ ہی تاریخ فلسفہ ہے؟ بعض فلسفی ایسا مانتے ہیں۔",
],
"pus": [
"فلسفه په لرغوني يونان کې د هستۍ له پوښتنې پيل شوه او لومړنيو فيلسوفانو هڅه کوله چې نړۍ له افسانې پرته تشريح کړي.",
"هيګل وايي چې شعور په يوه جدلي بهير کې له حسي باور څخه د مطلق پوهې خوا ته ځي.",
"مارکس دا طريقه له هيګل څخه واخيسته خو سرچپه يې کړه او ويې ويل چې جدليات د هيګل په اند پر سر ولاړ دي.",
"معاصرو څېړونکو لکه Robert Brandom او John McDowell د دې کتاب نوې لوستنې وړاندې کړې دي.",
"دا کتاب په ۱۸۰۷ کال کې خپور شو او اته څپرکي لري.",
"ايا ويلای شو چې فلسفه د فلسفې تاريخ دی؟ ځينې فيلسوفان داسې باور لري.",
],
}
def build_page(lines, png_path):
doc = fitz.open()
page = doc.new_page(width=595, height=842) # A4
css = "@font-face {font-family: f; src: url(%s);}" % FONT_FILE
html = (''
+ "".join(f"
{l}
" for l in lines) + "") page.insert_htmlbox(fitz.Rect(50, 60, 545, 800), html, css=css, archive=fitz.Archive(FONT_DIR)) pix = page.get_pixmap(dpi=300, colorspace=fitz.csGRAY) pix.save(png_path) doc.close() def ocr(png, lang, extra): with tempfile.TemporaryDirectory() as td: base = os.path.join(td, "o") cmd = [TESSERACT, png, base, "-l", lang, "--oem", "1", "--psm", "3", "-c", "tessedit_create_txt=1"] + extra if TESSDATA: cmd[3:3] = ["--tessdata-dir", TESSDATA] subprocess.run(cmd, capture_output=True) with open(base + ".txt", encoding="utf-8") as f: return f.read() def best_sim(gt_line, output): return max((difflib.SequenceMatcher(None, gt_line, o.strip()).ratio() for o in output.splitlines()), default=0.0) def main(): langs = [a for a in sys.argv[1:] if a in SAMPLES] or list(SAMPLES) reproduced = {} for lang in langs: lines = SAMPLES[lang] png = os.path.join(tempfile.gettempdir(), f"textord_{lang}.png") build_page(lines, png) default = ocr(png, lang, []) fixed = ocr(png, lang, ["-c", "textord_noise_rejrows=0", "-c", "textord_noise_rejwords=0"]) print(f"\n=== {lang} === page: {png}") print(f"{'line':>4} {'default':>8} {'rejrows=0':>9}") dropped = 0 for i, gt in enumerate(lines, 1): d, f = best_sim(gt, default), best_sim(gt, fixed) mark = "" if d < 0.5 <= f: dropped += 1 mark = " <-- DROPPED by default, recovered by the flag" print(f"{i:>4} {d:>8.0%} {f:>9.0%}{mark}") reproduced[lang] = dropped print(f"{lang}: lines silently dropped by default: {dropped}") print("\nsummary:", {k: v for k, v in reproduced.items()}) return 0 if any(reproduced.values()) else 1 if __name__ == "__main__": raise SystemExit(main()) ``` *(A Persian translation of this report follows in the first comment, for the primarily-affected user community.)*Source: tesseract-ocr/tesseract