[Parsing] Invisible text detected only via render mode 3; three other mechanisms leak into output

Description

should_skip_invisible() treats text as invisible only when its PDF text render mode is 3. That is one of several ways a PDF can carry text that is never painted to the page. Text hidden by a matching fill color, or covered by a later opaque object, is extracted and returned as ordinary body content.

The filter works correctly on the case it knows about. The issue is its scope.

What leaks

The attached probe.pdf places six lines on one page, each hidden a different way. Each line was checked twice: what lit parse returned, and whether the rendered page contains any pixel differing from that line's local background.

Case 4 is worth calling out separately. Nothing about that text object is unusual — plain black, full opacity, ordinary render mode. It is invisible purely because of paint order, so no property of the glyph or its graphics state can reveal it.

Cause

crates/liteparse/src/extract.rs, should_skip_invisible():

if ch.text_render_mode() == Some(3) {
    invisible += 1;
} else {
    visible += 1;
}

Render mode is the only signal consulted. In cases 2, 3 and 4 it is 0 (fill), so invisible stays at 0, the function returns false, and no filtering happens. The 30% visible/invisible ratio gate below it is never reached.

Real-world impact

Case 2 is exactly what the U.S. Government Publishing Office does, so this affects the congress.gov / govinfo bill corpus. Every bill PDF carries two stamps drawn with a white fill: a print-tracking line and an operator/workstation identifier. Both attached bills are affected on every page, and the 2008 one is shown in the Actual Output section below.

The two bills were chosen 18 years apart because they share no toolchain — ACOMP.exe WinVer 1c15 with Distiller 5.0.5 in 2008, against GPO G.S.D.D. with Distiller 26.0 / iText 9.4.0 in 2026. They produce the stamps identically, down to the same rotation matrix 0 5 -5 0 22 18 Tm, which suggests a long-standing convention rather than one bad export. I have tested only these two documents and have not swept the corpus, so treat the scope as "both documents tested, spanning 18 years" rather than a measured corpus-wide figure.

Notes toward a fix

I do not want to over-prescribe, because the general question — "was this glyph actually painted into the output?" — is a compositing question rather than a property of the text object, and case 4 shows it cannot be answered from the glyph alone. Some observations:

• The three cases are not equally hard. Case 2 is nearly free: white fill with nothing painted beneath it. Case 3 needs the fill compared against whatever was painted at that position, which brings in color spaces, alpha, blend modes and pattern fills. Case 4 needs paint order and coverage, which is effectively a rasterization question.
• A cheap partial step exists, with limits worth stating. TextItem already carries fill_color, and a library-API caller can filter on it today — filtering "ffffffff" is exact on the two attached GPO documents, with no false positives. But that is a document-specific workaround, not a general fix: it does nothing for case 3 or 4, and would wrongly drop white text on a dark background. There is also no equivalent handle through the CLI, since page.text and page.markdown arrive with the hidden text already merged in.
• This composes with #337 rather than conflicting with it. That request wants text color preserved as semantic markup for amendment PDFs. Anything done here should suppress text that was never painted, not text that is merely colored, so the two are compatible — and both rest on the same observation that fill_color is already present on TextItem and nothing currently acts on it.

Document

Three documents attached.

probe.pdf — the minimal reproduction. 1050 bytes, hand-built so the content stream operators are exactly as intended with no producer rewriting them. Six lines on one page, each hidden by a different mechanism. The script below regenerates it byte-for-byte with no dependencies, so the reproduction does not depend on the attachment surviving.

# No leading newline: the EOL after the `stream` keyword is a delimiter and is
# NOT part of the stream data, so counting it in /Length overruns by one byte.
CONTENT = b"""BT /F1 14 Tf 0 g 60 740 Td (CASE 1 visible black text on white) Tj ET

BT /F1 14 Tf 1 g 60 700 Td (CASE 2 white fill on unpainted page) Tj ET

1 0 0 rg 50 650 400 26 re f
BT /F1 14 Tf 1 0 0 rg 60 658 Td (CASE 3 red text on a red rectangle) Tj ET

BT /F1 14 Tf 0 g 60 610 Td (CASE 4 black text occluded by a later box) Tj ET
1 g 50 602 400 26 re f

BT /F1 14 Tf 0 g 3 Tr 60 560 Td (CASE 5 black text at render mode 3) Tj ET

BT /F1 14 Tf 0 g 0 Tr 60 520 Td (CASE 6 visible black text, second control) Tj ET
"""


def build(path):
    objs = [
        b"<< /Type /Catalog /Pages 2 0 R >>",
        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
        b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
        b"<< /Length " + str(len(CONTENT)).encode() + b" >>\nstream\n" + CONTENT + b"\nendstream",
        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
    ]
    out = bytearray(b"%PDF-1.4\n")
    offsets = []
    for i, body in enumerate(objs, start=1):
        offsets.append(len(out))
        out += str(i).encode() + b" 0 obj\n" + body + b"\nendobj\n"
    xref = len(out)
    n = len(objs) + 1
    out += b"xref\n0 " + str(n).encode() + b"\n0000000000 65535 f \n"
    for off in offsets:
        out += ("%010d 00000 n \n" % off).encode()
    out += (b"trailer\n<< /Size " + str(n).encode() + b" /Root 1 0 R >>\nstartxref\n"
            + str(xref).encode() + b"\n%%EOF\n")
    open(path, "wb").write(bytes(out))


build("probe.pdf")

BILLS-110hr7337ih.pdf and BILLS-119hr10136ih.pdf — real-world instances of case 2. U.S. Government Publishing Office bill PDFs from congress.gov, public domain. Produced 18 years apart by unrelated toolchains and affected identically on every page.

• https://www.congress.gov/119/bills/hr10136/BILLS-119hr10136ih.pdf
• https://www.congress.gov/110/bills/hr7337/BILLS-110hr7337ih.pdf

BILLS-119hr10136ih.pdf BILLS-110hr7337ih.pdf probe.pdf

Expected Output

probe.pdf — only the two lines actually painted to the page. Cases 2, 3 and 4 are present in the content stream but never rendered, so they should be filtered the same way case 5 already is.

--- Page 1 ---
CASE 1 visible black text on white

CASE 6 visible black text, second control

BILLS-110hr7337ih.pdf, page 2 — the bill text without the two white-filled GPO stamps.

--- Page 2 ---
       2
    1  Code, to disclose the nature and source of delays and can-
    2  cellations experienced by air travelers.
       Æ
    •HR 7337 IH

BILLS-119hr10136ih.pdf, page 1 (abridged — body text elided at [...]) — same two stamps removed, here bracketing the entire page.

--- Page 1 ---
    I

119TH CONGRESS  H. R 10136
  2D SESSION       .

[...]

    8      (a) DEFINITIONS.-In this section:

(The Æ on page 2 of the 2008 bill is correct and not part of this report: it is GPO's end-of-bill dingbat, drawn in a custom font whose /Differences names the slot AE and whose ToUnicode maps it to U+00C6. Poppler emits the same character.)

Actual Output

=== lit parse probe.pdf ===

--- Page 1 ---
CASE 1 visible black text on white

CASE 2 white fill on unpainted page

CASE 3 red text on a red rectangle

CASE 4 black text occluded by a later box




CASE 6 visible black text, second control


=== lit parse BILLS-110hr7337ih.pdf  (page 2 only) ===

--- Page 2 ---
jbell on PROD1PC69 with BILLS
       2
    1  Code, to disclose the nature and source of delays and can-
    2  cellations experienced by air travelers.
       Æ
    •HR 7337 IH
VerDate Aug 31 2005 05:35 Dec 11, 2008 Jkt 079200 PO 00000 Frm 00002 Fmt 6652 Sfmt 6301 E:\BILLS\H7337.IH H7337


=== lit parse BILLS-119hr10136ih.pdf  (page 1, abridged - body text elided) ===

--- Page 1 ---
ssavage on LAPJG3WLY3PROD with BILLS

    I

119TH CONGRESS  H. R 10136
  2D SESSION       .

[...]

    8      (a) DEFINITIONS.-In this section:

VerDate Sep 11 2014 01:51 Aug 21, 2026 Jkt 069200 PO 00000 Frm 00001 Fmt 6652 Sfmt 6201 E:\BILLS\H10136.IH H10136

Command Used

lit parse probe.pdf
lit parse BILLS-110hr7337ih.pdf
lit parse BILLS-119hr10136ih.pdf

LiteParse Version

2.13.0

Operating System

macOS (Apple Silicon)

Additional Context

How "painted to page" was determined

Pages were rendered with Ghostscript 10.07.1 at 150 dpi and sampled inside each text item's reported bounding box, comparing against the local background rather than against white — case 3 sits on red, so a white-only test would have misread it.

Every text item on every page of both GPO documents was audited this way, 556 in total. The split is clean with no exceptions: 486 items with fill_color ff000000, every one drawn; 70 items with ffffffff, every one at exactly zero differing pixels. No item fell on the wrong side in either direction.

This check reads TextItem.x/.y as PDF points, per the contract restored in #243. Confirmed to hold on 2.13.0 before relying on it: y spans 74.4 to 767.2 on a 792-point page, out of range for the grid line index that issue described. On a build predating that fix the bounding-box sampling will not reproduce, though the extraction defect itself still will.

Config options tried

No existing option affects any of this. skip_diagonal_text, preserve_very_small_text, keep_headers_footers and extract_blocks were each tried against the GPO documents with no change in output. skip_diagonal_text=True does not catch the rotated operator stamp either, since its 270 degree rotation is axis-aligned rather than diagonal.

Environment

No OCR server and no config file. OCR was disabled for the GPO measurements; the probe was run through the plain CLI with no flags. liteparse 2.13.0 (PyPI wheel, and lit 2.13.0 from Crates & CLI v2.13.0), macOS 26.5.1 / Darwin arm64, Python 3.12.13.