filterTinyText drops spaces, hyphens and em dashes at normal font sizes

Bug

TextProcessor.filterTinyText decides that text is too small to read by measuring the height of the chunk's bounding box. A bounding box covers the glyph's ink, and some ordinary characters draw little or none of it: a space draws nothing and reports height 0.0, while a hyphen and an em dash are thin strokes. At 12pt they are indistinguishable from a 2pt font by that measure, so they get dropped.

Because the filter runs before mergeCloseTextChunks, the chunks on either side are then merged with spacing inferred from geometry. A removed hyphen leaves a space behind, a removed space leaves nothing, so the damage goes in both directions:

Consumers read that as spelling errors in the source rather than as extraction damage.

Steps to reproduce

A Notion page exported to PDF reproduces it (Notion renders through Chromium, Skia/PDF m128). The file is attached; its content is fictional.

opendataloader-pdf --format json notion-export-spacing-repro.pdf

Every hyphen comes back as a space and every em dash is gone:

e mail, the EEA agreement, GDPR fines, the IT department, AI agents, AI systems, AI policy, AI generated summaries and ISO 27001 certification

Running the same file with --content-safety-off tiny returns all of them, which isolates this filter as the cause. On a separate 3-page document, instrumenting the filter showed 1098 chunks dropped: 1060 spaces, 21 hyphens and 17 em dashes, all at 12 to 16.5pt.

It also reproduces without a PDF:

List<IObject> contents = new ArrayList<>();
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 20.0, 10.0), " ", 12, 10.0));
TextProcessor.filterTinyText(contents);
// expected: the space survives, actual: contents.get(0) is null

Version

2.5.8

Java version

openjdk 17.0.20 (Temurin 17.0.20+8)

Suggested fix

Measure font size, which is what "too small to read" means. TextInfoChunk.getFontSize() already carries it:

if (textChunk.getFontSize() <= TEXT_MIN_FONT_SIZE) {

Text hidden at a fraction of a point is still dropped, including a normal font size scaled to near zero by the text matrix. I have this working with tests and am happy to open a PR.

Unrelated observation

With the filter corrected, a U+0000 becomes visible in JSON output where a hyphen has no Unicode mapping. MarkdownGenerator and HtmlGenerator already strip U+0000; the JSON writer does not. Pre-existing, just masked until now.

90f79d80-4d54-4b71-910d-f3b9c0a4c7b9_PDF_extraction_spacing_repro.pdf