PDF: `pdftotext -layout` interleaves columns on multi-column documents, and the word-based token estimate hides the 26x size blowup

Summary

extract_with_pdftotext() runs pdftotext -layout unconditionally. On a single-column document that is a good default — it keeps table columns aligned. On a multi-column document it is wrong twice over, and the second failure is silent.

Fully reproducible with a public-domain document, no copyrighted book needed: IRS Publication 17 (2025), 142 pages, three columns per page, https://www.irs.gov/pub/irs-pdf/p17.pdf.

1. Reading order — the one that matters

-layout emits each physical line across all three columns, so consecutive sentences from different columns get interleaved. Searching both extractions for the same phrase:

-layout        'standard deduction or if you         (QOF). Taxpayers who made a'

reading order  'Standard deduction amount increased. For 2025, the standard
                deduction amount has been increased for all filers'

The first is three columns welded together. Every downstream step — chapter detection, ToC detection, and the agent doing the distillation in Step 3 — reads that.

This is not a cosmetic problem. A skill distilled from interleaved prose will contain sentences that never existed in the book.

2. Size

-layout pads every line with spaces so columns line up visually. clean_pdftotext() strips repeated headers/footers and dehyphenates, but nothing ever collapses that padding.

The emitted full_text.txt is 26× larger than it needs to be, and 96.8% of it is spaces.

3. Why neither is visible

estimate_tokens() counts words:

return int(len(text.split()) / WORDS_PER_TOKEN)

Column padding does not change word count, and word count says nothing at all about ordering. So metadata.json reports:

"chars": 25264555,
"words": 165049,
"estimated_tokens": 220065,
"estimated_tokens_human": "~220K"

chars is right there and is 25M, but the CLI surfaces the token figure — so the run reports ~220K tokens for a file that costs ~6.3M to read. The number that gets checked and the number that gets paid are different numbers.

The same run also reported Chapters: 3 detected for a publication with dozens, and correctly warned that no ToC was found. I suspect both are downstream of the interleaving rather than separate bugs, though I have not proven that.

Repro

curl -sLO https://www.irs.gov/pub/irs-pdf/p17.pdf
python3 scripts/extract.py p17.pdf
# metadata.json: estimated_tokens ~220K, chars 25,264,555, chapters_detected 3

pdftotext -layout p17.pdf - | grep -o 'standard deduction[^\n]\{0,60\}' | head -1
pdftotext          p17.pdf - | grep -o 'Standard deduction[^\n]\{0,60\}' | head -1

Suggested fix

Detect columns and pick the mode. A crude signal works well: under -layout, count substantive lines containing a wide internal run of spaces (the gutter). On Pub 17 that is 85% of sampled lines; on a single-column PDF it is near zero.

GUTTER = re.compile(r"\S[ ]{4,}\S")

def looks_multicolumn(layout_text, threshold=0.35):
    lines = [ln for ln in layout_text.splitlines() if len(ln.strip()) > 40]
    if not lines:
        return False
    return sum(1 for ln in lines[:4000] if GUTTER.search(ln)) / len(lines[:4000]) >= threshold

Then use plain pdftotext (reading order) when multi-column, keep -layout otherwise. Both extractions are cheap — the whole thing took 11.7s on 142 pages — so running both and choosing is affordable.

Smaller changes that would each help on their own:

1. Estimate tokens from characters, not words, or report both. The word-based figure is defensible as a proxy for content volume but is not what the run will cost. This alone would have made the 26× blowup visible at conversion time.
2. Collapse the padding after clean_pdftotext() for the cases where -layout is still the right choice. Collapsing runs of 2+ spaces to one took Pub 17 from 25,264,555 to 970,243 chars with all 165,049 words intact — but note this makes interleaved text smaller, not correct, so it is not a substitute for fix (1) above.
3. Warn when -layout output looks multi-column, even without changing the mode. Consistent with the reasoning in #47 — surface it rather than swallow it.

Happy to open a PR for the detection approach if the shape looks right to you. I have it working locally against this document.

Environment

• book-to-skill 442aaaa2d21dbe5ae0f7c6a396585ef3a9d2534e (v1.3.0)
• Windows 11, Python 3.12
• poppler pdftotext, pypdf and pdfminer.six all installed; --check reports the PDF chain ready
• --mode text (default); docling not installed, so --mode technical was not exercised