[Bug] Silent content loss (rotated glyphs dropped, overflow text drawn outside box, LLM-empty-response batches dropped)

[Bug] Silent content loss (rotated glyphs dropped, overflow text drawn outside box, LLM-empty-response batches dropped)

babeldoc version: 0.6.4 (pip: babeldoc==0.6.4) Environment: macOS, Python 3.12, translating English → Vietnamese (target text ~20–40% longer than source), OpenAI-compatible endpoint (DeepSeek).

Summary

While integrating babeldoc into a PDF translation pipeline, we found several independent issues that cause silent content loss or layout corruption with no error/warning, all reproduced on real production PDFs and traced to specific source lines in the installed 0.6.4 package. Two small single-page repro PDFs (extracted from a real book, no hand-edited content) are available — happy to attach them to this issue or share via gist, whichever this tracker prefers.

Bug A — Glyphs with rotation angle outside 0°/90° (±0.1°) are silently dropped at parse time

format/pdf/document_il/frontend/il_creater.py, on_lt_char() around line 968-974: the function returns early (drops the character) if the text-matrix rotation angle is not within -0.1..0.1 or 89.9..90.1 degrees. There is no fallback path — the character never enters the IL, is never translated, and (because babeldoc rebuilds the content stream rather than patching it) the original glyph is also gone from the output. No log line is emitted.

Also confirmed the renderer (format/pdf/document_il/backend/pdf_creater.py:111-120) and PdfCharacter (il_version_1.py:627-663) have no rotation-angle field at all — so even loosening the threshold would only redraw the character horizontally, not preserve its original angle. This looks like a deliberate architectural limitation, not just a bad threshold.

Repro: rotated_text_p67_source.pdf (single real page containing a ~-11° rotated caption block, 16 lines — available on request, see Repro files section below). Running babeldoc --pages 1 rotated_text_p67_source.pdf against any translator: the rotated block is completely absent from the output PDF, with no error.

Measured impact on a 418-page real book: 7,832 characters across 19 pages use rotation outside the accepted range — all silently lost.

Suggested fix: at minimum, log a warning (logger.warning(...)) when a character is dropped for this reason (il_creater.py:973), so downstream tools/users can detect the loss instead of finding a blank region in the output with no diagnostic trail.

Bug B — Overflowing paragraphs are deliberately drawn outside their bounding box, overlapping the next block

format/pdf/document_il/midend/typesetting.py, around line 1349-1350: each paragraph is typeset starting from its own original bbox top (current_y = box.y2 - avg_height), with no awareness of how far the previous paragraph actually overflowed.

Around line 1440-1444: when a line would exceed the bottom of the box, the code sets all_units_fit = False but explicitly continues (a comment in the source states the overflow is intentional, in Chinese: “这里不要 break，继续排版剩余内容” — "don't break here, continue laying out the rest") — i.e. text is drawn past the box boundary on purpose.

Box expansion (get_max_bottom_space / get_max_right_space, ~line 1017-1062) is only attempted once font scale has already dropped below 0.7, and only expands into free space — in a fixed-size decorative box or table cell there is none, so the only remaining behavior is overflow onto the block below.

The one overlap-avoidance pass that exists (~line 1172-1211) shrinks the upper box from the bottom rather than pushing the lower block down — which reduces the upper box's own capacity and makes it more likely to overflow, the opposite of the intended effect.

Measured impact: on real production output translating EN→VI, overlapping text-block pairs (intersection >5% of the smaller block's area) went from a handful in the source PDF to several times more in the translated PDF, including on plain single-column prose pages that had zero overlaps in the source.

Suggested fix: this is presumably a deliberate tradeoff for text that never fits regardless of scale, but a per-paragraph option to hard-truncate or flag instead of overflow would let downstream consumers choose predictable behavior. Also — independently of the tradeoff — the overlap-avoidance pass shrinking the wrong box (upper instead of pushing the lower block down) looks like a straightforward logic bug worth revisiting.

Bug C — Reasoning-model / long responses silently drop whole paragraph batches via hardcoded max_tokens=2048

translator/translator.py (OpenAITranslator.do_llm_translate, ~line 324) and tools/executor/translator.py (~line 74) hardcode max_tokens=2048 for the LLM translate call, with no CLI flag to override it. Against an OpenAI-compatible reasoning model, the model can consume the entire 2048-token budget on reasoning content, returning an empty content string. format/pdf/document_il/midend/il_translator_llm_only.py (~line 852) then calls json.loads(""), which raises, and the exception handling drops the entire paragraph batch — after babeldoc has already removed the original glyphs from the content stream, so the result is a blank region with no error surfaced to the caller.

Suggested fix: expose max_tokens as a configurable value (CLI flag / translator config), or at minimum detect an empty completion and retry/flag rather than silently dropping the batch.

Repro files

• rotated_text_p67_source.pdf — repro for Bug A (single real page, rotation ~-11°)
• rotated_chart_p15_source.pdf — repro for Bug A on a rotated table (also loses ~39/58 blocks)

Both are single pages extracted directly from a real book with PyMuPDF (insert_pdf), no hand-edited content. Not attached to keep the initial report small — will attach on request or in a follow-up comment.

Happy to open small PRs for (a) the missing warning log in Bug A and (b) the overlap-avoidance direction fix in Bug B if that's useful — flagging here first in case there's already known context or a different intended design.