[M1] Cap page-render DPI to bound single-page peak memory (the real lever for #458)
Relates to: #458 (OOM), follow-up to #543 · related to #562 (native-resolution extraction)
Background
PR #543 (#458 hotfix) removed accumulation-driven heap growth: BufferedImage.flush()
after each write (M0) and dropping the retained ContrastRatioConsumer reference (M2).
Independent re-review (decompile of verapdf's render path + before/after build on a
public sample, PDFUA-Ref-2-09_Scanned.pdf) confirmed those help documents with many
images, but do not lower the single-page render peak, which is the dominant cost on
large / high-DPI scanned pages.
The #458 OOM stack is a single large allocation during full-page rendering, before any extraction or flush runs:
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferInt.<init>
at java.awt.image.Raster.createPackedRaster
at org.apache.pdfbox.pdmodel.graphics.color.PDDeviceRGB.toRGBImage
at org.apache.pdfbox.rendering.PDFRenderer.renderImageWithDPI ← peak allocation here
at org.verapdf...ContrastRatioConsumer.renderPage
at org.verapdf...ContrastRatioConsumer.getRenderPageflush() (M0) runs only after getPageSubImage() returns — i.e. after this allocation
has already happened. So #543 cannot move this peak. M1 is the actual memory lever for
#458.
Problem
StaticLayoutContainers.getContrastRatioConsumer(sourcePdfPath, password, enableAntialias, imagePixelSize)
already threads an imagePixelSize (Float) parameter into the verapdf
ContrastRatioConsumer constructor. Decompilation of wcag-algorithms 1.29.45 confirms
verapdf uses this value as a scale factor, not an absolute pixel cap — roughly
renderDPI ≈ imagePixelSize * max(pageWidth, pageHeight) / pageBounds. But every call
site passes null, so no scaling is ever applied:
ImagesUtils— image extraction (bothwriteImage/writePicturepaths) →..., false, null)HiddenTextProcessor(--filter-hidden-text) →..., false, null)(HiddenTextProcessor.java:45)
With imagePixelSize == null, verapdf renders the full page at its default DPI. A large
scanned page at default DPI is a multi-hundred-MB DataBufferInt, which is exactly the
allocation that OOMs in #458.
Proposal
- Route a non-null
imagePixelSizefrom the call sites intogetContrastRatioConsumer(...)so the render is bounded. Candidate target: ~144 DPI effective — note this is the current implicit default that #562 wants to raise (see Related), so M1 and #562 pull in opposite directions and must be reconciled. The exact value follows from the verapdf scale-factor formula above, not a hard DPI constant. - Translate the cap through verapdf's scale-factor semantics. Since
imagePixelSizeis a scale factor (confirmed by decompile), the implementation must compute theimagePixelSizethat yields the desired effective DPI for a given page size, rather than passing a DPI directly. Validate on a large page that the resulting raster size is bounded as intended before locking a value. - Make the cap configurable via a
--max-render-dpi-style flag (forward-compatible with S2--rendering-strategy, but does not require S2 to land), so users who need full-fidelity extraction can opt out — this is the seam that lets #562 coexist. - Add a local memory regression check at a low
-Xmxon a large-page fixture (forward-compatible with S6 CI matrix, but does not require the full S6 matrix).
Acceptance criteria
- Before implementation: document how
imagePixelSizemaps to effective DPI for a given page (the verapdf scale-factor formula), and theimagePixelSizevalue chosen to hit the target cap. Record in the PR description. - At least one call site passes a non-null
imagePixelSize; default behaviour applies a documented effective-DPI cap. - Reproducible memory check on
PDFUA-Ref-2-09_Scanned.pdf(or an equivalent large scanned page): - Baseline (main): OOMs or peak heap clearly elevated at a stated-Xmx(e.g. the-Xmxvalue that currently fails). - After fix: completes at that same-Xmx, with a measured peak-heap reduction. - Measurement method stated (e.g.jcmd <pid> GC.heap_info/ Java Flight Recorder), observed reduction reported in the PR. - No regression in extracted-image output for normal documents below the cap (byte-compare on the existing public sample).
- Cap is configurable via a flag, not hard-coded.
- Quality/accuracy impact of the cap on contrast-ratio (hidden-text) detection is assessed and documented.
Closing plan
Keep #458 open until this lands, then close them together. #543 alone is a scoped hotfix (accumulation + diagnostics) and #458 was intentionally kept open for this lever. Do not close #458 on #543's merge alone — that would force a close-then-reopen if M1 slips. The per-reporter heap estimates walked back in #543 (the "≈2 GB" numbers) depend on this cap landing. Reporters should see one coherent roadmap: "#458 → hotfixed by #543 (M0/M2/M3) + closed by M1 (this issue)."
Related
- #562 — "Extract images at native resolution instead of downscaling to 144 DPI."
This is the inverse pressure: #562 wants higher resolution for image quality, M1
wants a bounded render for memory. Both touch the same
imagePixelSizelever. The configurable--max-render-dpiflag (proposal #3) is the reconciliation point — it lets M1 cap by default for safety while letting #562's users opt into full fidelity. M1 and #562 should be designed with awareness of each other, but solve different problems and are not duplicates.
Out-of-scope references (from #543)
S1 (duplicate render under --filter-hidden-text), S2 (--rendering-strategy), S3 (LRU
eviction), S4 (explicit render phase), S9 (direct sub-image extraction). M1 is the
prerequisite for a meaningful memory ceiling; the S-items refine it.
Source: opendataloader-project/opendataloader-pdf