Parser subprocess OOMs on large SingleFile archives with many inline images (proposal: strip/downscale large data: images before parsing, like inline videos)

First off — thank you for Karakeep. It has become the system-of-record core of our personal knowledge pipeline (SingleFile → n8n → Karakeep + Wallabag fan-out), and the SingleFile integration in particular is exactly the workflow we wanted: one click in the browser, and a faithful archived copy lands in Karakeep with tagging, embeddings and search. The effort behind it is very much appreciated — this issue is a corner case, not a complaint.

Environment

• Karakeep: release image (0.33.x line)
• Ingest: SingleFile browser extension → POST /api/v1/bookmarks/singlefile (the official integration)
• Crawler settings: CRAWLER_PARSER_MEM_LIMIT_MB=3072, CRAWLER_PARSE_TIMEOUT_SEC=180, CRAWLER_JOB_TIMEOUT_SEC=240

What happens

SingleFile archives of image-heavy articles (tutorials with a screenshot per step, photo galleries) inline every image as a data: URI. The HTML itself is small, but the base64 payload is not:

The parse subprocess (metascraper + Readability) is killed by V8 OOM on the 56 MB archive even with a 3072 MB heap:

[Crawler][439:4] Parse subprocess OOM killed (exit code undefined).
Consider increasing CRAWLER_PARSER_MEM_LIMIT_MB (currently 3072MB).

Each job then burns its retries (we observed 6+ runs per bookmark) and the bookmark is left without readable content. With MAX_ASSET_SIZE_MB raised to 100 (as the SingleFile integration guide recommends), this class of archives now reliably reaches the parser.

For calibration, 20–29 MB archives parse fine within the same limits (~104 s for a 23.6 MB one — parse timeout also had to be raised from the 60 s default). The failure mode is specific to archives whose base64 payload dominates the file.

Why not just raise the limit

A 56 MB archive needs >3 GB of heap; extrapolating (DOM representation is a multiple of the source text), 80–100 MB archives would need 5+ GB. That is not a sustainable curve — the interesting property of these archives is that the bulk of the payload is image bytes that readability doesn't need.

Proposal

You already solve the exact same problem for inline videos: "Large singlefile archives (typically because of videos) used to fail metadata extraction as the parser used to OOM. We strip inline videos now to make the parsing succeed." (release notes, drop inline media blocks before metadata parsing).

Could the same pre-parse pass either:

1. strip large data: image URIs above a size threshold (e.g. configurable CRAWLER_MAX_INLINE_IMAGE_BYTES, defaulting to something like 512 KB — the same ballpark Readability uses to distinguish placeholders), or
2. replace them with their original URL when available: SingleFile's "save original URLs" option records the source address in data-sf-original-src next to the inlined src. Swapping src=data:... → src=<data-sf-original-src> before parsing shrinks the document by ~95% while keeping the reader fully functional (it already fetches remote banner images).

Option 2 matches what the wider ecosystem does (e.g. phpdocker-io/readability-js-server restores data-src/data-sf-original-src before running Readability for the same reason), and both keep the untouched archive as the stored asset — only the parsing/reader copy would be lighter.

Secondary note: if the parse subprocess is still on JSDOM, linkedom is reported to cut parsing memory ~3× (JSDOM measured ~1 GB heap on a 12 MB document); that alone would move the OOM threshold meaningfully.

Workarounds we use meanwhile

• SingleFile imageReductionFactor: 2 (halves image dimensions on save — archives shrink ~3×)
• Raised parser mem/time limits as above

Happy to turn this into a PR if the approach sounds acceptable — option 1 with a threshold looks self-contained (same place the video stripping happens).