0.2.4: one image-only page makes the whole PDF convert to nothing, including the pages that do have text

In 0.2.4 a PDF whose pages are mostly text but which contains one image-only page converts to nothing at all: the NeedsOcr error replaces the output instead of annotating it. The pages that have a text layer are lost with it.

Measured on 0.2.4 (Node, macOS arm64), same two text pages in both files:

CLI: anydoc mixed.pdf exits 3 with anydoc: page 3 of 3 needs OCR and writes nothing to stdout.

Adding a page that cannot be read removes the pages that can.

Why I think the previous behaviour was fixed in the wrong direction

The defect in ≤0.2.3 was not that the image-only page was missing from the markdown. Markdown of a page made of pixels is genuinely not available without OCR, and anydoc is right not to do OCR. The defect was that nothing said so: the output looked like a complete document, and a reader had no way to know a page was absent. I measured that on 0.2.3 — a 3-page mixed PDF converts, exit 0, no warning, page 3 simply not there.

An undeclared loss is cured by a declaration. 0.2.4 cures it by refusing, which trades a partial result for none. For a 400-page scanned-in-parts report, one photographed page now costs the other 399.

Proposal

1. Keep the hard error only where there is nothing to return — every page image-only. There the refusal is the honest answer and 0.2.4 already gets it right.
2. For a mixed document, convert and report. Return the markdown, and carry needsOcr with pages and pageCount on the result rather than on an error. CLI: exit 0, the notice on stderr, the markdown on stdout. The caller decides whether a partial document is usable — for many pipelines it is, and today they cannot even see it.
3. Put a marker in the markdown where the page was. Something like > [page 3 of 3: image-only, not converted]. This matters more than it looks: the markdown outlives the call that produced it. Once it is written to a file or pasted into a prompt, any result-level metadata is gone and the document looks complete again. A marker in the text is the only signal that travels with the text.

Point 3 is the part I would argue for hardest. It is also the cheapest: one line in the renderer, and it makes the omission legible to a human reading the output rather than only to the code that called the API.

A note on ocr: 'hosted'

Hosted OCR is a good option to have, but it cannot be the general answer to this, because for some callers sending the file anywhere is not a technical choice. Documents covered by a confidentiality obligation, or belonging to a third party, stay on the machine that converts them. Those callers need the partial conversion plus the declaration, which is exactly what proposal 2 and 3 give them.

Reproduction

// npm i @firecrawl/anydoc pdfkit
const PDFDocument = require('pdfkit');
const { createWriteStream, writeFileSync } = require('node:fs');
const { toMarkdown } = require('@firecrawl/anydoc');

// an 8x8 grey PNG, enough to make a page that holds no text
writeFileSync('px.png', Buffer.from(
  'iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAAAAADhZOFXAAAAAmJLR0QA/4ePzL8AAAAHdElNRQfqCBsUHRbc9Nv6AAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDI2LTA4LTI3VDIwOjI5OjIyKzAwOjAwzqQJtwAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyNi0wOC0yN1QyMDoyOToyMiswMDowML/5sQsAAAAodEVYdGRhdGU6dGltZXN0YW1wADIwMjYtMDgtMjdUMjA6Mjk6MjIrMDA6MDDo7JDUAAAAEElEQVQI12OsZ4AAJgaKGAAlwQCP/uWZYwAAAABJRU5ErkJggg==', 'base64'));

function build(file, withImage) {
  const doc = new PDFDocument({ autoFirstPage: false });
  const out = createWriteStream(file);
  const done = new Promise((r) => out.on('finish', r));
  doc.pipe(out);
  doc.addPage().font('Helvetica').fontSize(14).text('Page one: native text, readable without OCR.');
  doc.addPage().font('Helvetica').fontSize(14).text('Page two: more native text.');
  if (withImage) doc.addPage().image('px.png', 0, 0, { width: 612 });
  doc.end();
  return done;
}

(async () => {
  await build('text.pdf', false);
  await build('mixed.pdf', true);
  for (const f of ['text.pdf', 'mixed.pdf']) {
    try { console.log(f, '→', (await toMarkdown(f)).length, 'chars'); }
    catch (e) { console.log(f, '→ throws', e.code, JSON.stringify(e.pages), e.pageCount); }
  }
})();

text.pdf  → 77 chars
mixed.pdf → throws needsOcr [3] 3