Lazy two-phase batch visibility check to skip ctid decode using metadata

What feature are you requesting?

Currently, check_batch in the aggregate scan decodes the full ctid (via fast field lookup) for every document before checking visibility. This is expensive, the ctid is mostly a BitwiseLinear decode which involves bitpacked data access for every doc. When you're doing something like count(*) you don't actually need the ctid you just need to reason about document viz.

It turns out the way the BitwiseLinear is stored and the way ctid is structured (block is the leftmost bits) unlocks a cheap check just using metadata with no decode. This can be used for an alternative two-phase visibility check that can skip the expensive ctid decode entirely for some documents.

• Phase 1 (cheap bounds check): For each doc, use the codec's interpolation metadata (need to expose via Tantivy) to compute the possible heap block range (~5 arithmetic ops, no bitpacked I/O). If all blocks in that range are marked all-visible in the visibility map, mark the doc as visible immediately with a sentinel value — no ctid decode needed.
• Phase 2 (full decode): Only docs that couldn't be resolved in Phase 1 go through the existing path: full ctid decode, sort by ctid, and heap tuple visibility check with buffer pinning.

Motivation: On large tables where most heap blocks are all-visible (common after VACUUM), this should significantly reduce CPU time spent in the visibility check hot path during aggregate scans (I've seen around 40%).

Outstanding question: Is there a better way of doing this?

Why are you requesting this feature?

Performance

Full Name:

James Blackwood-Sewell

Affiliation:

ParadeDB