#9458·lance

feature: define descriptor schema transitions for mixed Blob versions

Author: XuanwoCreated Sep 22, 2026Updated Sep 22, 2026
Labelsfeature

Supporting legacy Blob and Blob v2 files in the same table leaves a user-visible question: when and how should the Arrow descriptor schema switch?

BlobFile APIs can hide the physical encoding, but Arrow scans expose it. Python to_table() and scanner() return Blob descriptions by default:

  • Legacy: struct<position: uint64, size: uint64>
  • Blob v2: struct<kind: uint8, position: uint64, size: uint64, blob_id: uint32, blob_uri: string>

A mixed scan needs a consistent descriptor structure. The proposed direction is to normalize old descriptors to the existing v2 Inline representation while leaving their payloads in the original files. This issue tracks the resulting schema transition, rather than mixed-Blob read/write support itself.

We need to define:

  • When the descriptor schema switches: for example, when the first v2 file is committed, or through a consistent reader-side projection policy.
  • Whether the output schema is determined by the dataset snapshot or by the selected fragments. Scanning only old fragments in a mixed table should not unexpectedly produce an incompatible batch schema.
  • How historical snapshots and legacy-only tables behave, and whether compaction can change the descriptor schema again.
  • How this descriptor projection relates to the table's logical schema and accepted write inputs; adding descriptor fields should not automatically imply a separate user-managed schema upgrade.

Compatibility considerations: named access to preserved fields may continue working, but positional access and exact Arrow schema comparisons can break. V2 also requires interpreting position together with kind and the object reference; it is no longer always an offset in the data file. BlobFile and materialized-byte APIs should retain their observable behavior.

The resolution should document the transition policy and cover it with integration tests across append, reopen, fragment selection, historical snapshots, and compaction.