bug: readers before v12.0.0 fail to open a dataset whose inline transaction they cannot decode
Opening a dataset eagerly decodes the manifest's inline transaction section purely to warm the session cache. Before #7740 the decode error propagated, so a transaction carrying an operation the reader does not know makes the whole dataset version unopenable — even though nothing about reading the data needs the transaction.
#7740 fixed this on main and first shipped in v12.0.0 (2026-09-02). It was never backported, so every maintained release line from release/v1.0 through release/v11.0 still has the failing form.
Why this is not hypothetical
Operations are added to the Transaction.operation oneof over time, and each addition breaks every reader released before it:
| Operation | First shipped | Readers that fail |
|---|---|---|
Clone, UpdateBases, UpdateMemWalState |
1.x line | earlier 1.x |
DataOverlay (field 115) |
v9.0.0 (#7381) | v1.0 – v8.x |
So a v8 reader opening a table whose newest version was written by v9+ with a DataOverlay operation already fails today. #7740's own commit message says as much ("This already applies to recently added operation types ... read by older 1.x releases").
The inline transaction section has existed since v1.0 (#4774), so the exposure spans the whole 1.0–11.0 range.
Mechanism
Dataset::load_manifest (rust/lance/src/dataset.rs, release/v11.0 line 804):
if let Some(transaction_offset) = manifest.transaction_section
&& manifest_size - transaction_offset <= last_block.len()
{
...
let transaction: Transaction =
lance_table::format::pb::Transaction::decode(message_data)?.try_into()?;
// ^ only to populate metadata_cacheTransaction::try_from returns Error::internal("Transaction message did not contain an operation") when prost decodes an unknown oneof variant to None, and the ? fails the open.
Reproduction
- Write a table with a version whose operation is newer than the reader, e.g. a
DataOverlaycommit from v9+, small enough to inline (underMAX_INLINE_TRANSACTION_BYTES). - Open it with a v8 (or any v1.0–v11.0) reader.
- Open fails. The same table opens fine on v12.0.0+.
Fix
Backport #7740 to release/v1.0 … release/v11.0. The change is small and self-contained: make the decode return Option, log a warning, and skip the cache warm-up. Paths that genuinely need the transaction (read_transaction, conflict resolution) still surface the error at their call sites.
Worth considering alongside it: a backport only helps readers that take the patch. For deployments pinned to an unpatched older version, the only lever is on the writer — not inlining a transaction whose operation postdates some compatibility floor, so transaction_section is absent and an old reader skips the block entirely. That is a bigger design question and does not need to block the backport.
Source: lance-format/lance