merge for adding large binary columns OOMs at full scale and can't be run incrementally, while merge_insert fails with a bitmask panic

Hi Lance team,

I'm building a 5TB dataset with large large_binary columns (pickle payloads) and need to repeatedly attach new binary columns to an existing dataset by matching on a join key. I've hit a wall with both merge and merge_insert and would like guidance on the intended workflow.

Environment:

• Lance version: 0.38.0
• Dataset size: ~5TB, target columns are large_binary
• Target table: ~3M rows, join key global_idx (int64)
• Input: many parquet files (10GB each), each containing global_idx + binary payload columns

What I've observed:

1. merge (add column) works for small inputs but OOMs at full scale.

When I run it on a small subset, it succeeds. But when I attempt to merge the full dataset, the process exhausts all available memory and gets OOM-killed.

2. merge cannot be run incrementally once the column exists.

After the first merge succeeds, subsequent calls fail with:

Invalid user input: Column boozer_payload exists in both sides of the dataset

So merge is a one-shot operation per column? If I need to update the values of an existing column later, merge is not applicabl?

3. merge_insert is the semantic match for updates, but is completely unusable in my workload.

merge_insert("global_idx").when_matched_update_all() is the only operation that supports updating existing rows. However, it fails with a panic in Arrow's bitmask handling:

thread 'lance_background_thread' panicked at arrow-buffer-58.4.0/src/util/bit_mask.rs:41:5:
assertion failed: offset_read.checked_add(len)... <= data.len() * 8

My questions:

1. Is merge intended to support incremental/batched operation for adding a large column? If a 5TB column can't fit in a single merge in memory, what is the recommended way to add it in pieces?

2. Is the merge_insert bitmask panic a known issue? It appears related to large_binary columns with nulls during the write path. Is there a workaround, or is this fixed in a later version?

3 Are there any plans to make merge support batched/incremental column addition, or to fix the merge_insert write path for large binary columns?

Minimal context: The join key is a simple int64. The binary columns are 5TB-scale. merge works at small scale but OOMs at full scale and can't be repeated; merge_insert fails with a bitmask panic. I'm looking for the intended path to attach and update large binary columns in an existing Lance dataset.

Thanks for any guidance.