`Table.update()` stops returning on the 15th call of a long-running process — twice, identical frame, 0 CPU, 0 I/O

Table.update() stops returning on the 15th call of a long-running process — twice, identical frame, 0 CPU, 0 I/O

Environment

• lancedb 0.38.0, pylance/lance 10.0.0, Python 3.13.12, Linux 7.1.13 (Debian), local filesystem (no object store)
• One table, ~7.67 M rows, 2560-dim float32 vector column plus scalar columns; on disk ~108 GB, 845 data fragments, 13 indices (IVF-HNSW-SQ on the vector, BTree scalar indices incl. on file_path / chunk_id), 590 versions and 766 deletion files at the time of writing (3,439 versions and 3,361 deletion files at the second stall, before an at-rest cleanup)
• Single writer process; one other process holds the table open read-only (a query server)

What the process does

An incremental indexing run: per file, merge_insert new rows (batches of a few hundred), a per-file update(where="chunk_id IN (…)", values_sql={…}) to refresh metadata on kept rows, a mid-run compaction every ~50 batches, and every 200 files one "stamp" update:

table.update(
    where=f"file_path IN ('a.md', 'b.md', … 200 paths)",
    values_sql={"mtime": "CASE file_path WHEN 'a.md' THEN 1.7e9 WHEN 'b.md' THEN … END"},
)

What happens

The 15th such stamp update() in a process never returns. Two occurrences, 2026-09-14 and 2026-09-17, on runs of 3,833 and 6,014 files respectively; runs small enough to make only 3–6 stamp calls have never stalled. The run's own summary both times: "2800 file(s) stamped in 14 table pass(es) costing 14 s (slowest pass 2.3 s)" — fourteen fast calls, then the fifteenth hangs. The same run had made ~1,500 per-file update() calls (0.5 s each) and ~30 compactions before the stall.

While stalled (measured over one minute, 09-17):

• 0 CPU-seconds, 0 bytes of block I/O, all 42 threads S (sleeping) — 8 lancedb-tokio-w, 15 lance-cpu, 8 lance_background
• /proc/<pid>/status: VmRSS 11 MiB, VmSwap 7.2 GiB with 19.5 GiB free — the process was idle long enough to be paged out whole and nothing woke it
• deletion-file count unchanged across the minute (3,361 → 3,361); _versions/ had a latest_version_hint.json written at the moment the call began and no new manifest after it
• Python faulthandler dump (main thread), identical on both dates:

File ".../threading.py", line 359 in wait
File ".../concurrent/futures/_base.py", line 451 in result
File ".../lancedb/background_loop.py", line 33 in run
File ".../lancedb/table.py", line 4075 in update
File "<our wrapper>.py" in update_mtimes

(09-14, lancedb of that day: table.py:3721 update, otherwise the same.)

Every other thread was in a wait/select. A second faulthandler dump 11 minutes later (09-14) showed the same frames. The Rust side is opaque from here — the dump ends at background_loop.run.

Signals

• SIGINT sent to the process changed nothing in three minutes (SigPnd clear — delivered, not acted on); SIGTERM (a Python-level handler that raises KeyboardInterrupt) had the interrupt propagate out of Future.result() within eight seconds and the process exit cleanly. The stalled future itself never completed.
• After a restart, the next run's update() calls succeed immediately, including on the same rows — the state is per process, not per table.

What was ruled out

• Not a slow scan: 0 CPU and 0 I/O for the whole window, and the predicate is index-served (the previous 14 calls on the same shape took 0.5–2.3 s each).
• Not memory pressure: 19.5 GiB free, the process paged out because idle.
• Not the mid-run compaction as a trigger: compactions run every 1–2 minutes in these runs, so every stall is "shortly after" one; small runs also compact and never stall.
• Not a reader: the read-only process was open in both cases and in every clean run.

Question

Is there a per-process resource in the update path — a background-loop slot, a commit-lock or retry state, a semaphore on the deletion-file writer — that a long sequence of update() calls (interleaved with merge_insert and compaction) can exhaust, so that the 15th stamp-shaped call parks forever with nothing running? I can run a large incremental under any tracing you suggest (RUST_LOG, LANCE_LOG, py-spy dump --native at the moment of the stall) — the stall reproduces on the 15th call of this shape in a run of ≥3,000 files on this table.