[BUG] Permanent deadlock on LMDB writer mutex: write txn held across .await in TxnRunner::try_run_once (v1.0.23)

Summary

On a long App.update() over a large corpus, the core deadlocks permanently on LMDB's writer mutex. A tokio-rt-worker blocks forever in __pthread_mutex_lock_full on the process-shared robust mutex in <state>/mdb/lock.mdb, while the mutex's owner is the other, still-alive tokio worker, parked in the scheduler. FUTEX_OWNER_DIED is not set, so LMDB's robust-mutex recovery path never runs and nothing ever breaks the lock. The process then makes no progress and no I/O: gRPC threads on condvars, the Python asyncio loop idle in select, executor threads idle.

Observed on v1.0.23 (current latest release), Python 3.13.13, Linux aarch64, in a container limited to --cpus 2 --memory 1g. Parked after ~41 minutes at 125,697 of ~134,000 points, with 130,753 chunks already embedded — i.e. it stops between embed and upsert.

Why I think this is a real bug and not just my workload

rust/core/src/state_store/storage.rs (v1.0.23), TxnRunner::try_run_once holds an LMDB write transaction across .await points by construction:

async fn try_run_once(&self, inputs: &[TxnBody]) -> Result<Vec<Box<dyn Any + Send>>> {
    let _read_guard = self.coord.read().await;
    let mut wtxn = WriteTxn::new(self.db_env.write_txn()?);
    for body in inputs {
        outputs.push(body(&mut wtxn).await?);
    }
    wtxn.into_inner().commit()?;
    Ok(outputs)
}

The doc comment on TxnBody in the same file states this intentionally: "Sync is required because try_run_once holds &[TxnBody] across await points; for &T to be Send, T must be Sync."

The compiler permits it because heed 0.22's RwTxn<'p> is a wrapper around RoTxn<'p, WithoutTls>, and heed declares unsafe impl Send for RoTxn<'_, WithoutTls> (there is an explicit rw_txns_are_send test in heed/src/txn.rs). But MDB_NOTLS only frees read transactions from thread affinity. A write transaction still takes env->me_wmutex, which on Linux is a process-shared, robust pthread mutex whose owner is recorded as the TID that called mdb_txn_begin. glibc refuses a pthread_mutex_unlock of a robust mutex from a non-owner thread (EPERM), and LMDB's UNLOCK_MUTEX does not check that return value.

So on a multi-thread tokio runtime, where a task may be polled by a different worker after each .await, the future in try_run_once can begin its write txn on worker A and finish/commit it on worker B. The unlock from B is a no-op error; the mutex stays locked with __owner = A forever, and every later mdb_txn_begin(write) blocks on it. That is exactly the state captured below: waiters present, owner alive but idle.

Separately — and independent of any migration — the same shape means the writer mutex is held for the entire duration of every awaited body in a batch, so any slow or wedged awaited work inside a body serialises and stalls all writers process-wide.

Captured evidence

Blocked worker (gdb -p <pid> -batch -ex "thread apply all bt", frames unsymbolised — see Limits):

Thread 13 (Thread 0xe73ae2b6f160 (LWP 4150079) "tokio-rt-worker"):
#0  0x0000e73afb7660e4 in __pthread_mutex_lock_full () from .../libc.so.6
#1  0x0000e73ae35f3c2c in ?? () from .../cocoindex/_internal/core.abi3.so
#2  0x0000e73ae35f4744 in ?? () from .../cocoindex/_internal/core.abi3.so
#3  0x0000e73ae30ad40c in ?? () from .../cocoindex/_internal/core.abi3.so
#4  0x0000e73ae2eeae50 in ?? () from .../cocoindex/_internal/core.abi3.so
#5  0x0000e73ae3016bc8 in ?? () from .../cocoindex/_internal/core.abi3.so
#6  0x0000e73ae2fbb324 in ?? () from .../cocoindex/_internal/core.abi3.so
#7  0x0000e73ae3311514 in ?? () from .../cocoindex/_internal/core.abi3.so
#8  0x0000e73ae330ee44 in ?? () from .../cocoindex/_internal/core.abi3.so
#9  0x0000e73ae330150c in ?? () from .../cocoindex/_internal/core.abi3.so
#10 0x0000e73ae330347c in ?? () from .../cocoindex/_internal/core.abi3.so
#11 0x0000e73ae32fb438 in ?? () from .../cocoindex/_internal/core.abi3.so
#12 0x0000e73afb762d80 in start_thread () from .../libc.so.6
#13 0x0000e73afb7d236c in thread_start () from .../libc.so.6

The mutex it is waiting on — x0 at the __pthread_mutex_lock_full frame is 0xe73afc103080, which falls inside the mapping

e73afc103000-e73afc106000 rw-s ... /<state>/cocoindex.db/mdb/lock.mdb

Its bytes (x/8xw $x0):

0xe73afc103080:  0x80000008  0x00000001  0x00000008  0x00000001
0xe73afc103090:  0x00000090  0x00000000  0xe295f240  0x0000e73a

i.e. __lock = 0x80000008 (owner TID 8 + FUTEX_WAITERS), __count = 1, __owner = 8, __nusers = 1, __kind = 0x90 = PTHREAD_MUTEX_PSHARED_BIT | PTHREAD_MUTEX_ROBUST_NORMAL_NP. FUTEX_OWNER_DIED (0x40000000) is not set.

The owner is alive. Two independent ways:

1. Mapping every live thread's NSpid shows container TID 8 is the other tokio-rt-worker, Thread 12 (LWP 4150080), parked in __syscall_cancel_arch/epoll_pwait — i.e. an idle tokio worker.
2. The robust-mutex list link in the same dump — the two words after __kind are 0xe73ae295f240 — points into Thread 12's stack region (0xe73ae295f160), which is the same thread reached the other way.

Everything else at the park is idle: 6 gRPC event_engine threads and lifeguard in absl ... FutexWaiter::WaitUntil, the other worker parked, and py-spy dump showing MainThread idle in selectors.select inside asyncio run_forever, both asyncio_N executor threads idle in concurrent.futures.thread._worker.

A sampler over /proc/<pid> shows the transition: at the park instant thread count collapsed 102 → 13 (tokio-rt-worker 91 → 2) and CPU ticks went to ~0 and stayed there until the run was killed 4 minutes later (cpu+1, then cpu+0, rss=764MiB, thr=13 unchanged across samples).

Reproduction recipe

Self-contained — no proprietary data and no hosted services:

1. Corpus: any git repo large enough to sustain concurrent write work; mine was ~15,300 files → ~134,000 chunks. git clone --mirror it and scan it over a file:// URL so no forge is involved.
2. Stub embedder instead of a real model, so embedding is not a variable: a tiny HTTP service answering POST /embed/documents and /embed/query with {"dimension": 768, "vectors": [...]}, deterministic L2-normalised vectors derived from each text's SHA-256, with per-batch (200 ms) and per-text (5 ms) delay knobs.
3. Scratch Qdrant qdrant/qdrant:v1.18.2 on its default ports (note qdrant-client(url=..., prefer_grpc=True) ignores the port in the URL for gRPC and always dials 6334).
4. Run the scan with a fresh state root and a fresh collection, constrained to --memory 1g --cpus 2, and disable any stall watchdog so it does not kill the park before you can capture it.
5. Detect the park by sampling /proc/<pid> CPU ticks, io, RSS, thread count and a per-thread wchan histogram every 30 s and flagging N consecutive zero-CPU samples. (/proc/<pid>/io reads as 0 for a container process from an unprivileged host user, so rely on CPU ticks.)
6. Capture py-spy dump, then gdb -p <pid> -batch -ex "thread apply all bt", then the mutex bytes at x0/rdi of the __pthread_mutex_lock_full frame plus /proc/<pid>/maps, then the NSpid mapping to decide whether the owner TID is alive.

It reproduced on the first full run of this shape; I have not attempted to establish a rate.

Suggested direction (yours to judge)

The invariant that seems to be missing is an LMDB write transaction must begin and end on the same OS thread. Options, roughly in increasing order of intrusiveness:

• Run the whole try_run_once body on a single dedicated writer thread (e.g. a spawn_blocking thread or a dedicated writer task on a LocalSet/current-thread runtime), with bodies handing over data rather than futures — i.e. no .await inside the txn's lifetime at all.
• If bodies must stay async, resolve everything they need before opening the write txn, then apply the batch synchronously and commit, so the txn's lifetime contains no yield point.
• Failing both, make the hazard loud: a debug assertion that the thread id at commit() matches the thread id at write_txn().

Worth noting that holding the writer mutex across awaited work is also a throughput ceiling independent of the deadlock, since it serialises all writers behind the slowest body in a batch.

Limits (stated plainly)

• n = 1 park. One reproduction, not a measured rate.
• aarch64 only, cocoindex 1.0.23, Python 3.13.13, glibc 2.42, container --cpus 2 --memory 1g. Not tried on x86_64 or with a different worker count.
• The frames are unsymbolised. core.abi3.so ships stripped, and the captured process is gone, so its load base is unrecoverable — those addresses can no longer be resolved by any later build. The call path above is named from reading v1.0.23 source, not from a symbolised frame.
• The specific migration is not proven. That RwTxn is Send and that try_run_once holds it across .await are facts from source; that this particular park was caused by the task resuming on a different worker (rather than by the owning task simply never being polled again while holding the txn) is not established from the capture. Both paths lead to the same permanently-held writer mutex, and both are consequences of the same shape.
• py-spy --native was not usable here: it fails on aarch64 with UNW_EBADREG.

I can run further experiments against a patched build if that would help — say what you would want measured.