MVCC passive checkpoint cleanup releases the WAL write lock twice after an I/O error
Problem
An error during an MVCC passive checkpoint can make checkpoint ownership differ from the WAL lock state.
The checkpoint state records an active pager write transaction. However, the pager commit already released the WAL write lock.
Statement cleanup then calls Pager::rollback_tx(). The rollback calls WalFile::end_write_tx() and triggers this assertion:
end_write_tx called while write lock not held according to connection stateThe panic occurs after the simulator exports the Elle history. Elle does not report a consistency error.
Reproduction
This seed fails on main at 9a082e5bc3:
cargo build -p turso_whopper
SEED=13150116157205938478 RUST_BACKTRACE=1 \
./target/debug/turso_whopper \
--elle rw-register \
--elle-output elle-history.edn \
--max-steps 100000 \
--enable-mvcc \
--enable-experimental-mvcc-passive-checkpoint \
--mvcc-checkpoint-threshold 1024The run reports:
3 allocation faults injected
905 checkpoint probes fired against suspended statements (all rejected)
Elle history exported to: elle-history.edn
thread 'main' panicked at core/storage/wal.rs:3569:9:
end_write_tx called while write lock not held according to connection stateThe important backtrace path is:
WalFile::end_write_tx
Pager::rollback_tx
CheckpointStateMachine::cleanup_after_external_io_error
CommitStateMachine::cleanup_mvcc_checkpoint_state
Program::abort
Statement::reset_best_effortCause
CheckpointStateMachine::BeginPagerTxn sets lock_states.pager_write_tx after it starts the pager write transaction.
Pager::commit_tx() releases the WAL write lock before its internal automatic checkpoint is complete. That automatic checkpoint can still return pending I/O.
In this seed, an injected allocation error interrupts that pending work. The outer checkpoint still has lock_states.pager_write_tx = true.
cleanup_after_external_io_error() trusts that flag and calls Pager::rollback_tx(). The rollback tries to release the WAL write lock again.
Relevant code:
Expected behavior
Checkpoint cleanup must release each pager and WAL lock once.
After an external error, the checkpoint ownership flags must agree with the actual pager and WAL state.
Related reports
- #8291 reported the same assertion from another cleanup path. That issue is closed.
- #8406 fixed a double release in
WalSession. It did not fix this MVCC checkpoint path. - PR #9104 reproduced this checkpoint path on
main. - PR #9107 produced the deterministic seed above in this CI job.
Source: tursodatabase/turso