`DeviceLogCommitCheckpointManager` 在并发读取过程中共享 I/O 完成操作,导致有效的检查点被跳过
作者: nixxcz创建于 2026年9月17日更新于 2026年9月17日
Describe the bug Concurrent checkpoint-metadata reads are not isolated in DeviceLogCommitCheckpointManager. Two concurrent INFO ALL requests can cause Garnet to: - log Skipping unreadable index checkpoint; - report Invalid metadata length 0; - return disk_checkpoint_entry:(empty) from one or both requests; even though the checkpoint metadata file is valid and a serial read immediately before and after the concurrent requests returns the same checkpoint successfully. This reproduces on a single fresh node after one checkpoint. It does not require replication, restart, failover, checkpoint transfer, concurrent writes, multiple checkpoints, or damaged files. The apparent cause is that every metadata operation shares the same instance-wide SemaphoreSlim and metadataIoErrorCode. ReadInto: 1. clears its leased buffer; 2. starts an asynchronous read; 3. waits on the shared semaphore; 4. copies and returns its buffer. Every operation uses the same IOCallback, which releases that shared semaphore without identifying which operation completed. With reads A and B in flight, A's completion can satisfy B's wait. B then copies its own still-zeroed buffer and returns it to the pool while B's I/O remains outstanding. The first four bytes therefore appear to be zero, and GetIndexCheckpointMetadata rejects the valid token as truncated or corrupt. The shared metadataIoErrorCode has the same ownership problem: one operation can reset or observe another operation's result. WriteInto uses the same semaphore, error field, callback, and buffer-lifetime pattern. The reproducer below demonstrates the concurrent-read failure only; it does not claim that the on-disk metadata was corrupted.
内容来源: microsoft/garnet