BUG: LRU block cache can spin forever in MaintainPoolSize

What happens

Our system shares LRU cache among multiple RocksDB instace and we noticed all RocksDB insteances can freeze at the same time.

What’s actually going on:

One thread is stuck in an infinite loop in LRUCacheShard::MaintainPoolSize() (cache/lru_cache.cc), still holding that shard’s mutex. Every other thread that needs the same cache shard blocks on that mutex (futex_wait). Because the cache is shared across many DBs, the whole process wedges, not just one database. This is not disk, compaction, or Raft. One core spins; everything else is waiting on one lock.

We hit this in production on RocksDB 10.9.1, with the default LRU settings (high_pri_pool_ratio = 0.5, low_pri_pool_ratio = 0), one LRU shared by ~128 DBs, and index/filter blocks in the cache.

Why the loop never stops

MaintainPoolSize has two loops. The second one demotes entries from the “low priority” pool until usage is under capacity:

while (low_pri_pool_usage_ > low_pri_pool_capacity_) {
  lru_bottom_pri_ = lru_bottom_pri_->next;
  assert(lru_bottom_pri_ != &lru_);        // gone in Release
  assert(lru_bottom_pri_->InLowPriPool()); // gone in Release
  ...
  low_pri_pool_usage_ -= lru_bottom_pri_->total_charge;  // unsigned; can wrap
}

With the default low_pri_pool_ratio = 0, capacity is 0. The loop must drain usage to exactly zero.

If it ever subtracts too much (walks past the low-pri region, or onto the list sentinel), size_t wraps to a huge number. After that, usage > 0 is always true, so the loop never exits and never drops the mutex.

The asserts that would catch this are compiled out in Release.

What we saw on a live stuck process

On the stuck shard:

• low_pri_pool_capacity_ was 0 (as expected).
• low_pri_pool_usage_ had wrapped to a number around 10^19.
• The high-pri counters still looked normal, so we were in the second loop, not the first.
• The LRU linked list itself was fine (no broken pointers). The markers (lru_low_pri_ vs lru_bottom_pri_) were in the wrong order, so the second loop was subtracting charges that had never been counted as low-pri.