[Bug] Multiple self-deadlocks in Memcached from re-locking non-recursive mutexes

Summary

Dear Memcached maintainers,

During our analysis of Memcached, we found 5 self-deadlock cases that appear to share the same root cause.

In the observed cases, a thread acquires a non-recursive mutex and later re-enters a path that attempts to acquire the same mutex again before the first acquisition is released, resulting in self-deadlock.

All 5 cases are single-thread lock re-entry deadlocks rather than multi-thread lock-order cycles.

Concrete example

One concrete case is worker_hang_lock in stop_threads().

In the current upstream tree, stop_threads() acquires worker_hang_lock in thread.c:217:

• thread.c:217 pthread_mutex_lock(&worker_hang_lock);

The same mutex is also acquired in the pause path:

• thread.c:169 pthread_mutex_lock(&worker_hang_lock);

and worker registration also synchronizes through it:

• thread.c:149 pthread_mutex_lock(&worker_hang_lock);

This lock is used to force worker threads into a blocked state during pause or shutdown coordination. If the shutdown path re-enters logic that tries to acquire worker_hang_lock again on the same thread before the first acquisition is released, the second acquisition blocks forever because the mutex is non-recursive.

We observed this as a self-loop deadlock associated with stop_threads().

Affected revision

Latest upstream revision analyzed:

• 3824603965ecbd38e552a725dbfc4ff38ab40cb2

Earlier revisions may also be affected, but we have not yet determined the exact introduction point.

Affected sites

1. lru_crawler_lock

   • crawler.c:719
   • lru_crawler_start() also acquires the same lock at crawler.c:833
   • observed trigger API: lru_crawler_crawl

2. lru_locks[id] / lru_locks[it->slabs_clsid]

   • items.c:1089
   • function: lru_pull_tail()

3. slabs_lock

   • slabs.c:808
   • function: slabs_locked_callback()

4. t->lock

   • slabs_mover.c:749
   • function: slabs_reassign()
   • the same lock is also taken again in nearby maintenance helpers such as slab_maintenance_pause() at slabs_mover.c:761

5. worker_hang_lock

   • thread.c:217
   • function: stop_threads()

Impact

Any affected path can hang the calling thread and potentially stall Memcached background work, slab maintenance, crawler activity, or shutdown handling, resulting in denial of service.

The worker_hang_lock case is especially concerning because it affects the thread stopping path and may interfere with clean shutdown or restart behavior.

Suggested direction

The fix likely needs to follow the same pattern at all sites:

• avoid invoking helpers or callbacks that can re-enter the same lock domain while these mutexes are held, or
• restructure the call paths so the same mutex cannot be acquired twice by the same thread, or
• narrow lock scope before secondary operations that may recurse into management or shutdown code, or
• only use recursive mutexes where re-entrancy is explicitly intended and proven safe

Reference

• Upstream tree: https://github.com/memcached/memcached/tree/3824603965ecbd38e552a725dbfc4ff38ab40cb2