[Bug] DistributedMutex flat combining crashes 100% under multi-threaded workloads on aarch64

Summary

DistributedMutex::lock_combine() crashes deterministically (100% reproduction rate) under multi-threaded workloads on aarch64 (ARM64) platforms. The same workload runs correctly on x86_64.

The crash occurs in the flat combining / CoalescedTask code path in DistributedMutex-inl.h. Replacing DistributedMutex with std::mutex eliminates the crash entirely.

The similar issue also found in CacheLib: https://github.com/facebook/CacheLib/issues/361

Environment

• Folly version: v2024.06.24.00
• Platform: aarch64 (AWS Graviton4 / ARM Neoverse V2)

Crash Symptoms

Two crash modes observed, both originating from the flat combining path:

Crash A: SIGILL (Illegal Instruction)

call_ in the Waiter struct's InlineFunctionRef jumps to a corrupted address. The callInline function pointer in storage_ was overwritten by a concurrent combiner thread writing to the overlapping metadata_ union fields. On aarch64, the corrupted address (0x47xxxx~0x6fxxxx) lands on the stack → illegal opcode → SIGILL.

Crash B: SIGSEGV (Invalid Pointer)

storage_ (first 16 bytes of Waiter) is corrupted by concurrent writes to waiters_/sleeper_ in the overlapping WakerMetadata. The this pointer captured in the lambda resolves to garbage → isValidSlab() fails → SIGSEGV.

Root Cause Analysis

The flat combining path in DistributedMutex-inl.h relies on non-atomic multi-byte reads/writes combined with timestamp-based spin heuristics to maintain mutual exclusion over a multi-purpose union. The Waiter struct (L260-310) reuses the same 24-byte region for three purposes:

1. metadata_ (WakerMetadata: waker_, waiters_, sleeper_) — written by the waking/combiner thread
2. function_ / storage_ via InlineFunctionRef — the callable stored by the waiting thread
3. Return value — coalesced result written back by TaskWithCoalesce::operator()

The invariant is: only one of these three uses is "active" at any given time. This invariant holds on x86 but breaks on aarch64.

Why it works on x86 but fails on aarch64

The code itself acknowledges the platform gap

At line 248:

// on x86, this gets optimized away to just a regular store, it might be
// needed on platforms where explicit acquire-release barriers are
// required for synchronization
//
// note that we release here at the end of the constructor because
// construction is complete here, any thread that acquires this release
// will see a well constructed wait node
futex_.store(futex, std::memory_order_release);

This comment shows the author was aware that non-x86 platforms need explicit barriers, but the subsequent code paths — particularly the TaskWithCoalesce union reuse and the lock_combine() return-value retrieval — do not add the necessary fences.

The critical crash path: lock_combine()

The entry point is lock_combine() (L1042-1078):

auto lock = dm.lock_combine([&] { /* user func */ return result; });

When the user's function is combined (executed by a remote combiner thread), the flow is:

1. Waiting thread stores its callable into Waiter::storage_ (via InlineFunctionRef)
2. Combiner thread reads storage_, executes the callable, writes the return value back into the same union region via TaskWithCoalesce::operator() (L479-486)
3. Waiting thread wakes and retrieves the return value via std::move(task).get() (L1077)

On aarch64, step 2's writes may not be visible to step 3's reads due to missing acquire-release synchronization on the union memory itself.

All problematic locations

Current Workaround

Replace DistributedMutex with std::mutex. This disables flat combining but eliminates the crash. The trade-off is reduced throughput under high lock contention (flat combining can provide 2-5× throughput improvement in contended scenarios).

Suggested Fix

1. Add acquire-release barriers around the non-atomic union field accesses that participate in cross-thread communication (especially the TaskWithCoalesce write-back and the lock_combine() return-value read)
2. Make critical union fields atomic with appropriate memory ordering
3. **Normalize **kScheduledAwaySpinThreshold to wall-clock time instead of raw tick counts, to avoid behavioral drift across architectures
4. Conservative option: #ifdef to disable flat combining on non-TSO architectures until the barriers are audited

Status

• Affected version: v2024.06.24.00 (confirmed)
• Latest main branch: same code, unchanged as of 2026-09-14
• Not fixed upstream

References

• Source file: folly/synchronization/DistributedMutex-inl.h (1745 lines)
• Flat Combining paper: Hendler, Incze, Shavit, Tzafrir. "Flat Combining and the Synchronization-Parallelism Tradeoff." SPAA 2010.