Memory Model Issues with Unaligned Loads and Stores Implementation

This is a summary of the issues I raised with @Sonicadvance1 on Discord. I'm not familiar enough with FEX to implement a patch, so I'm documenting them here for posterity.

There are two main problems with emulating memory models: ordering of modification visibility, and (single-copy) atomicity. The former is concerned with the order in which cores observe changes in variables, while the latter is concerned with having the observed change actually match the modification event.

It's usually assumed that as long as a store doesn't cross cache line boundaries, it should be single-copy atomic. This is architecturally guaranteed on x86 as per section 11.1.1 "Guaranteed Atomic Operations" of the latest Intel manuals (June 2026, volume 3, systems programming guide), which states that atomicity applies to:

Unaligned 16-, 32-, and 64-bit accesses to cached memory that fit within a cache line.

x86 is allowed to and will tear (observe only part of the bytes modified by a store instruction) for regular stores across cache line boundaries in existing implementations. This tearing can only be prevented on the store side with a LOCK prefix , because even a lock add of 0 by the reader can't guarantee the interconnect isn't locked down between the cache line updates. However, the situation on ARM is very different. The "Mixed-size Concurrency: ARM, POWER, C/C++11, and SC" by Shaked et al. covers it well:

Microarchitecturally, one would expect at least stores whose footprint spans a cache-line boundary to be split (otherwise one is in the realm of hardware transactional memory implementations, to provide atomic access to multiple cache lines while avoiding deadlock), but we also see splitting at finer granularities, and we are told of plausible implementation techniques for both ARM and POWER, which may be used in current implementations, which would lead (sometimes rarely) to such splitting. The architectures explicitly do not guarantee single-copy atomicity for misaligned accesses within cache lines, or indeed commit to any particular cache-line sizes, so programmers should not rely on that and our semantics should not guarantee it.

Even for ARM systems supporting FEAT_LSE2, the atomicity guarantees are only provided for accesses that don't cross 16-byte alignment boundaries (granules). The detailed experimental results tables reveal that the Cortex A53-based H955-A53 and OdroidC2 (which have 64-byte cache lines) tear on 16 and 32-byte boundaries as well, so this is not just a theoretical concern.

Unfortunately, barriers can't restore this lack of atomicity, and FEX currently relies on patching in barrier instructions for unaligned accesses when the atomic instructions trigger an alignment fault. This works well for maintaining ordering, but it doesn't guarantee that the observed synchronization variable will have the correct value - it will only guarantees that a thread will observe that the values changed after previous writes are already observable. Fortunately, this is sufficient to support a lot of standard SPSC queue implementations that implicitly or explicitly rely on equality comparisons between monotonically increasing counters to check whether the queue is empty or full, but it can't guarantee correctness for implementations that use caching optimizations. As per Sonicadvance1's comments, games like God of War 2018 crash without the barriers, though it's unknown whether they rely on the correct value being observed as well.

On that note, the current barrier instruction selection for unaligned stores is inefficient: It uses dmb ish instead of dmb ishst. Acquire load instructions and patched in dmb ishld for unaligned loads already cover the load-load, and load-store cases, so you only need a store-store barrier before unaligned writes. The reason why dmb ish was used is because that's the choice .NET made, but note that volatile stores have release semantics, so the devs needed a stronger barrier. ARM's (now Google's) Will Deacon recommended dmb ish over the equivalent dmb ishld; dmb ishst pair, so that's what the developers went for; GCC 16 recently changed their release fence mapping to the barrier pair, so it no longer follows the recommended C++11 mappings. The single-copy atomicity problem also applies to .NET, and it makes their instruction selection inefficient, but that's an issue I've raised separately with them. What's important for FEX is that it shouldn't concern itself with the efficiency of the barrier pair, because it already naturally creates the pair due to ordering emulation.

The only solution to this problem that doesn't require program-wide thread pausing is to use a double-word CAS (DWCAS), which ARM provides as casp. These are already used for RMW operations that emulate LOCK-prefixed and implicit LOCK (xchg with memory operand) instructions. It's important to note that the corresponding loads need a dummy CAS loop as well, because they need the guarantee that the store they've observed is correct, which can only be done after performing a successful CAS (even with a far atomic casp implementation, the home node is allowed to service loads while completing the instruction - it only needs to block new stores).

I'm not sure the DWCAS solution is worth implementing right now, it could be an optional setting for users to try when they have intermittent crashes, but the caching optimization is already somewhat rare, and it requires unaligned atomics, which anyone using languages based on the C++11 memory model should avoid due to undefined behaviour - and the write-up above explains exactly why the memory model architects made it undefined.