macOS 27: periodic Metal cache-clear drain hits kIOGPUCommandBufferCallbackErrorTimeout, then the SubmissionsIgnored penalty box the engine cannot self-recover from (0.5.7, M3 Ultra)

Summary

On macOS 27, the mx.synchronize() inside omlx/utils/metal_sync.py::_sync_and_clear_cache intermittently fails with kIOGPUCommandBufferCallbackErrorTimeout. After a small number of these, the Metal driver puts the process in its "excessive GPU errors" penalty box (kIOGPUCommandBufferCallbackErrorSubmissionsIgnored) and every subsequent command buffer from that process is rejected. The server stays up and keeps answering /v1/models, but cannot generate a single token.

It never self-recovers, and I think that part is a genuine bug rather than driver bad luck: the error-recovery path calls the same _sync_and_clear_cache that is failing (scheduler.py:8204), so recovery cannot succeed once the process is in the penalty box. The engine then loops — step fails, recovery fails, repeat every ~5-7 minutes — indefinitely. Only killing the process clears it.

Six timeouts in a single 10-hour window, and zero in the preceding three months of the same log file (156 engine starts) on macOS 26.5.1 with the same omlx build, model and settings.

Environment

The stack is identical for all six faults

omlx/engine_core.py", line 394, in _engine_loop
omlx/engine_core.py", line 344, in _step_burst
omlx/scheduler.py",   line 10732, in step
omlx/utils/metal_sync.py", line 62, in _sync_and_clear_cache
RuntimeError: [METAL] Command buffer execution failed: Caused GPU Timeout Error
             (00000002:kIOGPUCommandBufferCallbackErrorTimeout).

Line 62 is mx.synchronize() # default stream. So the fault is not in a prefill or decode kernel — it is the full-stream drain that precedes mx.clear_cache(). That is a hard wait for in-flight command buffers to complete, which makes it the operation in the engine most exposed to a completion-latency watchdog.

It is the periodic clear, not the memory-pressure path

scheduler.py:10714-10730 gives should_clear three triggers: _should_periodic_clear_cache(), the deferred post-completion clear, and _consume_pressure_clear() (hard-pressure reclaim requested by ProcessMemoryEnforcer).

I expected memory pressure and that appears to be wrong. There are zero Shrank … under memory pressure warnings in the entire log, so the hot-cache-shrink precursor to request_pressure_reclaim never fired. (Requested pressure cache reclaim is logged at INFO and this log is WARNING+, so I cannot fully exclude the pool-only branch — but its companion warning never appears.)

That leaves the periodic clear: an operation that had run harmlessly on this box for three months and began failing only after the OS upgrade.

Escalation behaviour

2026-09-15 20:19:38  ErrorTimeout            (absorbed, engine keeps serving)
2026-09-15 21:44:31  ErrorTimeout            (absorbed)
2026-09-15 22:20:21  ErrorTimeout  ->  SubmissionsIgnored in the same second
2026-09-15 22:25:21 … 23:07:36  SubmissionsIgnored every ~5-7 min, engine generation-dead
2026-09-15 23:26:38  ErrorTimeout
2026-09-16 02:07:02  ErrorTimeout  ->  SubmissionsIgnored in the same second
2026-09-16 03:22:40  ErrorTimeout

Two isolated timeouts were absorbed and the third tipped the process into the penalty box. Notably, after a process restart a single timeout tipped it immediately (02:07:02), so whatever budget the driver keeps is not simply per-process-lifetime.

Once in that state the log fills with:

omlx.scheduler - WARNING - Metal cache clear failed during error recovery: [METAL] Command buffer
execution failed: Ignored (for causing prior/excessive GPU errors)
(00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored).

which is scheduler.py:8204's except around the recovery _sync_and_clear_cache.

Why this matters even if the timeout itself is Apple's

The timeout may well be an OS-level watchdog change we cannot influence. But the recovery design turns a transient GPU fault into a permanent outage:

1. _sync_and_clear_cache fails;
2. recovery calls _sync_and_clear_cache;
3. it fails identically, because the process is now in the penalty box;
4. goto 1, forever.

The engine holds ~400 GB of wired memory throughout and reports healthy on /v1/models, so nothing short of an external generation probe notices. A detector that recognised SubmissionsIgnored as terminal and exited (letting the supervisor relaunch) would turn an unbounded outage into a bounded one.

Version bounds

The log covers 2026-06-20 → 2026-09-16 with 156 engine starts and contains no occurrence of either error class before 2026-09-15 20:19:38 — i.e. none on macOS 26.5.1, and the first one 2 h 23 m after the first boot on macOS 27. Nothing else changed: same keg, same model, same settings.

Suspected trigger — correlation only, not yet reproduced

All six timeouts fall inside one window during which a second, unrelated GPU process was resident and serving (a different local inference server holding a ~16 GB 4-bit model). In the ~15 hours since the last timeout there have been zero further ones — including a deliberate stress window on this same 0.5.7 keg running 4-way concurrent 16 K-token prefills, which loads our own GPU far harder than the traffic present during the faults.

That pattern fits "a second process interleaving command buffers inflates completion latency past the watchdog, and the periodic full-stream drain is where that surfaces" — and it fits the otherwise odd fact that the faults occurred while our own engine was nearly idle. But I want to be clear this is a hypothesis: n=1 window, and the controlled A/B is still pending on my side. I will report back either way.

Possibly related

• #2029 — same error class (ErrorTimeout killing the server) on M1/32 GB, macOS 26.5.1, omlx 0.4.4. No stack trace there; if it is the same call site, the trace above may be the missing piece. Different OS though, so possibly a different trigger.
• #1835 — the other open macOS 27 regression thread (long-context decode throughput). Different symptom, same OS boundary.

Happy to run diagnostics

The box is a single-tenant M3 Ultra I control, the failing window is reproducible-ish on demand, and I can rebuild the keg with patches or extra logging. Tell me what would be useful — a lower mlx_cache_cleanup_interval probe, Metal capture, MTL_DEBUG_LAYER, anything.