Decode throughput decays ~3.2x with server uptime on qwen4_exp (M3 Ultra): 20.5 tok/s after ~10h vs 66.5 tok/s on a fresh process

Summary

A long-running omlx-server decodes roughly 3.2x slower than a freshly started one, with identical weights, settings, and library versions. Restarting the process is by far the largest throughput lever I have found on this machine — larger than the mlx version bump and larger than the custom Metal kernels combined.

Nothing surfaces the loss: /api/status reports the lane healthy the whole time, and avg_generation_tps is a cumulative average so it slides down slowly rather than flagging a regression.

Environment

• Mac Studio M3 Ultra, 256 GiB (applegpu_g15d), macOS 26.6.2
• omlx 0.7.0.dev2 (editable install from the homebrew tap)
• mlx 0.32.2 / mlx-metal 0.32.2 / mlx-lm 0.31.3 / mlx-vlm 0.6.3 / nanobind 2.15.0
• Model: Qwen3.8-Flash-Next-Uncensored-oQ4e-mtp (qwen4_exp), 262144 ctx, Lightning MTP depth 3, qwen4_ple_ssd_offload: true, memory guard 190 GB
• Co-resident: Qwen3.8-27B-Uncensored-oQ8e-mtp (~29 GiB). Total resident ~102 GiB.
• Prefix cache on disk: ~182 GiB

Measurement

Streamed /v1/chat/completions, temperature=0, enable_thinking=false, 384-token cap. Decode rate computed as (completion_tokens - 1) / (t_last_token - t_first_token), so prefill is excluded. A UUID nonce in every prompt defeats the prefix cache. Prompt sizes ~3.1k / 6.2k / 12.4k / 49.6k. Pooled median of 12 runs per cell (3 reps x 4 sizes); single runs on this box vary by up to 38%, so medians are necessary.

Attribution: restart +224%, mlx 0.32.0→0.32.2 +7%, custom kernels +4%.

Only one variable changes between rows 1 and 2 — the process. Same mlx, same absent kernels, same model, same settings file.

What the degraded state looks like

MTP is not the problem; it stays healthy throughout. From the server's own log on the slow process:

MTP[130] finish=length tokens=384 cycles=120 tok/cycle=3.20 accept=265/292 (90.8%)
  depth[d1=109/117,d2=92/104,d3=64/71] d0=3
  timing[backbone=16180.7ms mtp=1811.0ms sample=5.6ms cache=23.6ms]

90.8% acceptance and 3.20 tokens/cycle — near the ceiling for depth 3. The cost is 90% in the backbone forward: ~135 ms per forward for a 6B-active model. On a fresh process the same request shape runs ~3x faster with the same acceptance rate, so the regression is in the backbone step, not in speculation, sampling, or cache handling.

The degraded state also produces intermittent outliers roughly half the already-reduced rate (9.01 / 9.34 / 15.42 tok/s against a ~20 tok/s median), which disappear after a restart. Run-to-run CV tightens from ~14% to ~10% post-restart.

Reproduction

1. Start omlx-server, load a qwen4_exp model, benchmark decode as above.
2. Leave it serving normally for ~10h (mine had handled ~85 requests, mixed 4k-190k ctx).
3. Benchmark again with the identical script → ~3x slower.
4. Restart the process, reload the same model, benchmark again → recovered.

I have not isolated which of these drives it, and would value a pointer:

• Metal heap/allocator fragmentation across many differently-shaped KV allocations
• prefix-cache growth (182 GiB on disk here) affecting the lookup or eviction path
• qwen4_ple_ssd_offload page-cache behaviour degrading as the page cache churns
• ArraysCache / ragged KV pool churn across long-context sessions

Happy to run instrumented builds, dump allocator stats, or bisect by disabling PLE offload / prefix cache / MTP on a long-lived process if that would help narrow it.

Workaround in use

A nightly job recycles the lane when uptime exceeds 6h. It works, but it is a workaround, and anyone running a 24/7 lane without one is likely losing most of their throughput without any signal that it is happening.

Possibly related

A user on X reported an M3 Ultra going from 20-30 tok/s to 50-70 tok/s on qwen3.8-flash-next-q4 and attributed it to upgrading to 0.7.0.dev1 — the before/after numbers match this decay profile closely, so it may have been a restart effect rather than a release improvement.