GLM-5.3-Flash-Q4_K Metal SSD-streaming (M4 Max, 128 GiB): host-side per-layer bookkeeping dominates token time (15 % GPU), and the async selected-id load is a no-op

Hi antirez, thanks for ds4 — it runs a 178 GiB GLM-5.3-Flash-Q4_K on a 128 GiB M4 Max, which is remarkable.

I spent a day measuring the GLM SSD-streaming decode path before touching it, and I think the findings below are worth an issue rather than a patch: I could not turn any of them into a net win. All numbers are from real runs on this machine; raw stderr is available on request.

Environment

• current fork build on top of origin/main @ 9139e2a, no local changes for these numbers
• Mac Studio M4 Max, 128 GiB, macOS 27; Metal 4 tensor API unavailable on this device (log: "Metal 4 tensor API disabled for pre-M5/pre-A19 devices")
• Model: GLM-5.3-Flash-Q4_K.gguf, 177.77 GiB, 1412 tensors, types f32/q8_0/q4_k/bf16 (129 q4_k = 42 routed layers x 3 expert tensors) — a uniform Q4_K expert layout, no Q5_K, no IQ2_XXS
• Protocol: --ssd-streaming --ctx 4096 --tokens 128 --temp 0, a prompt that actually generates 128 tokens, cold + warm run, values parsed from stderr with DS4_METAL_STREAMING_EXPERT_TIMING_SUMMARY=1 DS4_METAL_MEMORY_REPORT=1

Finding 1 — the timing summary is easy to misread, and 1-token runs measure the wrong thing

A prompt like "answer with exactly one word" yields a single-token answer: selected_calls=42 and read_avg=11.7 ms. That is 42 layers x 1 token, not a decode rate — the reported generation: 1.8 t/s is the first token, which carries the whole page-fault / cache-fill warmup. With a prompt that fills the token budget the same build reports:

run	prefill t/s	decode t/s	selected_calls	read_total
Q4 streaming, cold	2.60	6.21	5334	7813 ms
Q4 streaming, warm	2.64	6.22	5334	7793 ms

selected_calls = 42 x 127 confirms the counter is per layer x token. A sanity line printing the generated token count next to generation: t/s (or a doc note that the timing summary needs a prompt that exhausts --tokens) would prevent this class of misreading.

Finding 2 — the host readback is not the bottleneck; removing it entirely changes nothing

uniform_streaming_selected_cache and stream_addr_q4 are both true for this GGUF, so the decode already runs the address-table path (stage profile: pair=q4_stream_addr_swiglu, down_path=q4_stream_addr_down_simd), not a legacy path. I then removed the synchronous selected-id readback completely. It was gated twice: glm_graph_layer_uses_generic_routed_moe() (IQ2 only, from the initial GLM 5.2 commit 005afed) and, on Metal, an opt-in env var (ds4.c: return getenv("DS4_METAL_ENABLE_GLM_STREAMING_SELECTED_ASYNC_LOAD") != NULL;). With the gate opened for streaming_selected_cache and the flag set:

	baseline	async path
decode t/s	6.21 / 6.22	6.22 / 6.24
selected_calls / read_total	5334 / 7793 ms	0 / 0 ms
output (temp 0)	584 B	byte-identical

DS4_METAL_GLM_STREAMING_ASYNC_PROFILE=1 shows why: async_calls=5334 total=20007 ms avg=3.751 ms signal_start=17 ms flush_router=18 ms shared=42 ms flush_shared=7 ms finish=10945 ms routed=8974 ms. The work moves from the decode thread into the service thread, and the decode thread then waits in finish — no overlap is gained, because there is almost nothing left to overlap with (shared = 42 ms total).

Where the token time actually goes

Stage profile per layer x token in steady state: pair=0.31 ms down=0.25 ms -> GPU ~0.58 ms x 42 layers = ~24 ms of ~161 ms per token (15 %). The rest is host-side expert-cache bookkeeping per layer x token: read_total 61 ms (only without the async path), bind_total 25 ms, missing_bind (split path) 20 ms, readahead 22 ms (14769 calls in 127 tokens), buffer preparation ~11 ms. Cache pressure is real but not the lever either: 48 GB / 56 GB / 85 GB budgets give 5.3 / 5.3 / 4.6 t/s with hit_rate 0.74 / 0.77 / 0.84 — the automatic budget (70.1 GiB, hit_rate 0.83, 4575 evictions) is already the best, and 85 GB starts swapping.

Flags I tested (all neutral)

I also swept the knob surface around this path with the same 128-token protocol, two runs each, output byte-identical in every case: DS4_METAL_ENABLE_Q4_GROUPED_EXPERTS=1 (enable-only gate, default off), DS4_METAL_ENABLE_Q4_BATCH_EXPERT_TABLE=1, DS4_METAL_ENABLE_Q4_EXPERT_ADDRESS_TABLE=1 (already default on), DS4_METAL_DISABLE_STREAMING_EXPERT_READAHEAD=1 (slightly worse: 6.04 vs 6.22 t/s), DS4_METAL_DISABLE_STREAMING_EXPERT_SPLIT=1, DS4_METAL_DISABLE_STREAMING_EXPERT_HIT_VALIDATOR=1, --ssd-streaming-full-layers 2 and 4, --ssd-streaming-preload-experts 2000. All stay within noise of 6.2 t/s, and whatever is removed reappears elsewhere (readahead disabled -> bind_total +0.8 s). The per-layer chain looks serial rather than dominated by any single removable cost — which is why I am not sending a patch.

For completeness: --mtp on this path measures 5.28 / 5.22 t/s (i.e. ~15 % slower than without it) with byte-identical output, and I have switched my launcher default to MTP off for GLM streaming.

Related work in this tracker

• #437 (M4 Max 128 GiB SSD-streaming benchmark and alternative supply strategies) — same machine class, DeepSeek graph; my numbers are for the GLM path, which does per-layer host-side address resolution in ds4_gpu_glm_routed_moe_one_tensor.
• #840 (byte-identical streaming speedups on V4 Flash) — I could not reproduce a gain of that class on GLM Q4_K: see the sweep above.
• #636 (L+2 router lookahead from an offload prototype) — the per-layer chain described there matches what I measure here; a lookahead across layers is what I would try next.
• #810 (smaller expert cache faster on M5 Pro) — on M4 Max 128 GiB with GLM Q4_K the automatic budget was the best of the four budgets I tested.
• #495 (--mtp together with --ssd-streaming) — 15 % slower here, byte-identical output.

Question

For the DeepSeek/MXFP4 streaming path the same counters show read_avg=0.000 ms, bind_avg=0.444 ms at 15.6 t/s, i.e. per-layer address resolution looks much cheaper there. Is the intended direction for GLM to move this per-layer/per-token bookkeeping to the GPU (compact/validate style) instead of the host loop over routed experts? And is the Metal async selected-id load meant to stay opt-in for uniform Q2_K/Q4_K layouts, or is enabling it by default just untested?

I can provide raw stderr for all runs, the exact flag matrix, and the (correctness-neutral) patch if that helps. Thanks!