DeepSeek V4: incoherent GPU output on GB10 (sm_121, unified memory) from the routed-expert VRAM mirror cache

Summary

On a DGX Spark (GB10, compute capability 12.1 reported as sm_121, 130 GB unified memory), the DeepSeek V4 engine's GPU tier returns incoherent text when the routed-expert VRAM mirror cache is active. The same build produces correct text on the CPU, and correct text on the GPU once the mirror cache is either disabled or allowed to grow. The generic CUDA kernels themselves look correct; the fault is in how the mirror cache decides whether it may grow on a unified-memory device.

Environment

• GPU: NVIDIA GB10, sm_121, 130.7 GB unified memory (VRAM and system RAM are the same pool)
• CPU: Grace, aarch64, 20 cores
• CUDA 13.0 (nvcc V13.0.88), driver 580.142, Linux 6.17
• colibri commit f028d26b422144ed4a69ad9aeaee2553ce0f9572
• Build: make -f Makefile.deepseek-v4 -C c deepseek-v4 CUDA=1 CUDA_ARCH=native (generic kernels; DeepGEMM is sm_120a only and was not built)
• Model: DeepSeek V4 REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b, model_type deepseek_v4, 43 layers, 132 routed experts)

Symptom

The banner reports the tier is up, with no errors or NaN warnings:

[DSV4 CUDA] device 0: NVIDIA GB10 130.7 GB sm_121
v4_gpu tier=dense-matvec device=0

Output is garbled, and the same input gives the same garbled output every time.

Prompt (70 tokens): "You are a systems engineer. In four or five sentences, explain the practical trade-offs of running a large Mixture-of-Experts language model by streaming its expert weights from an NVMe SSD instead of holding the whole model in memory. Cover memory footprint, decode latency, and where a fast SSD helps versus where it does not."

Reproduction

Fails (mirror cache active, default reserve):

COLI_CUDA_ATTN_BATCH=1 COLI_CUDA_MOE_BATCH=1 \
DSV4_CUDA_EXPERT_MIRRORS=2048 DSV4_CUDA_VRAM_RESERVE_MB=12000 \
  ./c/deepseek_v4 <model> --prompt-file prompt.txt --max-tokens 64 --memory-gb 24
# -> "As asystems engineer details dealer, Mixture-of-Expert model, or AS the best
#     known system of other on various systems, the question par takes: ..."

CPU reference (correct):

DSV4_CUDA=0 ./c/deepseek_v4 <model> --prompt-file prompt.txt --max-tokens 16 --memory-gb 24
# -> "As a systems engineer, the core trade-off is shifting from a memory-bound to ..."

Two ways to get correct GPU output:

# 1) keep experts on the CPU (any value 1..7 turns the mirror cache off)
... DSV4_CUDA_EXPERT_MIRRORS=1 ...
# -> "As a systems engineer, the core trade-off is swapping memory ..."

# 2) lower the reserve so the mirror cache can grow instead of freezing
... DSV4_CUDA_EXPERT_MIRRORS=2048 DSV4_CUDA_VRAM_RESERVE_MB=256 ...
# -> "As a systems engineer, the core trade-off is swapping memory capacity for I/O"

Analysis

The corruption tracks the mirror cache's growth decision, which compares DSV4_CUDA_VRAM_RESERVE_MB against cudaMemGetInfo free memory. On this device cudaMemGetInfo free equals /proc MemFree, because the GPU shares the system memory pool. While the model streams, MemFree stays well below a multi-GB reserve, so the growth guard never opens and the cache freezes at one or two live entries. It then recycles those entries in place under LRU while the current layer's experts still reference them, so the layer is computed from the wrong weights. That matches the two observations above: the failure is deterministic and byte-stable, and dropping the reserve to its floor (256) fixes it by letting the cache grow past the frozen state.

Code that looks relevant, from reading rather than a proposed patch: v4_gpu_expert_attach_cached_ex() in c/deepseek_v4.c (the growth guard, and the "capacity < 8" early return, which checks the configured capacity rather than the live entry count), and dsv4_cuda_mem_free_mb() in c/backend_cuda_dsv4.cu, which returns cudaMemGetInfo free.

Two directions for the maintainer to weigh:

• Apply the "fewer than 8 entries" early return to the live entry count, not the configured capacity, and avoid recycling an entry that was attached since the last compute.
• On unified-memory devices (where cudaDevAttrPageableMemoryAccessUsesHostPageTables is set), do not treat cudaMemGetInfo free as VRAM headroom.

Workaround

Set DSV4_CUDA_VRAM_RESERVE_MB=256, or keep experts on the CPU with DSV4_CUDA_EXPERT_MIRRORS between 1 and 7.

Relation to earlier reports

This looks different from #76 (an oracle-file misuse) and #1450 (sm_86 generic DLL, where a sub-1% mirror hit rate would hide it). I did not find an existing report of this mirror-cache behaviour on unified memory.