test_deepseek41_metal: new router check fails on M1 Ultra (pre-M5); sqrt(softplus) compared at 3e-6 but 1.0f+exp is ~0.7% imprecise

Clarification: this report is about a failure in the model-free tests/test_deepseek41_metal router numerical check on our M1 Ultra. V4.1 inference works on this machine: we have successfully run server requests and inference benchmarks. We are not reporting an inference crash or inability to generate. The test failure also reproduces on pristine upstream a04f46f, without our KV compression PR (#186); see the fresh reproduction below in the comments.

tests/test_deepseek41_metal fails the new router check on M1 Ultra:

router n=256 mode=0 rows=1 row=0 expert=4 logit=-11.8886719 actual=0.00263455603 ref=0.00262947031
tests/test_deepseek41_metal.c:105: fabs(probs[t * n + e] - ref[e]) <= 3e-6 * (1 + ref[e])
ds4: Metal cleanup discarded 5 live tensor handles
make: *** [test-deepseek41-metal] Error 1

Repro

make tests/test_deepseek41_metal && ./tests/test_deepseek41_metal

No model or GGUF needed. Fails every run, identical values.

Environment

Not a regression: the check is new

check_router is added wholesale by a04f46f; every line of it is new in that commit's diff of tests/test_deepseek41_metal.c. Confirmed by bisect on this machine:

So the kernel behaviour is presumably unchanged; the new check is simply tighter than pre-M5 Apple GPUs can satisfy.

Root cause

The check compares two independent FP32 evaluations of sqrt(softplus(x)) for a logit of about -11.9, where that expression is ill-conditioned.

Kernel, metal/dsv4_misc.metal:5188 (and the same form at :5349):

const float4 sp = select(log(1.0f + exp(x)), x, x > 20.0f);

Host reference, tests/test_deepseek41_metal.c:98:

ref[e] = sqrtf(v > 20 ? (float)v : logf(1.0f + expf((float)v)));

For x = -11.8886719, exp(x) is about 6.87e-6. Adding that to 1.0f discards most of it: float resolution near 1.0 is 1.19e-7, so the addend carries roughly 0.68% relative error before the log. Against the true value:

Both sides are wrong by several times the tolerance, in the same direction, by different amounts. Their gap is 5.09e-6 against a bound of 3.01e-6. The check therefore only passes where the GPU happens to round this expression the same way the host does, which is presumably true on M5 Max and M3 Ultra and is not true here.

Note the second row of that table independently: the router probability the kernel produces is about 0.53% high for logits near -12. Switching the kernel to log1p(exp(x)) would remove essentially all of it.

Suggested fix

Two options, and the choice is yours since they differ in intent:

1. Fix the reference only. Use log1pf(expf(v)) in the test and widen the tolerance enough to cover the kernel's genuine FP32 error at small logits. Keeps kernel behaviour, makes the check portable. The comment at tests/test_deepseek41_metal.c:96 says the 1.0f + form was chosen deliberately to match Metal, so this is the smaller change.
2. Fix the kernel too. Use log1p(exp(x)) in metal/dsv4_misc.metal:5188 and :5349, then keep a tight tolerance against an accurate reference. This also removes the 0.53% routing-probability error above.

Scope of verification

Verified on Metal, M1 Ultra only. CUDA and ROCm untested. I do not have an M5 or M3 to confirm the check passes there.