BEAM=2 将 GPU 挂载在 gfx1201 (RDNA4) 上,使用 bf16 matmul,需要重置驱动程序

--- BEAM search on a bf16 matmul hangs the GPU on gfx1201. The kernel driver detects a hung queue and performs a MODE1 reset, losing VRAM. `BEAM=0` on the identical workload completes in 1.4s. Reproduced 4x from clean state, on both `5b05106` and current master `0319b1e`. ### Repro import time from tinygrad import Tensor, Device, dtypes from tinygrad.helpers import Context N = 2048 for beam in (0, 2): print(f"BEAM={beam} ...", end=" ", flush=True) t0 = time.perf_counter() try: with Context(TC=1, BEAM=beam): a = Tensor.rand(N, N).cast(dtypes.bfloat16).contiguous().realize() b = Tensor.rand(N, N).cast(dtypes.bfloat16).contiguous().realize() (a @ b).realize() print(f"OK in {time.perf_counter()-t0:.1f}s") except Exception as e: print(f"FAILED after {time.perf_counter()-t0:.1f}s: {type(e).__name__}: {e}") $ AMD=1 python3 repro.py BEAM=0 ... OK in 1.4s BEAM=2 ... FAILED after 30.5s: RuntimeError: Wait timeout: 30000 ms! (the signal is not set to 27, but 26) Raised from `runtime/support/hcq.py:287` in `HCQSignal.wait`, via `HCQCompiled.synchronize` at `hcq.py:427`. ### dmesg amdgpu 0000:23:00.0: GPU reset begin!. Source: 3 amdgpu 0000:23:00.0: MES(1) failed to respond to msg=REMOVE_QUEUE amdgpu 0000:23:00.0: failed to unmap legacy queue amdgpu 0000:23:00.0: MODE1 reset amdgpu 0000:23:00.0: GPU reset succeeded, trying to resume amdgpu 0000:23:00.0: VRAM is lost due to GPU reset! amdgpu 0000:23:00.0: GPU reset(3) succeeded! amdgpu 0000:23:00.0: [drm] device wedged, but recovered through reset The GPU recovers each time; no reboot required. Any concurrent GPU work dies. ### Environment GPU AMD Radeon RX 9070 XT (device 0x7550, gfx1201 / RDNA4) renderer AMDLLVMRenderer tensor WMMA_16_16_16_half_float, WMMA_16_16_16_half_half, cores WMMA_16_16_16___bf16_float, WMMA_16_16_16___bf16___bf16 ROCm 7.2.4 kernel 7.0.0-30-generic OS Ubuntu 24.04.4 LTS Python 3.12.3 tinygrad master 0319b1e (also reproduces on 5b05106) ### Secondary: TC=1 is slower than TC=0 for half precision Possibly the same root cause, so noting rather than filing separately. Same 2048x2048 matmul, `BEAM=0`, TFLOP/s from wall clock and `2*n^3` (not `GlobalCounters`), median of 5, fresh operands each iteration: | dtype | TC=1 | TC=0 | TC=0 / TC=1 | --- | --- | --- | --- | fp32 | 10.43 | 10.27 | 0.99x | fp16 | 3.76 | 9.38 | **2.49x** | bf16 | 2.24 | 3.74 | **1.67x** | fp32 is unaffected, as expected. Both half precisions are meaningfully **slower** with tensor cores enabled — and `TC` defaults to `1`, so this is the out-of-box path on this hardware. AMD documents ~191 TFLOP/s dense bf16 WMMA for RDNA4; we measure 2.24. ### Note `test/opt/test_tensor_cores.py:84` still carries `# TODO: don't skip bf16 for real device (METAL, AMD)` on master, so this path appears to be untested on real AMD hardware — consistent with it going …