Q8_0 Qwen 27B decode is ~85x below isolated GEMV bandwidth
Author: PKD667Created Aug 28, 2026Updated Aug 28, 2026
The tinygrad engine decodes tokens from Qwen3.8-27B at 1 < tok/s bs=1 (19.4 GB/s streaming).
Measured bandwidth references on the same idle GPU (GH200) is around 2.7 TB/s for a plain f16 stream-add, and 1.6 TB/s for an M=1 GEMV against the biggest layer (lm_head, 5120×248320).
The integrated decode is around 85× below what the GEMV kernels achieve in isolation, with the Q8_0→f16 expansion redone inside every kernel launch of every step.
Reference ~50/s on a (probably very unoptimized) ollama setup
Source: tinygrad/tinygrad