#17800·tinygrad

Q8_0 Qwen 27B decode is ~85x below isolated GEMV bandwidth

Author: PKD667Created Aug 28, 2026Updated Aug 28, 2026

The tinygrad engine decodes tokens from Qwen3.8-27B at 1 < tok/s bs=1 (19.4 GB/s streaming).

Measured bandwidth references on the same idle GPU (GH200) is around 2.7 TB/s for a plain f16 stream-add, and 1.6 TB/s for an M=1 GEMV against the biggest layer (lm_head, 5120×248320).

The integrated decode is around 85× below what the GEMV kernels achieve in isolation, with the Q8_0→f16 expansion redone inside every kernel launch of every step.

Reference ~50/s on a (probably very unoptimized) ollama setup