#617·BitNet

[Bug]: I2_S GEMM fast path produces garbage for multi-token prompts on AVX-only CPUs

Author: puddintaim1975-devCreated Aug 26, 2026Updated Aug 26, 2026

Description

On CPUs without AVX2 (e.g., Intel Xeon E5-2690 v2, Ivy Bridge, AVX-only), the I2_S GEMM fast path in ggml-cpu.c:1492 produces corrupt output when processing prompts with more than ~3 tokens. Single-token generation (GEMV path) works correctly.

Symptoms

  • 1-3 token prompts → coherent output
  • 5+ token prompts → ?????? or garbled output
  • The corruption affects the KV cache: even after the prompt is processed, subsequent generation tokens are garbled

Root Cause

The GEMM fast path (ggml_gemm_i2_i8_s) is called for multi-token prompt evaluation (when src1 has multiple columns). The scalar fallback implementation has a bug in how it indexes the I2_S weight matrix and/or activation matrix for column strides > 1.

Workaround

Disabling the GEMM fast path forces I2_S through the dequantize-then-float-matmul path:

c
// ggml-cpu.c:1492 — change from:
if (src0->type == GGML_TYPE_I2_S && ggml_n_dims(src0) == 2) {
// to:
if (false && src0->type == GGML_TYPE_I2_S && ggml_n_dims(src0) == 2) {

This produces correct results but is slower (~0.6 tok/s prompt eval vs ~26 tok/s for F16 on the same hardware).

Key Distinction from #547

This is distinct from #547/PR #580 which covers the empty-body scalar fallback for ggml_vec_dot_i2_i8_s_* kernels. After applying those fixes (or equivalent scalar implementations), the GEMM path still produces wrong results for multi-token prompts.

The GEMV path (line ~1217) works correctly for single-token generation. The issue is specifically in ggml_gemm_i2_i8_s when called with nr > 1 (multiple activation columns).

Reproduction

bash
# Build with AVX2 disabled (forces scalar fallback)
cmake -B build -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF
cmake --build build --target llama-server -j8

# Short prompt works
curl http://localhost:8081/v1/completions \
  -d '{"model":"bitnet","prompt":"Hello","max_tokens":20}'
# → " there! I'm happy to help" ✓

# Longer prompt fails
curl http://localhost:8081/v1/completions \
  -d '{"model":"bitnet","prompt":"The weather today is","max_tokens":20}'
# → "?????" ✗

Also requires fixes #588 (ReLU²) and PR #616 (weight scale direction) for coherent F16 output.

Environment

  • CPU: Intel Xeon E5-2690 v2 (Ivy Bridge, AVX only, no AVX2)
  • OS: Ubuntu 24.04 LTS
  • Compiler: Clang 18.1.3
  • CMake flags: -DBITNET_ARM_TL1=OFF -DBITNET_X86_TL2=OFF
  • Model: BitNet-b1.58-2B-4T (I2_S format)
  • BitNet commit: 390c30775