CPU `gemm_4bit_forward` kernel is requested without `backend="cpu"`, so it never loads on a CUDA torch build

bitsandbytes/backends/cpu/ops.py requests the fused CPU 4-bit gemm kernel without specifying a backend. kernels therefore infers the backend from the installed torch build, which on any CUDA wheel is a CUDA backend — and kernels-community/quantization-bitsandbytes publishes CPU-only build variants. Every variant is rejected, get_kernel raises, and the except Exception handler logs a warning on every process start on any host whose CPU reports AVX512-BF16.

This is separate from #1972, which fixed the repo id and the missing version in the same call. With 0.50.2 the id and version are correct; the backend mismatch remains.

Versions

• bitsandbytes 0.50.2
• kernels 0.16.1
• torch 2.12.1+cu130 (torch.version.cuda == "13.0", compiled_with_cxx11_abi() == True)
• Python 3.12, Linux x86_64
• CPU reports avx512_bf16, so has_avx512bf16() is True and the guarded block executes

What happens

Failed to load CPU gemm_4bit_forward from kernels-community:
Cannot find a build variant for this system in kernels-community/quantization-bitsandbytes
(revision: 4fc7d1e1abacca3985171c276723e1c4b4f30a81):
torch212-cxx11-cpu-x86_64-linux: backend (cpu) does not match ...

Reproduction

from kernels import get_kernel, has_kernel
from kernels.backends import _select_backend

REPO = "kernels-community/quantization-bitsandbytes"
print(_select_backend(None).variant_str)      # -> cu130
print(has_kernel(REPO, version=1))            # -> False

get_kernel(REPO, version=1)                   # -> FileNotFoundError, backend (cpu) does not match
get_kernel(REPO, version=1, backend="cpu")    # -> loads torch212-cxx11-cpu-x86_64-linux cleanly

The second call is the whole report: the variant exists and is loadable; only the inferred backend is wrong for it.

Why the inference cannot be right here

The kernel is registered for the CPU device — @register_kernel("bitsandbytes::gemv_4bit", "cpu") — so the backend it needs is known statically at the call site. kernels cannot infer it, because it infers from the torch build rather than from the device the caller intends. A CUDA torch build is the normal case for anyone who also uses this library on a GPU, so on those installations the CPU kernel is unreachable by construction.

Suggested fix

Pass the backend explicitly at the call site:

gemm_4bit_forward_kernel = get_kernel(
    "kernels-community/quantization-bitsandbytes", version=1, backend="cpu"
)

Impact

Cosmetic but persistent: the fused path silently falls back to the unfused reference implementation, and the warning appears on every process start — including in headless/containerised runs, where it lands in every log a batch job produces. Downstream projects that use bitsandbytes only for CUDA quantization end up suppressing the logger, which also hides bitsandbytes.cextension's CUDA-setup warnings unless they take care to target the submodule.

Remark

This issue was formated and investigated with the help of AI.