[Bug]: bf16 MatMul/GEMM returns corrupted output (cos 0.28-0.65, 5-135x magnitude) - f16 and f32 controls are correct
OpenVINO Version
2026.4.0
Operating System
Ubuntu 20.04 (LTS)
Device used for inference
CPU
Framework
PyTorch
Model used
Qwen3-Reranker-0.6B
Issue description
The OpenVINO CPU plugin computes incorrect results for bf16 MatMul/GEMM. A single hand-written bf16 MatMul (no tracing, no fusion, no model download) returns an output with cosine ~0.28 and magnitude ~5.5x the correct value, while the identical graph in f16 and f32 is correct (cos 0.999997). On a real 0.6B cross-encoder (Hugging Face Qwen/Qwen3-Reranker-0.6B) traced through the same path, the bf16 model's final hidden state is cos 0.633 vs the float32 reference, whereas the f16 model is cos 0.9999.
torch's own bf16 forward on the same model and same CPU is cos 0.9999, so the corruption is isolated to the OpenVINO CPU bf16 GEMM data path, not to bf16 in general and not to model conversion.
Any bf16-activation model is therefore silently corrupted when run on the CPU plugin. f16 is the correct/usable activation for CPU inference on this hardware.
Environment
| item | value |
|---|---|
| OpenVINO | 2026.4.0 (pip wheel 2026.4.0-22959-99c81491cc3-releases/2026/4), Python API (openvino, openvino.opset13) |
| OS | Ubuntu 24.04 LTS, x86_64, kernel 7.2.5 |
| CPU | AMD Ryzen 9 9950X (16 cores / 32 threads) |
| CPU ISA | AVX512-F, AVX512-BF16 (avx512_bf16), AVX512-VNNI, F16C; no Intel AMX (no amx_tile/amx_bf16/amx_int8), no AVX10.2 |
| Python deps | torch 2.14.0+cpu, numpy 2.4.6, ml_dtypes (real-model reference only) |
The CPU is AMD, so the bf16 GEMM exercises the AVX512-BF16 oneDNN path (the Intel AMX bf16 path is not present on this hardware). This may be relevant to where the bug lives.
Minimal reproduction (no model download, ~5 s)
A single hand-written bf16 MatMul with a correct bf16 weight constant, compared against a float32 numpy reference:
import numpy as np
import ml_dtypes
import openvino as ov
import openvino.opset13 as op
core = ov.Core()
rng = np.random.default_rng(11)
O, K = 1024, 512
Wf = (rng.standard_normal((O, K)) * 0.02).astype(np.float32)
x_in = (rng.standard_normal((64, K)) * 0.02).astype(np.float32)
ref = x_in @ Wf.T # float32 reference GEMM (no bias)
def cos(a, b):
return float((a.ravel() @ b.ravel()) / (np.linalg.norm(a) * np.linalg.norm(b)))
def run(y_op, param, feed):
model = ov.Model([y_op], [param])
cm = core.compile_model(model, "CPU")
req = cm.create_infer_request()
req.infer({list(cm.inputs)[0]: feed})
return req.get_output_tensor(0).data.astype(np.float32)
# correct bf16 weight constant (verified by running, NOT by get_data() readback)
Wconst = op.constant(Wf.astype(ml_dtypes.bfloat16), ov.Type.bf16)
x = op.parameter((64, K), ov.Type.bf16)
out1 = run(op.matmul(x, Wconst, False, True), x, x_in.astype(ml_dtypes.bfloat16))
print(f"bf16 in -> bf16 out : cos={cos(out1, ref):.6f} mag={np.abs(out1).mean()/np.abs(ref).mean():.1f}x")
xf = op.parameter((64, K), ov.Type.f32)
out2 = run(op.matmul(op.convert(xf, ov.Type.bf16), Wconst, False, True), xf, x_in)
print(f"f32 in -> cast bf16: cos={cos(out2, ref):.6f} mag={np.abs(out2).mean()/np.abs(ref).mean():.1f}x")
x16 = op.parameter((64, K), ov.Type.f16)
out3 = run(op.matmul(x16, op.constant(Wf.astype(np.float16), ov.Type.f16), False, True), x16, x_in.astype(np.float16))
print(f"f16 in -> f16 out : cos={cos(out3, ref):.6f} (control: expect ~0.999)")
x3 = op.parameter((64, K), ov.Type.f32)
out4 = run(op.matmul(x3, op.constant(Wf, ov.Type.f32), False, True), x3, x_in)
print(f"f32 in -> f32 out : cos={cos(out4, ref):.6f} (control: expect ~1.0)")Expected: a clean bf16 GEMM matches the f32 reference to cos ~0.999 (bf16 keeps ~3 significant digits). All four lines ~0.999.
Actual (OpenVINO 2026.4.0, CPU):
bf16 in -> bf16 out : cos=0.275838 mag=5.5x
f32 in -> cast bf16: cos=0.856568 mag=134.6x
f16 in -> f16 out : cos=0.999997 (control - correct)
f32 in -> f32 out : cos=0.999997 (control - correct)The output-magnitude factor is data-dependent and inconsistent (5.5x here, 75x on a 3072x1024 GEMM, 1.4x on the full model) - a hallmark of a scrambled or mis-scaled weight operand in the GEMM, not of floating-point rounding.
Real-model confirmation (Qwen3-Reranker-0.6B)
Same conclusion on Qwen/Qwen3-Reranker-0.6B (a real 28-layer cross-encoder; hidden 1024, inter 3072, vocab 151669, tied embeddings). The float32 reference is AutoModel.from_pretrained("Qwen/Qwen3-Reranker-0.6B", dtype=torch.float32); the OV models are the same weights re-loaded as dtype=torch.float16 / dtype=torch.bfloat16, traced with ov.convert_model, reshaped to dynamic, then compared against the float32 reference (B=1, T=32, seed 1337):
| path | cos vs float32 reference | mean abs |
|---|---|---|
| torch bf16 forward (same CPU) | 0.999857 | 1.27 (ref) |
| OpenVINO f16 | 0.999928 | 1.27 |
| OpenVINO bf16 | 0.632893 | 1.82 (~1.4x too large) |
torch bf16 being correct on this hardware rules out "bf16 is just imprecise" - the corruption is specific to the OpenVINO CPU bf16 GEMM.
What I ruled out (to confirm it is not my conversion code)
- Not a save/reload or serialization artifact. A graph built directly in the opset API (no
convert_model, nosave_model) reproduces it (the minimal repro above). - Not the input dtype. Both "bf16 in -> bf16 out" and "f32 in -> cast bf16" are wrong.
- Not weight storage. The bf16 weight constant is created via the known-correct
astype(ml_dtypes.bfloat16)path and verified by running (see the readout caveat below). - Not a plain bitcast. The output magnitude is ~5-135x off and data-dependent - a genuine numerical error, not a reinterpretation of bits.
- Not a fixed permutation. The error varies with layer size and data.
Related readout caveat (secondary issue, worth calling out)
Constant.get_data() on a bf16 constant returns the constant re-decoded as float16 (a bitcast of the bf16 bits), so reading back a correct bf16 constant yields values that look wrong: the returned dtype is float16, and the readback is cos ~0.86 vs the source (first values ~1.0 instead of ~0.02):
get_data() dtype=float16 shape=(1024, 512)
get_data first8 : [ 0.7749 1.2178 1.1963 -1.0381 -0.9702 ...]
source first8 : [ 0.0007 0.0272 0.0245 -0.0102 -0.0059 ...]This is a separate readout quirk, but it means bf16 constants cannot be validated by readback - you must validate by running the model. It makes the primary bug easy to misattribute to "wrong weights", so it is worth fixing or documenting.
No workaround found
- ISA cap (
ONEDNN_MAX_CPU_ISA). The bf16 GEMM is broken at every bf16-capable ISA (defaultandAVX512_CORE_BF16: cos 0.53, 75x at O=3072/K=1024), and the bf16 path errors out entirely atAVX2(no correct fallback). - CPU plugin config sweep.
defaultandENABLE_HYPER_THREADING=falseboth give cos 0.65; the bf16/dnnl-related keys (CPU_INFERENCE_PRECISION_HINT=bf16,CPU_DNNL_AOT=true,CPU_DNNL_MULTI_STREAM=false,SCHEDULING_CORE_TYPE=performance) are rejected as invalid config keys. No config recovers a correct bf16 GEMM.
Broader context: why I was here (the quantization investigation)
This bug was found while benchmarking quantization options for a CPU cross-encoder (a Qwen3-Reranker-0.6B query-docs re-scorer) and looking for the fastest correct configuration. The headline findings, for context:
- f16 is the fastest correct activation. bf16 is not usable (this bug); f16 and f32 controls are correct. So any fast CPU config must use f16 activations.
- f16-weights vs int8-weights inference speed. With f16 activations, int8 weights are dequantized to f16 and folded into the f16 GEMM at compile time (the arithmetic is identical). On this CPU, full-model inference at the token counts a re-scorer actually runs (M ~ 512-2048, batch x seq-len) is memory-bandwidth-bound on the weights, so int8 (2x smaller weights) is 12-22% faster than f16 weights and 2x the smaller file:
- Qwen3-Reranker-0.6B, 32 threads: M=512 int8 79.9 ms vs f16 101.8 ms (21.5%); M=1024 int8 151.1 ms vs f16 183.2 ms (17.5%); M=2048 int8 314.7 ms vs f16 358.1 ms (12.1%).
- Tradeoff: end-to-end cos ~0.99 (int8) vs ~0.999 (f16 weights); ranking is preserved.
- A named-config tool. I built a config system where the activation is always f16 (the fastest correct activation) and the weight precision is a named choice; the default is the fastest correct config (f16 activations + int8 weights). bf16 is deliberately not offered, because defaulting to it would ship a silently-corrupted model.
The bf16 GEMM bug is the reason bf16 had to be excluded from that "fastest" search - earlier "bf16 is faster" timings were measured on corrupted outputs.
Suggested next steps
- Isolate the CPU bf16 GEMM kernel on the AMD AVX512-BF16 path. The minimal repro above is a 5-second, dependency-light way to reproduce without a model.
- Check whether the bf16 weight-reorder / GEMM data path for the non-AMX AVX512-BF16 case is where the bug lives; the inconsistent 5-135x magnitude suggests a mis-scaled or permuted weight operand in the reorder.
- If a bf16 GEMM on a CPU without a native bf16 GEMM is expected to fall back to a dequant-to-f16 path, that fallback is not happening (the op runs a bf16 GEMM that produces garbage). The contrast between the
AVX2error and theAVX512_CORE_BF16garbage points at that boundary.
Reproducibility
- The minimal repro needs only
numpy,ml_dtypes, and OpenVINO 2026.4.0 (no torch, no model download, ~5 seconds). - The real-model confirmation needs
torch+transformers+ the Qwen3-Reranker-0.6B weights (~1.2 GB download).
Step-by-step reproduction
The OpenVINO CPU plugin computes incorrect results for bf16 MatMul/GEMM. A single hand-written bf16 MatMul (no tracing, no fusion, no model download) returns an output with cosine ~0.28 and magnitude ~5.5x the correct value, while the identical graph in f16 and f32 is correct (cos 0.999997). On a real 0.6B cross-encoder (Hugging Face Qwen/Qwen3-Reranker-0.6B) traced through the same path, the bf16 model's final hidden state is cos 0.633 vs the float32 reference, whereas the f16 model is cos 0.9999.
torch's own bf16 forward on the same model and same CPU is cos 0.9999, so the corruption is isolated to the OpenVINO CPU bf16 GEMM data path, not to bf16 in general and not to model conversion.
Any bf16-activation model is therefore silently corrupted when run on the CPU plugin. f16 is the correct/usable activation for CPU inference on this hardware.
Environment
| item | value |
|---|---|
| OpenVINO | 2026.4.0 (pip wheel 2026.4.0-22959-99c81491cc3-releases/2026/4), Python API (openvino, openvino.opset13) |
| OS | Ubuntu 24.04 LTS, x86_64, kernel 7.2.5 |
| CPU | AMD Ryzen 9 9950X (16 cores / 32 threads) |
| CPU ISA | AVX512-F, AVX512-BF16 (avx512_bf16), AVX512-VNNI, F16C; no Intel AMX (no amx_tile/amx_bf16/amx_int8), no AVX10.2 |
| Python deps | torch 2.14.0+cpu, numpy 2.4.6, ml_dtypes (real-model reference only) |
The CPU is AMD, so the bf16 GEMM exercises the AVX512-BF16 oneDNN path (the Intel AMX bf16 path is not present on this hardware). This may be relevant to where the bug lives.
Minimal reproduction (no model download, ~5 s)
A single hand-written bf16 MatMul with a correct bf16 weight constant, compared against a float32 numpy reference:
import numpy as np
import ml_dtypes
import openvino as ov
import openvino.opset13 as op
core = ov.Core()
rng = np.random.default_rng(11)
O, K = 1024, 512
Wf = (rng.standard_normal((O, K)) * 0.02).astype(np.float32)
x_in = (rng.standard_normal((64, K)) * 0.02).astype(np.float32)
ref = x_in @ Wf.T # float32 reference GEMM (no bias)
def cos(a, b):
return float((a.ravel() @ b.ravel()) / (np.linalg.norm(a) * np.linalg.norm(b)))
def run(y_op, param, feed):
model = ov.Model([y_op], [param])
cm = core.compile_model(model, "CPU")
req = cm.create_infer_request()
req.infer({list(cm.inputs)[0]: feed})
return req.get_output_tensor(0).data.astype(np.float32)
# correct bf16 weight constant (verified by running, NOT by get_data() readback)
Wconst = op.constant(Wf.astype(ml_dtypes.bfloat16), ov.Type.bf16)
x = op.parameter((64, K), ov.Type.bf16)
out1 = run(op.matmul(x, Wconst, False, True), x, x_in.astype(ml_dtypes.bfloat16))
print(f"bf16 in -> bf16 out : cos={cos(out1, ref):.6f} mag={np.abs(out1).mean()/np.abs(ref).mean():.1f}x")
xf = op.parameter((64, K), ov.Type.f32)
out2 = run(op.matmul(op.convert(xf, ov.Type.bf16), Wconst, False, True), xf, x_in)
print(f"f32 in -> cast bf16: cos={cos(out2, ref):.6f} mag={np.abs(out2).mean()/np.abs(ref).mean():.1f}x")
x16 = op.parameter((64, K), ov.Type.f16)
out3 = run(op.matmul(x16, op.constant(Wf.astype(np.float16), ov.Type.f16), False, True), x16, x_in.astype(np.float16))
print(f"f16 in -> f16 out : cos={cos(out3, ref):.6f} (control: expect ~0.999)")
x3 = op.parameter((64, K), ov.Type.f32)
out4 = run(op.matmul(x3, op.constant(Wf, ov.Type.f32), False, True), x3, x_in)
print(f"f32 in -> f32 out : cos={cos(out4, ref):.6f} (control: expect ~1.0)")Expected: a clean bf16 GEMM matches the f32 reference to cos ~0.999 (bf16 keeps ~3 significant digits). All four lines ~0.999.
Actual (OpenVINO 2026.4.0, CPU):
bf16 in -> bf16 out : cos=0.275838 mag=5.5x
f32 in -> cast bf16: cos=0.856568 mag=134.6x
f16 in -> f16 out : cos=0.999997 (control - correct)
f32 in -> f32 out : cos=0.999997 (control - correct)The output-magnitude factor is data-dependent and inconsistent (5.5x here, 75x on a 3072x1024 GEMM, 1.4x on the full model) - a hallmark of a scrambled or mis-scaled weight operand in the GEMM, not of floating-point rounding.
Real-model confirmation (Qwen3-Reranker-0.6B)
Same conclusion on Qwen/Qwen3-Reranker-0.6B (a real 28-layer cross-encoder; hidden 1024, inter 3072, vocab 151669, tied embeddings). The float32 reference is AutoModel.from_pretrained("Qwen/Qwen3-Reranker-0.6B", dtype=torch.float32); the OV models are the same weights re-loaded as dtype=torch.float16 / dtype=torch.bfloat16, traced with ov.convert_model, reshaped to dynamic, then compared against the float32 reference (B=1, T=32, seed 1337):
| path | cos vs float32 reference | mean abs |
|---|---|---|
| torch bf16 forward (same CPU) | 0.999857 | 1.27 (ref) |
| OpenVINO f16 | 0.999928 | 1.27 |
| OpenVINO bf16 | 0.632893 | 1.82 (~1.4x too large) |
torch bf16 being correct on this hardware rules out "bf16 is just imprecise" - the corruption is specific to the OpenVINO CPU bf16 GEMM.
What I ruled out (to confirm it is not my conversion code)
- Not a save/reload or serialization artifact. A graph built directly in the opset API (no
convert_model, nosave_model) reproduces it (the minimal repro above). - Not the input dtype. Both "bf16 in -> bf16 out" and "f32 in -> cast bf16" are wrong.
- Not weight storage. The bf16 weight constant is created via the known-correct
astype(ml_dtypes.bfloat16)path and verified by running (see the readout caveat below). - Not a plain bitcast. The output magnitude is ~5-135x off and data-dependent - a genuine numerical error, not a reinterpretation of bits.
- Not a fixed permutation. The error varies with layer size and data.
Related readout caveat (secondary issue, worth calling out)
Constant.get_data() on a bf16 constant returns the constant re-decoded as float16 (a bitcast of the bf16 bits), so reading back a correct bf16 constant yields values that look wrong: the returned dtype is float16, and the readback is cos ~0.86 vs the source (first values ~1.0 instead of ~0.02):
get_data() dtype=float16 shape=(1024, 512)
get_data first8 : [ 0.7749 1.2178 1.1963 -1.0381 -0.9702 ...]
source first8 : [ 0.0007 0.0272 0.0245 -0.0102 -0.0059 ...]This is a separate readout quirk, but it means bf16 constants cannot be validated by readback - you must validate by running the model. It makes the primary bug easy to misattribute to "wrong weights", so it is worth fixing or documenting.
No workaround found
- ISA cap (
ONEDNN_MAX_CPU_ISA). The bf16 GEMM is broken at every bf16-capable ISA (defaultandAVX512_CORE_BF16: cos 0.53, 75x at O=3072/K=1024), and the bf16 path errors out entirely atAVX2(no correct fallback). - CPU plugin config sweep.
defaultandENABLE_HYPER_THREADING=falseboth give cos 0.65; the bf16/dnnl-related keys (CPU_INFERENCE_PRECISION_HINT=bf16,CPU_DNNL_AOT=true,CPU_DNNL_MULTI_STREAM=false,SCHEDULING_CORE_TYPE=performance) are rejected as invalid config keys. No config recovers a correct bf16 GEMM.
Broader context: why I was here (the quantization investigation)
This bug was found while benchmarking quantization options for a CPU cross-encoder (a Qwen3-Reranker-0.6B query-docs re-scorer) and looking for the fastest correct configuration. The headline findings, for context:
- f16 is the fastest correct activation. bf16 is not usable (this bug); f16 and f32 controls are correct. So any fast CPU config must use f16 activations.
- f16-weights vs int8-weights inference speed. With f16 activations, int8 weights are dequantized to f16 and folded into the f16 GEMM at compile time (the arithmetic is identical). On this CPU, full-model inference at the token counts a re-scorer actually runs (M ~ 512-2048, batch x seq-len) is memory-bandwidth-bound on the weights, so int8 (2x smaller weights) is 12-22% faster than f16 weights and 2x the smaller file:
- Qwen3-Reranker-0.6B, 32 threads: M=512 int8 79.9 ms vs f16 101.8 ms (21.5%); M=1024 int8 151.1 ms vs f16 183.2 ms (17.5%); M=2048 int8 314.7 ms vs f16 358.1 ms (12.1%).
- Tradeoff: end-to-end cos ~0.99 (int8) vs ~0.999 (f16 weights); ranking is preserved.
- A named-config tool. I built a config system where the activation is always f16 (the fastest correct activation) and the weight precision is a named choice; the default is the fastest correct config (f16 activations + int8 weights). bf16 is deliberately not offered, because defaulting to it would ship a silently-corrupted model.
The bf16 GEMM bug is the reason bf16 had to be excluded from that "fastest" search - earlier "bf16 is faster" timings were measured on corrupted outputs.
Suggested next steps
- Isolate the CPU bf16 GEMM kernel on the AMD AVX512-BF16 path. The minimal repro above is a 5-second, dependency-light way to reproduce without a model.
- Check whether the bf16 weight-reorder / GEMM data path for the non-AMX AVX512-BF16 case is where the bug lives; the inconsistent 5-135x magnitude suggests a mis-scaled or permuted weight operand in the reorder.
- If a bf16 GEMM on a CPU without a native bf16 GEMM is expected to fall back to a dequant-to-f16 path, that fallback is not happening (the op runs a bf16 GEMM that produces garbage). The contrast between the
AVX2error and theAVX512_CORE_BF16garbage points at that boundary.
Reproducibility
- The minimal repro needs only
numpy,ml_dtypes, and OpenVINO 2026.4.0 (no torch, no model download, ~5 seconds). - The real-model confirmation needs
torch+transformers+ the Qwen3-Reranker-0.6B weights (~1.2 GB download).
Relevant log output
Issue submission checklist
- I'm reporting an issue. It's not a question.
- I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
- There is reproducer code and related data files such as images, videos, models, etc.
Source: openvinotoolkit/openvino