#32496·onnxruntime

[Feature Request] Add Fusion Transformers for WebNN EP Decomposed LM Operators

Author: Yuhengwe1Created Sep 9, 2026Updated Sep 16, 2026
Labelsfeature requestmodel:transformerplatform:webep:WebNNep:WebGPU

Describe the feature request

We propose adding fusion transformers to reconstruct LM operators decomposed during WebNN EP graph processing: GroupQueryAttention, MatmulNBits, RMSNorm and GatherBlockQuantized.

These high-level operators are already present in the original ONNX model, but the WebNN EP decomposes them into primitive operations to meet WebNN API constraints. This decomposition affects both performance and numerical accuracy: native EPs such as WebGPU can no longer take advantage of their specialized kernels for these high-level operators.

Describe scenario use case

The target scenario is on-device LM inference through WebNN, using WebNN's ORT backend with native WebGPU EP. In phi4-mini demo, reconstructing these operators reduces graph complexity:

Operator Node reduction
GroupQueryAttention About 3,030 GQA-related primitive nodes → 32 GQA nodes
MatMulNBits 644 → 161 nodes
RMSNorm 390 → 65 nodes
GatherBlockQuantized 2 → 1 node, avoiding a roughly 1.145 GiB dequantized embedding table per invocation

And these fusions improve decode IPOT from ~2000ms/token to ~60ms/token. The upstream path also exhibited accuracy issues; restoring RMSNorm with FP32 statistics resolved the generation correctness issue observed in our demo.