[Umbrella] ONNX gaps for storage-faithful and efficient GGUF/LLM interchange

Summary

ONNX can represent most GGUF model mathematics with primitive operators, but it often cannot simultaneously preserve:

1. the checkpoint's exact compact storage,
2. portable numerical semantics, and
3. an optimization boundary that runtimes can execute efficiently.

This umbrella issue collects the remaining standard-level gaps observed while implementing 148 GGUF architecture routes and 25 active stored qtypes in onnxruntime/mobius. It links narrower existing proposals rather than replacing them.

[!IMPORTANT] Core gap: ONNX can usually express the mathematics, but often cannot preserve the checkpoint's native compact bytes and an efficient runtime boundary at the same time.

Target: preserve math + storage + execution semantics, with a normative portable fallback and fail-closed handling for unknown codecs.

Three distinct questions

• Can ONNX compute it? — Mathematical representability: primitive graphs can usually compute the result.
• Can ONNX preserve it? — Storage fidelity: canonical TensorProto encoding often cannot reuse the original GGUF bytes.
• Can runtimes execute it efficiently? — Execution fidelity: a primitive fallback can force full dequantization, compute every expert, or produce tens of thousands of nodes.

️ At a glance

A successful standard should preserve all three dimensions where possible while retaining a well-defined portable fallback.

✅ Current ONNX coverage

Recent ONNX additions cover important parts of this space:

• INT4/UINT4, FLOAT4E2M1, FP8 variants, FLOAT8E8M0, and INT2/UINT2 data types
• blocked DequantizeLinear
• Attention with MHA/GQA/MQA and cache inputs/outputs
• RotaryEmbedding
• opset 27 LinearAttention, including linear, gated, delta, and gated_delta recurrent updates with past_state/present_state
• functional state using ordinary graph inputs/outputs
• external tensor data using one location plus optional offset and length

These are valuable, but they do not describe many native GGUF records or provide standard sparse/weight-only consumers.

Evidence from real GGUF imports

Mobius currently classifies each import separately as byte-preserved, numerically lossless repack, lossy requantization, explicit float, or rejected. See:

• truthful GGUF quantization reporting, PR #668
• qtype/role runtime capability matrix, PR #661
• exact-code FP8 QDQ and large PLE evidence, PR #658
• Nemotron-H MoE fail-closed evidence, PR #674
• Qwen3.5-MoE explicit-float runtime evidence, PR #672
• generated GGUF support table

Examples:

1️⃣ Encoded/block-quantized tensor storage

Why affine QDQ is insufficient: blocked DequantizeLinear describes uniform affine blocks. Native GGUF formats additionally use:

• hierarchical super-block and sub-block scales,
• scale plus minimum encodings,
• nonlinear/shared codebooks,
• mixed bit widths or formats within one logical tensor,
• interleaved scale/code records,
• expert-major packed blocks,
• codec-specific nibble/bit ordering,
• 1-, 3-, 5-, and 6-bit codes,
• importance/imatrix-derived encoding metadata.

Representing each format as a new primitive element-type enum would not scale. A more general option is a first-class encoded tensor view that separates logical tensor semantics from physical storage:

logical_shape
logical_element_type
storage_resource / external byte ranges
codec domain + version
block/super-block geometry
code, scale, zero/minimum and codebook descriptors
byte/bit order
integrity digest

The codec must have normative decode semantics or a standard Function-equivalent fallback. Unknown codecs must fail closed.

Related issues:

• #7691 — non-uniform GGUF K-quant/super-block quantization
• #8359 — externally stored block-quantized tensors without re-encoding

2️⃣ Low-bit weight-only consumers

ONNX has quantized operators for several INT8/UINT8/FP8 cases, but no standard equivalent of runtime-specific weight-only operators such as com.microsoft::MatMulNBits.

The portable fallback is commonly:

packed codes -> unpack/Cast -> DequantizeLinear -> MatMul

This preserves mathematics, but an implementation is not required to fuse it and may allocate the full dense weight. We need standard optimization boundaries with exact fallback semantics:

• BlockQuantizedMatMul
• BlockQuantizedConv
• BlockQuantizedGather / quantized embedding lookup

[!WARNING] Quantized Gather is the acute memory gap: ordinary Gather cannot request decode of only selected rows.

BlockQuantizedGather is particularly important. Ordinary Gather does not provide a standard way to gather codes and their block metadata and decode only selected rows.

Concrete evidence: a Qwen4-Exp per-layer embedding table would expand to approximately 95 GiB BF16 / 191 GiB FP32 if reconstructed as one dense table. Mobius must keep 128 shards separate and perform full-shard DQ before token-sized Gather because no standard quantized Gather exists.

A standard consumer should:

• support canonical and extensible encoded layouts,
• define exact logical equivalence to decode/DQ plus the float operator,
• permit selected-block/selected-row decoding,
• support tied embedding/output weights,
• expose accumulator/output type explicitly,
• avoid requiring a full dense intermediate.

3️⃣ Sparse MoE execution and expert storage

[!NOTE] Do not standardize a model-specific mega-op. Keep routing composable; standardize grouped sparse execution and encoded expert-bank consumption.

A rank-3 stacked expert tensor is legal ONNX storage, but ONNX lacks a standard sparse dispatch/combine or grouped expert matmul boundary.

Primitive TopK/Gather/Scatter/Loop graphs can encode the mathematics, but implementations often either compute every expert or generate one static subgraph per expert per layer.

Real routing variants require more than Softmax -> TopK:

• probabilities and expert-selection scores may be different tensors,
• sigmoid rather than softmax routing,
• correction bias affecting selection but not mixing weights,
• optional normalization and routed scaling,
• shared experts outside sparse dispatch,
• latent projections around routed experts,
• activations including squared ReLU (relu2),
• mixed quantization formats across gate/up/down projections,
• deterministic tie-breaking/capacity/token-drop semantics.

Nemotron-H 30B evidence demonstrates the cost of the current standard-ONNX explicit-float fallback:

• 128 experts, top-6, 23 MoE layers
• 37,142 optimized nodes
• 6,255 initializers
• 6,028 MatMul nodes
• 18.0 GB IQ2_XXS source versus 63.2 GB FP16 / 126.3 GB FP32 explicit-storage equivalents

Interpret the large numbers correctly: the 63.2/126.3 GB figures are storage-equivalent sizes for that explicit-float fallback, not a claim that every runtime must allocate the entire amount simultaneously. ORT's vendor-domain com.microsoft::MatMulNBits can avoid full float materialization after conversion to its affine packed-code + per-block scale/zero-point contract. However, that contract does not natively describe IQ2_XXS's codec/codebook semantics, so such a conversion is not currently proven byte-preserving or value-exact and may require lossy requantization. MatMulNBits also consumes one logical 2-D weight matrix per node; it does not by itself provide a rank-3 expert-bank contract, dynamic top-k expert selection, sparse dispatch/combine, or grouped expert execution. Expanding one node per expert/projection/layer therefore retains the graph-scale problem even when it reduces weight storage.

A bounded parse of the pinned GGUF header gives 31,577,940,288 logical parameters. The following estimates quantify the lossy MatMulNBits alternative rather than leaving "smaller" qualitative. They convert the 31,224,668,160 parameters used by linear projections to symmetric affine packed weights with FP16 per-block scales, split rank-3 expert banks into independent 2-D expert matrices, and keep the remaining 353,272,128 embedding, causal-convolution, state, and normalization parameters in FP16. They exclude optional explicit zero points, container metadata/alignment, and EP-specific prepacking:

For comparison, the pinned mixed-format GGUF is 18,010,755,296 bytes (18.011 GB / 16.774 GiB). Thus affine Q2 can be materially smaller but changes the weights; affine Q4 may be close to or even larger than the source while still losing IQ/codebook semantics. These are storage estimates, not a runtime-support claim: current MatMulNBits still requires one node per 2-D matrix, and EP/kernel availability must be established separately.

The ideal portable representation would keep each expert bank in its native compact IQ2_XXS-style encoded storage, express the router's correction-biased sigmoid selection separately from its unbiased mixing weights, dispatch only the selected top-6 experts, execute their gate/up/down projections through grouped quantized consumers, apply relu2, and combine the routed result with the shared expert. The graph should contain a bounded number of nodes per MoE layer rather than one static subgraph per expert, and a conforming implementation should not need to materialize the 63.2 GB FP16 or 126.3 GB FP32 expert banks. A normative Function fallback may be slower, but must preserve exactly the same routing, activation, and mixing semantics.

The minimum clean standardization split is:

1. Keep routing composable with existing primitives (Sigmoid/Softmax, correction bias, TopK) so selection scores and mixing weights may remain distinct.
2. Standardize the GroupedMatMul shape proposed in #7902: input[B,M,K], weights[E,K,N], and group_indices[B,M,k]. Because the expert indices are direct inputs, this single boundary already represents dynamic top-k dispatch over a rank-3 expert bank; a separate mandatory router or dispatch op is not required for the common case.
3. Add a way for GroupedMatMul to consume encoded/block-quantized expert banks without first materializing weights[E,K,N] as float. The preferred general solution is the encoded-tensor storage contract from Gap 1 plus defined consumer semantics. If that cannot be integrated cleanly, a narrower new BlockQuantizedGroupedMatMul operator with exact decode-plus-grouped-matmul fallback semantics is needed.
4. Keep activation, routed-weight multiplication/reduction, and shared experts composable, while allowing runtimes to fuse the recognized end-to-end pattern.

Therefore the missing primitive is not a model-specific NemotronMoE monolith. #7902 covers the float grouped-execution boundary; the unresolved new standard work is its storage-faithful quantized/encoded counterpart.

⭐ Why #7902 is pivotal

#7902 is the key near-term standard proposal for MoE efficiency. Its weights[E,K,N] plus group_indices[B,M,k] contract replaces thousands of statically expanded per-expert MatMul branches with a bounded number of nodes per layer. More importantly, it gives runtimes an unambiguous semantic boundary: these are dynamically selected expert GEMMs, not an accidental pattern of Gather/Reshape/MatMul nodes. That signal is what permits safe end-to-end MoE optimization without hard-coding individual model architectures.

Standardizing #7902 would immediately improve portable float MoE graphs and establish the consumer shape that encoded/quantized expert-bank storage should target. It does not by itself solve native IQ2_XXS storage, but it prevents the storage proposal from having to solve sparse execution and graph explosion at the same time.

⚡ Runtime work still required

A schema boundary enables optimization; it does not supply the kernel. Runtime implementations still need to:

• Dispatch only selected experts: compact/permute tokens by group_indices, handle empty and uneven groups, and avoid reading or decoding unselected expert blocks.
• Fuse the expert MLP: combine dispatch, grouped gate/up GEMMs, activation (relu2, SiLU, GELU, etc.), grouped down GEMM, unpermute, routed-weight reduction, and shared-expert addition where legal.
• Fuse decode with compute: consume encoded/IQ/K-quant expert blocks directly inside grouped GEMM instead of creating a dense FP16/FP32 expert bank or per-expert dequantized intermediates.
• Optimize both decode and prefill: use persistent/small-M kernels for token-by-token decode and high-throughput grouped kernels for variable-size prefill groups.
• Prepack and cache safely: reuse EP-specific packed expert weights without duplicating the whole bank, preserve external-data integrity, and keep workspace proportional to dispatched tokens/experts.
• Control routing deterministically: preserve selection-versus-mixing score semantics, tie-breaking, normalization, capacity/token-drop policy, and numerical accumulator/output types.
• Scale across devices: support expert parallelism, overlap all-to-all token exchange with grouped compute, and avoid unnecessary host transfers.
• Retain a correctness path: execute the normative Function/decomposition when no fused kernel is available, while reporting capability/performance separately from schema validity.

These kernel, fusion, scheduling, and distributed-execution items belong in individual runtime projects after ONNX provides the portable semantic boundary. They are listed here to make clear what #7902 unlocks and why merely serializing a rank-3 float tensor is not sufficient.

Related: #7902 (GroupedMatMul for MoE).

4️⃣ Segmented external storage and integrity

Current external data works well when a logical tensor maps to one contiguous byte range whose bytes already use canonical ONNX tensor encoding. It does not portably describe one logical tensor spread across multiple files or discontiguous ranges.

This matters for sharded checkpoints and very large embedding/expert tensors. Concatenating them can require another full-size destination.

Please consider ordered external segments such as:

{ location, file_offset, length, logical_byte_offset, digest }

Requirements:

• one logical tensor may have multiple ordered segments,
• segments may live in different files,
• no mandatory concatenated copy,
• normative bounds/overlap validation,
• strong integrity digests (for example SHA-256),
• clear interaction with encoded tensor codecs.

The legacy TensorProto.Segment field does not currently provide interoperable external assembly semantics; see #2630.

5️⃣ Remaining recurrent/SSM operators, sparse attention, and state bindings

ONNX opset 27 now standardizes LinearAttention with linear, gated, delta, and gated_delta update rules, GQA/MQA, and recurrent state I/O. It also standardizes CausalConvWithState for the depthwise causal convolution used by Gated DeltaNet and Mamba-family preprocessing. These close the standard representation gaps for original DeltaNet/Gated DeltaNet-style layers, including the form used by Qwen3.5, and their stateful causal convolution. Runtime kernel availability and performance for these new operators are implementation concerns, not ONNX schema gaps.

Remaining model families and newer variants still require other optimization boundaries where their equations do not match LinearAttention-27, including:

• Gated DeltaNet-2-style independently parameterized erase/write updates (#8027),
• Mamba/selective scan,
• sparse selected-token attention,
• compressed sparse attention,
• paged/absorbed latent attention.

Scan, Loop, and primitive tensor operations can express many of these, but do not necessarily give runtimes a stable fusion target.

Related: #7689 delivered the opset 27 LinearAttention schema; #8027 tracks a recurrence not covered by that schema.

Separately, ONNX can expose state as graph inputs/outputs but lacks a structured model-level declaration for how a runtime should carry those tensors across calls. A hypothetical StateBindingProto (the name is illustrative, not an existing schema) or equivalent metadata contract could declare:

• the feedback edge from a current-call output to a next-call input,
• state role (KV cache, recurrent/SSM, convolution window, page table, or another extensible role),
• initialization policy (zero, initializer-backed, optional, or caller-required),
• semantic axes such as batch, beam, layer, head, sequence, and capacity,
• update policy such as append, replacement, recurrent update, or ring buffer,
• capacity and growth policy,
• beam reorder behavior, including the axis that must be gathered,
• rollback behavior: truncation where valid, checkpoint/restore, recomputation, or explicitly unsupported,
• lifetime/sharing scope and heterogeneous per-layer or per-component membership.

Illustratively:

StateBinding {
  name
  input_value_name
  output_value_name
  role
  axis_roles[]
  initialization
  update_policy
  reorder_policy
  rollback_policy
  group
}

For example, a LinearAttention binding would connect present_state to the next invocation's past_state, identify the batch/beam axis for generation-time reorder, and normally require checkpoint-or-recompute rather than claiming that an arbitrary token rewind is possible. A CausalConvWithState binding would make its fixed convolution-history window discoverable. KV bindings could instead declare sequence-axis append/truncation and capacity grow