#1370 Perplexity vs Ollama's Q4_K_M on the same tokens: our int4 experts lose 2.5–3.4 %, the int8 trunk buys nothing — two questions about the format · colibri

I measured output quality against Ollama for the first time, on Qwen3.6-35B-A3B, and the result raises two questions about the container format that only you can answer. Data first, then the questions. Nothing here is a request to change course; it is a request for your read.

The measurement

wikitext-2-raw test split, identical tokens on both sides (297,054 with the container's tokenizer.json, 297,054 with llama-tokenize on Ollama's GGUF, same ids), scored the way llama-perplexity scores: 16 chunks of 512, first half context, second half scored, natural log. Colibrì side through qwen36's PPL=1 teacher-forcing mode, one chunk per process; Ollama's own Q4_K_M blob through a llama.cpp CPU build (which needed four local loader patches to read Ollama's GGUF — Ollama's converter targets Ollama's engine; patches listed at the end, none of them touch arithmetic).

	perplexity	scored tokens
Ollama Q4_K_M (`qwen3.6:35b`, 23.9 GB)	7.147	4080
Colibrì gs64 int4 experts, int8 trunk (22 GB)	7.325	4096
Colibrì per-row int4 experts, int8 trunk (20 GB)	7.388	4096
Colibrì per-row int4 experts, f32 trunk (`COLI_DENSE_I8=0`)	7.389	4096

Paired over the same 16 chunks: Q4_K_M is better in 15 of 16 against gs64 and 14 of 16 against per-row; mean NLL difference +0.025 nats (standard error 0.006) and +0.033 (0.010). The int8 trunk against the f32 trunk: −0.0001 ± 0.0025, i.e. nothing.

Sanity checks before trusting it: the patched llama.cpp reports the right hyperparameters (40 layers, 16/2 heads, 256 experts top-8, DeltaNet 128/32/16, 35.95 B params, Q4_K – Medium), and its greedy continuation of my standard prompt begins word for word like Colibrì's and diverges only after ~25 tokens, as two quantizations should. A misloaded model would not score 7.1.

What I read from it, and where I was wrong

The int8 trunk is quality-neutral and bandwidth-expensive. It reads 1 byte per weight where Q4_K_M reads ~0.56, which my R4 measurements had already identified as the largest term in the speed gap against Ollama (1.23× on one 8 GB card after the placement work in #1361). I had kept it as "the precision we pay for". It buys no perplexity. That was my design choice in #712, so this is a correction of my own assumption, not of yours.
The experts are where the gap sits, for two reasons, and I had the second one wrong until I looked at Ollama's tensor types. Q4_K_M is not "the same bits": ffn_gate/up_exps are Q4_K (4.5 bpw, 576 KiB per expert) but ffn_down_exps and output.weight are Q6_K (6.6 bpw, 840 KiB per expert). The experts average 5.2 bpw against gs64's 4.56; the file is 23.9 GB against 22. So part of the 2.5 % is more bits where it matters most, and part is the quantizer itself — asymmetric with a min per 32-block and a super-block scale, against our symmetric absmax per 64. Which part dominates I have not separated; an A/B with down at int6/int8 in our converter would.
gs64 over per-row (−0.85 %) confirms the August A/B at the perplexity level; Z5 (gs64 as the converter default) now has its number.

The two questions

Q1 — quantizer. Would you want the converter (and the CPU/CUDA/Metal/Vulkan expert kernels) to move to a K-quant-style scheme — a min and a scale per 32 with a super-block scale, and more bits on down than on gate/up? It is a project-wide format change, which is why I am asking before writing a line. My expectation is that it closes the quality gap at equal size and lets the trunk drop to the Q4 class as well, which would also close part of the speed gap. I can run the experiment first (down at higher precision, then the min/scale quantizer) and bring numbers rather than an opinion, if that is the order you prefer.

Q2 — GGUF as a source. In #1163 you wrote that GGUF cannot serve colibrì's per-expert pread because it is a single file meant to be mapped whole and "where quantization blocks straddle expert boundaries that is not possible at all". I checked that against Ollama's file: Q4_K blocks are 256 weights, the inner dimension is 2048, so every expert in ffn_*_exps is one contiguous, integral byte range (576 KiB or 840 KiB), exactly as addressable by offset as a safetensors tensor is — the per-expert stream would work unchanged. What GGUF would cost is real and I do not want to understate it: K-quant dequant kernels for Q4_K/Q5_K/Q6_K/Q8_0 on every backend, a second reader beside st.h, and converter-dependent tensor naming (four patches for Ollama's file alone). What it would buy is the whole GGUF ecosystem without conversion, including blobs Ollama has already downloaded on the same machine. Was the block-alignment point the deciding reason, or were there others — keeping the format under your control, not carrying K-quant kernels, the direction toward official checkpoints as the source (Kimi's MXFP4, Qwen3.8's FP8)? Knowing which would tell me whether Q1 should be "K-quant scheme in our container" or "read GGUF".

Either answer is fine by me; the measurement is the same either way, and I would rather build what fits the project than what fits my curiosity.

Reproducibility

Harness: tools-free Python driver that cuts the chunks and drives PPL=1 (I can PR it under c/tools/ if useful); per-chunk numbers for all four runs are in the table I can attach.
llama.cpp 7620399, CPU build, local patches to load Ollama's GGUF: accept rope.dimension_sections of length 3; fall back to blk.N.ssm_dt without .bias; normalise the per-layer head_count_kv array (zeros on DeltaNet layers) to its maximum; load partially past the v.*/mtp.* tensors. None touch the math; the same binary generates coherent text.
Same prompt, same box and thread settings as the R6 calibration in #1361.