POC for qwen3.5 / qwen3.6 support optimized for AGX Orin

https://github.com/alansrobotlab2/mlc-llm/tree/qwen3_5 https://github.com/alansrobotlab2/mlc-llm/blob/qwen3_5/qwen3_5.md https://github.com/alansrobotlab2/mlc-llm/blob/qwen3_5/worklog.md https://github.com/alansrobotlab2/mlc-llm/tree/qwen3_5/.claude/plans

I had a need to get qwen3.6-35b-a3b working on my orin agx. Here are the results from claude code hacking away at it for about 3 days. If there's anything here worthwhile you're welcome to it.

approx 2x performance compared to unsloth q4 35b model, 1.3x unsloth 0.8b model. Probably also benefits the other qwen3.5 and qwen3.6 models. ymmv.

qwen3.6-35b-a3b benchmark results

tg	MLC q4f16_1 v2+FI	llama.cpp Q4_K_S¹	llama.cpp Q4_K_XL²	ratio (vs Q4_K_XL)
512	54.46	29.19	28.26	1.927×
1024	54.30	29.30	28.21	1.925×
2048	54.07	29.31	28.13	1.922×
4096	53.69	29.04	28.07	1.913×
8192	53.00	28.48	27.86	1.902×
Δ tg512→tg8192	−2.7 %	−2.4 %	−1.4 %	flat

qwen3.5-0.8b benchmark results

tg	llama.cpp Q4_K_XL (pure tg)	MLC q4f16_g16e + FI	ratio
512	100.3	134.82	1.345×
1024	100.1	134.29	1.341×
2048	99.7	133.54	1.340×
4096	98.0	132.17	1.349×
8192	96.5	129.59	1.343×

Claude's summary

Starting point (~10 tps). Bench harness was reporting blended pp+tg numbers, batch_decode was being routed through the wrong path, and the dlight-default GEMM grid was tuned for Hopper.

Phase 1–4 — kernel & router fixes (10 → 52.6 tps). Pinned batch_decode batch_size=1 to unlock the gemv MoE path (+128%); replaced the serial top-k softmax router with a parallel kernel (+6.3%); fixed the sm_87 dlight GEMV tile (+6.9%); register-cached gdn_func state (+2.4%). MTP self-spec + B-ext spec decode were both empirically ruled out — token-agreement collapsed under fp16 drift.

Phase 5–7 — KV cache experiments. fp8 was a structural loss on Orin; int8 shipped throughput-neutral as a capacity lever; mxfp4 lost to LUT cost. Plumbing kept for sm_89+ ports.

Phase 8 — hybrid prefix cache. TTFT 17.7× on the 35B. Spec-batch hang fixed en route.

Phase 9b — the prefill unlock. Stage 9.2's tile-tuning lever returned only +2.9%, confirming the hand-schedule was near-locally-optimal. The real win was Stage 2d: a tensor-core (wmma) MoE GEMM with int4-dequant and lookup-table dispatch, hitting 16.9 TFLOPS in the hottest kernel vs ~0.5 TFLOPS scalar. Combined with re-enabling FlashInfer (the Phase 8 rebuild had silently flipped flashinfer=0), pp512 went 207.95 → 561.16 tps (2.70×) and tg recovered to 54.35.

End state: pp 561.5 ± 0.4, tg flat from 54.46 → 53.00 across 512 → 8K depth, 1.85× over llama.cpp Q4_K_XL at every depth. The old §14.1 long-ctx crossover is closed; both stacks are now weight-BW bound with near-identical decay shape.

It includes changes to TVM as well as exploring:

• mtp>1 (net negative on orin)
• fp8 kv cache (net negative on orin)
• int8 kv cache (possibly improves long contexts)
• mxfp4 kv cache
• improved prefill 2x over baseline