[Feature Request] Speculative decoding (draft model + batched verification) in the web runtime

Feature request: speculative decoding (draft + batched verification) in the web runtime

Motivation

The mlc-llm serve engine already supports speculative decoding (speculative_mode in cpp/serve/config.h / python/mlc_llm/serve/config.py), but web-llm does not expose it. In-browser serving is exactly where it pays most: WebGPU inference is memory-bandwidth-bound at batch 1, which is the regime where draft/verify buys the largest wall-clock wins.

Our use case: a small draft model (Qwen3.5-0.8B) paired with a ~2B verifier from the same tokenizer family, both already running individually under web-llm as q4f16 MLC builds. Server-side measurements with this model family show >2x decode speedup at K=5 in reasoning-heavy generation, with draft acceptance rates around 0.82 — so the browser win is sitting on the table.

What's missing in the web runtime

Rolling this in user space isn't currently possible because the API doesn't expose the needed primitives:

1. Per-position logits from a prefill/verify pass — verifying K drafted tokens in one forward pass requires reading logits at each of the K positions, not just the final one.
2. KV-cache rollback — on rejection at position i, the verifier's cache must rewind to i.

Without these, "verification" degenerates into the verifier regenerating the same tokens at full cost.

Ask

Either of these would unblock the use case:

• Preferred: expose the engine's existing speculative mode in web-llm (e.g., an optional draft_model in the engine config, mirroring the serve engine's interface), or
• Minimal: expose the two primitives above (multi-position logits from a forward pass + KV rollback) so applications can implement draft/verify themselves.

Happy to test builds against our draft/verifier pair and report acceptance/latency numbers on real workloads.