Roadmap

Function calling

• Integrate with XGrammar's structural tag: https://github.com/mlc-ai/xgrammar/pull/162, and enable reliable tool use with small models in WebLLM
• Add an E2E MCP-like example, using the structural tag / tool-use stated above

Models

• Support Phi-4
• Support Gemma3 https://github.com/mlc-ai/mlc-llm/pull/3429
• Support Gemma3n

Modalities

• Add reliable image input feature (currently have initial support, performance and correctness need to be further investigated)
• Add other modality input (e.g. audio) https://github.com/mlc-ai/web-llm/pull/793

Performance

• Profile existing performance, identify bottlenecks and address them if exist
• Switch some existing CPU workload (e.g. sampling) to GPU if performance improves
  • https://github.com/mlc-ai/web-llm/pull/697
  • https://github.com/mlc-ai/web-llm/pull/737
  • Sampling on GPU for logprobs == false is blocked since it uses kernels that use i8s. We should have TVM generate kernels that avoid using i8s when the backend is WebGPU.
• Subgroup operation support https://github.com/apache/tvm/pull/18823
• Reduce sync frequency https://github.com/mlc-ai/web-llm/pull/779
• Reduce end-to-end generation time through asynchronous generation and flushing
• Reduce model load time on crash
• Replay conversations and checkpoint state for crash recovery

Others

• Better WASM conversion experience (e.g. hosting a huggingface space, so users do not need to set up environment)
• Qwen2.5's 1.5B webgpu-only correctness issue—very obvious with DeepSeek-R1-Distill
• Parse thinking/non-thinking tokens when returning completion response
• Cross-Origin Storage https://github.com/mlc-ai/web-llm/pull/748
• Integrity verification for model artifacts https://github.com/mlc-ai/web-llm/pull/787