Model Runner V2 Remaining TODOs

Remaining gaps relative to Model Runner V1, replaces this doc. Might not be Exhaustive.

See also https://github.com/vllm-project/vllm/issues/41286 which is tracking the incremental effort to make MRV2 default for more models.

Core

• Sequence Parallelism @WoosukKwon - https://github.com/vllm-project/vllm/pull/46789
• Reasoning Budget - https://github.com/vllm-project/vllm/pull/46727
• Expert Indices - https://github.com/vllm-project/vllm/pull/50721
• Prompt embeds - https://github.com/vllm-project/vllm/pull/42963
• Fast prefill for gemma-style kv sharing cases - https://github.com/vllm-project/vllm/pull/56145
• Non-pinned-mem compatibility - https://github.com/vllm-project/vllm/pull/56908
• Custom Logits Processors - https://github.com/vllm-project/vllm/pull/56497
• STOCK_TORCH_COMPILE mode - https://github.com/vllm-project/vllm/pull/59079

Multi-modal

• NGramPerReq for OCR - see https://github.com/vllm-project/vllm/pull/46564#issuecomment-4824395098 @sawsa307
  • https://github.com/vllm-project/vllm/pull/57540
• ViT full CG - https://github.com/vllm-project/vllm/pull/49852
• E/P/D disaggregation

Spec Decoding

• Dynamic Spec Decoding - https://github.com/vllm-project/vllm/pull/45953
• NGram - https://github.com/vllm-project/vllm/pull/40704
• Draft Model - https://github.com/vllm-project/vllm/pull/43091 / https://github.com/vllm-project/vllm/pull/55990
  • TLI (universal draft) - TBD
• Suffix decoding - TBD
• p-eagle TBD - https://github.com/vllm-project/vllm/pull/44816
• Custom/plugin - @njhill
• Decide on default rejection sampling approach

Pooling models

• Encoder-only attention https://github.com/vllm-project/vllm/pull/49331
• Full compatibility

Distributed

• Elastic EP - https://github.com/vllm-project/vllm/pull/53934
• Dual Batch Overlap
  • https://github.com/vllm-project/vllm/pull/50945
  • https://github.com/vllm-project/vllm/pull/51700

Some new MRV2-Only features/fixes

• https://github.com/vllm-project/vllm/pull/46781
• https://github.com/vllm-project/vllm/pull/46995
• https://github.com/vllm-project/vllm/pull/46994 / https://github.com/vllm-project/vllm/pull/50514