[RFC]: Extended online quantization roadmap
Motivation.
Continuation of https://github.com/vllm-project/vllm/issues/32029.
Online quantization API is now relatively stable at https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online and documented at https://docs.vllm.ai/en/stable/features/quantization/online/.
This RFC summarizes and motivates proposed features that would be helpful to make online quantization to be more flexible and more widely compatible with models, including pre-quantized ones.
Proposed Change.
1. Support activation quant key override
Implemented in https://github.com/vllm-project/vllm/pull/51808.
This is necessary to dispatch on specific linear/moe backends that can be picked only through the activation_quant_key (see e.g. https://github.com/vllm-project/vllm/blob/c4e969294ecab9ffefb995b734303f13f62b723f/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L441-L445)
2. [done] Targeted online quantization configuration based on user patterns
Implemented in https://github.com/vllm-project/vllm/pull/51285, RFC in https://github.com/vllm-project/vllm/issues/50281.
The goal is simply to extend the UI to support e.g. --quantization-config.targets '{"re:.*o_proj.*":"mxfp8"}', e.g. to target a single layer similar to e.g. https://github.com/vllm-project/vllm/blob/dc9f845ddc54c1df38fdbce5afe03f9fd15813bd/vllm/model_executor/layers/attention/mla_attention.py#L935-L948 (that is currently hard-coded and may be causing accuracy issues if done silently https://github.com/vllm-project/vllm/pull/48051), or e.g. --quantization-config.targets '{"re:.*shared_expert.*":"fp8_per_block"}' to enable fused shared expert.
3. [done] Support online quantization with partially pre-quantized checkpoints
Implemented in https://github.com/vllm-project/vllm/pull/51392.
This is necessary to combine online quantization with already partially quantized checkpoints, which is notoriously the case recently. This is e.g. useful to test the accuracy/speed tradeoff of quantizing certain additional layers, or useful to e.g. release a single model compatible with fused shared expert / unfused shared expert.
This has e.g. been requested in separate PRs:
4. Requantize already-quantized layers to different precision (different weight quant key)
Implemented in https://github.com/vllm-project/vllm/pull/55684
This is similar to https://github.com/sgl-project/sglang/pull/28291 & https://github.com/sgl-project/sglang/pull/29328 on sglang side.
This idea here would be to allow loading through a certain quantization method, dequantize and override the quantization method on request.
This has e.g. been requested in:
5. Adopt online/* API for Humming online quantization, torchao online quantization
Currently, Humming (https://github.com/vllm-project/vllm/pull/34556) and torchao (https://github.com/vllm-project/vllm/pull/32133, https://github.com/vllm-project/vllm/pull/23014) have their own online quantization logic:
In contrary proposed online quantization for AMD/Quark in vLLM that intended to plug similarly in existing quant method was rejected: https://github.com/vllm-project/vllm/issues/31028 - so we ended up having a plugin (and eventually adopting upstream online/* API now).
These torchao/humming logics could be moved to use the online quantization frontend.
cc @jerryzh168 @kylesayrs @jinzhen-lin
Feedback Period.
One month
CC List.
@BowenBao @mgoin @kylesayrs @vkuzo @maeehart @tanpinsiang @fanxingran @access2rohit
Any Other Things.
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
Source: vllm-project/vllm