DSpark Qwen3 版本的模型默认忽略 `partial_rotary_factor`, 导致检查点的配置与训练权重不匹配(导致 vLLM 服务出现问题)
作者: arthurgao2003创建于 2026年7月22日更新于 2026年7月23日
- L34-L40 — the custom
apply_rotary_pos_embappliescos/sinacross the full head dim, with no rotary/pass-through split:
def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
...
q_embed = (q * cos[..., -q_len:, :]) + (rotate_half(q) * sin[..., -q_len:, :])
k_embed = (k * cos) + (rotate_half(k) * sin)
- L239 — `self.rotary_emb = Qwen3RotaryEmbedding(config)`. With transformers 5.13.1 and a config declaring `partial_rotary_factor: 0.25`, `head_dim: 128`, this still yields **128-dim** cos/sin (`inv_freq` len 64), so the declared factor never takes effect at training time either way.内容来源: deepseek-ai/DeepSpec