#369·Qwen3-TTS

25Hz 分词器: qkv_attention_manual 从不对填充的关键字进行掩码处理 (bool masked_fill),短注意力窗口会产生错误的输出

作者: raghav-rathi创建于 2026年9月13日更新于 2026年9月13日

**What happens:** `attn_mask` is a `torch.bool` tensor, and `masked_fill` keeps the dtype of `self`, so `-3.4e38` is cast to `True`. After that line the mask is all `True`, and `attn_scores + attn_mask` adds `1.0` to every score, which softmax ignores. Padded keys are never masked. Because `k_padded` / `v_padded` are zero-initialised, every padded slot receives weight `exp(1)/Σ` in the softmax, which shrinks the weights of the real keys. No error or warning is raised. The cast behaves the same on every PyTorch release I checked (1.5 to 2.14; the CPU and CUDA kernels both do `value.to<bool>()`).

内容来源: QwenLM/Qwen3-TTS