Fine-tuned 1.7B voice: short/opening utterances unstable (wrong timbre, occasional gender flip) while long-form is stable

Description

Thanks for Qwen3-TTS — long-form fine-tuning results are genuinely excellent, which is what makes this issue frustrating to isolate.

A LoRA fine-tune of the 1.7B base on ~21 min of clean single-speaker English produces stable, on-target speech for long/multi-clause generations, but short generations and the opening ~1–2 seconds of any generation are unstable: wrong timbre, occasional apparent gender flip, sometimes a whispered or deep-male rendering. The generation often "settles" into the correct voice partway through, so the instability is concentrated at generation onset. Under greedy decoding it's deterministic per input, affecting roughly 15% of short utterances.

I've ruled out my own serving/transcode path by capturing the raw model PCM before any resampling — the instability is present in the raw model output. I've also swept lora_scale (0.1–0.6; 0.3–0.4 best), compared epoch-9 vs epoch-19 checkpoints (earlier was worse, so not overfitting), forced greedy on both the main talker and the code-predictor, and tried leading/trailing text padding (leading made it worse). Adding targeted short-phrase and bridge-phrase training data helped marginally but didn't resolve it.

Core question: is onset/short-utterance instability a known limitation of the current fine-tuning path, and has anyone solved it specifically (as opposed to the end-of-generation acceleration discussed in #39)? Would trailing silence on training clips, or a more long-form-weighted data balance, be expected to help onset stability?

Reproduction

1. LoRA fine-tune Qwen3-TTS-12Hz-1.7B-Base on ~21 min single-speaker English (mix of short phrases, medium "bridge" sentences, and long-form), LR 2e-6, batch_size 2, grad-accum 4, 20 epochs, attn_implementation=sdpa.
2. Merge the epoch-19 adapter at lora_scale 0.4; bake the speaker embedding into the servable model.
3. Serve on vLLM-Omni with greedy decoding (generation_config: temperature 0, top_k 1; code-predictor forced greedy via local patch).
4. Generate a short standalone utterance, e.g. "Great — thanks so much for calling in today.", repeatedly.
5. Observe: long-form generations render correctly and on-target; a subset (~15%) of short generations render with wrong timbre / gender flip / whisper, concentrated in the first ~1–2 s, sometimes recovering mid-utterance. Behavior is deterministic per input under greedy (identical output across repeats of the same text).

Note: the same text generated as part of a longer sentence renders correctly — only the short/standalone or onset case fails.

Logs

Environment Information

• Model: Qwen3-TTS-12Hz-1.7B-Base (LoRA fine-tune, single-speaker, lora_rank 16)
• Training: LR 2e-6, batch_size 2, gradient_accumulation 4, 20 epochs, attn_implementation=sdpa
• Training data: ~21 min, single speaker, English, 24 kHz, clean
• Serving: vLLM-Omni, greedy decoding (main talker temperature 0 / top_k 1; code-predictor patched to greedy locally)=
• GPU: NVIDIA A10 (24 GB)
• OS: Ubuntu 24.04
• Driver: 595.71.05
• CUDA: 13.2 (driver) / 13.0 (torch built against)
• Python: 3.12
• torch: 2.11.0
• torchaudio: 2.11.0
• transformers: 5.13.0
• vllm: 0.24.0
• vllm-omni: 0.24.0

Known Issue

• The issue hasn't been already addressed in Documentation, Issues, and Discussions.