Fine-tuning Qwen3-TTS-12Hz-1.7B-Base for Bengali: Foreign/Hindi Accent Bias and Lack of Naturalness

Description

Fine-tuning Qwen3-TTS-12Hz-1.7B-Base for Bengali: Foreign/Hindi Accent Bias and Lack of Naturalness

Problem

I fine-tuned Qwen3-TTS-12Hz-1.7B-Base on a Bengali / Bangla dataset of around 7 hours from a single speaker. I am planning to train on 150+ hours of multi-speaker Bengali data later.

For inference, I used the custom_voice type model functions. The fine-tuning process itself runs successfully after applying a few patches, and the trained model is able to generate Bengali speech.

However, the generated speech does not sound like native Bengali. It sounds more like Bengali spoken with a Foreign/Hindi accent or Hindi-influenced prosody. The pronunciation is somewhat Bengali-like, but the accent, rhythm, intonation, and speaking style are noticeably Hindi-influenced and not natural for Bengali.

Dataset

• Language: Bengali / Bangla
• Dataset size: around 7 hours
• Number of samples: 8k+
• Speaker setup: single Bengali female speaker
• Text format: Bengali letters and punctuation
• Audio code fields: prepared using the Qwen3 TTS tokenizer

We also tried IPA / romanized phoneme input. Interestingly, the Hindi accent issue was reduced with phoneme-style input, but the generated speech became more robotic and less natural compared to using Bengali script. Because of that, we are keeping the phoneme approach aside for now and may revisit it later with a larger dataset.

Training Setup

• Base model: Qwen3-TTS-12Hz-1.7B-Base
• Speaker name: bengali_female
• GPU: RTX 3090
• Batch size: 2
• Learning rate: 2e-6
• Higher learning rate used for:
  • text embeddings
  • codec embeddings
  • some upper / adaptation layers
• Goal of higher LR on selected layers: stronger Bengali accent, tone, and prosody adaptation
• Epochs: 10
• Warmup: 3%
• Scheduler: Sequential Linear
• Validation ratio: 0.01
• Inference language: Auto

Observed Behavior

• Training loss decreases normally at the beginning.
• After around 2–3 epochs, the loss becomes mostly stuck.
• After that point, there is not much improvement in speech quality.
• In some cases, quality starts to degrade with further training.
• The model generates Bengali speech, but there are still some pronunciation issues.
• The main problem is that the accent does not sound like native Bengali.
• Prosody, rhythm, and speaking style sound closer to Hindi than Bengali.
• The generated speech lacks naturalness compared to native Bengali speech.

Questions

1. Does Qwen3-TTS-12Hz-1.7B-Base have a strong prior bias toward Hindi or other Indic-language prosody?
   As far as I know, the base TTS model may not have seen Hindi speech directly, but the underlying LLM may have Hindi text bias. Since Hindi and Bengali scripts / phonetics have some overlap, could the model be mapping Bengali text through Hindi-like phonetic or prosodic patterns?

2. What is the recommended way to achieve stronger Bengali accent adaptation and more natural Bengali prosody?
   Are there specific layers, learning-rate strategies, prompt formats, speaker conditioning methods, or data-preparation changes that help with native accent adaptation for a new language?

3. Should we add an explicit Bengali language tag during fine-tuning and inference?
   For example, would adding something like <lang:bn> or another Bengali language identifier help the model separate Bengali from Hindi / other Indic-language priors?

4. Are there recommended fine-tuning parameters or guideline for unsupported or low-resource languages?

5. Is there a recommended Bengali G2P / phoneme / romanization pipeline for Qwen3 TTS fine-tuning?
   Raw Bengali script gives more natural speech but Hindi-influenced accent. IPA / romanized phoneme input reduces the Hindi accent but sounds more robotic. Is there a better Bengali phoneme representation, G2P pipeline, or hybrid text + phoneme approach recommended for new-language fine-tuning?

Additional Notes

The main goal is to fine-tune Qwen3 TTS for native-sounding Bengali speech with natural Bengali accent, rhythm, and intonation. Any guidance on dataset preparation, Bengali text normalization, phoneme design, language tagging, or fine-tuning strategy would be very helpful.

Thanks in advance!