LoRA language adaptation pipeline for Chatterbox Nano + training/inference contract findings

Hi!

I wanted to share a small project that may be useful to the Chatterbox Nano team.

I originally started this because I needed a lightweight German TTS model for a local assistant, and Chatterbox Nano was a very good fit in terms of size and runtime requirements.

While trying to adapt Nano to German with LoRA, I ran into an issue where an early training setup produced plausible training losses, but failed badly during real autoregressive inference.

After tracing the Nano inference path, I rebuilt the training setup around a stricter causal contract that more closely matches production inference.

The main changes in the V2 training contract are:

• independent conditioning and target clips from the same speaker
• no target/conditioning self-pairs
• production-compatible causal speech framing
• speech-only supervision
• explicit speech-start / speech-stop handling
• adapter reload validation
• fresh-process inference traces
• ASR-based text-control checks

The final German adaptation uses LoRA on the T3 transformer projections (c_attn, c_proj, c_fc) and produces an adapter of roughly 9 MB.

Runtime speaker conditioning is still preserved. In testing, I also noticed that accent characteristics can transfer from the reference recording, so a native German reference works best for German output.

The project is not intended as a replacement for any official training pipeline. I mainly wanted to share the training/inference mismatch I encountered and the approach that ended up working for me, in case any of it is useful upstream.

Training pipeline: https://github.com/Pseud0naut/chatterbox-nano-language-training

German LoRA: https://huggingface.co/Pseudonaut/chatterbox-nano-de-lora

The training contract is documented here: https://github.com/Pseud0naut/chatterbox-nano-language-training/blob/main/docs/training_inference_contract.md

Thanks for making Chatterbox Nano open source. It was a fun model to experiment with.