FireRedTTS2

Overview

FireRedTTS‑2 is a long-form streaming TTS system for multi-speaker dialogue generation, delivering stable, natural speech with reliable speaker switching and context-aware prosody.

Highlight

• Long Conversational Speech Generation: It currently supports 3 minutes dialogues with 4 speakers and can be easily scaled to longer conversations with more speakers by extending training corpus.
• Multilingual Support: It supports multiple languages including English, Chinese, Japanese, Korean, French, German, and Russian. Support zero-shot voice cloning for cross-lingual and code-switching scenarios.
• Ultra-Low Latency: Building on the new 12.5Hz streaming speech tokenizer, we employ a dual-transformer architecture that operates on a text–speech interleaved sequence, enabling flexible sentence-bysentence generation and reducing first-packet latency，Specifically, on an L20 GPU, our first-packet latency as low as 140ms while maintaining high-quality audio output.
• Strong Stability：Our model achieves high similarity and low WER/CER in both monologue and dialogue tests.
• Random Timbre Generation:Useful for creating ASR/speech interaction data.

Demo Examples

Random Timbre Generation & Multilingual Support

Zero-Shot Podcast Generation

Speaker-Specific Finetuned Podcast Generation

⚠️ Speaker voices: hosts "肥杰" and "惠子" from the podcast "肥话连篇". Use without authorization is forbidden.

⚠️ 声音来源：播客 "肥话连篇" 主播 "肥杰" 和 "惠子"，未经授权不能使用。

For more examples, see demo page.

News

• [2025/10/26] We have released comprehensive fine-tuning code and tutorials (based on the LJSpeech dataset). You can easily adapt this foundation to fine-tune models for multilingual or even conversational datasets. With conversational data, you can achieve podcast generation results comparable to our demo showcase.
• [2025/10/11] We now support streaming dialogue generation.
• [2025/09/28] Supports bf16 inference, reducing VRAM usage from 14GB to 9GB and enabling consumer-grade GPU deployment.
• [2025/09/12] We have added a UI tool to the dialogue generation.
• [2025/09/08] We release the pre-trained checkpoints and inference code.
• [2025/09/02] We release the technical report and demo page

Roadmap

• 2025/09

  • Release the pre-trained checkpoints and inference code.
  • Add web UI tool.

• 2025/10

  • Release a base model with enhanced multilingual support.
  • Provide fine-tuning code & tutorial for specific dialogue/multilingual data.
  • End-to-end text-to-podcast pipeline.

Install & Model Download

Clone and install

• Clone the repo

  bash复制

  git clone https://github.com/FireRedTeam/FireRedTTS2.git
  cd FireRedTTS2

• Create env:

  Setup environment with Conda:

  bash复制

  conda create --name fireredtts2 python==3.11
  conda activate fireredtts2
  
  # Step 1. PyTorch Installation (if required)
  pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
  
  # Step 2. Install Dependencies
  pip install -e .
  pip install -r requirements.txt

  or use Docker:

  bash复制

  # Build docker image
  docker build -t fireredtts2:v1.0 docker
  
  # Launch docker container
  docker run -v=${PWD}:/workspace/FireRedTTS2 --ipc=host --net=host --gpus=all -it fireredtts2:v1.0 bash

• Model download

  bash复制

  git lfs install
  git clone https://huggingface.co/FireRedTeam/FireRedTTS2 pretrained_models/FireRedTTS2

Basic Usage

Dialogue Generation with Web UI

Generate dialogue through an easy-to-use web interface that supports both voice cloning and randomized voices.

python gradio_demo.py --pretrained-dir "./pretrained_models/FireRedTTS2"

Dialogue Generation

…

Dialogue Generation (Sreaming)

NOTE: Each audio chunk is 0.08 seconds, except the first (a little shorter) and last (a little longer).

…

Monologue Generation

…

Acknowledgements

• We thank Moshi and Sesame CSM for their novel dual-transformer approach. Additionally, we adapted Sesame CSM's structure and core inference code.

• We referred to Qwen2.5-1.5B text tokenizer solution.

• We referred to Xcodec2 Vocos-based acoustic decoder.

⚠️ Usage Disclaimer ❗️❗️❗️❗️❗️❗️

• The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended solely for academic research purposes.
• DO NOT use this model for ANY illegal activities❗️❗️❗️❗️❗️❗️
• The developers assume no liability for any misuse of this model.
• If you identify any instances of abuse, misuse, or fraudulent activities related to this project, please report them to our team immediately.

Star History