MOSS-TTSD 是一种用于表达性多语音合成的对话生成模型,具有长上下文建模、灵活的语音控制和高效的语音合成。
MOSS-TTSD is the long-form dialogue specialist within our open-source [MOSS‑TTS Family](https://github.com/OpenMOSS/MOSS-TTS). While foundational models typically prioritize high-fidelity single-speaker synthesis, MOSS-TTSD is architected to bridge the gap between isolated audio samples and cohesive, continuous human interaction. The model represents a paradigm shift from "text-to-speech" to "script-to-conversation." By prioritizing the flow and emotional nuances of multi-party engagement, MOSS-TTSD transforms static dialogue scripts into dynamic, expressive oral performances. It is designed to serve as a robust backbone for creators and developers who require a seamless transition between distinct speaker personas without sacrificing narrative continuity. Whether it is capturing the spontaneous energy of a live talk show or the structured complexity of a multilingual drama, MOSS-TTSD provides the stability and expressive depth necessary for professional-grade, long-form content creation in an open-source framework. ## Highlights - **From Monologue to Dialogue**: Unlike traditional TTS which optimizes for reading, MOSS-TTSD masters the rhythm of conversation. It supports 1 to 5 speakers with flexible control, handling natural turn-taking, overlapping speech patterns, and distinct persona maintenance. - **Extreme Long-Context Modeling**: moving beyond short-sentence generation, the model is architected for stability over long durations, supporting up to 60 minutes of coherent audio in a single session with consistent identity. - **Diverse Scenario Adaptation**: fine-tuned for high-variability scenarios including conversational media (AI Podcasts), dynamic commentary (Sports/Esports), and entertainment (Audiobooks, Dubbing, and Crosstalk). - **Multilingual & Zero-Shot Capabilities**: features state-of-the-art zero-shot voice cloning requiring only short reference audio, with robust cross-lingual performance across major languages including Chinese, English, Japanese, and European languages. ## News - **[2026-03-18]** We support efficient end-to-end SGLang inference for MOSS-TTSD v1.0. - **[2026-03-06]** We added end-to-end SGLang inference support for MOSS-TTSD v0.7. For detailed instructions, please see the [legacy v0.7 docs](./legacy/v0.7/README.md). - **[2026-02-10]** MOSS-TTSD v1.0 is released! MOSS-TTSD v1.0 is officially released! This milestone version redefines long-form synthesis with 60-minute single-session context and support for multi-party interactions. It significantly expands multilingual capabilities and diverse usage scenarios. Earlier updates - **[2025-11-01]** MOSS-TTSD v0.7 is released! v0.7 significantly improves audio quality, voice cloning capability, and stability, adds support for 32 kHz high‑quality output, greatly extends single‑pass generation length (960s→1700s). - **[2025-09-09]** We supported SGLang inference engine to accelerate model inference by up to **16x**. - **[2025-08-25]** We released the 32khz version of XY-Tokenizer. - **[2025-08-12]** We add support for streaming inference in MOSS-TTSD v0.5. - **[2025-07-29]** We provide the SiliconFlow API interface and usage examples for MOSS-TTSD v0.5. - **[2025-07-16]** We open-source the fine-tuning code for MOSS-TTSD v0.5, supporting full-parameter fine-tuning, LoRA fine-tuning, and multi-node training. - **[2025-07-04]** MOSS-TTSD v0.5 is released! v0.5 has enhanced the accuracy of timbre switching, voice cloning capability, and model stability. - **[2025-06-20]** MOSS-TTSD v0 is released! Moreover, we provide a podcast generation pipeline named Podever, which can automatically convert PDF, URL, or long text files into high-quality podcasts. **Note:** For MOSS-TTSD v0.7 (including end-to-end SGLang inference), please refer to the [legacy v0.7 docs](./legacy/v0.7/README.md) for detailed instructions. ## Supported Languages MOSS-TTSD currently supports **20 languages**: | Language | Code | Flag | Language | Code | Flag | Language | Code | Flag | |---|---|---|---|---|---|---|---|---| | Chinese | zh | | English | en | | German | de | | | Spanish | es | | French | fr | | Japanese | ja | | | Italian | it | | Hebrew | he | | Korean | ko | | | Russian | ru | | Persian (Farsi) | fa | | Arabic | ar | | | Polish | pl | | Portuguese | pt | | Czech | cs | | | Danish | da | | Swedish | sv | | Hungarian | hu | | | Greek | el | | Turkish | tr | | | | | ## Installation To run MOSS-TTSD, you need to install the required dependencies. You can use pip and conda to set up your environment. ### Using conda ```bash conda create -n moss_ttsd python=3.12 -y && conda activate moss_ttsd pip install -r requirements.txt pip install flash-attn ``` ## Usage ### Quick Start MOSS-TTSD uses a **continuation** workflow: provide reference audio for each speaker, their transcripts as a prefix, and the dialogue text to generate. The model continues in each speaker's identity. ``` … ``` ### Batch Inference You can use the provided inference script for batch inference. The script automatically uses all visible GPUs. You can control GPU visibility via `export CUDA_VISIBLE_DEVICES=`. ```bash python inference.py \ --model_path OpenMOSS-Team/MOSS-TTSD-v1.0 \ --codec_model_path OpenMOSS-Team/MOSS-Audio-Tokenizer \ --input_jsonl /path/to/input.jsonl \ --save_dir outputs \ --mode voice_clone_and_continuation \ --batch_size 1 \ --text_normalize ``` Parameters: - `--model_path`: Path or HuggingFace model ID for MOSS-TTSD. - `--codec_model_path`: Path or HuggingFace model ID for MOSS-Audio-Tokenizer. - `--input_jsonl`: Path to the input JSONL file containing dialogue scripts and speaker prompts. - `--save_dir`: Directory where the generated audio files will be saved. - `--mode`: Inference mode. Choices: `generation`, `continuation`, `voice_clone`, `voice_clone_and_continuation`. We recommend using `voice_clone_and_continuation` for the best voice cloning experience. - `--batch_size`: Number of samples per batch (default: `1`). - `--max_new_tokens`: Maximum number of new tokens to generate. Controls total generated audio length (1s ≈ 12.5 tokens). - `--temperature`: Sampling temperature (default: `1.1`). - `--top_p`: Top-p sampling threshold (default: `0.9`). - `--top_k`: Top-k sampling threshold (default: `50`). - `--repetition_penalty`: Repetition penalty (default: `1.1`). - `--text_normalize`: Normalize input text (**recommended to always enable**). - `--sample_rate_normalize`: Resample prompt audios to the lowest sample rate before encoding (**recommended when using 2 or more speakers**). #### JSONL Input Format The input JSONL file should contain one JSON object per line. MOSS-TTSD supports 1 to 5 speakers per dialogue. Use `[S1]`–`[S5]` tags in the `text` field and provide corresponding `prompt_audio_speakerN` / `prompt_text_speakerN` pairs for each speaker: ```json { "base_path": "/path/to/audio/files", "text": "[S1]Speaker 1 dialogue[S2]Speaker 2 dialogue[S3]...[S4]...[S5]...", "prompt_audio_speaker1": "path/to/speaker1_audio.wav", "prompt_text_speaker1": "Reference text for speaker 1 voice cloning", "prompt_audio_speaker2": "path/to/speaker2_audio.wav", "prompt_text_speaker2": "Reference text for speaker 2 voice cloning", "...": "...", "prompt_audio_speaker5": "path/to/speaker5_audio.wav", "prompt_text_speaker5": "Reference text for speaker 5 voice cloning" } ``` ### Accelerate Inference with SGLang MOSS-TTSD v1.0 supports running the fused MOSS-TTSD and MOSS-Audio-Tokenizer model with the deeply extended [SGLang](https://github.com/OpenMOSS/sglang) from OpenMOSS, enabling efficient inference for audio generation. #### Environment Setup First, clone the SGLang branch compatible with MOSS-TTSD v1.0. ```bash git clone https://github.com/OpenMOSS/sglang -b moss-ttsd-v1.0-with-cat ``` ##### Using venv ```bash python -m venv moss_ttsd_sglang source moss_ttsd_sglang/bin/activate pip install ./sglang/python[all] ``` ##### Using conda ```bash conda create -n moss_ttsd_sglang python=3.12 conda activate moss_ttsd_sglang pip install ./sglang/python[all] ``` #### End-to-End Inference Service ##### Start the inference server Before starting the service, first download [MOSS-TTSD-v1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) and [MOSS-Audio-Tokenizer](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer). ```bash git clone https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0 git clone https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer ``` Or: ```bash hf download OpenMOSS-Team/MOSS-TTSD-v1.0 --local-dir ./MOSS-TTSD-v1.0 hf download OpenMOSS-Team/MOSS-Audio-Tokenizer --local-dir ./MOSS-Audio-Tokenizer ``` After the download is complete, run the following command to fuse MOSS-TTSD v1.0 and MOSS-Audio-Tokenizer into a single-directory model that can be loaded by SGLang. After fusion, the model uses `voice_clone_and_continuation` inference mode by default: ```bash python scripts/fuse_moss_tts_delay_with_codec.py \ --model-path \ --codec-model-path \ --save-path ``` Then start the inference server with: ```bash sglang serve \ --model-path \ --delay-pattern \ --trust-remote-code \ --port 30000 --host 0.0.0.0 ``` > The first service startup may take longer due to compilation. Once you see `The server is fired up and ready to roll!`, the service is ready. The first request after startup may still trigger a lengthy compilation, which is expected behavior, so please be patient. > **Tip:** The end-to-end inference service may cause some VRAM fragmentation during runtime. If GPU memory is tight, we recommend using `--mem-fraction-static` when starting SGLang to reserve enough space for intermediate tensors. ##### Send a generation request The service API is compatible with the standard multimodal text-generation interface. The `text` field in the returned JSON contains the base64-encoded WAV audio. The repository currently provides a minimal request example script: ```bash python scripts/request_sglang_generation.py ``` This script will: - send requests to `http://localhost:30000/generate` by default - use `asset/reference_02_s1.wav` and `asset/reference_02_s2.wav` in the repository as reference audio - save the returned audio to `outputs/output.wav` If you need to change the reference audio, input text, sampling parameters, or server URL, you can directly edit the corresponding constants in `scripts/request_sglang_generation.py`. ## Evaluation ### Objective Evaluation(TTSD-eval) We introduce a robust evaluation framework leveraging MMS-FA for word-level alignment and utterance segmentation and wespeaker for embedding extraction to derive Speaker Attribution Accuracy (ACC) and Speaker Similarity (SIM). Please refer to [TTSD-eval](https://github.com/OpenMOSS/TTSD-eval) for the code and data.
暂无开放 Issues，或尚未同步最近议题。