Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
D

Diffusion-Speech-Tokenizer

> 编程语言
Open source

This repository contains a series of works on diffusion-based speech tokenizers, including the official implementation of the paper: "TaDiCodec: Text-aware Diff

198 stars0 likes1 views
WebsiteGitHub

About

This repository contains a series of works on diffusion-based speech tokenizers, including the official implementation of the paper: "TaDiCodec: Text-aware Diff



Overview

This repository is designed to provide comprehensive implementations for our series of diffusion-based speech tokenizer research works. Currently, it primarily features TaDiCodec, with plans to include additional in-progress works in the future. Specifically, the repository includes:

  • A simple PyTorch implementation of the TaDiCodec tokenizer
  • Token-based zero-shot TTS models based on TaDiCodec:
  • Autoregressive based TTS models
  • Masked diffusion (a.k.a. Masked Genrative Model (MGM) based TTS models
  • ️ Training scripts for tokenizer and TTS models
  • Hugging Face and ModelScope (to be updated) for easy access to pre-trained models

Short Intro on TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling:

We introduce the Text-aware Diffusion Transformer Speech Codec (TaDiCodec), a novel approach to speech tokenization that employs end-to-end optimization for quantization and reconstruction through a diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression. TaDiCodec achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS).


News & Updates

  • ** [2025-09-19]** TaDiCodec is accepted by NeurIPS 2025!
  • ** [2025-08-25]** We release the offical implementation of TaDiCodec and the TTS models based on TaDiCodec.
  • ** [2025-08-25]** TaDiCodec paper released! Check out our arXiv preprint
  • ** [2025-08-25]** Added auto-download functionality from Hugging Face for all models!

Development Roadmap & TODO List

Core TaDiCodec Implementation

  • ️ Repository Structure Setup
  • Documentation Framework
  • TaDiCodec Model Architecture
    • NAR Llama-style transformers for encoder and decoder architectures
    • text-aware flow matching (diffusion) decoder
    • vocoder for mel2wav
  • ⚡ Inference Pipeline
    • Basic inference pipeline
    • Auto-download from Hugging Face
    • Add auto-ASR for text input

Training Infrastructure

  • ️ TaDiCodec Training Scripts
  • Dataset and Dataloader

Text-to-Speech Models

  • Autoregressive Models
    • Model architecture
    • Pre-training models loading and inference
    • Training scripts
  • Masked Diffusion Models
    • Model architecture
    • Pre-training models loading and inference
    • Training scripts

Evaluation

  • Add evaluation scripts

Future Works

  • Diffusion-based Speech Tokenizer without text conditioning

Pre-trained Models

Model Zoo - Ready to Use!

Download our pre-trained models for instant inference

TaDiCodec

Model Hugging Face Status
** TaDiCodec** ✅
** TaDiCodec-old**

Note: TaDiCodec-old is the old version of TaDiCodec, the TaDiCodec-TTS-AR-Phi-3.5-4B is based on TaDiCodec-old.

TTS Models

Model Type LLM Hugging Face Status
** TaDiCodec-TTS-AR-Qwen2.5-0.5B** AR Qwen2.5-0.5B-Instruct ✅
** TaDiCodec-TTS-AR-Qwen2.5-3B** AR Qwen2.5-3B-Instruct ✅
** TaDiCodec-TTS-AR-Phi-3.5-4B** AR Phi-3.5-mini-instruct
** TaDiCodec-TTS-MGM** MGM - ✅
  • ModelScope will be updated soon.

Quick Model Usage

…

Quick Start

Installation

Conda Linux

Select one of the 2 pytorch lines depending on your hardware

bash
# Clone the repository
git clone https://github.com/HeCheng0625/Diffusion-Speech-Tokenizer.git
cd Diffusion-Speech-Tokenizer

# Install dependencies
conda create -n tadicodec python=3.10
conda activate tadicodec
pip install setuptools wheel psutil packaging ninja numpy hf_xet

# pytorch
# CUDA
pip install torch==2.8.0 torchaudio --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu128
# OR CPU only
pip install torch==2.8.0 torchaudio

pip install flash_attn==2.7.4.post1
pip install -r requirements.txt

Conda Windows

This assumes you are using powershell Select one of the 2 pytorch lines depending on your hardware Select one of the 2 flash_attn sections on if you want to use a pre-build whl or to compile your own

…

UV Linux

Select one of the 2 pytorch lines depending on your hardware

bash
# Clone the repository
git clone https://github.com/HeCheng0625/Diffusion-Speech-Tokenizer.git
cd Diffusion-Speech-Tokenizer

# Install python and dependencies
uv python install 3.10
uv venv --python 3.10
uv pip install setuptools wheel psutil packaging ninja numpy hf_xet

# pytorch
# CUDA
pip install torch==2.8.0 torchaudio --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu128
# CPU only
pip install torch==2.8.0 torchaudio

pip install flash_attn==2.7.4.post1
pip install -r requirements.txt

UV Windows

This assumes you are using powershell Select one of the 2 pytorch lines depending on your hardware Select one of the 2 flash_attn sections on if you want to use a pre-build whl or to compile your own

…

Auto-Download from Hugging Face

All models support automatic download from Hugging Face! Simply use the Hugging Face model ID instead of local paths:

python
# Models will be automatically downloaded on first use
from models.tts.tadicodec.inference_tadicodec import TaDiCodecPipline
from models.tts.llm_tts.inference_llm_tts import TTSInferencePipeline

# Auto-download TaDiCodec
tokenizer = TaDiCodecPipline.from_pretrained("amphion/TaDiCodec")

# Auto-download TTS pipeline (downloads both TaDiCodec and LLM)
pipeline = TTSInferencePipeline.from_pretrained(
    tadicodec_path="amphion/TaDiCodec",
    llm_path="amphion/TaDiCodec-TTS-AR-Qwen2.5-0.5B"
)

Note: Models are cached locally after first download for faster subsequent use.

Basic Usage

Please refer to the use_examples folder for more detailed usage examples.

bash
cd Diffusion-Speech-Tokenizer

# conda linux and windows
conda activate tadicodec
# UV linux
./venv/Scripts/activate.sh
# UV powershell
.\.venv\Scripts\activate.ps1

# download models
python test_auto_download.py

cd use_examples
python test_auto_download.py
python test_llm_tts.py
python test_mgm_tts.py
python test_rec.py

Speech Tokenization and Reconstruction

…

Zero-shot TTS with TaDiCodec

…

Performance

TaDiCodec Performance Metrics

TTS Model Performance Metrics


️ Training Examples

Training TaDiCodec

  • To be updated

Training TTS Models

  • To be updated

Citation

If you find this repository useful, please cite our paper:

TaDiCodec:

bibtex
@article{tadicodec2025,
  title={TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling},
  author={Yuancheng Wang, Dekun Chen, Xueyao Zhang, Junan Zhang, Jiaqi Li, Zhizheng Wu},
  journal={arXiv preprint},
  year={2025},
  url={https://arxiv.org/abs/2508.16790}
}

Amphion:

bibtex
@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}

MaskGCT:

bibtex
@inproceedings{wang2024maskgct,
  author={Wang, Yuancheng and Zhan, Haoyue and Liu, Liwei and Zeng, Ruihong and Guo, Haotian and Zheng, Jiachen and Zhang, Qiang and Zhang, Xueyao and Zhang, Shunsi and Wu, Zhizheng},
  title={MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

License

TaDiCodec is licensed under the Apache 2.0.

Acknowledgments

  • MGM-based TTS is built upon MaskGCT.

  • Vocos vocoder is built upon Vocos.

  • NAR Llama-style transformers is built upon transformers.

  • (Binary Spherical Quantization) BSQ is built upon vector-quantize-pytorch and bsq-vit.

  • Training codebase is built upon Amphion and accelerate.


Building the future of speech synthesis, one token at a time


Stay Updated: Watch this repository for the latest developments in diffusion-based speech tokenization!

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 18, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言