百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

Diffusion-Speech-Tokenizer

> 编程语言
开源

此仓库包含了一系列基于扩散的语音标记器的作品,其中包括该论文的官方实现:"TaDiCodec: Text-aware Diff

198 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

此仓库包含了一系列基于扩散的语音标记器的作品,其中包括该论文的官方实现:"TaDiCodec: Text-aware Diff



Overview

This repository is designed to provide comprehensive implementations for our series of diffusion-based speech tokenizer research works. Currently, it primarily features TaDiCodec, with plans to include additional in-progress works in the future. Specifically, the repository includes:

  • A simple PyTorch implementation of the TaDiCodec tokenizer
  • Token-based zero-shot TTS models based on TaDiCodec:
  • Autoregressive based TTS models
  • Masked diffusion (a.k.a. Masked Genrative Model (MGM) based TTS models
  • ️ Training scripts for tokenizer and TTS models
  • Hugging Face and ModelScope (to be updated) for easy access to pre-trained models

Short Intro on TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling:

We introduce the Text-aware Diffusion Transformer Speech Codec (TaDiCodec), a novel approach to speech tokenization that employs end-to-end optimization for quantization and reconstruction through a diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression. TaDiCodec achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS).


News & Updates

  • ** [2025-09-19]** TaDiCodec is accepted by NeurIPS 2025!
  • ** [2025-08-25]** We release the offical implementation of TaDiCodec and the TTS models based on TaDiCodec.
  • ** [2025-08-25]** TaDiCodec paper released! Check out our arXiv preprint
  • ** [2025-08-25]** Added auto-download functionality from Hugging Face for all models!

Development Roadmap & TODO List

Core TaDiCodec Implementation

  • ️ Repository Structure Setup
  • Documentation Framework
  • TaDiCodec Model Architecture
    • NAR Llama-style transformers for encoder and decoder architectures
    • text-aware flow matching (diffusion) decoder
    • vocoder for mel2wav
  • ⚡ Inference Pipeline
    • Basic inference pipeline
    • Auto-download from Hugging Face
    • Add auto-ASR for text input

Training Infrastructure

  • ️ TaDiCodec Training Scripts
  • Dataset and Dataloader

Text-to-Speech Models

  • Autoregressive Models
    • Model architecture
    • Pre-training models loading and inference
    • Training scripts
  • Masked Diffusion Models
    • Model architecture
    • Pre-training models loading and inference
    • Training scripts

Evaluation

  • Add evaluation scripts

Future Works

  • Diffusion-based Speech Tokenizer without text conditioning

Pre-trained Models

Model Zoo - Ready to Use!

Download our pre-trained models for instant inference

TaDiCodec

Model Hugging Face Status
** TaDiCodec** ✅
** TaDiCodec-old**

Note: TaDiCodec-old is the old version of TaDiCodec, the TaDiCodec-TTS-AR-Phi-3.5-4B is based on TaDiCodec-old.

TTS Models

Model Type LLM Hugging Face Status
** TaDiCodec-TTS-AR-Qwen2.5-0.5B** AR Qwen2.5-0.5B-Instruct ✅
** TaDiCodec-TTS-AR-Qwen2.5-3B** AR Qwen2.5-3B-Instruct ✅
** TaDiCodec-TTS-AR-Phi-3.5-4B** AR Phi-3.5-mini-instruct
** TaDiCodec-TTS-MGM** MGM - ✅
  • ModelScope will be updated soon.

Quick Model Usage

…

Quick Start

Installation

Conda Linux

Select one of the 2 pytorch lines depending on your hardware

bash
# Clone the repository
git clone https://github.com/HeCheng0625/Diffusion-Speech-Tokenizer.git
cd Diffusion-Speech-Tokenizer

# Install dependencies
conda create -n tadicodec python=3.10
conda activate tadicodec
pip install setuptools wheel psutil packaging ninja numpy hf_xet

# pytorch
# CUDA
pip install torch==2.8.0 torchaudio --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu128
# OR CPU only
pip install torch==2.8.0 torchaudio

pip install flash_attn==2.7.4.post1
pip install -r requirements.txt

Conda Windows

This assumes you are using powershell Select one of the 2 pytorch lines depending on your hardware Select one of the 2 flash_attn sections on if you want to use a pre-build whl or to compile your own

…

UV Linux

Select one of the 2 pytorch lines depending on your hardware

bash
# Clone the repository
git clone https://github.com/HeCheng0625/Diffusion-Speech-Tokenizer.git
cd Diffusion-Speech-Tokenizer

# Install python and dependencies
uv python install 3.10
uv venv --python 3.10
uv pip install setuptools wheel psutil packaging ninja numpy hf_xet

# pytorch
# CUDA
pip install torch==2.8.0 torchaudio --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu128
# CPU only
pip install torch==2.8.0 torchaudio

pip install flash_attn==2.7.4.post1
pip install -r requirements.txt

UV Windows

This assumes you are using powershell Select one of the 2 pytorch lines depending on your hardware Select one of the 2 flash_attn sections on if you want to use a pre-build whl or to compile your own

…

Auto-Download from Hugging Face

All models support automatic download from Hugging Face! Simply use the Hugging Face model ID instead of local paths:

python
# Models will be automatically downloaded on first use
from models.tts.tadicodec.inference_tadicodec import TaDiCodecPipline
from models.tts.llm_tts.inference_llm_tts import TTSInferencePipeline

# Auto-download TaDiCodec
tokenizer = TaDiCodecPipline.from_pretrained("amphion/TaDiCodec")

# Auto-download TTS pipeline (downloads both TaDiCodec and LLM)
pipeline = TTSInferencePipeline.from_pretrained(
    tadicodec_path="amphion/TaDiCodec",
    llm_path="amphion/TaDiCodec-TTS-AR-Qwen2.5-0.5B"
)

Note: Models are cached locally after first download for faster subsequent use.

Basic Usage

Please refer to the use_examples folder for more detailed usage examples.

bash
cd Diffusion-Speech-Tokenizer

# conda linux and windows
conda activate tadicodec
# UV linux
./venv/Scripts/activate.sh
# UV powershell
.\.venv\Scripts\activate.ps1

# download models
python test_auto_download.py

cd use_examples
python test_auto_download.py
python test_llm_tts.py
python test_mgm_tts.py
python test_rec.py

Speech Tokenization and Reconstruction

…

Zero-shot TTS with TaDiCodec

…

Performance

TaDiCodec Performance Metrics

TTS Model Performance Metrics


️ Training Examples

Training TaDiCodec

  • To be updated

Training TTS Models

  • To be updated

Citation

If you find this repository useful, please cite our paper:

TaDiCodec:

bibtex
@article{tadicodec2025,
  title={TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling},
  author={Yuancheng Wang, Dekun Chen, Xueyao Zhang, Junan Zhang, Jiaqi Li, Zhizheng Wu},
  journal={arXiv preprint},
  year={2025},
  url={https://arxiv.org/abs/2508.16790}
}

Amphion:

bibtex
@inproceedings{amphion,
    author={Xueyao Zhang and Liumeng Xue and Yicheng Gu and Yuancheng Wang and Jiaqi Li and Haorui He and Chaoren Wang and Ting Song and Xi Chen and Zihao Fang and Haopeng Chen and Junan Zhang and Tze Ying Tang and Lexiao Zou and Mingxuan Wang and Jun Han and Kai Chen and Haizhou Li and Zhizheng Wu},
    title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
    booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
    year={2024}
}

MaskGCT:

bibtex
@inproceedings{wang2024maskgct,
  author={Wang, Yuancheng and Zhan, Haoyue and Liu, Liwei and Zeng, Ruihong and Guo, Haotian and Zheng, Jiachen and Zhang, Qiang and Zhang, Xueyao and Zhang, Shunsi and Wu, Zhizheng},
  title={MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer},
  booktitle    = {{ICLR}},
  publisher    = {OpenReview.net},
  year         = {2025}
}

License

TaDiCodec is licensed under the Apache 2.0.

Acknowledgments

  • MGM-based TTS is built upon MaskGCT.

  • Vocos vocoder is built upon Vocos.

  • NAR Llama-style transformers is built upon transformers.

  • (Binary Spherical Quantization) BSQ is built upon vector-quantize-pytorch and bsq-vit.

  • Training codebase is built upon Amphion and accelerate.


Building the future of speech synthesis, one token at a time


Stay Updated: Watch this repository for the latest developments in diffusion-based speech tokenization!

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言