AuK

> 编程语言

Free

AuK: An Open-Source Foundational Model for Speech Generation and Editing

818 stars0 likes1 views

Website GitHub

About

AuK: An Open-Source Foundational Model for Speech Generation and Editing

Try our model on the HuggingFace Space · ModelScope Space!

News

[2026/09/16] Reduced encoder memory by ~7.5 GiB, making local inference more accessible on consumer GPUs. See PR #19.
[2026/09/13] ️ AuK now officially supports MLX inference on Apple Silicon (available on the feat/mlx-apple-silicon branch) and CPU offload for CUDA inference.
[2026/09/13] AuK serves as the end-to-end baseline for the Single Model Track of the ICASSP 2027 Audio Editing Challenge.
[2026/09/09] Thanks to SGLang-Omni for Day 0 support for AuK and AuK-Flash! Check out the SGLang-Omni cookbook to get started.
[2026/09/09] We open-source AuK. Code and model weights are publicly available. Try it on the Demo Space or the ModelScope Space!

Demo

English

https://github.com/user-attachments/assets/d07332fc-5f69-4f16-9d00-a7443cc19e6a

中文

https://github.com/user-attachments/assets/c532bbdb-e6ce-4434-a9a5-16f29a8d4135

News
Introduction
Performance
Model Architecture
Supported Tasks
Quick Start
Fine-tuning
Contributing
Acknowledgements
Citation
License

Introduction

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface. AuK has two variants:

Model	Description	Weight
AuK	Base model for high-quality generation	Hugging Face · ModelScope
AuK-Flash	Distilled model for fast 4-step inference	Hugging Face · ModelScope

Performance

Model Architecture

Supported Tasks

AuK exposes every task through the same natural-language instruction interface. The table below groups the supported tasks by category, with a short description and a link to its section in the Cookbook, where you'll find instruction templates plus CLI and Python examples.

  Category
  Task
  Description
  Cookbook






  Speech Generation
  Zero-shot TTS
  Speak the target text in the voice of the reference audio.
  Zero-shot TTS




  Instruct TTS
  Generate speech from a voice description alone — no reference audio.
  Instruct TTS




  Content Editing
  Speech Content Editing
  Rewrite what is said — replace, insert, or remove text.
  Speech Content Editing




  Lyric Editing
  Rewrite lyrics in a singing recording while preserving the melody and voice.
  Lyric Editing




  Acoustic Editing
  Pitch Editing
  Raise or lower the pitch by semitones.
  Pitch Editing




  Speed Editing
  Adjust the speaking rate; output length scales with the speed factor.
  Speed Editing




  Volume Editing
  Raise or lower the volume by decibels.
  Volume Editing




  Paralinguistic Editing
  Emotion
  Change the emotion while preserving content and voice.
  Emotion




  Timbre
  Change the timbre to a description while keeping the content unchanged.
  Timbre




  De-accent
  Remove a regional accent while preserving the speaker's voice and content.
  De-accent




  Nonverbal Editing
  Remove or add nonverbal sounds such as breaths, laughs, or coughs.
  Nonverbal Editing




  Whisper Conversion
  Convert between normal speech and whisper while preserving speaker and content.
  Whisper Conversion




  Enhancement &amp; Separation
  Speech Enhancement
  Denoise, dereverberate, or restore natural, clear speech.
  Speech Enhancement




  Speech Separation
  Keep one speaker by talking order and remove the others.
  Speech Separation




  Music Separation
  Extract the singing voice from a mix, or keep all human voices.
  Music Separation




  Target Speaker Extraction
  Keep the target speaker identified by what they say.
  Target Speaker Extraction

Quick Start

Installation

Clone the repository, then choose either uv or Conda to create an isolated Python 3.10 environment.

bash

git clone https://github.com/Tencent-Hunyuan/AuK
cd AuK

uv

bash

# Create and activate a project-local environment.
uv venv --python 3.10
source .venv/bin/activate

# Choose one installation target:
# Core inference and CLI only
uv pip install -e .

# Core inference + Gradio + Prompt Enhancer + ASR
uv pip install -e ".[gradio]"

# Core inference + ComfyUI nodes + Prompt Enhancer + ASR
uv pip install -e ".[comfyui]"

# Core inference + fine-tuning
uv pip install -e ".[train]"

# Everything
uv pip install -e ".[gradio,train]"

Conda

bash

conda create -n auk python=3.10 -y
conda activate auk

# Choose one installation target:
# Core inference and CLI only
pip install -e .

# Core inference + Gradio + Prompt Enhancer + ASR
pip install -e ".[gradio]"

# Core inference + ComfyUI nodes + Prompt Enhancer + ASR
pip install -e ".[comfyui]"

# Core inference + fine-tuning
pip install -e ".[train]"

# Everything
pip install -e ".[gradio,train]"

The default installation includes PyTorch, TorchAudio, and TorchVision. If your platform requires a specific CPU or CUDA build, install a matching PyTorch stack for your platform first, then install AuK with either command above.

Download the weights

** HuggingFace**

bash

pip install -U "huggingface_hub[cli]"

# AuK-Base
hf download tencent/AuK --local-dir ./ckpts/AuK

# AuK-Flash (4-step distilled) 
hf download tencent/AuK-Flash --local-dir ./ckpts/AuK-Flash

# MLLM Encoder
hf download Qwen/Qwen2.5-Omni-3B  --local-dir ./ckpts/Qwen2.5-Omni-3B

** ModelScope**

bash

pip install -U modelscope

# AuK-Base
modelscope download --model Tencent-Hunyuan/AuK --local_dir ./ckpts/AuK

# AuK-Flash (4-step distilled)
modelscope download --model Tencent-Hunyuan/AuK-Flash  --local_dir ./ckpts/AuK-Flash

# MLLM Encoder
modelscope download --model Qwen/Qwen2.5-Omni-3B --local_dir ./ckpts/Qwen2.5-Omni-3B

The expected directory structure is:

ckpts/
├── AuK/
├── AuK-Flash/          # optional
└── Qwen2.5-Omni-3B/

The model checkpoint contains the diffusion transformer and layer-fusion weights. The MLLM encoder and VAE are loaded from separate files at runtime, so missing text_encoder.* keys during checkpoint loading are expected.

Command-line inference

[!TIP] When starting from a free-form request, we recommend using Prompt Enhancer. It prepares the model instruction, target duration, and any required audio preprocessing, then prints a ready-to-run one-line auk-infer command.

Prompt Enhancer

PE uses the same OpenAI-compatible LLM environment variables described above. Load them from .env, then run:

bash

set -a
source ./.env
set +a

python src/auk/infer/pe.py \
  --audio assets/demo-input-audio/whisper/wh-w2n-zh-input.wav \
  --instruction "Convert this whisper into normal speech while preserving the speaker and content." \
  --asr auto

PE prints the generated command:

bash

auk-infer \
  --audio assets/after_pe/wh-w2n-zh-input.wav \
  --instruction 'Convert this whispered speech into normal speech.' \
  --output assets/after_pe/wh-w2n-zh-input.output.wav \
  --gen_seconds 8.58

The terminal also shows the detected task and target duration, and writes a compact JSON manifest under assets/after_pe/.

CLI Examples

All tasks use the same message-based interface. An --instruction is always required, while source or reference --audio is optional depending on the task. The examples below are just a taste — for the full instruction templates and per-task CLI examples, see the Cookbook.

Content editing

Rewrite what is said by describing the change in the instruction:

bash

auk-infer \
    --audio assets/demo-input-audio/content-edit/content.wav \
    --instruction "Replace 'but accepting what we cannot have' with 'and living well with dreams unmet'." \
    --output out_content_edit.wav \
    --gen_seconds 7.0

Speech enhancement / separation

Denoising, enhancement, and source separation are the same message-driven call — just say what to keep or remove:

bash

auk-infer \
    --audio assets/demo-input-audio/vocal-extraction/vocal-1-input.wav \
    --instruction "请将这段音频恢复成纯净人声版本：保留原本所有说话人，并去除其中的噪声和混响，输出等长的纯净语音。" \
    --output out_denoise.wav

Zero-shot TTS

Write the target text into the instruction, then hint the duration with --gen_text (+ optional --ref_text, the reference transcript) or an explicit --gen_seconds:

bash

auk-infer \
    --audio assets/demo-input-audio/zero-shot-tts/ref.wav \
    --instruction "Say the following with the same voice: 'Ladies and gentlemen, it's an honor to have the opportunity to address such a distinguished audience'" \
    --output out_tts.wav \
    --gen_seconds 6.0

To use AuK-Flash, set:

bash

--ckpt ckpts/AuK-Flash/auk_flash.safetensors

AuK-Flash use 4 fixed time steps and set CFG=0.

Lower VRAM usage (CUDA only)

Add --cpu_offload to auk-infer or auk-gradio, set cpu_offload=True when constructing AukInfer, or enable cpu_offload in ComfyUI's AuK Model Loader. It is disabled by default and works with both AuK variants.

Peak VRAM measured on one NVIDIA A800-SXM4-80GB with bf16 inference:

Model	Input	CPU offload disabled	CPU offload enabled	VRAM saved
AuK	Text only, 1.5 s output	24.78 GiB	16.75 GiB	8.03 GiB (32.4%)
AuK	5 s reference audio	25.00 GiB	16.98 GiB	8.02 GiB (32.1%)
AuK-Flash	Text only, 1.5 s output	24.77 GiB	16.75 GiB	8.02 GiB (32.4%)
AuK-Flash	5 s reference audio	24.97 GiB	16.98 GiB	7.99 GiB (32.0%)

The table reports torch.cuda.max_memory_allocated; actual usage depends on input length, dtype, hardware, and software versions.

Interactive Gradio demo

Install the Gradio dependencies with pip install -e ".[gradio]" (or the equivalent uv pip install command above) before starting the demo.

Prompt Enhancer requires an OpenAI-compatible LLM:

LLM: Tencent Cloud TokenHub
Optional cloud ASR: Tencent Cloud Recording File Recognition

Export the credentials before starting Gradio:

bash

# Required when Prompt Enhancer is enabled
export LLM_API_KEY="your-llm-api-key"
export LLM_BASE_URL="https://tokenhub.tencentmaas.com/v1"
export LLM_MODEL_NAME="hy3"

# Optional c