ThinkSound

ThinkSound

English | 简体中文 | 繁體中文 | Español | Français | 日本語

       

If you find this project useful,
a star ⭐ on GitHub would be greatly appreciated!

--- ## Repository layout This **ThinkSound** GitHub repository hosts two related projects on separate branches: | Branch | Project | Documentation | |--------|---------|----------------| | **`master`** | **ThinkSound** (NeurIPS 2025) — unified Any2Audio generation with CoT-guided flow matching | This file: **`README.md`** | | **`prismaudio`** | **PrismAudio** — follow-up work (ICLR 2026) on video-to-audio with multi-dimensional CoT-RL | **`README.md`** on the [`prismaudio`](https://github.com/liuhuadai/ThinkSound/tree/prismaudio) branch | For **ThinkSound**, use branch **`master`** (this README). For **PrismAudio**, check out **`prismaudio`** and follow **`README.md`** there. --- **ThinkSound** is a unified Any2Audio generation framework with flow matching guided by Chain-of-Thought (CoT) reasoning. PyTorch implementation for multimodal audio generation and editing: generate or edit audio from video, text, and audio, powered by step-by-step reasoning from Multimodal Large Language Models (MLLMs). --- ## News - **2026.03.24**   **PrismAudio** is released in the same repo on branch [`prismaudio`](https://github.com/liuhuadai/ThinkSound/tree/prismaudio) — see **`README.md`** there for setup and models. - **2026.01.26**   PrismAudio accepted to **ICLR 2026 Main Conference** (code/docs on `prismaudio`). - **2025.11.25**   [Online PrismAudio Demo](http://prismaudio-project.github.io/) is live. - **2025.11.25**   [PrismAudio paper](https://arxiv.org/pdf/2511.18833) on arXiv — multi-dimensional CoT-RL for video-to-audio. - **2025.09.19**   **ThinkSound** accepted to the **NeurIPS 2025 Main Conference**! - **2025.09.01**   Our AudioCoT dataset is now open-sourced and available on [Hugging Face](https://huggingface.co/datasets/liuhuadai/AudioCoT)! - **2025.07.17**   Finetuning enabled: training and finetuning code is now publicly available, along with clear usage instructions to help you customize and extend ThinkSound with your own data. - **2025.07.15**   Simplified installation and usability: dependencies on PyPI for easy cross-platform setup; Windows `.bat` scripts automate environment creation and script running. - **2025.07.08**    Major update: model lightweighted and optimized memory and GPU usage, now supports high-throughput audio generation at scale! - **2025.07.01**   Online demo on [Hugging Face Spaces](https://huggingface.co/spaces/FunAudioLLM/ThinkSound) and [ModelScope](https://modelscope.cn/studios/iic/ThinkSound) for interactive experience! - **2025.07.01**   Released inference scripts and web interface; - **2025.06**   [ThinkSound paper](https://arxiv.org/pdf/2506.21448) released on arXiv! - **2025.06**   [Online Demo](http://thinksound-project.github.io/) is live - try it now! --- --- ## Features - **Any2Audio**: Generate audio from arbitrary modalities — video, text, audio, or their combinations. - **Video-to-Audio SOTA**: Achieves state-of-the-art results on multiple V2A benchmarks. - **CoT-Driven Reasoning**: Chain-of-Thought reasoning for compositional and controllable audio generation via MLLMs. - **Interactive Object-centric Editing**: Refine or edit specific sound events by clicking on visual objects or using text instructions. - **Unified Framework**: One foundation model supports generation, editing, and interactive workflow. --- ## ✨ Method Overview ThinkSound decomposes audio generation and editing into three interactive stages, all guided by MLLM-based Chain-of-Thought (CoT) reasoning: 1. **Foley Generation:** Generate foundational, semantically and temporally aligned soundscapes from video. 2. **Object-Centric Refinement:** Refine or add sounds for user-specified objects via clicks or regions in the video. 3. **Targeted Audio Editing:** Modify generated audio using high-level natural language instructions. --- ## ⚡ Quick Start **Environment Preparation:** ``` … ``` > ✅ **Windows Tip:** > Windows users can simply run `setup_windows.bat` (or double-click it) to automatically create the conda environment, install all dependencies (including FFmpeg), and download the pretrained model — no manual setup required. > Make sure `conda` and `git` are installed and available in your system PATH before running the script. ### ▶️ Run the Demo #### **Linux/macOS** ```bash chmod +x scripts/demo.sh ./scripts/demo.sh <CoT description> [use-half] ``` #### **Windows** You can use the provided `.bat` script instead: ```bash .\scripts\demo.bat <path-to-your-demo-video> <title> <CoT description> [use-half] ``` **Note:** * `<path-to-your-demo-video>`: The path to a single video * `[use-half]` (optional): Add use-half at the end to enable half precision feature extraction. --- ### Batch Inference #### **Linux/macOS** ```bash chmod +x scripts/eval_batch.sh ./scripts/eval_batch.sh <video_path> <csv_path> <save_path (optional)> [use-half] ``` #### **Windows** Use the equivalent `.bat` script: ```bash .\scripts\eval_batch.bat <video_path> <csv_path> <save_path (optional)> [use-half] ``` **Note:** * `<video_path>`: Path to the root directory containing all .mp4 videos to be processed (all videos must be of equal duration). * `<csv_path>`: A CSV file with text prompts for each video (see `demo_test.csv` for format). * `<save_path>` (optional): Where to save generated audio. Defaults to `results/features`. * `[use-half]` (optional): Add use-half at the end to enable half precision feature extraction. --- ### Web Interface Usage For an interactive experience, launch the Gradio web interface: ```bash python app.py ``` ## ️ Train the Model See [`Training.md`](docs/Training.md) --- ## License This project is released under the Apache 2.0 License. > **Note:** > The code, models, and dataset are **for research and educational purposes only**. > **Commercial use is NOT permitted.** > For commercial licensing, please contact the authors. ** Third-Party Components** * **Stable Audio Open VAE** (by Stability AI): This repository includes a fine-tuned VAE from [Stable Audio Open](https://huggingface.co/stabilityai/stable-audio-open-1.0/), licensed under the [Stability AI Community License](./third_party/LICENSE_StabilityAI.md). **Commercial use and redistribution require prior permission from Stability AI.** * **All other code and models** are released under the Apache License 2.0. --- ## Acknowledgements Many thanks to: * **stable-audio-tools** (by Stability AI): For providing an easy-to-use framework for audio generation, as well as the VAE module and weights. * **MMAudio**: For the implementation of the MM-DiT backbone in the audio domain. --- ## Citation If you find our project useful in your research or work, please cite our paper: ``` … ``` --- ## Contact ✨ Feel free to [open an issue](https://github.com/liuhuadai/ThinkSound/issues) or contact us via email ([[email protected]](mailto:[email protected])) if you have any questions or suggestions!