Step-Audio2

# Step-Audio 2 ## News!! * Sep 15, 2025: We release [Step-Audio 2 mini Think](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Think) and its corresponding [examples](examples-think.py). * Sep 3, 2025: We release our [vLLM backend](https://github.com/stepfun-ai/vllm/tree/step-audio2-mini) and corresponding [examples](examples-vllm.py). * Aug 29, 2025: We are pleased to open-source [Step-Audio 2 mini](https://huggingface.co/stepfun-ai/Step-Audio-2-mini), [Step-Audio 2 mini Base](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Base) and their corresponding inference [examples](examples.py). [Technical report](https://arxiv.org/pdf/2507.16632) is also updated. * Jul 24, 2025: We release [demonstration videos for Step-Audio 2](https://www.stepfun.com/docs/step-audio2). * Jul 23, 2025: We release our benchmark for paralinguistic information understanding, [StepEval-Audio-Paralinguistic](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-Paralinguistic). * Jul 23, 2025: We release our benchmark for tool calling, [StepEval-Audio-Toolcall](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-Toolcall). * Jul 23, 2025: We release the technical report of [Step-Audio 2](https://arxiv.org/pdf/2507.16632). WeChat Developer Group ## Introduction Step-Audio 2 is an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. - **Advanced Speech and Audio Understanding**: Promising performance in ASR and audio understanding by comprehending and reasoning semantic information, para-linguistic and non-vocal information. - **Intelligent Speech Conversation**: Achieving natural and intelligent interactions that are contextually appropriate for various conversational scenarios and paralinguistic information. - **Emotional Reasoning**: Analyzing user's paralinguistic information such as age and emotion, leading to more accurate and intelligent interpretation of the audio context. - **Tool Calling and Multimodal RAG**: By leveraging tool calling and RAG to access real-world knowledge (both textual and acoustic), Step-Audio 2 can generate responses with fewer hallucinations for diverse scenarios, while also having the ability to switch timbres based on retrieved speech. - **State-of-the-Art Performance**: Achieving state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. (See [Evaluation](#evaluation) and [Technical Report](https://arxiv.org/pdf/2507.16632)). + **Open-source**: [Step-Audio 2 mini](https://huggingface.co/stepfun-ai/Step-Audio-2-mini), [Step-Audio 2 mini Base](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Base) and [Step-Audio 2 mini Think](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Think) are released under [Apache 2.0](LICENSE) license. ## Model Download | Models | Hugging Face | ModelScope | |-------|-------|-------| | Step-Audio 2 mini | [stepfun-ai/Step-Audio-2-mini](https://huggingface.co/stepfun-ai/Step-Audio-2-mini) | [stepfun-ai/Step-Audio-2-mini](https://www.modelscope.cn/models/stepfun-ai/Step-Audio-2-mini) | | Step-Audio 2 mini Base | [stepfun-ai/Step-Audio-2-mini-Base](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Base) | [stepfun-ai/Step-Audio-2-mini-Base](https://www.modelscope.cn/models/stepfun-ai/Step-Audio-2-mini-Base) | | Step-Audio 2 mini Think | [stepfun-ai/Step-Audio-2-mini-Think](https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Think) | [stepfun-ai/Step-Audio-2-mini-Think](https://www.modelscope.cn/models/stepfun-ai/Step-Audio-2-mini-Think) | ## Model Usage ### Dependencies and Installation - Python >= 3.10 - [PyTorch >= 2.3-cu121](https://pytorch.org/) - [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) ```bash conda create -n stepaudio2 python=3.10 conda activate stepaudio2 pip install transformers==4.49.0 torchaudio librosa onnxruntime s3tokenizer diffusers hyperpyyaml git clone https://github.com/stepfun-ai/Step-Audio2.git cd Step-Audio2 git lfs install git clone https://huggingface.co/stepfun-ai/Step-Audio-2-mini # git clone https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Base ``` ### vLLM docker image We highly recommend using our vLLM backend for faster and streaming inference, also deploying across multiple GPUs. ``` … ``` ### Inference Scripts ```bash python examples.py # python examples-base.py # python examples-vllm.py # python examples-think.py ``` ### Local web demonstration ```bash pip install gradio python web_demo.py # python web_demo_vllm.py ``` ## Online demonstration ### StepFun realtime console - Both Step-Audio 2 and Step-Audio 2 mini are available in our [StepFun realtime console](https://realtime-console.stepfun.com/) with web search tool enabled. - You will need an API key from the [StepFun Open Platform](https://platform.stepfun.com/). ### StepFun AI Assistant - Step-Audio 2 is also available in our StepFun AI Assistant mobile App with both web and audio search tools enabled. - Please scan the following QR code to download it from your app store then tap the phone icon in the top-right corner. ## WeChat group You can scan the following QR code to join our WeChat group for communication and discussion. ## Evaluation ### Automatic speech recognition CER for Chinese, Cantonese and Japanese and WER for Arabian and English. N/A indicates that the language is not supported.

Category	Test set	Doubao LLM ASR	GPT-4o Transcribe	Kimi-Audio	Qwen-Omni	Step-Audio 2	Step-Audio 2 mini
English	Common Voice	9.20	9.30	7.83	8.33	5.95	6.76
FLEURS English	7.22	2.71	4.47	5.05	3.03	3.05
LibriSpeech clean	2.92	1.75	1.49	2.93	1.17	1.33
LibriSpeech other	5.32	4.23	2.91	5.07	2.42	2.86
Average	6.17	4.50	4.18	5.35	3.14	3.50
Chinese	AISHELL	0.98	3.52	0.64	1.17	0.63	0.78
AISHELL-2	3.10	4.26	2.67	2.40	2.10	2.16
FLEURS Chinese	2.92	2.62	2.91	7.01	2.68	2.53
KeSpeech phase1	6.48	26.80	5.11	6.45	3.63	3.97
WenetSpeech meeting	4.90	31.40	5.21	6.61	4.75	4.87
WenetSpeech net	4.46	15.71	5.93	5.24	4.67	4.82
Average	3.81	14.05	3.75	4.81	3.08	3.19
Multilingual	FLEURS Arabian	N/A	11.72	N/A	25.13	14.22	16.46
Common Voice yue	9.20	11.10	38.90	7.89	7.90	8.32
FLEURS Japanese	N/A	3.27	N/A	10.49	3.18	4.67
In-house	Anhui accent	8.83	50.55	22.17	18.73	10.61	11.65
Guangdong accent	4.99	7.83	3.76	4.03	3.81	4.44
Guangxi accent	3.37	7.09	4.29	3.35	4.11	3.51
Shanxi accent	20.26	55.03	34.71	25.95	12.44	15.60