SpeechCraft 数据集的官方仓库,这是一个具有自然语言描述的大规模表达性双语语音数据集。
This is the official repository of the ACM Multimedia 2024 paper "SpeechCraft: A Fine-Grained Expressive Speech Dataset with Natural Language Description".
For details of the pipeline and dataset, please refer to our Paper and Demo Page
[2024-09-26]: Structured metadata (pitch, energy, speed, age, gender, emotion tone, emphasis, topic/category, and transcript) has been made available to facilitate further enhancements and augmentations of the dataset.
[2024-12-20]: Code and checkpoint of the Annotation pipeline are released.
| Language | Speech Corpus | #Duration | #Clips |
|---|---|---|---|
| ZH | Zhvoice | 799.68h | 1,020,427 |
| ZH | AISHELL-3 | 63.70h | 63,011 |
| EN | GigaSpeech-M | 739.91h | 670,070 |
| EN | LibriTTS-R | 548.88h | 352,265 |
| Description | Instruction | Labels | |
|---|---|---|---|
| ZH | download | download | download |
| EN | download | download | download |
--gender: Male, Female --age: Child, Teenager, Youth adult, Middle-aged, Elderly--pitch: low, normal, high--speed: slow, normal, fast--volume: low, normal, high--emotion (English): Fearful, Happy, Disgusted, Sad, Surprised, Angry, Neutral--emphasis: Non-label words--transcript: Non-label sentence--LLM Prompt:…--年龄:儿童,少年,青年,中年,老年--性别:男,女--语速:快,中,慢--音高:高,中,低--音量:高,中,低--重读:无标签,字词--语气:无标签,自然语句--文本:无标签,自然语句--LLM Prompt:请参照以下转换案例,使用中文自然语言描述一个人按照给定风格属性,如音高、音量、年龄、性别、语调,来说文本中的话。注意,仅描述说话风格,不需要描述肢体动作或心理状态,不要重复输入的内容。
*示例输入-示例输出*
现在尝试处理以下句子,根据示例直接输出转换后的句子,不要遗漏任何标签。Since we do not own the copyright of the original audio files, for researchers and educators who wish to use the audio files for non-commercial research and/or educational purposes, we can provide access to our regenerated version under certain conditions and terms. To apply for the AISHELL-3 and LibriTTS-R with fine-grained keyword emphasis, please fill out the EULA form at Emphasis-SpeechCraft-EULA.pdf and send the scanned form to [email protected]. Once approved, you will be supplied with a download link. ([2024-09-26]: With metadata updated!)
Please first refer to some emphasis examples provided here. We are actively working on improving methods for large-scale fine-grained data construction that align with human perception.
| Language | Speech Corpus | #Duration | #Clips |
|---|---|---|---|
| ZH | AISHELL-3-stress | 50.59h | 63,258 |
| EN | LibriTTS-R-stress | 148.78h | 75,654 |
Download models for speech style recognition.
Models from :
llama_base_model = "baichuan-inc/Baichuan2-13B-Base"
gender_model_path = "alefiury/wav2vec2-large-xlsr-53-gender-recognition-librispeech"
age_model_path = "audeering/wav2vec2-large-robust-24-ft-age-gender"
asr_path = "openai/whisper-medium" / "openai/whisper-large-v3"Model from funasr (for English emotion classification):
emotion_model = "iic/emotion2vec_base_finetuned"Prepare SECap from here (for Chinese emotion captioning).
Create conda environment
conda env create -f ./requirements.yaml
mv ./AutomaticPipeline/models/SECap/model2.py $your_SECap_dirDownload the lora ckpt from here as ./llama-ft/finetuned-llama/ for description rewriting.
Remember to change the path of LLM ckpt at "base_model_name_or_path" in ./llama-ft/finetuned-llama/adapter_config.json.
Get the scp file with raw scores for the audio corpus.
cd ./AutomaticPipeline
python AutoPipeline.pyGet the json file with classified result prepared for the description rewriting.
python Clustering.pycd ../llama-ft
python llama_infer.pyPlease cite our paper if you find this work useful:
@inproceedings{jin2024speechcraft,
title={Speechcraft: A fine-grained expressive speech dataset with natural language description},
author={Jin, Zeyu and Jia, Jia and Wang, Qixin and Li, Kehan and Zhou, Shuoyi and Zhou, Songtao and Qin, Xiaoyu and Wu, Zhiyong},
booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
pages={1255--1264},
year={2024}
}暂无开放 Issues,或尚未同步最近议题。