百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
S

SpeechCraft

> 编程语言
开源

SpeechCraft 数据集的官方仓库,这是一个具有自然语言描述的大规模表达性双语语音数据集。

198 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

SpeechCraft 数据集的官方仓库,这是一个具有自然语言描述的大规模表达性双语语音数据集。

SpeechCraft

This is the official repository of the ACM Multimedia 2024 paper "SpeechCraft: A Fine-Grained Expressive Speech Dataset with Natural Language Description".

For details of the pipeline and dataset, please refer to our Paper and Demo Page

News

[2024-09-26]: Structured metadata (pitch, energy, speed, age, gender, emotion tone, emphasis, topic/category, and transcript) has been made available to facilitate further enhancements and augmentations of the dataset.

[2024-12-20]: Code and checkpoint of the Annotation pipeline are released.

SpeechCraft Dataset

1. Download Speech Corpus

Language Speech Corpus #Duration #Clips
ZH Zhvoice 799.68h 1,020,427
ZH AISHELL-3 63.70h 63,011
EN GigaSpeech-M 739.91h 670,070
EN LibriTTS-R 548.88h 352,265

2. Download Speech Annotation

Description Instruction Labels
ZH download download download
EN download download download

3. Labels and Prompts

EN Version

  • --gender: Male, Female
  • --age: Child, Teenager, Youth adult, Middle-aged, Elderly
  • --pitch: low, normal, high
  • --speed: slow, normal, fast
  • --volume: low, normal, high
  • --emotion (English): Fearful, Happy, Disgusted, Sad, Surprised, Angry, Neutral
  • --emphasis: Non-label words
  • --transcript: Non-label sentence
  • --LLM Prompt:
…

ZH Version

  • --年龄:儿童,少年,青年,中年,老年
  • --性别:男,女
  • --语速:快,中,慢
  • --音高:高,中,低
  • --音量:高,中,低
  • --重读:无标签,字词
  • --语气:无标签,自然语句
  • --文本:无标签,自然语句
  • --LLM Prompt:
请参照以下转换案例,使用中文自然语言描述一个人按照给定风格属性,如音高、音量、年龄、性别、语调,来说文本中的话。注意,仅描述说话风格,不需要描述肢体动作或心理状态,不要重复输入的内容。

*示例输入-示例输出*

现在尝试处理以下句子,根据示例直接输出转换后的句子,不要遗漏任何标签。

4. Request Access to Emphasis Speech Dataset

Since we do not own the copyright of the original audio files, for researchers and educators who wish to use the audio files for non-commercial research and/or educational purposes, we can provide access to our regenerated version under certain conditions and terms. To apply for the AISHELL-3 and LibriTTS-R with fine-grained keyword emphasis, please fill out the EULA form at Emphasis-SpeechCraft-EULA.pdf and send the scanned form to [email protected]. Once approved, you will be supplied with a download link. ([2024-09-26]: With metadata updated!)

Please first refer to some emphasis examples provided here. We are actively working on improving methods for large-scale fine-grained data construction that align with human perception.

Language Speech Corpus #Duration #Clips
ZH AISHELL-3-stress 50.59h 63,258
EN LibriTTS-R-stress 148.78h 75,654

Annotation Pipeline

Step 0 : Installation

  1. Download models for speech style recognition.

    Models from :

    llama_base_model = "baichuan-inc/Baichuan2-13B-Base"
    gender_model_path = "alefiury/wav2vec2-large-xlsr-53-gender-recognition-librispeech"
    age_model_path = "audeering/wav2vec2-large-robust-24-ft-age-gender"
    asr_path = "openai/whisper-medium" / "openai/whisper-large-v3"

    Model from funasr (for English emotion classification):

    emotion_model = "iic/emotion2vec_base_finetuned"

    Prepare SECap from here (for Chinese emotion captioning).

  2. Create conda environment

    conda env create -f ./requirements.yaml
    mv ./AutomaticPipeline/models/SECap/model2.py $your_SECap_dir
  3. Download the lora ckpt from here as ./llama-ft/finetuned-llama/ for description rewriting.

    Remember to change the path of LLM ckpt at "base_model_name_or_path" in ./llama-ft/finetuned-llama/adapter_config.json.

Step 1 : Labeling with the Automatic Annotation Pipeline

  1. Get the scp file with raw scores for the audio corpus.

    bash
    cd ./AutomaticPipeline
    python AutoPipeline.py
  2. Get the json file with classified result prepared for the description rewriting.

    bash
    python Clustering.py

Step 2 : Rewriting with the Finetuned Llama

bash
cd ../llama-ft
python llama_infer.py

Citation

Please cite our paper if you find this work useful:

@inproceedings{jin2024speechcraft,
  title={Speechcraft: A fine-grained expressive speech dataset with natural language description},
  author={Jin, Zeyu and Jia, Jia and Wang, Qixin and Li, Kehan and Zhou, Shuoyi and Zhou, Songtao and Qin, Xiaoyu and Wu, Zhiyong},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={1255--1264},
  year={2024}
}

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言