Data-Speech Data-Speech是一套用于标记语音数据集的实用脚本. 它的目的是提供一个简单,干净的代码库 一个..
Data-Speech Data-Speech是一套用于标记语音数据集的实用脚本. 它的目的是提供一个简单,干净的代码库 一个..
Data-Speech is a suite of utility scripts designed to tag speech datasets.
Its aim is to provide a simple, clean codebase for applying audio transformations (or annotations) that may be requested as part of the development of speech-based AI models, such as text-to-speech engines.
Its primary use is to reproduce the annotation method from Dan Lyth and Simon King's research paper Natural language guidance of high-fidelity text-to-speech with synthetic annotations, that labels various speaker characteristics with natural language descriptions.
Applying these tools allows us to prepare and release tagged versions of LibriTTS-R, and of the English version of MLS.
This repository is designed to accompany the Parler-TTS library, which contains the inference and training code for Parler-TTS, a new family of high-quality text-to-speech models.
You first need to clone this repository before installing requirements.
git clone [email protected]:huggingface/dataspeech.git
cd dataspeech
pip install -r requirements.txtIn the following examples, we'll load 30 hours of audio data from the Jenny TTS dataset, a high-quality mono-speaker TTS dataset, from an Irish female speaker named Jenny.
The aim here is to create an annotated version of Jenny TTS, in order to fine-tune the Parler-TTS v1 checkpoint on this dataset.
Thanks to a script similar to what's described in the FAQ, we've uploaded the dataset to the HuggingFace hub, under the name reach-vb/jenny_tts_dataset.
Feel free to follow the link above to listen to some samples of the Jenny TTS dataset thanks to the hub viewer.
[!IMPORTANT] Refer to the section Annotating datasets from scratch for more detailed explanations of what's going on under-the-hood.
We'll:
We'll use main.py to get the following continuous variables:
- Speaking rate (nb_phonemes / utterance_length)
- Scale-Invariant Signal-to-Distortion Ratio (SI-SDR)
- Reverberation
- Speech monotony
python main.py "reach-vb/jenny_tts_dataset" \
--configuration "default" \
--text_column_name "transcription" \
--audio_column_name "audio" \
--cpu_num_workers 8 \
--rename_column \
--repo_id "jenny-tts-tags-v1" \
--apply_squim_quality_estimationNote that the script will be faster if you have GPUs at your disposal. It will automatically scale-up to every GPUs available in your environnement.
The resulting dataset will be pushed to the HuggingFace hub under your HuggingFace handle. Mine was pushed to ylacombe/jenny-tts-tags-v1.
Since the ultimate goal here is to fine-tune the Parler-TTS v1 checkpoint on the Jenny dataset, we want to stay consistent with the text bins of the datasets on which the latter model was trained.
This is easy to do thanks to the following command:
python ./scripts/metadata_to_text.py \
"ylacombe/jenny-tts-tags-v1" \
--repo_id "jenny-tts-tags-v1" \
--configuration "default" \
--cpu_num_workers "8" \
--path_to_bin_edges "./examples/tags_to_annotations/v02_bin_edges.json" \
--path_to_text_bins "./examples/tags_to_annotations/v02_text_bins.json" \
--avoid_pitch_computation \
--apply_squim_quality_estimationThanks to v02_bin_edges.json, we don't need to recompute bins from scratch and the above script takes a few seconds.
The resulting dataset will be pushed to the HuggingFace hub under your HuggingFace handle. Mine was push to ylacombe/jenny-tts-tags-v1.
You can notice that text bins such as slightly slowly, very monotone have been added to the samples.
Now that we have text bins associated to the Jenny dataset, the next step is to create natural language descriptions out of the few created features.
Here, we decided to create prompts that use the name Jenny, prompts that'll look like the following:
In a very expressive voice, Jenny pronounces her words incredibly slowly. There's some background noise in this room with a bit of echo.'
This step generally demands more resources and times and should use one or many GPUs.
run_prompt_creation_jenny.sh indicates how to run it on the Jenny dataset:
python ./scripts/run_prompt_creation.py \
--speaker_name "Jenny" \
--is_single_speaker \
--is_new_speaker_prompt \
--dataset_name "ylacombe/jenny-tts-tags-v1" \
--dataset_config_name "default" \
--model_name_or_path "mistralai/Mistral-7B-Instruct-v0.2" \
--per_device_eval_batch_size 128 \
--attn_implementation "sdpa" \
--output_dir "./tmp_jenny" \
--load_in_4bit \
--push_to_hub \
--hub_dataset_id "jenny-tts-tagged-v1" \
--preprocessing_num_workers 24 \
--dataloader_num_workers 24As usual, we precise the dataset name and configuration we want to annotate. model_name_or_path should point to a transformers model for prompt annotation. You can find a list of such models here. Here, we used a version of Mistral's 7B model.
[!NOTE] If you want to use this on a multi-speaker dataset, you'll have to adapt the logic of the script. First, you need to remove the
--is_single_speakerand--speaker_name "Jenny"flags.Then, there's two cases:
- In case you want to associate names to some speakers, you need to pass the speaker id column name, and a JSON file which maps the speaker ids to these names. For example,
--speaker_id_column "speaker_id" --speaker_ids_to_name_json ./examples/prompt_creation/speaker_ids_to_names.json. Feel free to take a look at speaker_ids_to_names.json to get inspiration.- In case you don't want to associate names to speakers, you don't have to do anything else.
In the following examples, we'll load 1,000 hours of labelled audio data from the LibriTTS-R dataset and add annotations using the dataspeech library. The resulting dataset is complete with discrete annotation tags, as well as a coherent audio description of the spoken audio characteristics.
There are 3 steps to be completed in order to generate annotations:
(nb_phonemes / utterance_length)For the time being, main.py can be used to generate speaking rate, SNR, reverberation, PESQ, SI-SDR and pitch estimation.
To use it, you need a dataset from the datasets library, either locally or on the hub.
python main.py "blabble-io/libritts_r" \
--configuration "dev" \
--output_dir ./tmp_libritts_r_dev/ \
--text_column_name "text_normalized" \
--audio_column_name "audio" \
--cpu_num_workers 8 \
--rename_column \
--apply_squim_quality_estimationHere, we've used 8 processes for operations that don't use GPUs, namely to compute the speaking rate. If GPUs were present in the environnement, the operations that can be computed on GPUs - namely pitch, SNR and reverberation estimation - will use every GPUs available in the environnement.
You can learn more about the arguments you can pass to main.py by passing:
python main.py --helpIn /examples/tagging/run_main_1k.sh, we scaled up the initial command line to the whole dataset. Note that we've used the repo_id argument to push the dataset to the hub, resulting in this dataset.
The dataset viewer gives an idea of what has been done, namely:
utterance_pitch_std: Gives a measure of the standard deviation of pitch in the utterance.utterance_pitch_mean: Gives a measure of average pitch in the utterance.snr: Speech-to-noise ratioc50: Reverberation estimationspeaking_ratephonemes: which was used to compute the speaking ratepesq and si-sdr: which measure intelligibility and a proxy of noise, as indicated hereThe next step is to map the continuous annotations from the previous steps to key-words. To do so, continous annotations are mapped to categorical bins that are then associated to key-words. For example, the speaking rate can be associated to 7 text bins which are: "very slowly", "quite slowly", "slightly slowly", "moderate speed", "slightly fast", "quite fast", "very fast".
scripts/metadata_to_text.py computes bins on aggregated statistics from multiple datasets:
python ./scripts/metadata_to_text.py "ylacombe/libritts-r-text-tags-v3+ylacombe/libritts-r-text-tags-v3" \
--configuration "clean+other" \
--output_dir "./tmp_tts_clean+./tmp_tts_other" \
--cpu_num_workers "8" \
--leading_split_for_bins "train" \
--plot_directory "./plots/" \
--path_to_text_bins "./examples/tags_to_annotations/v02_text_bins.json" \
--apply_squim_quality_estimation \Note how we've been able to pass different datasets with different configurations by separating the relevant arguments with "+".
By passing `--repo_id parler-tts/libritts-r-tag
暂无开放 Issues,或尚未同步最近议题。