自 SpeechBrain 1.1.0 以来,训练效率下降

Describe the bug

Context We are performing a custom ASR training based on the CommonVoice/CTC recipe. For this test, the dataset is composed of 25k records for Train, 3k records for Val, 3,2k records for Test (dynamic_batch_length 80). Our complete dataset is composed of 220k records. We are training on 3 x A100 40Gb. Problem With Speechbrain 1.0.2

• using 1 GPU, each epoch last around 2:30, and val 1:20:

    100%|██████████| 523/523 [02:35<00:00,  3.37it/s, train_loss=2.72]
    100%|██████████| 417/417 [01:20<00:00,  5.17it/s]

• using 3 GPUs, each epoch last also 1:10, and val 1:15s (usually 2.5 time faster as expected):

    100%|██████████| 175/175 [01:10<00:00,  2.49it/s, train_loss=3.36]
    100%|██████████| 417/417 [01:14<00:00,  5.62it/s]

With Speechbrain 1.1.0

• using 1 GPU, each epoch last around 3:45, and val 0:40:

    100%|██████████| 523/523 [03:44<00:00,  2.33it/s, train_loss=2.72]
    100%|██████████| 417/417 [00:38<00:00, 10.80it/s]

• using 3 GPUs, each epoch last also 3:45, and val 0:40 (no change!):

    100%|██████████| 175/175 [03:47<00:00,  1.30s/it, train_loss=3.37]
    100%|██████████| 417/417 [00:36<00:00, 11.45it/s]

Conclusion Although the validation step appears more efficient, single-GPU training performance has decreased by 50%, and multi-GPU training has become inefficient. This issue prevents us from upgrading to SpeechBrain 1.1.0, as our training times are three times longer compared to SpeechBrain 1.0.2 with the same dataset, recipe, hyperparameters, and torch/transformers versions.

Expected behaviour

Equal or better training performance

To Reproduce

Training command line (for multi-GPU) CUDA_VISIBLE_DEVICES=0,1,2 torchrun --nproc_per_node=3 --master_port=25000 /data/sb/wav2vec2/train_with_wav2vec.py /data/sb/wav2vec2/hyperparams_train_w2v2.yaml --wav2vec2_folder=/data/wav2vec2_checkpoints/ --seed=1234 --skip_prep=True --precision=bf16 --wav2vec2_hub=microsoft/wavlm-large --language=en --max_batch_length_train=80.0 --max_batch_length_val=20.0 --number_of_epochs=2 --test_beam_search={kenlm_model_path: "/data/lm/lm_train.arpa"} --token_type=bpe --avoid_if_longer_than=10.0 --data_folder=/data/dataset_test --train_csv=/data/dataset_test/dataset_train.csv --valid_csv=/data/dataset_test/dataset_val.csv --test_csv=/data/dataset_test/dataset_test.csv --output_neurons=60 --output_folder=/data/output --find_unused_parameters

hyperparams_train file:

# ################################
# Model: wav2vec2 + DNN + CTC
# Augmentation: SpecAugment
# Authors: Titouan Parcollet 2021
# ################################
# Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1234
__set_seed:
…