Full fine-tuning on ~4K hours (1.87M samples) is taking 10 days to reach step 0 — how to speed up the full-finetuning process?
Author: mukherjeesougata-erosCreated Sep 28, 2026Updated Sep 30, 2026
I'm running full fine-tuning of VoxCPM2 on a large corpus:
~4,000 hours of audio
1,871,561 training samples
73,535 validation samples
This dataset was assembled from several sources. Some sources' metadata provide per-file duration information, and others don't. When I include a duration field in the JSON objects for the audio files whose sources provide it, while leaving that field absent for the rest, the following error appears:
(VoxCPM) sougata.mukherjee@ai-core-team-3:/devwork/Sougata/TTS/VoxCPM2/VoxCPM$ python scripts/train_voxcpm_finetune.py --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml
Detected architecture: voxcpm2 -> VoxCPM2Model
/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
WeightNorm.apply(module, name, dim)
Loading AudioVAE from pytorch: /devwork/Sougata/TTS/VoxCPM2/model/PT/audiovae.pth
Running on device: cuda, dtype: bfloat16
Loading model from safetensors: /devwork/Sougata/TTS/VoxCPM2/model/PT/model.safetensors
Generating train split: 163081 examples [00:00, 455258.78 examples/s]
Traceback (most recent call last):
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/builder.py", line 1871, in _prepare_split_single
writer.write_table(table)
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/arrow_writer.py", line 643, in write_table
pa_table = table_cast(pa_table, self._schema)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/table.py", line 2293, in table_cast
return cast_table_to_schema(table, schema)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/table.py", line 2241, in cast_table_to_schema
raise CastError(
datasets.table.CastError: Couldn't cast
audio: string
text: string
ref_audio: string
dataset_id: int64
duration: double
to
{'audio': Value(dtype='string', id=None), 'text': Value(dtype='string', id=None), 'dataset_id': Value(dtype='int64', id=None), 'ref_audio': Value(dtype='string', id=None)}
because column names don't match
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/devwork/Sougata/TTS/VoxCPM2/VoxCPM/scripts/train_voxcpm_finetune.py", line 837, in <module>
train(**yaml_args)
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/argbind/argbind.py", line 159, in cmd_func
return func(*cmd_args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/Sougata/TTS/VoxCPM2/VoxCPM/scripts/train_voxcpm_finetune.py", line 111, in train
train_ds, val_ds = load_audio_text_datasets(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/argbind/argbind.py", line 159, in cmd_func
return func(*cmd_args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/Sougata/TTS/VoxCPM2/VoxCPM/src/voxcpm/training/data.py", line 34, in load_audio_text_datasets
dataset_dict: DatasetDict = load_dataset("json", data_files=data_files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/load.py", line 2084, in load_dataset
builder_instance.download_and_prepare(
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/builder.py", line 925, in download_and_prepare
self._download_and_prepare(
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/builder.py", line 1001, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/builder.py", line 1742, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/devwork/MiniConda/miniconda3/envs/VoxCPM/lib/python3.11/site-packages/datasets/builder.py", line 1873, in _prepare_split_single
raise DatasetGenerationCastError.from_cast_error(
datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset
All the data files must have the same columns, but at some point there are 1 new columns ({'duration'})
This happened while the json dataset builder was generating data using
/workspace/Sougata/VoxCPM/data/train.jsonl
Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)Dropping the duration field completely it takes roughly 10 days before the first training update (step 0) begins. Is there a way to speed up the process?
Source: OpenBMB/VoxCPM