📝 Originally published (in Japanese) at forge.workstyle.tech.
Automated Pipeline for Generating Training Corpora for TTS Voice Models I've been operating a pipeline that automatically generates training corpora for voice synthesis models.
The process involves having TTS read text, transcribing it with Whisper, comparing the transcription to the script, and saving only the clips that pass the quality check as training material.
It's a straightforward setup.
While running a batch to produce 12 voices, two presenters failed to generate properly.
There were 20 and 18 rejected clips respectively, while other voices had only 0–10 rejections.
Something was clearly wrong.
After digging deeper, I discovered that the trimming process I had written to prevent tail hallucinations wasn't doing anything when hallucinations actually occurred.
I hadn't noticed this issue for months.
Short Sentences Were Being Rejected When I listed the rejected sentences, a pattern emerged.
The base corpus consists of 65 sentences with mora counts ranging from 13 (minimum) to 33 (maximum), with a median of
21.
All the rejected sentences were on the shorter end.
I reproduced the issue locally using the presenter's caption ("clear and expressive speech with emphasis on important points while addressing the audience") and processed it through Whisper.
After the script ended, meaningless speech continued.
This is a classic case of tail hallucination.
When I tried it twice, the output was identical.
Since the diffusion TTS I'm using returns deterministic audio when the caption and seed are the same, this wasn't a fluke.
The same input would always produce the same hallucination.
The Countermeasure Was Already in the Code The tricky part was that I had already implemented a countermeasure for this phenomenon.
The clip generation retry system has three fallback levels: Generate → Whisper → Compare to script.
If it passes, save the clip If it fails, change and retry (keeping caption and seed fixed so the voice doesn't change, only the diffusion trajectory is reset) If it still fails, trim at Whisper's segment end and re-evaluate The third step is specifically designed to handle tail hallucinations.
The intention was clear, the implementation worked, and unit tests would pass.
Yet the hallucinations weren't being removed.
Was the Culprit When I looked at what Whisper was actually returning, I understood why.
Whisper was returning the hallucination as a separate segment.
The script ended at at 3.50 seconds, while the hallucination continued until 5.10 seconds in .
And my code was using —which is 5.10 seconds, the very end of the audio.
The trimming wasn't cutting anything.
More precisely, it was only removing trailing silence after the last segment with 40ms of padding left.
Since no audio was being removed, the clip would fail again and be excluded.
Another example: The script ended at 2.50 seconds, followed by nearly 1.5 seconds of silence before the hallucination began.
Despite this clear separation, choosing the wrong trimming position made the entire countermeasure ineffective.
The frustrating part was that this implementation worked correctly when no hallucination occurred.
If there's only one segment, trimming the trailing silence is a harmless operation.
It only fails when hallucinations occur—precisely when the countermeasure is needed.
Find Where the Script Content Ends The correct approach isn't to use "the end of the last segment" but to find "where the script content ends." By summing segments from the beginning and selecting the boundary where the match rate with the script is maximized, we can solve this.
For , adding up to results in a perfect match with the script (ratio 1.00), while including mixes in the hallucination, dropping the ratio to 0.73.
The maximum value is selected at 3.50 seconds.
The Full Retry Ladder Here's the corrected generation logic.
The key points are that there are three fallback levels and each handles different types of failur