📝 Originally published (in Japanese) at forge.workstyle.tech.
Quality Control for TTS Models: Why Transcription Isn’t Enough I used to perform quality control (QC) for TTS models using this process: Have the model read probe sentences Transcribe with Whisper Compare against the script to check accuracy and trailing elongation Analyze the waveform for utterance duration, sound pressure, and F0 to detect abnormalities I created 12 voices and passed all of them through this QC.
Whisper got 4/4 accuracy, no trailing elongation, and sound pressure was within normal range.
I reported 100% pass rate.
Later, when I rechecked from a different angle, 4 of them still had defects.
These were invisible to STT-based inspection due to fundamental limitations.
STT Drops Short Sounds The first clue came when I received this report: After finishing the script, there’s a full 0.5-second silence followed by a 0.1–0.3 second utterance.
This isn’t trailing resonance—the model is producing sounds not in the script (the root cause was training corpus contamination: "3 characters" allowed by the quality gate became verbal tics).
The reason my initial inspection missed this is simple: Whisper dropped these sounds.
A 0.28-second utterance doesn’t appear in the transcription at all.
Short sounds that aren’t meaningful words may not appear in STT output.
As long as you’re only looking at transcriptions, this defect doesn’t exist.
I even concluded, “STT got 0/6, so no extra sounds,” mistaking the blind spot of my measurement method for a property of the target.
Seeing Through Waveform Envelope What’s actually being output appears in the waveform.
By extracting voiced blocks from the RMS envelope and examining their sequence, we can detect these artifacts.
Why Double Thresholds Matter The part is subtly critical.
Relative threshold alone fails for low-volume voices.
If the overall volume is quiet, the maximum value is small, causing noise floor to be misclassified as voiced.
Absolute threshold alone fails for high-volume voices.
Breathing or lip smacks get classified as voiced.
The 12 voices had sound pressure ranging from −13.3 to −18.8 dB, so neither threshold alone could work across all voices.
Discarding blocks under 30ms is also necessary.
Without this, lip noise or quantization noise appears as many tiny blocks, breaking downstream logic.
Detection Criteria Once voiced blocks are extracted, we check: “Is there sufficient silence before the final block, and does that block have sufficient duration?” separates natural trailing resonance or pauses from clearly separated utterances.
Measured artifacts had silence gaps of 0.26–0.91 seconds, so 0.25 is sufficient. avoids catching fade-out tails.
Measured artifacts were 0.07–0.36 seconds long.
Commas Caused False Positives in All 12 Models In my first scan, all 12 models triggered the detector.
The probe sentence contained this: This was a pause after a comma.
After “では、” there’s a gap, then “始めます。” follows.
The final block is part of the script itself, yet it perfectly matches the detection condition (gap + subsequent utterance).
The condition “there’s utterance after the final gap” will always produce false positives for sentences containing commas.
That’s because it doesn’t consider script structure.
There are two fixes: Limit probes to single sentences.
If you exclude sentences with commas, any utterance after the body can be definitively identified as an artifact.
This is what I adopted—simple implementation and no dependency on the script.
Align with script end position.
Derive the script end position from Whisper segments and check if energy exists beyond that point.
This is more general but reintroduces STT dependency.
If artifacts don’t appear in Whisper segments, the end position might be incorrectly determined.
After removing false positives: Model Artifacts Male Narrator 6/6 Female Operator 4/6 Female Presenter 4/6 Male Presenter 2/6 Remaining 8 0/6 Had I reported the initial results as-is, I wo