📝 Originally published (in Japanese) at forge.workstyle.tech.
Building an Emotion-Expressive TTS Model: How Quality Gates Can Backfire I was working on an automated pipeline to generate training data for an emotion-expressive TTS model.
For each of 12 emotions (joy, sadness, anger, fear, etc.), I prepared several audio clips with the target emotion applied.
Naturally, this required quality control.
I used Whisper to transcribe the generated audio and only kept clips that matched the script exactly.
The resulting model ended up sounding flat and monotone.
The issue turned out to be the quality control process itself.
Switching Emotion Styles Still Sounds Like the Same Voice This model has distinct "styles" for each emotion—joy style, sadness style, etc.—and can synthesize speech while switching between them.
However, when switching styles, the perceived difference in the audio is minimal.
The numbers made this painfully clear.
I measured the cosine similarity between each emotion-style synthesis and a neutral style.
If the emotion was properly applied, the similarity should decrease (i.e., the value should be smaller).
A cosine similarity near 0.9 means that even when using the "joy" style, the output sounds almost identical to the neutral voice.
The emotion styles were effectively non-functional.
Honestly, when I first listened to the samples, I thought, "Eh, it's fine." It wasn't until the numbers showed 0.9 that I realized something was seriously wrong.
Three Root Causes Two of these were configuration issues, but the third is the real culprit.
1.
Speaker CFG Settings Cause Interjections to Sound Like a Different Person When scripts started with interjections like "Waa!" or "Eh!", the beginning of the clip would sound like a completely different voice.
The parameter controlling fidelity to the reference audio was breaking speaker consistency when emotion was applied.
2.
Emojis Get Read as Audio I had included emojis in the script as emotion markers, but they were either being read aloud ("happy face") or triggering unintended sound effects.
These markers should have been placed outside the text.
3.
Whisper-Only Validation Favors Flat Takes This was the main issue.
Validation Rejects Emotion-Rich Audio The pipeline generates multiple candidate clips and keeps only those that pass Whisper validation.
Here's what happens: Emotion-rich audio tends to have: Trembling voice (fear) Extended or rising pitch at the end (joy) Volume distortion (anger) Fading endings (sadness) All of these make recognition harder for Whisper, causing the transcription to deviate from the script and fail validation.
Flat, monotone takes, on the other hand, are clear and easy to recognize.
They pass validation effortlessly.
So when filtering clips based on "does it match the script?", emotion-sparse takes are disproportionately selected.
The stricter the gate, the stronger this bias becomes.
The better the gate works, the flatter the corpus becomes.
The goal (training an emotionally expressive voice) and the method (selecting based on script fidelity) were directly at odds.
And since every component was functioning correctly, no errors were thrown—making the issue hard to detect.
Split Validation into Two Stages: Filtering and Ranking The solution was to split the selection process into two stages.
Stage ① is just a quality filter—it doesn't determine ranking.
Stage ② uses a separate metric to rank candidates.
How to Measure "Emotion Applied" To measure whether emotion is applied, we use style embedding distance between the synthesized clip and a neutral version of the same speaker.
Since the TTS model can extract style vectors from audio, we leverage that.
If embedding extraction isn't available, you can approximate using combinations of: Median F0 F0 range (in semitones) RMS variance The key idea is to measure relative deviation from neutral, not absolute values.
Trying to define absolute thresholds like "joy should sound bright" forces you to adjust