一个粗糙的片段会破坏整个风格——如何选择五个好片段

📝 Originally published (in Japanese) at forge.workstyle.tech. The TTS we're using creates emotional styles from "a few representative clips." For a joyful style, for example, passing a few joyful audio clips registers an average style vector. At first, I simply passed the first five clips from each group: Synthesizing with this style resulted in hoarse voices. Every single sentence, without fail. One out of 200 makes a difference The cause was that one or more rough clips were mixed in with the five I passed. Generated audio has a certain probability of being low quality. Voices can sound rough, pitches can fluctuate unstable, or octaves can jump mid-sentence. These still pass the quality gate (script matching) because the content is read correctly, so the roughness of the sound goes undetected. The entire corpus contains about 200 clips, so even if a few rough ones are mixed in, their impact on the overall learning is minimal. However, only five clips are used for style registration. If one of those five is rough, its impact becomes 20%. And dilution into the average doesn't work as well as expected. Style vectors are averages in the embedding space, but rough audio often lies far outside the normal range in the direction of "hoarseness." When averaging four normal clips with one outlier, the center of gravity is pulled significantly toward the outlier. As a result, all audio synthesized with that style carries a hoarse quality. The roughness of a single clip propagates to all outputs of that style. What makes a clip "rough"? Breaking down what feels "rough" when listening, I found two main factors: Jitter (period fluctuation): Unstable vocal cord vibration periods, resulting in a rough impression. Octave jumps: F0 estimation jumps to double or half between adjacent frames. This can be due to actual voice cracking or estimator errors, but both manifest as "unstable sound." I measure these two factors and select clips with the lowest values. Octave jumps are weighted more heavily (×2.0) because they have a greater impact on perception. Jitter is a continuous roughness, while jumps are discrete breaks that stand out as "voice cracking." The registration code was changed to: This eliminated the hoarseness. Excluding clips with short voiced segments The part is subtly important. Short interjections ("Yes!", "Okay!") or sentences with many unvoiced sounds ("desu shi", "shikkari") may have only a few voiced frames. Calculating jitter from these is meaningless and could incorrectly identify them as the most stable due to chance. Unjudgable clips are excluded from the candidates. If fewer than five clips remain, we select as many as possible. ⚠️ Jitter is overestimated in lower voices This metric has a known weakness: F0 estimation using autocorrelation is more error-prone for lower voices. Harmonics may be mistaken for the fundamental frequency, or estimation may jump between frames. These errors are counted as jitter. In practice, male voices showed jitter values of 31-56%. Since normal human voice jitter is less than 1%, measurement error clearly dominates. Therefore: It can be used for relative comparisons within the same speaker. Errors are uniformly applied, so rankings are meaningful. It cannot be used for comparisons between speakers, especially between genders. Lower voices are uniformly disadvantaged. Since style registration involves selecting from clips of the same speaker, this metric works fine for our purposes. When using the same metric with weighted values for a different purpose (selecting good voices from multiple candidates), all male candidates received negative scores (see [[screening-voices-by-metrics-not-ears|Having a Machine Select "Narrator-like Voices" from 24 Candidates]]). This was a case of misapplying the metric, and we adjusted the weights and incorporated other judgment criteria. Separating quality gates This issue highlighted that "correct content" and "clean sound" require separate gates. Gate Checks Rejects Script Matching (Whisper) Content match, out-of-script sounds Hallucinations, misreads, omissions Ending Fidelity Unscripted ending elongations Clips with ingrained ending habits Acoustic Normalization Frequency characteristics, sound pressure Channel characteristic variations Stability Selection Jitter, octave jumps Rough sounds, hoarseness Script matching only checks content, so rough audio passes through. Acoustic normalization (see [[tts-changes-recording-room-every-time|TTS That Changes "Recording Room" Every Generation]]) standardizes spectral shapes but cannot fix temporal instability (jitter). Each gate examines a different aspect and cannot substitute for another. Stability selection is unique in that it selects rather than rejects. While other gates exclude failures, this one chooses the top performers. Resources used in small quantities require selection, not just exclusion. Determining strictness based on impact range Generalizing, the approach is: The greater the impact of a single clip, the stricter the selection. Training corpus of 200 clips → 0.5% impact per clip. Quality gate filtering is sufficient. Style registration with 5 clips → 20% impact per clip. Filtering is insufficient; top selection is needed. Even from the same material pool, required quality differs based on usage. Clips suitable for the corpus and clips suitable as style representatives are selected using different criteria. I initially overlooked this because I assumed that "passing the quality gate means it's usable." Gates guarantee a minimum usability, not suitability as a representative. Summary Outliers in small resources can ruin the whole set. One out of five has a 20% impact, which cannot be diluted by averaging. Roughness can be measured by jitter and octave jumps. Jumps have a greater perceptual impact and are weighted more heavily. Clips with short voiced segments are excluded as unjudgable. They might be incorrectly identified as most stable. Jitter is overestimated in lower voices. Use only for within-speaker comparisons. Exclusion gates and selection gates are different. Small, elite resources require selection. Determine strictness based on impact range. Required quality differs based on usage, even for the same material. Series: Mass-Producing Practical Voices from Diffusion TTS This series documents the process of designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is part of Section 2: Manufacturing. ← Previous: [[tts-changes-recording-room-every-time|TTS That Changes "Recording Room" Every Generation]] → Next: [[where-did-the-elongated-ending-come-from|Where Did the AI's Habit of Stretching "Konnichiwa" Come From?]] All 18 Articles in the Series [[diffusion-tts-too-slow-for-conversation|The TTS Chosen for Quality Was Too Slow for Conversation]] [[deterministic-voice-gacha-and-design-ledger|Drawing Voices Like Gacha]] [[screening-voices-by-metrics-not-ears|Having a Machine Select "Narrator-like Voices" from 24 Candidates]] [[quality-gate-selection-bias-flat-takes|The Stricter the Quality Gate, the More Monotone Voices Survive]] [[speaking-style-is-baked-into-the-corpus|Speaking Speed Cannot Be Changed After Training]] [[tts-changes-recording-room-every-time|TTS That Changes "Recording Room" Every Generation]] 7. [[one-rough-clip-ruins-the-whole-style|One Rough Clip Can Make the Entire Style Hoarse]] ← You are here [[where-did-the-elongated-ending-come-from|Where Did the AI's Habit of Stretching "Konnichiwa" Come From?]] [[the-character-that-broke-the-tts-input|"Shoshou" Becomes "Shomo" — Character Allowlist Was Trimming Japanese]] [[hallucination-guard-that-never-fired|Hallucination Guard Code Only Activated During Hallucinations]] [[three-chars-became-a-verbal-tic|Three Characters Allowed by the Quality Gate Became the Model's Catchphrase]] [[measuring-factory-defects-as-product-traits|Rejecting Candidates Based on Fixable Defec