A single rough clip can ruin the entire style — How to choose 5 good ones

📝 Originally published (in Japanese) at forge.workstyle.tech. The TTS we're using creates emotional styles from "a few representative clips." For a joyful style, for example, passing a few joyful audio clips registers an average style vector. At first, I simply passed the first five clips from each group: Synthesizing with this style resulted in hoarse voices. Every single sentence, without fail. One out of 200 makes a difference The cause was that one or more rough clips were mixed in with the five I passed. Generated audio has a certain probability of being low quality. Voices can sound rough, pitches can fluctuate unstable, or octaves can jump mid-sentence. These still pass the quality gate (script matching) because the content is read correctly, so the roughness of the sound goes undetected. The entire corpus contains about 200 clips, so even if a few rough ones are mixed in, their impact on the overall learning is minimal. However, only five clips are used for style registration. If one of those five is rough, its impact becomes 20%. And dilution into the average doesn't work as well as expected. Style vectors are averages in the embedding space, but rough audio often lies far outside the normal range in the direction of "hoarseness." When averaging four normal clips with one outlier, the center of gravity is pulled significantly toward the outlier. As a result, all audio synthesized with that style carries a hoarse quality. The roughness of a single clip propagates to all outputs of that style. What makes a clip "rough"? Breaking down what feels "rough" when listening, I found two main factors: Jitter (period fluctuation): Unstable vocal cord vibration periods, resulting in a rough impression. Octave jumps: F0 estimation jumps to double or half between adjacent frames. This can be due to actual voice cracking or estimator errors, but both manifest as "unstable sound." I measure these two factors and select clips with the lowest values. Octave jumps are weighted more heavily (×2.0) because they have a greater impact on perception. Jitter is a continuous roughness, while jumps are discrete breaks that stand out as "voice cracking." The registration code was changed to: This eliminated the hoarseness. Excluding clips with short voiced segments The part is subtly important. Short interjections ("Yes!", "Okay!") or sentences with many unvoiced sounds ("desu shi", "shikkari") may have only a few voiced frames. Calculating jitter from these is meaningless and could incorrectly identify them as the most stable due to chance. Unjudgable clips are excluded from the candidates. If fewer than five clips remain, we select as many as possible. ⚠️ Jitter is overestimated in lower voices This metric has a known weakness: F0 estimation using autocorrelation is more error-prone for lower voices. Harmonics may be mistaken for the fundamental frequency, or estimation may jump between frames. These errors are counted as jitter. In practice, male voices showed jitter values of 31-56%. Since normal human voice jitter is less than 1%, measurement error clearly dominates. Therefore: It can be used for relative comparisons within the same speaker. Errors are uniformly applied, so rankings are meaningful. It cannot be used for comparisons between speakers, especially between genders. Lower voices are uniformly disadvantaged. Since style registration involves selecting from clips of the same speaker, this metric works fine for our purposes. When using the same metric with weighted values for a different purpose (selecting good voices from multiple candidates), all male candidates received negative scores (see [[screening-voices-by-metrics-not-ears|Having a Machine Select "Narrator-like Voices" from 24 Candidates]]). This was a case of misapplying the metric, and we adjusted the weights and incorporated other judgment criteria. Separating quality gates This issue highlighted that "correct content" and "clean sound" require separate gates. Gate Chec