18 Insights from Mass-Producing Voice Models — From Diffusion TTS Voice Design to Training Corpus Creation and Quality Gate Pitfalls

📝 Originally published (in Japanese) at forge.workstyle.tech. This is a record of designing voices from single-line captions, automatically creating a learning corpus, and passing all 12 role-specific voices (narrator/counselor/sales/presenter/operator/MC for both men and women) through full inspection. I wrote about the failures I encountered during approximately one month of actual work, divided into 18 articles. This article is the table of contents. The Conclusion Upfront Voice design, voice manufacturing, and voice operation are different technologies with different failures. Design uses diffusion TTS. The voice is determined by the caption and random seed, making it fully reproducible. Manufacturing is primarily about corpus generation. The design of the quality gate directly determines the voice quality. Operation relies on lightweight pre-trained models. Diffusion TTS is too slow for conversation (2.5 times slower on the same GPU). The biggest lesson boils down to one point: Having a quality gate and it being effective are two different things. Six of these 18 articles are about gates that existed but weren’t effective. Reading Order The articles are arranged in the order of design → manufacturing → inspection → operation. Reading from the top will take you through the journey of a single voice being created and deployed into production. Chapter 1: Design — How to Determine the Voice The TTS Chosen for Sound Quality Was Too Slow for Conversation A 2.5x real-time factor (RTF) difference. How we settled on a two-stage approach: designing voices with diffusion TTS and using pre-trained models for speech. Drawing Voices Like a Gacha Voices are determined by captions and random seeds. By keeping a ledger of design values, voices can be recreated even if the model is lost. Letting a Machine Choose "Narrator-like Voices" from 24 Candidates Listening to all candidates is unsustainable. Automatically measure speech rate, intonation, and stability to only listen to the top candidates. Also, the story of how all male candidates were eliminated due to metric limitations. Chapter 2: Manufacturing — Corpus Quality Directly Becomes the Voice The Stricter the Quality Gate, the More Monotonous the Takes Survive The reason all emotion-infused corpora ended up monotonous was the quality gate itself. This is the central story of this series. Speech Rate Cannot Be Changed After Training Endings and speech rate are baked into the corpus. They cannot be adjusted with synthesis parameters. TTS That Changes "Recording Location" Every Time Even with the same model and speaker, frequency characteristics differ for each clip. If not standardized, style switching results in inconsistent sound quality. One Rough Clip Ruins the Entire Style If one out of five clips is rough, the entire style becomes hoarse. Dilution into the average doesn’t help. Chapter 3: Inspection — Having a Gate and It Being Effective Are Different Where Did the AI’s Habit of Stretching "Hello" Come From? Verification discarded long vowels due to kana normalization, making the defect fundamentally undetectable. "A Little" Becomes "Shomo" The model and parameters were fine, but the input text was broken. The symbol removal list was dropping characters like "々", "〆", and "髙". The Hallucination Guard Code Only Failed During Hallucinations A single-line bug made the guard ineffective only in necessary situations. The "Three Characters" Allowed by the Quality Gate Became a Verbal Tic The size passing through the gate matched the size of the sound reproduced by the model. Rejecting Candidates for Fixable Defects Is the metric measuring the product’s characteristics or the process’s state? Defects Unseen in Transcription STT-only inspection misses 0.1-second additive sounds with silence in between. Capture them with waveform envelopes. Chapter 4: Operation — Running as a Factory 70 Minutes of Training Material Lost to a Network Blink One disconnection led to 205 retries. Finding area