📝 Originally published (in Japanese) at forge.workstyle.tech.
I Found a Diffusion TTS That Generates Voices from Just a Caption Pass this description, and it will speak in exactly that voice.
You can generate any Japanese voice without preparing even one second of speaker audio.
Moreover, the same caption and random seed will always produce the same voice.
I thought this could be useful for interactive avatars—eliminating the hassle of preparing voices for each character.
In short: it didn’t work for conversations.
But it was too valuable to discard, so I repurposed it.
Benchmark: 2.5x Slower on the Same GPU Our existing system used a pre-trained TTS (Style-Bert-VITS2-based).
We synthesized the same 7.5-second sentence on the same GPU slice to compare.
Engine Conditions Generation Time RTF Diffusion TTS 40 steps 2g.20gb 1.9–2.1s 0.25–0.28 Diffusion TTS 24 steps 2g.20gb 1.4–2.0s 0.19–0.27 Diffusion TTS 16 steps 2g.20gb 1.1–1.4s 0.15–0.19 Pre-trained Model 2g.20gb 0.74–0.81s 0.115–0.127 RTF (Real Time Factor) = generation time ÷ audio length.
Lower is faster.
At 40 steps, it’s 2.5× slower.
Even cutting steps to 16 still leaves a 1.5× gap.
And below 16 steps, quality collapses (6 steps was immediately rejected on listening).
On a smaller GPU slice, the gap widens further.
Engine Conditions Generation Time RTF Diffusion TTS 40 steps 1g.10gb 3.4–4.0s 0.46–0.54 Diffusion TTS 16 steps 1g.10gb 2.0s 0.27 In conversation, audio is synthesized and played sentence-by-sentence.
The delay until the first sentence plays determines the user experience.
A 1–2 second lag per turn is prohibitive.
Upgrading GPUs Didn’t Solve It I considered allocating more GPU resources—from 2g.20gb to 7g.80gb (4× the compute).
But measurements showed a fixed overhead of ~1.1 seconds.
Even cutting steps to 12, generation time bottomed out at 1.1–1.4 seconds.
This is due to model loading and text processing—parts that don’t benefit from parallelism.
So even with 4× the resources, the fastest possible generation would be ~1.0–1.2 seconds—still slower than the pre-trained model’s 0.8 seconds.
The cost multiplies.
We decided not to adopt it.
A useful takeaway: measure fixed overhead first.
By reducing steps to the point where performance stops improving, you can estimate how much of the bottleneck is parallelizable.
Splitting the Workflow Speed was the only downside.
But diffusion TTS offers something pre-trained models can’t: generating voices from captions alone.
Pre-trained models can only reproduce voices seen during training.
So we split the pipeline: During design, diffusion TTS creates “this kind of voice.” We record ~200 lines with that voice to build a training corpus, then train a lightweight model for runtime use.
Since diffusion TTS runs only once per voice, its slowness isn’t a problem.
Generating one voice takes ~70 minutes, but it’s a batch process.
Conditions for This Split to Work Deterministic generation.
The same caption and seed must always produce the same voice.
Without this, we couldn’t reproduce the designed voice, and every retraining would yield a different speaker.
Thanks to this property, we only need to store the design ledger to recreate the model.
Even if we lose the trained model file, we can regenerate the exact voice from the caption and seed. (We once lost the driver script, but recovered the voice by restoring the caption and seed from logs: [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]).
Audio quality must survive training.
Since generated audio becomes training data, diffusion TTS’s audio quality sets the ceiling.
We verified this by actually training and listening—no issues here.
Speaker identity must remain consistent.
Over 200 clips, the voice must stay consistent.
This was tricky.
Starting sentences with exclamations sometimes changed the speaker at the beginning, and strong emotional expressions could break speaker identity.
We solved this by carefully managing reference audio—but that’s