📝 Originally published (in Japanese) at forge.workstyle.tech.
Understanding "Voice" — Breaking Down the Components When trying to convert your recorded voice into someone else's, the first question that comes to mind is: What exactly is a "voice"?
Even if the same words are spoken, different speakers produce different sounds.
Even the same speaker produces different sounds depending on what they're saying.
Adding intonation changes it further.
In other words, speech is a signal composed of multiple independent pieces of information mixed together — at least: who is speaking (speaker identity), what is being said (content), and how it's being said (prosody).
The core challenge of voice conversion (VC) lies here.
If you naively process speech to replace only the speaker identity, the content and intonation often get altered as well.
In this article, we'll explain how Seed-VC, which we adopted for the backend of our "voice design" app, solves this problem — by separating and handling information using four modules, while walking through the actual model loading and inference code.
Design Philosophy: Delegate Separable Information to Dedicated Modules At the heart of Seed-VC is the idea: "Don't make a monolithic model do everything." Instead, decompose the components of speech by type and assign each to a dedicated module, then recombine them at the end.
The model loading process at startup looks like this: While seven components are returned, they can be grouped into four functional layers: whisper () — Extracts content (what is being said) as semantic features campplus () — Encodes speaker identity (who is speaking) into a single vector embedding CFM/DiT () — A diffusion model that generates a mel-spectrogram conditioned on the above two BigVGAN () — Converts the mel-spectrogram back into an audible waveform Each role — "extract meaning", "extract speaker", "draw the spectrogram", "convert to sound" — is cleanly separated.
This division of labor is why we can replace only the speaker identity while keeping everything else intact.
Let's go through each step.
Step 1: whisper — Extract Only "What Is Being Said" For content extraction, we use the encoder of the speech recognition model Whisper.
We don't use the decoder (transcription).
Instead, we take the intermediate representation (a sequence of semantic features) directly as a feature vector.
A key point: the input is resampled to 16kHz before processing.
Since Whisper is designed for 16kHz audio, even if the main conversion model operates at 44.1kHz, we always downsample to 16kHz before semantic extraction.
The resulting sequence contains little to no information about pitch or timbre — it represents a representation aligned with the spoken content.
This is the prerequisite that allows us to freely replace speaker identity later.
Note: Whisper has a limitation — it can only process up to 30 seconds of audio at a time.
For longer audio, chunking is required.
This constraint itself was a major pitfall (see our other article: "The Culprit Behind the 'Slow Speech' Bug in Voice Conversion Was Whisper's 30-Second Limit").
Step 2: campplus — Condense "Who Is Speaking" into a 192-Dimensional Vector Speaker identity extraction is handled by CAMPPlus, a speaker embedding model.
It takes audio, computes fbank features, and outputs a single 192-dimensional vector.
The fact that "speaker identity = a single vector" is crucial to our "voice design" app.
Because vectors can be added, blended, or interpolated, we precompute embeddings from clean recordings of 18 speakers as "anchors", then blend them using weighted averages based on slider inputs.
We define sliders for interpretable axes like "age", "pitch", and "huskiness", compute weights , and take a weighted average of anchor embeddings.
This "design by blending voices" operation is only possible because speaker identity is modularized as a standalone vector.
If speaker identity were entangled with content, blending would corrupt the spok