Training setup for the zero-shot voice prompt (audio prefix) in TTS

Thanks for releasing Pocket-TTS and the CALM paper — really impressive work, especially the 1-step LSD quality on CPU.

I'm trying to reproduce the zero-shot (voice-cloning) training, but I couldn't quite find the details of how the voice prompt is constructed during training (I may well have missed them). In the meantime, I followed the zero-shot training scheme from SALAD (Turetzky et al., 2024 — one of your references): using speaker-id labels to sample the speaker prompt from a different utterance of the same speaker ("a three-second speaker prompt … taken randomly from another utterance of the same speaker"). With that, I was able to train the model successfully.

That said, I don't think this is how you trained on the full mix: several corpora in your 88k-hour data (GigaSpeech, SPGISpeech, EARNINGS22) don't seem to provide per-speaker labels, which would rule out same-speaker sampling. (They do expose a recording/call-level id — e.g. audio_id / file_id — but segments sharing the same id aren't guaranteed to be the same speaker) So I suspect your prompt construction is different, and I'd love to understand it.

Also, from the public inference code, the voice cloning doesn't seem to follow the VALL-E / CosyVoice style of in-context prompting (where the reference audio tokens and the reference transcript are prepended in the same token space and the model continues from there). The audio prompt is provided, but as a dedicated projection of its VAE latents (speaker_proj) prepended as a prefix — and the text stream contains only the target text, with no reference transcript. That's part of why the training-time prompt construction isn't obvious to me.

From the paper I gather: text is a SentencePiece prefix (§5.2), voice conditioning is an "audio prefix" (§E.5), and eval uses a 3s prompt from ground-truth audio (§G). Could you clarify the training-time setup:

1. Prompt source Is the audio prefix taken from the same target utterance (a slice of it), or from a different same-speaker utterance (as in SALAD)?

2. If cut from a single utterance Is the cut made along a forced-alignment (e.g. MFA) word boundary, or is it a random-position/length slice? And if it's a random cut, are the prompt frames excluded from the training loss (e.g. masked out of the FM/LSD loss)?

3. No-speaker-label corpora Does the fact that GigaSpeech / SPGISpeech / EARNINGS22 have no speaker labels imply the prompt is always drawn from the target utterance itself (i.e. no same-speaker sampling)?

Even a short description would help a lot. Thanks!