OmniVoice Voice Cloning / Pronunciation Accuracy Issue

Checks

• This template is only for bug reports, usage problems go with 'Help Wanted'.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

Environment:

• OS: Ubuntu 24.04.4 LTS
• GPU: NVIDIA L40S 46 GB
• NVIDIA Driver: 580.173.02
• CUDA Version: 13.0
• Python: 3.12.3
• PyTorch: 2.11.0
• vLLM: 0.26.0
• vLLM-Omni: 0.26.0rc2.dev22+g18be2ba3d
• Model: OmniVoice

Serving Command:

vllm serve OmniVoice \
  --omni \
  --port 8000 \
  --trust-remote-code

API Endpoint:

/v1/audio/speech

Steps to Reproduce

1. Start the OmniVoice model using vLLM:

vllm serve OmniVoice \
  --omni \
  --port 8000 \
  --trust-remote-code

1. Send a request to the /v1/audio/speech endpoint using a reference audio for voice cloning.

2. Test voice cloning using reference audio without providing the reference transcript.

3. Observe that the generated voice has poor voice similarity and/or pronunciation accuracy.

4. Repeat the test by providing both:

   • Reference audio
   • Reference audio transcript

5. With simple sentences, the generated speech is relatively good.

6. Test sentences containing structured text such as:

   • Email addresses
   • Phone numbers
   • Integer numbers
   • Decimal numbers
   • Dates
   • URLs
   • Alphanumeric values

7. Observe that these structured inputs are frequently pronounced incorrectly, skipped, or generated unnaturally.

8. Compare the output with a simple sentence using the same reference audio and transcript.

✔️ Expected Behavior

I expect OmniVoice to generate speech with good speaker similarity and accurate pronunciation.

Specifically:

1. Voice cloning should work reasonably well when only reference audio is provided, if reference-audio-only cloning is supported by the model.

2. When a reference transcript is provided, the generated speech should maintain the speaker's voice characteristics while accurately reproducing the input text.

3. Structured text should be pronounced correctly, including:

   • Email addresses
   • Phone numbers
   • Integer numbers
   • Decimal numbers
   • Dates
   • URLs
   • Alphanumeric values

4. The pronunciation accuracy should be comparable to that of normal/simple sentences.

❌ Actual Behavior

There are two main issues observed.

1. Reference audio without transcript

When only the reference audio is provided without its transcript, the generated speech has poor voice-cloning accuracy and pronunciation. The generated voice does not reliably reproduce the expected speaker characteristics.

2. Reference audio with transcript

When both reference audio and its transcript are provided, simple sentences generally produce acceptable results.

However, the model has significant difficulty with structured text such as email addresses, numbers, decimals, phone numbers, dates, URLs, and alphanumeric values.

For example, simple sentences work relatively well:

Hello, how are you today?

But sentences containing structured information may have incorrect pronunciation, missing content, or unnatural speech:

My email address is [email protected] and my account number is 123456.78.

The issue appears to be more noticeable with emails, numbers, decimals, and other non-standard text formats.

I would like to know whether this is an expected limitation of OmniVoice, a text-normalization/tokenization issue, or something related to the vLLM/vLLM-Omni serving implementation.