Concurrent TTS Performance Observation (VoxCPM2 + vLLM-Omni)

Hi Team,

I tested the /api/tts_direct_stream endpoint under concurrent load and observed the following behavior.

Environment:

Server: AWS EC2 (L40S GPU) Serving Stack: vLLM-Omni v0.20.0 Endpoint: /api/tts_direct_stream Client: Python (requests + ThreadPoolExecutor) for concurrent load testing

Model Serving Command:

vllm-omni serve $model_path --omni --served-model-name tts-service --host 0.0.0.0 --port 8000 --enforce-eager --dtype bfloat16

Test Results:

1 concurrent request Latency: ~0.59s TTFB: ~0.11s

10 concurrent requests Average latency: ~3.08s Maximum latency: ~4.46s TTFB increases from ~0.49s for the first few requests to ~3.53s for the last requests.

It appears that requests are being processed in batches rather than all 10 starting simultaneously. The first few requests begin processing immediately, while the remaining requests wait before inference starts, resulting in higher TTFB and overall latency.

Could you please confirm:

-Is there a concurrency limit (or maximum number of parallel inference slots) configured in the current vLLM-Omni/VoxCPM2 setup? -Is this expected behavior for the current deployment, or is there any server-side configuration that can improve concurrent throughput? -What's the minimum GPU for getting concurrency. -Are there recommended settings (e.g., scheduler, batching, max_num_seqs, or deployment configuration) to handle higher concurrent TTS requests?

Thanks.