Multi-concurrency inference not supported

Description

When I load a single Qwen3TTSModel instance and call generate_custom_voice() from multiple Python threads at the same time, inference becomes extremely slow — much slower than sequential calls, and much slower than passing a list of texts for batch inference. Increasing the thread count does not improve throughput; total latency just keeps growing. It seems a single model instance does not handle concurrent calls efficiently (likely serialized / contending on internal state), and there is no documented way to do concurrent local inference with one instance. Could you clarify whether single-instance multithreaded inference is supported, and what the recommended concurrency pattern is (queue + single worker, dynamic batching, multiple instances, or vLLM-Omni)?

Reproduction

import os, time, sys from concurrent.futures import ThreadPoolExecutor import torch from qwen_tts import Qwen3TTSModel

MODEL_PATH = os.getenv("MODEL_PATH", "/home/miner/data/model/Qwen3-TTS-12Hz-1.7B-Base") REF_AUDIO = "tools/video_choises/女声01中文样本.wav" REF_TEXT = "这是一段用于声音克隆的参考音频文本。" MAXTOK = 512 TEXTS = [ "今天天气非常好，我们一起去公园散步吧。", "人工智能正在改变我们的生活方式。", "请记得晚上八点参加线上会议。", "学习新技能需要持续练习和耐心。", ] N = len(TEXTS) LOG = "bench_result.txt"

def log(msg): with open(LOG, "a") as f: f.write(msg + "\n"); f.flush() print(msg, flush=True)

open(LOG, "w").close() log(f"加载模型 {MODEL_PATH} ...") model = Qwen3TTSModel.from_pretrained(MODEL_PATH, device_map="cuda:0", dtype=torch.bfloat16, attn_implementation="sdpa") prompt = model.create_voice_clone_prompt(ref_audio=REF_AUDIO, ref_text=REF_TEXT) log("模型就绪")

def run_one(text): model.generate_voice_clone(text=text, language="Auto", voice_clone_prompt=prompt, max_new_tokens=MAXTOK, do_sample=True, temperature=0.9, top_p=1.0, repetition_penalty=1.05)

log("预热..."); run_one(TEXTS[0]); torch.cuda.synchronize(); log("预热完成")

t0 = time.perf_counter() for t in TEXTS: run_one(t) torch.cuda.synchronize(); seq = time.perf_counter() - t0 log(f"A 单线程串行 : {seq:.2f}s ({seq/N:.2f}s/条)")

t0 = time.perf_counter() with ThreadPoolExecutor(max_workers=N) as ex: list(ex.map(run_one, TEXTS)) torch.cuda.synchronize(); thr = time.perf_counter() - t0 log(f"B 多线程并发x{N}: {thr:.2f}s ({thr/N:.2f}s/条)")

t0 = time.perf_counter() model.generate_voice_clone(text=TEXTS, language=["Auto"]N, voice_clone_prompt=promptN, max_new_tokens=MAXTOK, do_sample=True, temperature=0.9, top_p=1.0, repetition_penalty=1.05) torch.cuda.synchronize(); bat = time.perf_counter() - t0 log(f"C batch一次 : {bat:.2f}s ({bat/N:.2f}s/条)")

log("="*40) log(f"多线程/串行 = {thr/seq:.2f}x -> {'多线程更慢' if thr>seq else '多线程更快'}") log(f"batch/串行 = {bat/seq:.2f}x -> {'batch更快' if bat<seq else 'batch更慢'}")

Logs

Environment Information

pip list Package Version Editable project location

accelerate 1.12.0 aiofiles 24.1.0 aiohappyeyeballs 2.6.1 aiohttp 3.13.3 aiosignal 1.4.0 airportsdata 20250909 annotated-doc 0.0.4 annotated-types 0.7.0 anthropic 0.76.0 antlr4-python3-runtime 4.9.3 anyio 4.12.1 apache-tvm-ffi 0.1.8.post2 astor 0.8.1 attrs 25.4.0 audioread 3.1.0 blake3 1.0.8 brotli 1.2.0 cache_dit 1.2.0 cachetools 6.2.5 cbor2 5.8.0 certifi 2026.1.4 cffi 2.0.0 charset-normalizer 3.4.4 click 8.3.1 cloudpickle 3.1.2 coloredlogs 15.0.1 compressed-tensors 0.13.0 cryptography 46.0.3 cuda-bindings 13.1.1 cuda-pathfinder 1.3.3 cuda-python 13.1.1 cupy-cuda12x 13.6.0 dashscope 1.25.9 decorator 5.2.1 depyf 0.20.0 diffusers 0.36.0 dill 0.4.1 diskcache 5.6.3 distro 1.9.0 dnspython 2.8.0 docstring_parser 0.17.0 einops 0.8.2 email-validator 2.3.0 fa3_fwd 0.0.1 fastapi 0.128.0 fastapi-cli 0.0.20 fastapi-cloud-cli 0.11.0 fastar 0.8.0 fastrlock 0.8.3 ffmpy 1.0.0 filelock 3.20.3 flash_attn 2.8.3.post1 flashinfer-python 0.6.1 flatbuffers 25.12.19 frozenlist 1.8.0 fsspec 2026.1.0 gguf 0.17.1 gradio 5.50.0 gradio_client 1.14.0 groovy 0.1.2 grpcio 1.76.0 grpcio-reflection 1.76.0 h11 0.16.0 hf-xet 1.2.0 httpcore 1.0.9 httptools 0.7.1 httpx 0.28.1 httpx-sse 0.4.3 huggingface-hub 0.36.0 humanfriendly 10.0 idna 3.11 ijson 3.4.0.post0 ImageIO 2.37.2 imageio-ffmpeg 0.6.0 importlib_metadata 8.7.1 iniconfig 2.3.0 interegular 0.3.3 Jinja2 3.1.6 jiter 0.12.0 jmespath 1.1.0 joblib 1.5.3 jsonschema 4.26.0 jsonschema-specifications 2025.9.1 lark 1.2.2 lazy_loader 0.4 librosa 0.11.0 llguidance 1.3.0 llvmlite 0.44.0 lm-format-enforcer 0.11.3 loguru 0.7.3 markdown-it-py 4.0.0 MarkupSafe 3.0.3 mcp 1.26.0 mdurl 0.1.2 mistral_common 1.8.8 model-hosting-container-standards 0.1.13 more-itertools 10.8.0 mpmath 1.3.0 msgpack 1.1.2 msgspec 0.20.0 multidict 6.7.1 nest-asyncio 1.6.0 networkx 3.6.1 ninja 1.13.0 numba 0.61.2 numpy 2.2.6 nvidia-cublas-cu12 12.8.4.1 nvidia-cuda-cupti-cu12 12.8.90 nvidia-cuda-nvrtc-cu12 12.8.93 nvidia-cuda-runtime-cu12 12.8.90 nvidia-cudnn-cu12 9.10.2.21 nvidia-cudnn-frontend 1.17.0 nvidia-cufft-cu12 11.3.3.83 nvidia-cufile-cu12 1.13.1.3 nvidia-curand-cu12 10.3.9.90 nvidia-cusolver-cu12 11.7.3.90 nvidia-cusparse-cu12 12.5.8.93 nvidia-cusparselt-cu12 0.7.1 nvidia-cutlass-dsl 4.3.5 nvidia-ml-py 13.590.48 nvidia-nccl-cu12 2.27.5 nvidia-nvjitlink-cu12 12.8.93 nvidia-nvshmem-cu12 3.3.20 nvidia-nvtx-cu12 12.8.90 omegaconf 2.3.0 onnxruntime 1.23.2 openai 2.15.0 openai-harmony 0.0.8 openai-whisper 20250625 opencv-python-headless 4.13.0.90 orjson 3.11.5 outlines 0.1.11 outlines_core 0.2.11 packaging 26.0 pandas 2.3.3 partial-json-parser 0.2.1.1.post7 pillow 11.3.0 pip 25.0.1 platformdirs 4.5.1 pluggy 1.6.0 pooch 1.8.2 prometheus_client 0.24.1 prometheus-fastapi-instrumentator 7.1.0 propcache 0.4.1 protobuf 6.33.4 psutil 7.2.1 py-cpuinfo 9.0.0 py-spy 0.4.2 pybase64 1.4.3 pybind11 3.0.1 pycountry 24.6.1 pycparser 3.0 pydantic 2.12.3 pydantic_core 2.41.4 pydantic-extra-types 2.11.0 pydantic-settings 2.12.0 pydub 0.25.1 Pygments 2.19.2 PyJWT 2.10.1 pytest 9.0.2 python-dateutil 2.9.0.post0 python-dotenv 1.2.1 python-json-logger 4.0.0 python-multipart 0.0.22 pytz 2025.2 PyYAML 6.0.3 pyzmq 27.1.0 qwen-tts 0.0.4 /mnt/data/miner/supy/Qwen3-TTS ray 2.53.0 referencing 0.37.0 regex 2026.1.15 requests 2.32.5 resampy 0.4.3 rich 14.3.1 rich-toolkit 0.17.1 rignore 0.7.6 rpds-py 0.30.0 ruff 0.14.14 safehttpx 0.1.7 safetensors 0.7.0 scikit-learn 1.8.0 scipy 1.17.0 semantic-version 2.10.0 sentencepiece 0.2.1 sentry-sdk 2.50.0 setproctitle 1.3.7 setuptools 80.10.2 shellingham 1.5.4 six 1.17.0 sniffio 1.3.1 soundfile 0.13.1 sox 1.5.0 soxr 1.0.0 sse-starlette 3.2.0 starlette 0.50.0 supervisor 4.3.0 sympy 1.14.0 tabulate 0.9.0 threadpoolctl 3.6.0 tiktoken 0.12.0 tokenizers 0.22.2 tomlkit 0.13.3 torch 2.9.1+cu128 torchaudio 2.9.1+cu128 torchsde 0.2.6 torchvision 0.24.1+cu128 tqdm 4.67.1 trampoline 0.1.2 transformers 4.57.6 triton 3.5.1 typer 0.21.1 typer-slim 0.21.1 typing_extensions 4.15.0 typing-inspection 0.4.2 tzdata 2025.3 urllib3 2.6.3 uvicorn 0.40.0 uvloop 0.22.1 vllm 0.14.0rc2.dev335+g566cdb6cf vllm-omni 0.14.0rc1 /mnt/data/Qwen3-TTS/vllm-omni-main watchfiles 1.1.1 websocket-client 1.9.0 websockets 15.0.1 xformers 0.0.28.post3 xgrammar 0.1.29 yarl 1.22.0 zipp 3.23.0

Known Issue

• The issue hasn't been already addressed in Documentation, Issues, and Discussions.