InferenceRequest.async_exec(decoupled=True) intermittently raises `RuntimeError: Invalid argument` under concurrent in-flight BLS to a decoupled model

Environment

• Triton Inference Server version: 2.70.0 (NVIDIA Release 26.06, build 55746352)
• Python backend: triton_python_backend_utils from libtriton_python.so (build 2025-06-24)
• Python runtime: 3.12.3
• Backends loaded: identity, python, pytorch, vllm
• GPU: NVIDIA A100-SXM4-80GB, driver 595.71.05 (CUDA 13.2)
• Topology hosting the bug:
  复制

  ensemble   (Python backend, KIND_CPU, 8 instances, async BLS orchestrator)
    ├─ ...
    ├─ ...
    └─ whisper_vllm (vllm backend, decoupled)  <-- BLS target

Summary

When an async Python model (async def execute) issues BLS requests to a decoupled model using pb_utils.InferenceRequest.async_exec(decoupled=True), and more than a handful of these are in flight concurrently, the call intermittently fails with:

RuntimeError: Invalid argument

raised from inside concurrent/futures/thread.py (the worker-thread self.fn(*self.args, **self.kwargs) dispatched by async_exec). The failure is intermittent — it appears to depend on the number of simultaneously in-flight decoupled BLS streams, not on the request contents or shapes.

Expected behavior

Per the Python backend docs (Business Logic Scripting section, "Decoupled Models" -> "Using BLS with Decoupled Models"), the documented pattern is:

import triton_python_backend_utils as pb_utils
import asyncio

class TritonPythonModel:
    async def execute(self, requests):
        inference_request = pb_utils.InferenceRequest(
            model_name='whisper_vllm',
            requested_output_names=['text_output'],
            inputs=[...])

        infer_response_awaits = [
            inference_request.async_exec(decoupled=True)
            for _ in range(4)
        ]
        async_responses = await asyncio.gather(*infer_response_awaits)

        for infer_responses in async_responses:
            for infer_response in infer_responses:
                assert not infer_response.has_error()
                ...

This should return one Awaitable per call, each resolving to an iterator of the decoupled responses, and should be safe to fan out concurrently.

Actual behavior

With the exact pattern above, the code intermittently crashes with RuntimeError: Invalid argument:

File ".../whisper_vllm.py", line 240, in _run_single_whisper
    iterator = await request.async_exec(decoupled=True)
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
RuntimeError: Invalid argument

Observations:

• RuntimeError surfaces from a ThreadPoolExecutor worker thread — i.e. async_exec internally schedules the blocking exec onto a worker thread and returns a Future, and the wrapped C++ call returns EINVAL.
• Raising concurrency increases the probability of failure; a single sequential call rarely fails.
• Workaround that is reliable in 2.70.0: run the synchronous request.exec(decoupled=True) inside asyncio.to_thread(...) with a bounded semaphore (i.e. offload the documented stable sync API to a worker thread instead of using async_exec). This confirms the problem is isolated to the async_exec(decoupled=True) path, not to concurrency against the decoupled model in general.

# reliable workaround in 2.70.0
iterator = await asyncio.to_thread(request.exec, decoupled=True)
for response in iterator:
    ...

Minimal reproducer

A minimal two-model repo pair ("orchestrator" = async Python backend model, "leaf" = decoupled Python backend model) that fans out N concurrent async_exec(decoupled=True) BLS calls would reproduce the intermittent RuntimeError. I have not yet packaged a fully standalone repo, but the essential ingredients are:

1. A decoupled leaf Python model whose execute writes a response and returns None (model_transaction_policy: { decoupled: True }).
2. An async orchestrator Python model that, per request, issues several await asyncio.gather(*[req.async_exec(decoupled=True) for _ in range(N)]) with N >= 8.
3. A client that bursts many concurrent orchestrator requests.

If useful I can provide the full model definitions, but the three ingredients above drive 100% of the behaviour.

Impact

• Any production policy that relies on async Python-model orchestration over a decoupled backend (e.g. an ensemble/stub that diarizes -> transcribes via a streaming/decoupled ASR or LLM) is at risk of nondeterministic per-request failures from this single API path.
• The failure is a hard RuntimeError that aborts the whole BLS request (all gathered sub-requests), not a graceful per-response error.