Triton TRT-LLM Multimodal guide outdated

Page

https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Popular_Models_Guide/Llava1.5/llava_trtllm_guide.html

Source: Popular_Models_Guide/Llava1.5/llava_trtllm_guide.md in triton-inference-server/tutorials.

The LLaVA-1.5 guide is using TRTLLM engine backend, which is deprecated. Thus, updating the guide to LLM API, with a currently supported vision model such as Qwen2.5-VL.

While attempting that update, we found there is no way to pass an image through the Triton llmapi backend to TRTLLM. The PyTorch/LLM API path on Triton is triton_backend/all_models/llmapi/tensorrt_llm/ (config.pbtxt + 1/model.py). It is text-only.

I think the gap is entirely in config.pbtxt and model.py, thus opening this PR: https://github.com/NVIDIA/TensorRT-LLM/pull/18381

1. Can someone help check if editing these 2 files in TRTLLM is the right direction?
2. Is there a supported way today to serve a multimodal model on Triton with the TensorRT-LLM PyTorch backend?

Tested on TensorRT-LLM v1.2.1 (376f7e1bd8) and nvcr.io/nvidia/tritonserver:26.07-trtllm-python-py3 (Triton 2.71.0 / TRT-LLM 1.2.1 / CUDA 13.1).

Details

1. No media input tensor. config.pbtxt declares text_input, streaming, stop, 11 sampling_param_* and 3 return_* inputs. None can carry an image.
2. The prompt is passed as a bare str. _convert_request (1/model.py:482-511) reads only text_input and returns a decoded string, which _execute_single_request passes to generate_async(prompt, SamplingParams(...), streaming) (1/model.py:415). The LLM API only enters its multimodal branch when inputs is a dict carrying multi_modal_data (tensorrt_llm/llmapi/llm.py:480), so a string can never reach it.
3. No processor or HF config is loaded at startup. initialize starts the engine only; the AutoProcessor and PretrainedConfig needed for chat-template rendering and placeholder fusion are absent.

What we verified about the path that does work

trtllm-serve already serves images on v1/chat/completions, using only public helpers in tensorrt_llm/. Nothing in the library is missing; the Triton backend simply never calls any of it. For reference, the sequence in OpenAIServer.openai_chat (tensorrt_llm/serve/openai_server.py:547-570) is:

1. parse_chat_messages_coroutines(messages, model_config, multimodal_server_config) (tensorrt_llm/serve/chat_utils.py:268) parses OpenAI content parts (text / image_url / video_url / audio_url / image_embeds) and returns the conversation, a set of lazily-awaited media coroutines, and per-model placeholder counts.
2. async_load_image (tensorrt_llm/inputs/utils.py:150) does the fetch — http(s) via aiohttp, data: base64, or a local path — returning RGB, format="pt" by default.
3. MultimodalDataTracker + add_multimodal_placeholders (tensorrt_llm/inputs/utils.py:451, :518) inject the model-specific placeholder tokens from MULTIMODAL_PLACEHOLDER_REGISTRY at the registered position.
4. apply_chat_template (tensorrt_llm/inputs/utils.py:601) renders the prompt, preferring processor.chat_template.
5. awaiting the coroutines yields prompt["multi_modal_data"], which flows into generate_async → create_input_processor_with_hash (tensorrt_llm/inputs/registry.py:647) → the model's input processor.

The init-time state this needs — tokenizer, AutoProcessor.from_pretrained, and load_pretrained_config — is built once in OpenAIServer.__init__ (tensorrt_llm/serve/openai_server.py:81-120).

A few things we learned that would matter to anyone implementing this:

• _execute_single_request already runs on the engine event loop, via asyncio.run_coroutine_threadsafe(coro, self._event_loop) (1/model.py:378), so awaiting the media coroutines requires no new threading — _convert_request just becomes async.
• parse_chat_message_content_part swallows media-load failures into None (tensorrt_llm/serve/chat_utils.py:129-133), so a bad URL silently degrades to a text-only request. A backend should compare the returned multi_modal_data against the number of media items supplied and raise rather than inherit that behavior.
• self.model_config in 1/model.py:145 is already the Triton model config from json.loads(args["model_config"]); the HF PretrainedConfig needs a different attribute name.
• default_multimodal_input_loader (tensorrt_llm/inputs/utils.py:656) looks like a shortcut, but it calls AutoProcessor.from_pretrained and ModelLoader.load_hf_tokenizer on every invocation, so it is not viable for a per-request serving path.

Questions

1. Is there a supported way today to serve a multimodal model on Triton with the TensorRT-LLM PyTorch backend? If we are missing an existing mechanism for getting an image through the llmapi backend, we would rather use it than add one.