TensorRT backend leaks GPU resources after model initialization failure; unload does not recover memory

Description GPU memory accumulates when TensorRT model loading fails after partial initialization. Explicitly unloading the failed model does not release the retained memory. Successful load/unload cycles remain stable, and no inference requests are needed to reproduce the problem.

The attached reproducer generates a valid identity engine and then deliberately sets an incorrect output datatype in the model configuration. Triton correctly rejects the configuration, but GPU memory usage increases with repeated failed load/unload cycles. Restoring the correct configuration and successfully loading/unloading the model does not reclaim the earlier retained memory.

Triton Information

• Official, unmodified nvcr.io/nvidia/tritonserver:26.07-py3 image, pinned to digest sha256:551599c0812bb706121410a71abda07f9920d25e6c66edf1ec07a375e49f5f90.
• Triton 2.71.0; TensorRT 11.1.0.106; CUDA compiler 13.3.73.
• Tested on one RTX 3080 Ti, 12 GiB, driver 580.173.02.
• Linux host with Docker and NVIDIA Container Toolkit; one otherwise idle GPU.
• The image supplies the compiler, TensorRT headers/libraries, Python 3 and nvidia-smi needed by the reproducer.

To Reproduce Download the attached repro.py. The script generates a weight-free FP32 identity engine with one input and one output, each shaped [16777216]. These dimensions are test parameters, not an identified condition or threshold for the bug.

The script performs the following sequence:

1. Build the engine in a separate process that exits before Triton starts.
2. Run three successful model load/unload cycles to establish a memory baseline.
3. Change only the configured output datatype from TYPE_FP32 to TYPE_INT32.
4. Run six failed model loads, explicitly unloading after each failure.
5. Restore TYPE_FP32, successfully load/unload the model, and measure again.
6. Stop Triton and measure memory after process exit.

From the directory containing the extracted repro.py, run:

REPRO_GPU=0
REPRO_IMAGE=nvcr.io/nvidia/tritonserver:26.07-py3@sha256:551599c0812bb706121410a71abda07f9920d25e6c66edf1ec07a375e49f5f90

docker run --rm --gpus "device=${REPRO_GPU}" \
  --user "$(id -u):$(id -g)" --shm-size=1g \
  -v "$PWD:/work" --entrypoint python3 \
  "$REPRO_IMAGE" /work/repro.py

The client and server run inside the container without publishing host ports. The script writes server.log and results.json to the working directory. It sends only standard HTTP model-repository requests, with auto-complete disabled, and verifies that each failed load reports the intended error:

unexpected datatype TYPE_FP32 for inference output 'OUTPUT', expecting TYPE_INT32

Observed behavior

The following readings are from the tested environment. Each post-unload reading is the median of three GPU-memory samples on an otherwise idle GPU.

These values describe this model and environment; the report does not assume a fixed amount of memory retained per failure. The problem is continued memory growth after failed initialization and explicit unload, while successful-load controls remain stable.

Expected behavior The invalid configuration should be rejected and resources allocated during partial initialization should be released. Repeated failed load/unload cycles should not accumulate GPU memory or require restarting Triton to reclaim it.

repro.py