TensorRT

[](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html) [](documents/tensorrt_roadmap_2026q3.pdf) # :mega::mega: Announcement :mega::mega: TensorRT 11.X is now released with powerful new capabilities designed to accelerate your AI inference workflows. With this major version bump, TensorRT's API has been streamlined and a few legacy features from 10.X have been removed. Below provides migration guides for the following features: - Weakly-typed networks and related APIs have been removed, replaced by [Strongly Typed Networks](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/advanced.html#strongly-typed-networks). - Implicit quantization and related APIs have been removed, replaced by [Explicit Quantization](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-quantized-types.html#explicit-quantization) - IPluginV2 and related APIs have been removed, replaced by [IPluginV3](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/extending-custom-layers.html#migrating-v2-plugins-to-ipluginv3) - TREX tool has been removed, replaced by [Nsight Deep Learning Designer](https://docs.nvidia.com/nsight-dl-designer/UserGuide/index.html#visualizing-a-tensorrt-engine) - Python bindings for Python 3.9 and older versions have been removed. RPM packages for RHEL/Rocky Linux 8 and RHEL/Rocky Linux 9 now depend on Python 3.12. # TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. It includes the sources for TensorRT plugins and ONNX parser, as well as sample applications demonstrating usage and capabilities of the TensorRT platform. These open source software components are a subset of the TensorRT General Availability (GA) release with some extensions and bug-fixes. - For step-by-step walkthroughs of the TensorRT import paths (ONNX, Torch-TensorRT, HuggingFace/Optimum, Network Definition API) with examples and tooling tips, see the [Import Workflows Guide](documents/import_workflows.md). - For the per-model support matrix across import paths (LLM, encoder-NLP, vision, audio, diffusion, multimodal), see [Supported Models](documents/supported_models.md). - For code contributions to TensorRT-OSS, please see our [Contribution Guide](CONTRIBUTING.md) and [Coding Guidelines](CODING-GUIDELINES.md). - For a summary of new additions and updates shipped with TensorRT-OSS releases, please refer to the [Changelog](CHANGELOG.md). - For business inquiries, please contact [[email protected]](mailto:[email protected]) - For press and other inquiries, please contact Hector Marinez at [[email protected]](mailto:[email protected]) Need enterprise support? NVIDIA global support is available for TensorRT with the [NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/launchpad/ai/ai-enterprise/) for free access to a set of hands-on labs with TensorRT hosted on NVIDIA infrastructure. Join the [TensorRT and Triton community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and stay current on the latest product updates, bug fixes, content, best practices, and more. # Agentic Coding Skills Various skills related to TensorRT usage and benchmarking are available [here](.agents/skills). For installation, refer to the instructions of your preferred coding agent. # Prebuilt TensorRT Python Package We provide the TensorRT Python package for an easy installation. \ To install: ```bash pip install tensorrt ``` You can skip the **Build** section to enjoy TensorRT with Python. # Build ## Prerequisites To build the TensorRT-OSS components, you will first need the following software packages. **TensorRT GA build** - TensorRT v11.2.1.2 - Available from direct download links listed below **System Packages** - [CUDA](https://developer.nvidia.com/cuda-toolkit) - Recommended versions: - cuda-13.3.0 - cuda-12.9.0 - [CUDNN (optional)](https://developer.nvidia.com/cudnn) - cuDNN 8.9 - [GNU make](https://ftp.gnu.org/gnu/make/) >= v4.1 - [cmake](https://github.com/Kitware/CMake/releases) >= v3.31 - [python](https://www.python.org/downloads/) >= v3.10, <= v3.14.x - [pip](https://pypi.org/project/pip/#history) >= v19.0 - Essential utilities - [git](https://git-scm.com/downloads), [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/), [wget](https://www.gnu.org/software/wget/faq.html#download) **Optional Packages** - [NCCL](https://developer.nvidia.com/nccl/nccl-download) >= v2.19, < v3.0 — only when building with multi-device support (`-DTRT_BUILD_ENABLE_MULTIDEVICE=ON`) for the `sampleDistCollective` sample. - Containerized build - [Docker](https://docs.docker.com/install/) >= 19.03 - [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker) - PyPI packages (for demo applications/tests) - [onnx](https://pypi.org/project/onnx/) - [onnxruntime](https://pypi.org/project/onnxruntime/) - [tensorflow-gpu](https://pypi.org/project/tensorflow/) >= 2.5.1 - [Pillow](https://pypi.org/project/Pillow/) >= 9.0.1 - [pycuda](https://pypi.org/project/pycuda/) < 2021.1 - [numpy](https://pypi.org/project/numpy/) - [pytest](https://pypi.org/project/pytest/) - Code formatting tools (for contributors) - [Clang-format](https://clang.llvm.org/docs/ClangFormat.html) - [Git-clang-format](https://github.com/llvm-mirror/clang/blob/master/tools/clang-format/git-clang-format) > NOTE: [onnx-tensorrt](https://github.com/onnx/onnx-tensorrt), [cub](http://nvlabs.github.io/cub/), and [protobuf](https://github.com/protocolbuffers/protobuf.git) packages are downloaded along with TensorRT OSS, and not required to be installed. ## Downloading TensorRT Build 1. #### Download TensorRT OSS ```bash git clone -b main https://github.com/nvidia/TensorRT TensorRT cd TensorRT git submodule update --init --recursive ``` 2. #### (Optional - if not using TensorRT container) Specify the TensorRT GA release build path If using the TensorRT OSS build container, TensorRT libraries are preinstalled under `/usr/lib/x86_64-linux-gnu` and you may skip this step. Else download and extract the TensorRT GA build from [NVIDIA Developer Zone](https://developer.nvidia.com) with the direct links below: - [TensorRT 11.2.1.2 for CUDA 13.3, Linux x86_64](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/11.2.1/tars/TensorRT-Enterprise-11.2.1.2-Linux-x86_64-cuda-13.3-Release-external.tar.zst) - [TensorRT 11.2.1.2 for CUDA 12.9, Linux x86_64](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/11.2.1/tars/TensorRT-Enterprise-11.2.1.2-Linux-x86_64-cuda-12.9-Release-external.tar.zst) - [TensorRT 11.2.1.2 for CUDA 13.3, Windows x86_64](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/11.2.1/zip/TensorRT-Enterprise-11.2.1.2-Windows-amd64-cuda-13.3-Release-external.zip) - [TensorRT 11.2.1.2 for CUDA 12.9, Windows x86_64](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/11.2.1/zip/TensorRT-Enterprise-11.2.1.2-Windows-amd64-cuda-12.9-Release-external.zip) **Example: Ubuntu 22.04 on x86-64 with cuda-13.3** ```bash cd ~/Downloads tar --zstd -xvf TensorRT-Enterprise-11.2.1.2-Linux-x86_64-cuda-13.3-Release-external.tar.zst export TRT_LIBPATH=`pwd`/TensorRT-11.2.1.2/lib ``` **Example: Windows on x86-64 with cuda-12.9** ```powershell Expand-Archive -Path TensorRT-Enterprise-11.2.1.2-Windows-amd64-cuda-12.9-Release-external.zip $env:TRT_LIBPATH="$pwd\TensorRT-11.2.1.2\lib" ``` ## Setting Up The Build Environment For Linux platforms, we recommend that you generate a docker container for building TensorRT OSS as described below. For native builds, please install the [prerequisite](#prerequisites) _System Packages_. 1. #### Generate the TensorRT-OSS build container. **Example: Ubuntu 24.04 on x86-64 with cuda-13.3 (default)** ```bash ./docker/build.sh --file docker/ubuntu-24.04.Dockerfile --tag tensorrt-ubuntu24.04-cuda13.3 ``` **Example: Rockylinux8 on x86-64 with cuda-13.3** ```bash ./docker/build.sh --file docker/rockylinux8.Dockerfile --tag tensorrt-rockylinux8-cuda13.3 ``` **Example: Ubuntu 24.04 cross-compile for Jetson (aarch64) with cuda-13.3 (JetPack SDK)** ```bash ./docker/build.sh --file docker/ubuntu-cross-aarch64.Dockerfile --tag tensorrt-jetpack-cuda13.3 ``` **Example: Ubuntu 24.04 on aarch64 with cuda-13.3** ```bash ./docker/build.sh --file docker/ubuntu-24.04-aarch64.Dockerfile --tag tensorrt-aarch64-ubuntu24.04-cuda13.3 ``` 2. #### Launch the TensorRT-OSS build container. **Example: Ubuntu 24.04 build container** ```bash ./docker/launch.sh --tag tensorrt-ubuntu24.04-cuda13.3 --gpus all ``` > NOTE: >
1. Use the `--tag` corresponding to build container generated in Step 1. >
2. [NVIDIA Container Toolkit](#prerequisites) is required for GPU access (running TensorRT applications) inside the build container. >
3. `sudo` password for Ubuntu build containers is 'nvidia'. >
4. Specify port number using `--jupyter ` for launching Jupyter notebooks. >
5. Write permission to this folder is required as this folder will be mounted inside the docker container for uid:gid of 1000:1000. ## Building TensorRT-OSS - Generate Makefiles and build **Example: Linux (x86-64) build with default cuda-13.3** ```bash cd $TRT_OSSPATH mkdir -p build && cd build cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out make -j$(nproc) ``` **Example: Linux (aarch64) build with default cuda-13.3** ```bash cd $TRT_OSSPATH mkdir -p build && cd build cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64-native.toolchain make -j$(nproc) ``` **Example: Native build on Jetson Thor (aarch64) with cuda-13.3** ```bash cd $TRT_OSSPATH mkdir -p build && cd build cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DTRT_PLATFORM_ID=aarch64 CC=/usr/bin/gcc make -j$(nproc) ``` > NOTE: C compiler must be explicitly specified via CC= for native aarch64 builds of protobuf. **Example: Ubuntu 24.04 Cross-Compile for Jetson Thor (aarch64) with cuda-13.3 (JetPack)** ```bash cd $TRT_OSSPATH mkdir -p build && cd build cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64_cross.toolchain make -j$(nproc) ``` **Example: Ubuntu 24.04 Cross-Compile for DriveOS (aarch64) with cuda-13.3** ```bash cd $TRT_OSSPATH mkdir -p build && cd build cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64_dos_cross.toolchain make -j$(nproc) ``` **Example: Native builds on Windows (x86) with cuda-13.3** ```bash cd $TRT_OSSPATH New-Item -ItemType Directory -Path build cd build cmake .. -DTRT_LIB_DIR="$env:TRT_LIBPATH" -DTRT_OUT_DIR="$pwd\\out" msbuild TensorRT.sln /property:Configuration=Release -m:$env:NUMBER_OF_PROCESSORS ``` > NOTE: The default CUDA version used by CMake is 13.3. To override this, for example to 12.9, append `-DCUDA_VERSION=12.9` to the cmake command. - Required CMake build arguments are: - `TRT_LIB_DIR`: Path to the TensorRT installation directory containing libraries. - `TRT_OUT_DIR`: Output directory where generated build artifacts will be copied. - Optional CMake build arguments: - `CMAKE_BUILD_TYPE`: Specify if binaries generated are for release or debug (contain debug symbols). Values consists of [`Release`] | `Debug` - `CUDA_VERSION`: The version of CUDA to target, for example [`12.9.9`]. - `CUDNN_VERSION`: The version of cuDNN to target, for example [`8.9`]. - `PROTOBUF_VERSION`: The ver