Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
B

backend

> 编程语言
Open source

Common source, scripts and utilities for creating Triton backends.

377 stars0 likes0 views
WebsiteGitHub

About

Common source, scripts and utilities for creating Triton backends.

Triton Inference Server Backend

A Triton backend is the implementation that executes a model. A backend can be a wrapper around a deep-learning framework, like PyTorch, TensorFlow, TensorRT or ONNX Runtime. Or a backend can be custom C/C++ logic performing any operation (for example, image pre-processing).

This repo contains documentation on Triton backends and also source, scripts and utilities for creating Triton backends. You do not need to use anything provided in this repo to create a Triton backend but you will likely find its contents useful.

Frequently Asked Questions

Full documentation is included below but these shortcuts can help you get started in the right direction.

Where can I ask general questions about Triton and Triton backends?

Be sure to read all the information below as well as the general Triton documentation available in the main server repo. If you don't find your answer there you can ask questions on the main Triton issues page.

Where can I find all the backends that are available for Triton?

Anyone can develop a Triton backend, so it isn't possible for us to know about all available backends. But the Triton project does provide a set of supported backends that are tested and updated with each Triton release.

TensorRT: The TensorRT backend is used to execute TensorRT models. The tensorrt_backend repo contains the source for the backend.

ONNX Runtime: The ONNX Runtime backend is used to execute ONNX models. The onnxruntime_backend repo contains the documentation and source for the backend.

TensorFlow: The TensorFlow backend is used to execute TensorFlow models in both GraphDef and SavedModel formats. The same backend is used to execute both TensorFlow 1 and TensorFlow 2 models. The tensorflow_backend repo contains the documentation and source for the backend.

PyTorch: The PyTorch backend is used to execute PyTorch models in both TorchScript and PyTorch 2.0 formats. The pytorch_backend repo contains the documentation and source for the backend.

OpenVINO: The OpenVINO backend is used to execute OpenVINO models. The openvino_backend repo contains the documentation and source for the backend.

Python: The Python backend allows you to write your model logic in Python. For example, you can use this backend to execute pre/post processing code written in Python, or to execute a PyTorch Python script directly (instead of first converting it to TorchScript and then using the PyTorch backend). The python_backend repo contains the documentation and source for the backend.

DALI: DALI is a collection of highly optimized building blocks and an execution engine that accelerates the pre-processing of the input data for deep learning applications. The DALI backend allows you to execute your DALI pipeline within Triton. The dali_backend repo contains the documentation and source for the backend.

FIL: The FIL (Forest Inference Library) backend is used to execute a variety of tree-based ML models, including XGBoost models, LightGBM models, Scikit-Learn random forest models, and cuML random forest models. The fil_backend repo contains the documentation and source for the backend.

TensorRT-LLM: The TensorRT-LLM backend allows you to serve TensorRT-LLM models with Triton Server. Check out the Triton TRT-LLM user guide for more information. The tensorrtllm_backend repo contains the documentation and source for the backend.

vLLM: The vLLM backend is designed to run supported models on a vLLM engine. This backend depends on python_backend to load and serve models. The vllm_backend repo contains the documentation and source for the backend.

Important Note! Not all the above backends are supported on every platform supported by Triton. Look at the Backend-Platform Support Matrix to learn about the same.

How can I develop my own Triton backend?

First you probably want to ask on the main Triton issues page to make sure you are not duplicating a backend that already exists. Then follow the tutorial to learn how to create your first simple Triton backend and incrementally improve it to add more features. You should also read the complete documentation on Triton backends.

Can I add (or remove) a backend to an existing Triton installation?

Yes. See Backend Shared Library for general information about how the shared library implementing a backend is managed by Triton, and Triton with Unsupported and Custom Backends for documentation on how to add your backend to the released Triton Docker image. For a standard install the globally available backends are in /opt/tritonserver/backends.

What about backends developed using the "legacy custom backend" API.

The legacy custom API is removed from Triton. If you have custom backends that you developed using this older API you must port them to the new Triton Backend API.

Backends

A Triton backend is the implementation that executes a model. A backend can be a wrapper around a deep-learning framework, like PyTorch, TensorFlow, TensorRT, ONNX Runtime or OpenVINO. A backend can also implement any functionality you want as long as it adheres to the backend API. Triton uses this API to send requests to the backend for execution and the backend uses the API to communicate with Triton.

Every model must be associated with a backend. A model's backend is specified in the model's configuration using the backend setting. For using TensorRT backend, the value of this setting should be tensorrt. Similarly, for using PyTorch, ONNX and TensorFlow backends, the backend field should be set to pytorch, onnxruntime or tensorflow respectively. For all other backends, backend must be set to the name of the backend. Some backends may also check the platform setting for categorizing the model, for example, in TensorFlow backend, platform should be set to tensorflow_savedmodel or tensorflow_graphdef according to the model format. Please refer to the specific backend repository on whether platform is used.

Backend Shared Library

Each backend must be implemented as a shared library and the name of the shared library must be libtriton_.so. For example, if the name of the backend is "mybackend", a model indicates that it uses the backend by setting the model configuration 'backend' setting to "mybackend", and Triton looks for libtriton_mybackend.so as the shared library that implements the backend. The tutorial shows examples of how to build your backend logic into the appropriate shared library.

For a model, M that specifies backend B, Triton searches for the backend shared library in the following places, in this order:

  • /M//libtriton_B.so

  • /M/libtriton_B.so

  • /B/libtriton_B.so

Where \ is by default /opt/tritonserver/backends. The --backend-directory flag can be used to override the default.

Typically you will install your backend into the global backend directory. For example, if using Triton Docker images you can follow the instructions in Triton with Unsupported and Custom Backends. Continuing the example of a backend names "mybackend", you would install into the Triton image as:

/opt/
  tritonserver/
    backends/
      mybackend/
        libtriton_mybackend.so
        ... # other files needed by mybackend

Starting from 24.01, the default backend shared library name can be changed by providing the runtime setting in the model configuration. For example,

runtime: "my_backend_shared_library_name.so"

A model may choose a specific runtime implementation provided by the backend.

Triton Backend API

A Triton backend must implement the C interface defined in tritonbackend.h. The following abstractions are used by the API.

TRITONBACKEND_Backend

A TRITONBACKEND_Backend object represents the backend itself. The same backend object is shared across all models that use the backend. The associated API, like TRITONBACKEND_BackendName, is used to get information about the backend and to associate a user-defined state with the backend.

A backend can optionally implement TRITONBACKEND_Initialize and TRITONBACKEND_Finalize to get notification of when the backend object is created and destroyed (for more information see backend lifecycles).

TRITONBACKEND_Model

A TRITONBACKEND_Model object represents a model. Each model loaded by Triton is associated with a TRITONBACKEND_Model. Each model can use the TRITONBACKEND_ModelBackend API to get the backend object representing the backend that is used by the model.

The same model object is shared across all instances of that model. The associated API, like TRITONBACKEND_ModelName, is used to get information about the model and to associate a user-defined state with the model.

Most backends will implement TRITONBACKEND_ModelInitialize and TRITONBACKEND_ModelFinalize to initialize the backend for a given model and to manage the user-defined state associated with the model (for more information see backend lifecycles).

The backend must take into account threading concerns when implementing TRITONBACKEND_ModelInitialize and TRITONBACKEND_ModelFinalize. Triton will not perform multiple simultaneous calls to these functions for a given model; however, if a backend is used by multiple models Triton may simultaneously call the functions with a different thread for each model. As a result, the backend must be able to handle multiple simultaneous calls to the functions. Best practice for backend implementations is to use only function-local and model-specific user-defined state in these functions, as is shown in the tutorial.

TRITONBACKEND_ModelInstance

A TRITONBACKEND_ModelInstance object represents a model instance. Triton creates one or more instances of the model based on the instance_group settings specified in the model configuration. Each of these instances is associated with a TRITONBACKEND_ModelInstance object.

The only function that the backend must implem

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

C++

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言