Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
The Triton backend for Python. The goal of Python backend is to let you serve models written in Python by Triton Inference Server without having to write any C++ code.
docker run --shm-size=1g --ulimit memlock=-1 -p 8000:8000 -p 8001:8001 -p 8002:8002 --ulimit stack=67108864 -ti nvcr.io/nvidia/tritonserver:-py3Replace \ with the Triton version (e.g. 21.05).
git clone https://github.com/triton-inference-server/python_backend -b rcd python_backend
mkdir -p models/add_sub/1/
cp examples/add_sub/model.py models/add_sub/1/model.py
cp examples/add_sub/config.pbtxt models/add_sub/config.pbtxttritonserver --model-repository `pwd`/modelsdocker run -ti --net host nvcr.io/nvidia/tritonserver:-py3-sdk /bin/bashgit clone https://github.com/triton-inference-server/python_backend -b rpython3 python_backend/examples/add_sub/client.pypip3 install numpyOn Ubuntu or Debian you can use the command below to install rapidjson, libarchive, and zlib:
sudo apt-get install rapidjson-dev libarchive-dev zlib1g-devmkdir build
cd build
cmake -DTRITON_ENABLE_GPU=ON -DTRITON_BACKEND_REPO_TAG= -DTRITON_COMMON_REPO_TAG= -DTRITON_CORE_REPO_TAG= -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
make installThe following required Triton repositories will be pulled and used in the build. If the CMake variables below are not specified, "main" branch of those repositories will be used. \ should be the same as the Python backend repository branch that you are trying to compile.
-DTRITON_BACKEND_REPO_TAG=-DTRITON_COMMON_REPO_TAG=-DTRITON_CORE_REPO_TAG=Set -DCMAKE_INSTALL_PREFIX to the location where the Triton Server is
installed. In the released containers, this location is /opt/tritonserver.
mkdir -p models/add_sub/1/
cp examples/add_sub/model.py models/add_sub/1/model.py
cp examples/add_sub/config.pbtxt models/add_sub/config.pbtxt/opt/tritonserver/bin/tritonserver --model-repository=`pwd`/modelspython3 examples/add_sub/client.pyIn order to use the Python backend, you need to create a Python file that has a structure similar to below:
…Every Python backend can implement the following main functions:
auto_complete_configauto_complete_config is called only once when loading the model assuming
the server was not started with
--disable-auto-complete-config.
Implementing this function is optional. No implementation of
auto_complete_config will do nothing. This function can be used to set
max_batch_size,
dynamic_batching,
input
and
output
properties of the model using set_max_batch_size, set_dynamic_batching,
add_input, and add_output. These properties will allow Triton to load the
model with
minimal model configuration
in absence of a configuration file. This function returns the
pb_utils.ModelConfig object with these properties. You can use the as_dict
function to gain read-only access to the pb_utils.ModelConfig object.
The pb_utils.ModelConfig object being returned from here will be used as the
final configuration for the model.
In addition to minimal properties, you can also set model_transaction_policy
through auto_complete_config using set_model_transaction_policy.
For example,
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
@staticmethod
def auto_complete_config(auto_complete_model_config):
...
transaction_policy = {"decoupled": True}
auto_complete_model_config.set_model_transaction_policy(transaction_policy)
...Note: The Python interpreter used to invoke this function will be destroyed
upon returning from this function and as a result none of the objects
created here will be available in the initialize, execute, or finalize
functions.
initializeinitialize is called once the model is being loaded. Implementing
initialize is optional. initialize allows you to do any necessary
initializations before execution. In the initialize function, you are given
an args variable. args is a Python dictionary. Both keys and
values for this Python dictionary are strings. You can find the available
keys in the args dictionary along with their description in the table
below:
| key | description |
|---|---|
| model_config | A JSON string containing the model configuration |
| model_instance_kind | A string containing model instance kind |
| model_instance_device_id | A string containing model instance device ID |
| model_repository | Model repository path |
| model_version | Model version |
| model_name | Model name |
executeexecute function is called whenever an inference request is made. Every
Python model must implement execute function. In the execute function you
are given a list of InferenceRequest objects. There are two modes of
implementing this function. The mode you choose should depend on your use case.
That is whether or not you want to return decoupled responses from this model
or not.
This is the most generic way you would like to implement your model and
requires the execute function to return exactly one response per request.
This entails that in this mode, your execute function must return a list of
InferenceResponse objects that has the same length as requests. The work
flow in this mode is:
execute function receives a batch of pb_utils.InferenceRequest as a
length N array.
Perform inference on the pb_utils.InferenceRequest and append the corresponding pb_utils.InferenceResponse to a response list.
Return back the response list.
The length of response list being returned must be N.
Each element in the list should be the response for the corresponding element in the request array.
Each element must contain a response (a response can be either output tensors or an error); an element cannot be None.
Triton checks to ensure that these requirements on response list are satisfied and if not returns an error response for all inference requests. Upon return from the execute function all tensor data associated with the InferenceRequest objects passed to the function are deleted, and so InferenceRequest objects should not be retained by the Python model.
Starting from 24.06, models may choose to send the response using the
InferenceResponseSender as illustrated on Decoupled mode.
Since the model is in default mode, it must send exactly one response per
request. The pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL flag must be sent
either with the response or as a flag only response afterward.
In case one of the requests has an error, you can use the TritonError object
to set the error message for that specific request. Below is an example of
setting errors for an InferenceResponse object:
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
...
def execute(self, requests):
responses = [No open issues yet, or sync has not completed.