[Question] Reloading models and latency spikes

Hi! I've been using NVIDIA Triton Inference server at the company I work for and we have a use-case where it is necessary to load new versions of a torchscript model every couple minutes.

The problem is that using the load model API causes significant spikes in response times of inference requests both observed on the client side as well on the server side (verified via the available prometheus metrics), that cause a lot of requests to be cancelled by the client due to timeouts.

My question is: is this intended behaviour? if yes is there a way to not block the traffic to the model that's being reloaded?

Steps to reproduce:

1. Deploy any torchscript model with the config below
2. Simulate sufficient traffic to the model
3. While traffic is being simulated, send a load request
4. The client should observe a significant increase in response times

Related issue: https://github.com/triton-inference-server/server/issues/5757 And yes I tried the --model-load-thread-count as mentioned in the issue above, and it didn't help either :(

My setup is as follows: I'm using the load api (https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/protocol/extension_model_repository.html#load) along with explicit model control and an appropriate version policy (approximate config.pbtxt below)

name: "some_model"
backend: "pytorch"
max_batch_size: 16

version_policy {
    latest: {
        num_versions: 1
    }
}

input [
  {
    name: "in_0"
    data_type: TYPE_FP32
    dims: [ -1 ]
  }
]

output [
  {
    name: "out_0"
    data_type: TYPE_FP32
    dims: [ -1 ]
  }
]

dynamic_batching {}

instance_group [
  {
    count: 4
    kind: KIND_GPU
  }
]

model_warmup [
  {
    name: "random data"
    batch_size: 8
    count: 1
    inputs: {
      key: "input_0"
      value: {
        data_type: TYPE_FP32
        dims: [1024]
        random_data: true
      }
    }
  }
]

The load model request is sent without any additional parameters (like below)

curl -X POST .../v2/repository/models/some_model/load