k8s-onprem autoscaling metric avg_time_queue_us under-reports queue time by roughly the number of loaded models

Description

deploy/k8s-onprem/values.yaml defines the prometheus-adapter rule behind the avg_time_queue_us HPA metric:

metricsQuery: 'avg(delta(nv_inference_queue_duration_us{<<.LabelMatchers>>}[30s])/(1+delta(nv_inference_request_success{<<.LabelMatchers>>}[30s]))) by (<<.GroupBy>>)'

The README says this scales "based on the average queue time". It does not compute that. Two independent errors push the value down, and both delay scale-up.

1. avg(...) by (pod) takes an unweighted mean over per-model ratios. nv_inference_queue_duration_us and nv_inference_request_success are per-model counters, so the inner division yields one series per (model, version) on the pod. A model with no traffic contributes a ratio of exactly 0 and counts as an equal term. A pod that loads N models and serves traffic to one reports roughly 1/N of the true queue wait.

2. 1 + delta(...) divides the accumulated queue time by n+1 requests instead of n. At low request counts the error is large: 2 requests in the window reports 2/3 of the true mean wait.

Both under-report, so the HPA scales up later than the configured target implies, and the reported number stays plausible.

This is the default deployment shape. Triton loads every model in --model-repository, which is what the chart passes. The README's own quickstart repository holds two models, so a stock install halves the metric.

Triton Information

Not specific to a server version. The defect is in the chart configuration at deploy/k8s-onprem/values.yaml, present on main as of 26.08. Both counters are per-model in every release that exposes them.

To Reproduce

The HPA supplies only a namespace matcher and a pod-name matcher, so <<.LabelMatchers>> expands to namespace="default",pod=~"triton-a" and <<.GroupBy>> to pod. Label filters written in seriesQuery are not carried across: prometheus-adapter uses seriesQuery for discovery only, and builds LabelMatchers from the namespace, the selected resource names, and any HPA metric selector (pkg/naming/metrics_query.go, metricsQuery.Build).

Evaluate the expanded query with promtool test rules. One pod, two models, traffic to one: 100 requests per 30 s, each waiting 1000 us.

tests:
  - interval: 15s
    input_series:
      - series: 'nv_inference_queue_duration_us{namespace="default",pod="triton-a",model="densenet_onnx",version="1"}'
        values: '0+50000x40'
      - series: 'nv_inference_request_success{namespace="default",pod="triton-a",model="densenet_onnx",version="1"}'
        values: '0+50x40'
      - series: 'nv_inference_queue_duration_us{namespace="default",pod="triton-a",model="inception_onnx",version="1"}'
        values: '0+0x40'
      - series: 'nv_inference_request_success{namespace="default",pod="triton-a",model="inception_onnx",version="1"}'
        values: '0+0x40'
    promql_expr_test:
      - expr: 'avg(delta(nv_inference_queue_duration_us{namespace="default",pod=~"triton-a"}[30s])/(1+delta(nv_inference_request_success{namespace="default",pod=~"triton-a"}[30s]))) by (pod)'
        eval_time: 5m
        exp_samples:
          - labels: '{pod="triton-a"}'
            value: 1000

The test fails. The rule returns 495.05 against a true mean wait of 1000 us. Adding a third idle model drops it to 330.03.

Expected behavior

The metric should report the mean time a request spent queued on that pod, across the models it serves. Sum both counters over the pod before dividing:

metricsQuery: 'sum(increase(nv_inference_queue_duration_us{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>) / clamp_min(sum(increase(nv_inference_request_success{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>), 1)'

On the input above this returns 1000.

If the intent is to scale when any single model backs up, take the worst model rather than the mean. This also returns 1000 on that input:

metricsQuery: 'max(increase(nv_inference_queue_duration_us{<<.LabelMatchers>>}[30s]) / clamp_min(increase(nv_inference_request_success{<<.LabelMatchers>>}[30s]), 1)) by (<<.GroupBy>>)'

Two changes apply to either form. clamp_min(x, 1) replaces 1 + x: it guards the same division by zero without shifting the result. increase replaces delta, because the Prometheus documentation states delta "should only be used with gauges" while increase adjusts for "counter resets due to target restarts" — and these are counters that reset when Triton restarts. The recording rules under Deployment/Kubernetes/TensorRT-LLM_Autoscaling_and_Load_Balancing in triton-inference-server/tutorials already use clamp_min this way.

What was verified, and what was not

The expansion of <<.LabelMatchers>> comes from reading pkg/naming/metrics_query.go and pkg/naming/metric_namer.go in kubernetes-sigs/prometheus-adapter. The numbers come from evaluating the expanded queries with promtool 3.14.0, not from a deployed HPA on a live cluster. The delta versus increase point rests on the documented function semantics rather than on a test.

Separately, while this file is open: seriesQuery hardcodes namespace="default", so the metric is never discovered when the chart is installed into another namespace. That looks like the cause of #6247.