Random latency regression on L40S nodes after new pod creation — node becomes persistently "bad"

Description

We're serving several Ads GPU models with Triton on AWS g6e instances (NVIDIA L40S), one Triton pod per node. We're seeing a random pattern where a pod scheduled onto a node comes up with elevated p99 latency (~100+ ms vs. a healthy baseline of ~30 ms) and never recovers. The node itself appears to become "bad" — replacement pods on the same node hit the same latency floor, but the node was healthy before some earlier pod transition. Draining and replacing the node is currently our only remediation.

Nsight profiles on a bad node vs. a healthy node running identical workloads show GPU-side execution is fine (kernel durations identical, cudaStreamSynchronize at 1×) while every host-side CUDA API is systematically slower on the bad node (cudaLaunchHostFunc 11.6×, cudaLaunchKernel 1.24×, etc.). CPU utilization on the bad node is also higher than on healthy peers. This points to a host-side CPU/scheduling issue rather than a GPU issue, but we'd like Nvidia's input on whether this pattern has been seen before and what host/driver diagnostics to collect next.

Triton Information

• Triton version: tritonserver:26.05-py3

To Reproduce

We do not have deterministic reproduction steps — the degradation is triggered by pod scheduling on some (not all) nodes, and we cannot force it on demand. The pattern is:

1. Deploy Triton pod (one pod per node) on an AWS g6e instance (L40S GPU) in a Kubernetes cluster.
2. Serve production traffic (ONNX backend) against the pod for hours to days.
3. At some pod transition (pod delete + reschedule onto the same node), the newly created pod comes up with p99 ≈ 100+ ms vs. the ~30 ms baseline. Latency never recovers over the pod's lifetime.
4. Deleting the affected pod does not help — a fresh pod scheduled onto the same node exhibits the same elevated latency. Only draining/replacing the node clears the condition.
5. The same node may serve healthy pods for hours or days before turning "bad." We have not observed a running pod's latency degrade in place — the transition always happens at pod creation.

Model / workload:

• Framework: ONNX (ONNX Runtime backend)

Observations that narrow the problem:

• GPU metrics are clean on bad nodes — utilization, memory, temperature, clocks all in expected ranges, no throttle reasons active.
• CPU utilization is elevated on bad nodes vs. healthy peers running identical workloads.
• Nsight Systems profiles on a bad node and a healthy node (30 s window, same workload):

Headline comparison

Metric	Unhealthy	Healthy	Delta
Wall time captured	30,045 ms	30,053 ms	–
Kernels launched (same window)	2,487,957	2,738,746	−9.2 % throughput
GPU busy time	9,284 ms	10,388 ms	−10.6 %
GPU utilization	30.9 %	34.6 %	−3.7 pp
Avg kernel duration	3.73 µs	3.79 µs	identical
Avg cudaLaunchKernel	6.69 µs	5.40 µs	+24 %
Avg cudaLaunchHostFunc	458.26 µs	39.57 µs	11.6×

Host-side CUDA APIs are systematically slower on the bad node — GPU side is fine

Runtime API	Unhealthy µs	Healthy µs	Ratio
cudaLaunchHostFunc	458.26	39.57	11.58×
cudaMemcpyAsync	9.77	7.20	1.36×
cuStreamGetCaptureInfo	—	—	1.35×
cudaMemsetAsync	8.21	6.48	1.27×
cudaLaunchKernel	6.69	5.40	1.24×
cuLaunchKernel	6.44	5.56	1.16×
cudaLaunchKernelExC	6.39	5.57	1.15×
cuKernelGetName	0.22	—	1.12×
cudaEventRecord	1.74	—	1.04×
cudaStreamSynchronize	4.11	4.09	1×

Expected behavior

Newly scheduled Triton pods on a given node should exhibit the same p99 latency as pods running on other healthy nodes with the same workload (~30 ms). Node condition should not degrade permanently as a result of pod scheduling / teardown activity.

Questions for the Triton team

1. Has this pattern (per-node persistent latency regression triggered by pod churn, GPU-side clean, host-side CUDA APIs uniformly inflated) been reported before on L40S or other cards?
2. What diagnostics would you recommend collecting on a "bad" node before we cycle it? (e.g. specific DCGM fields, Triton-side profiling flags, CUDA driver state dumps.)
3. Are there known interactions between Triton pod teardown and residual GPU/driver state on the host we should look at?

Thanks!