Suggest a new scaler: NVIDIA GPU / AI Inference Metrics

Proposal

We're running large-scale AI inference workloads on Kubernetes (vLLM, Triton Inference Server) and the biggest gap we've hit with KEDA is scaling GPU-backed deployments. Standard CPU/memory HPA doesn't work — our GPU nodes sit at 10% CPU while the GPUs are saturated, or we have 200 pending requests in the vLLM queue and HPA has no idea.

Right now we work around this by pointing the Prometheus scaler at DCGM-exported metrics or vLLM's /metrics endpoint, but it's brittle:

• You need to know the exact PromQL query for each serving framework
• Different inference servers expose different metric names (vllm:num_requests_waiting, nv_inference_queue_duration_us, DCGM_FI_DEV_GPU_UTIL)
• There's no standard way to express "scale when GPU utilization > 80% AND request queue > 50"
• Teams copy-paste ScaledObject configs from Stack Overflow and get the thresholds wrong

A dedicated AI inference scaler would give users a clean interface that understands the common metrics these systems expose, without needing to be a PromQL expert.

Scaler Source

NVIDIA DCGM Exporter metrics (GPU utilization, memory, power) and inference server metrics from: - vLLM (/metrics endpoint) - NVIDIA Triton Inference Server - Text Generation Inference (TGI) These all expose Prometheus-compatible metrics endpoints. The scaler would query them directly or via a Prometheus server.

Scaling Mechanics

The scaler would support composite scaling triggers. Example ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
    - type: nvidia-gpu-inference
      metadata:
        # GPU utilization threshold (from DCGM Exporter)
        gpuUtilizationThreshold: "80"
        # Pending requests in serving queue
        pendingRequestsThreshold: "50"
        # Inference server type: vllm | triton | tgi
        serverType: "vllm"
        # Metrics endpoint (direct scrape) or Prometheus server
        metricsEndpoint: "http://vllm-service:8000/metrics"
        # Optional: scale based on KV cache utilization (vLLM specific)
        kvCacheUtilizationThreshold: "90"

Key scaling signals:

• GPU utilization (DCGM_FI_DEV_GPU_UTIL) — most universal, works across all GPU workloads
• Pending/waiting requests (vllm:num_requests_waiting, nv_inference_request_duration_us) — inference-specific queue pressure
• GPU memory utilization (DCGM_FI_DEV_MEM_COPY_UTIL) — important for large model serving where OOM kills pods
• KV cache usage (vLLM-specific) — when the KV cache is full, new requests block

The scaler would support AND/OR logic across these signals, similar to how you'd combine triggers in a ScaledObject today, but with inference-aware defaults.

Authentication Source

For direct metrics scrape: no auth needed (cluster-internal endpoints) or bearer token for secured endpoints. For Prometheus-based: reuse existing KEDA Prometheus authentication (same TriggerAuthentication as the Prometheus scaler).

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

I'm happy to implement this. I've been working on AI infrastructure tooling in the CNCF ecosystem (recently contributed the hf:// and modelscope:// backends to Dragonfly for P2P model distribution) and this is the natural next piece — once models are distributed to nodes, they need to scale properly.

I'm also speaking at HPSF Conference 2026 on "GitOps for HPC" and "DevOps for Scientific Software" — both talks deal directly with scaling Kubernetes for AI/ML and HPC workloads, which is where this gap keeps coming up in practice.

The AI inference scaling problem is only getting worse as more teams deploy LLMs on Kubernetes. Every platform team I've talked to is building their own custom PromQL queries for this. Having it as a first-class KEDA scaler would save a lot of duplicated effort.

If the maintainers prefer, I could also see this working as an add-on scaler (per #7470) rather than a built-in. Happy to discuss the right approach.