GPU-labeled unmanaged node without GPU capacity aborts cluster-wide scale-up

What happened?

In an AKS cluster, an unschedulable CPU workload correctly triggered Cluster Autoscaler processing for an autoscaled CPU node group. The node group was below its configured maximum size, but no automatic scale-up request was issued.

The cluster also contained a separate NVIDIA GPU node group that was not enabled for Cluster Autoscaler. Its nodes were Kubernetes Ready and had the Azure NVIDIA GPU label, but they did not report nvidia.com/gpu in status.capacity or status.allocatable.

While processing the unschedulable CPU workload, Cluster Autoscaler failed when creating the cluster-wide quota tracker. The scale-up loop stopped before evaluating the unrelated autoscaled CPU node group.

All customer and cluster identifiers in this report have been anonymized.

Environment

• Cloud provider: Azure / AKS
• Kubernetes version: v1.35.7
• Cluster Autoscaler version: Managed by AKS and not directly exposed
• Autoscaled CPU node group:
  • Current size: 2
  • Maximum size: 4
• Non-autoscaled GPU node group:
  • Node count: 4
  • VM size: Standard_NV12ads_A10_v5
  • Node condition: Ready=True
  • NVIDIA device plugin was not deployed
  • Node labels included:

accelerator=nvidia
kubernetes.azure.com/accelerator=nvidia

The GPU nodes did not report the following extended resource in either Capacity or Allocatable:

nvidia.com/gpu

Relevant logs

Cluster Autoscaler detected the workload as unschedulable and confirmed that the CPU node group had two ready nodes:

Found 216 pods in the cluster: 215 scheduled, 1 unschedulable

Pod example-namespace/example-pod is unschedulable

newNodes: 0, currentTarget: 2, deallocated: 0,
readinessReady: 2, readinessUnready: 0
for nodeGroup example-cpu-vmss

During the same autoscaler loop, all GPU nodes were treated as having an unready GPU:

Overriding status of node example-gpu-vmss000000, which seems to have unready GPU
Overriding status of node example-gpu-vmss000001, which seems to have unready GPU
Overriding status of node example-gpu-vmss000002, which seems to have unready GPU
Overriding status of node example-gpu-vmss000003, which seems to have unready GPU

Scale-up then stopped with the following error:

Failed to scale up: could not create quotas tracker:
failed to get custom resources:
node without with gpu label, without capacity not belonging to autoscaled node group

The same error was recorded on every autoscaler loop while the workload remained pending. There were no subsequent logs showing node-group selection or a resize request.

Manually increasing the CPU node group size allowed the workload to schedule immediately on the newly added node.

Actual behavior

A node with the cloud provider's GPU label, but without a corresponding GPU value in Allocatable, is processed by the GPU custom-resource processor.

Because this node belongs to a node group that is not managed by Cluster Autoscaler, NodeGroupForNode returns no node group. The GPU processor then returns an error while the cluster-wide quota tracker is being created.

This error terminates the entire scale-up orchestration before any expansion option is evaluated. Consequently, an invalid GPU resource state in a non-autoscaled node group can prevent unrelated autoscaled CPU node groups from scaling.

Manual node pool scaling is not affected.

Expected behavior

An invalid or temporarily incomplete custom-resource state on a node outside an autoscaled node group should not abort scale-up evaluation for every autoscaled node group in the cluster.

Possible behaviors could include:

• Treat the node as unmanaged and use its currently reported resources.
• Ignore the missing GPU custom resource when calculating quota usage for an unmanaged node.
• Exclude the inconsistent node from custom-resource quota accounting and emit a warning.
• Restrict the failure to GPU-related workloads or node groups instead of aborting unrelated CPU scale-up.

At minimum, the condition should generate a clear warning or status signal indicating that cluster-wide automatic scale-up has stopped.

Relevant upstream code path

The observed behavior appears to follow this upstream path.

1. The quota tracker is created before expansion options are calculated

https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.35.0/cluster-autoscaler/core/scaleup/orchestrator/orchestrator.go#L108-L133

2. Quota usage calculation processes the cluster nodes and resolves their node groups

https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.35.0/cluster-autoscaler/resourcequotas/usage.go#L67-L93

3. The GPU processor detects a GPU label without GPU Allocatable capacity

When nodeGroup == nil, it returns the error observed in the logs:

https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.35.0/cluster-autoscaler/processors/customresources/gpu_processor.go#L81-L119

The effective call path appears to be:

ScaleUp
  -> NewQuotasTracker
  -> calculateUsages
  -> NodeGroupForNode returns nil for the unmanaged GPU node
  -> GetNodeResourceTargets
  -> GPU label is present but GPU Allocatable is absent
  -> GPU processor returns an error
  -> quota tracker creation fails
  -> ScaleUp returns before node-group selection

Related upstream work

This appears related to the quota-tracker failure-isolation problem discussed in:

• https://github.com/kubernetes-sigs/cluster-autoscaler/issues/56
• https://github.com/kubernetes-sigs/cluster-autoscaler/pull/103

However, PR #103 appears to address errors returned directly by NodeGroupForNode.

In this Azure scenario, the unmanaged node resolves to:

nodeGroup == nil
error == nil

The GPU processor subsequently returns the fatal custom-resource error. Therefore, the change in PR #103 does not appear to cover this path.

Workaround

Installing the NVIDIA device plugin causes the GPU nodes to advertise nvidia.com/gpu, avoiding this specific inconsistent state.

If a node is not intended to provide GPUs, removing an incorrect or stale GPU label would also avoid the GPU processor path.

These actions correct the node state, but they do not isolate unrelated scale-up operations if a similar inconsistent node state occurs again.

Questions

1. Is this GPU-specific quota tracker failure a known issue?
2. Should the GPU processor treat nodeGroup == nil as an unmanaged node instead of returning a fatal cluster-wide scale-up error?
3. Would maintainers accept a change that falls back to the node's reported resources, or skips GPU quota calculation for this node while emitting a warning?
4. Is there an existing fix or release target that covers this specific path?