externalgrpc: MaxNodeStartupTime has no proto field, so every provider implementing NodeGroupGetOptions silently gets 0

Which component are you using?:

/area cluster-autoscaler

cluster-autoscaler, specifically cloudprovider/externalgrpc.

What version of the component are you using?:

Component version: reproduced on cluster-autoscaler-1.35.2 and cluster-autoscaler-1.36.1, and still present on master. 1.34.x is not affected, because the field did not exist yet.

What k8s version are you using (kubectl version)?:

The Kubernetes version does not matter here. The bug is internal to cluster-autoscaler and reproduces on any cluster.

What environment is this in?:

Self-managed clusters running an in-house CloudProvider gRPC service via --cloud-provider=externalgrpc.

What did you expect to happen?:

When an external gRPC cloud provider implements NodeGroupGetOptions, it should still get MaxNodeStartupTime from cluster-autoscaler's node group defaults (--max-node-startup-time, 15m), because the proto gives it no way to send a value of its own. That is what happens for a provider that returns Unimplemented.

What happened instead?:

MaxNodeStartupTime is always 0 for externalgrpc providers on 1.35 and later.

1.35 added MaxNodeStartupTime to config.NodeGroupAutoscalingOptions, but nothing added a matching field to cloudprovider/externalgrpc/protos/externalgrpc.proto. On master the NodeGroupAutoscalingOptions message still carries only fields 1, 2, 6, 7, 8, 9 and 10:

double scaleDownUtilizationThreshold = 1;
double scaleDownGpuUtilizationThreshold = 2;
bool zeroOrMaxNodeScaling = 6;
bool ignoreDaemonSetsUtilization = 7;
google.protobuf.Duration scaleDownUnneededDuration = 8;
google.protobuf.Duration scaleDownUnreadyDuration = 9;
google.protobuf.Duration MaxNodeProvisionDuration = 10;

With no field to read, GetOptions in cloudprovider/externalgrpc/externalgrpc_node_group.go builds the options struct without it and leaves the zero value:

opts := &config.NodeGroupAutoscalingOptions{
	ScaleDownUtilizationThreshold:    pbOpts.GetScaleDownUtilizationThreshold(),
	ScaleDownGpuUtilizationThreshold: pbOpts.GetScaleDownGpuUtilizationThreshold(),
	ScaleDownUnneededTime:            scaleDownUnneededTime,
	ScaleDownUnreadyTime:             scaleDownUnreadyTime,
	MaxNodeProvisionTime:             maxNodeProvisionTime,
	ZeroOrMaxNodeScaling:             pbOpts.GetZeroOrMaxNodeScaling(),
	IgnoreDaemonSetsUtilization:      pbOpts.GetIgnoreDaemonSetsUtilization(),
	// MaxNodeStartupTime is never set
}
return opts, nil

Cluster-autoscaler does send its defaults to the provider in the request, but it never merges them back into the response. DelegatingNodeGroupConfigProcessor reads them only when the provider returns nil or ErrNotImplemented (processors/nodegroupconfig/node_group_config_processor.go):

func (p *DelegatingNodeGroupConfigProcessor) GetMaxNodeStartupTime(nodeGroup cloudprovider.NodeGroup) (time.Duration, error) {
	ngConfig, err := nodeGroup.GetOptions(p.nodeGroupDefaults)
	...
	if ngConfig == nil || err == cloudprovider.ErrNotImplemented {
		return p.nodeGroupDefaults.MaxNodeStartupTime, nil   // 15m
	}
	return ngConfig.MaxNodeStartupTime, nil                  // 0
}

Any provider that implements the RPC, for any option at all, takes the second branch and gets 0.

clusterstate.go is where that lands. It loads the 15m package default, then overwrites it with the per-node-group value whenever the processor returns no error:

maxNodeStartupTime := MaxNodeStartupTime // 15 * time.Minute
update := func(current Readiness, node *apiv1.Node, nr kube_util.NodeReadiness) Readiness {
	nodeGroup, errNg := csr.cloudProvider.NodeGroupForNode(node)
	if errNg == nil && nodeGroup != nil {
		if startupTime, err := csr.nodeGroupConfigProcessor.GetMaxNodeStartupTime(nodeGroup); err == nil {
			maxNodeStartupTime = startupTime // 0
		}
	}
	...
	} else if node.CreationTimestamp.Time.Add(maxNodeStartupTime).After(currentTime) {
		current.NotStarted = append(current.NotStarted, node.Name)
	} else {
		current.Unready = append(current.Unready, node.Name)

At maxNodeStartupTime = 0, CreationTimestamp.Add(0).After(now) is false for any node created in the past. A node that has registered but has not gone Ready skips the NotStarted bucket and lands in Unready.

That bucket feeds node group and cluster health, scale-up backoff, and the --max-total-unready-percentage and --ok-total-unready-count accounting. So every node from a scale-up counts as unready between registration and readiness. With CNI and CSI init that is often a minute or more, and cluster-autoscaler can call the node group or the cluster unhealthy for that whole window.

How to reproduce it (as minimally and precisely as possible):

1. Run cluster-autoscaler 1.35 or later with --cloud-provider=externalgrpc and -v=5.

2. In the gRPC service, implement NodeGroupGetOptions and return a populated NodeGroupAutoscalingOptions. The fields do not matter, since the message cannot express MaxNodeStartupTime either way:

   go复制

   func (s *server) NodeGroupGetOptions(ctx context.Context, req *protos.NodeGroupAutoscalingOptionsRequest) (*protos.NodeGroupAutoscalingOptionsResponse, error) {
       return &protos.NodeGroupAutoscalingOptionsResponse{
           NodeGroupAutoscalingOptions: &protos.NodeGroupAutoscalingOptions{
               ScaleDownUtilizationThreshold: 0.5,
           },
       }, nil
   }

3. Scale a node group up.

4. The log reads Node <name>: using maxNodeStartupTime = 0s, and the new node shows as Unready rather than NotStarted for the whole interval between registration and readiness.

5. Change the RPC to return status.Error(codes.Unimplemented, "") and repeat. The same log line now reads using maxNodeStartupTime = 15m0s.

Anything else we need to know?:

Returning Unimplemented is the only workaround I found, and it costs every other per-node-group option the provider might want to set.

AllowNonAtomicScaleUpToMax has the same gap. It has been in config.NodeGroupAutoscalingOptions since 1.34 and has never had a proto field or an assignment in GetOptions. It hurts less, because false is the intended default, but it is the same bug.