Karpenter Node drift towards smaller instances with Version 1.14
Description
Observed Behavior:
After upgrading from Karpenter 1.13 to Karpenter 1.14 the cluster drifted towards smaller instances. Since we have a fixed daemonset configuration this lead to higher costs. We were able to mitigate this behaviour by excluding the smaller instance sizes for the NodePool. In our case the large and xlarge types on AWS. After applying the mitigation the cluster costs/usage returned to pre Karpenter 1.14 values.
Expected Behavior:
Selection of nodes does not prefer less efficient size of node.
Reproduction Steps (Please include YAML):
I am not sure how to reproduce this. Here is the (cleaned) NodePool configuration. This config includes the mitigation. Usually it would have large and xlarge as karpenter.k8s.aws/instance-size:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: default
spec:
disruption:
budgets:
- duration: 1m0s
nodes: "1"
schedule: 0 * * * *
- duration: 9m0s
nodes: "0"
schedule: 1 * * * *
- duration: 1m0s
nodes: "1"
schedule: 10 * * * *
- duration: 9m0s
nodes: "0"
schedule: 11 * * * *
- duration: 1m0s
nodes: "1"
schedule: 20 * * * *
- duration: 9m0s
nodes: "0"
schedule: 21 * * * *
- duration: 1m0s
nodes: "1"
schedule: 30 * * * *
- duration: 9m0s
nodes: "0"
schedule: 31 * * * *
- duration: 1m0s
nodes: "1"
schedule: 40 * * * *
- duration: 9m0s
nodes: "0"
schedule: 41 * * * *
- duration: 1m0s
nodes: "1"
schedule: 50 * * * *
- duration: 9m0s
nodes: "0"
schedule: 51 * * * *
consolidateAfter: 1m
consolidationPolicy: WhenEmptyOrUnderutilized
limits:
cpu: 6000
memory: 12000Gi
template:
spec:
expireAfter: 720h
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: kubernetes.io/os
operator: In
values:
- linux
- key: karpenter.sh/capacity-type
operator: In
values:
- on-demand
- key: karpenter.k8s.aws/instance-category
operator: In
values:
- c
- m
- r
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values:
- "5"
- key: karpenter.k8s.aws/instance-size
operator: In
values:
- 2xlarge
- 4xlarge
terminationGracePeriod: 1hWe saw this on a cluster with around 800 cores. Node count was increased by 60% and through the higher ratio of daemonset resources the total core count was increased by a value about 20%.
The cluster (with larger nodes, thus less daemonsets) is running ~1600 pods.
Versions:
- Chart Version: 1.14.0
- Kubernetes Version (
kubectl version): v1.35.4
Let me know if anything is missing or additional information that would be helpful to diagnose this, thanks
- Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Source: aws/karpenter-provider-aws