Incorrect ephemeral storage capacity in NodeClaim

Author: chaosun-abnormalsecurityCreated Mar 21, 2025Updated Sep 10, 2026

Labelsbugtriage/needs-investigation

Description

Observed Behavior: We use a custom AMI in one of our environments, which has the following block devices:

/dev/sda1=snap-072c0faac36719073:20:true:gp2:encrypted
/dev/sdb=snap-085429bd12a18879a:50:true:gp3:encrypted
/dev/sdc=ephemeral1

And we configure our EC2NodeClass to override the first one to be:

yaml

  blockDeviceMappings:
  - deviceName: /dev/sda1
    ebs:
      deleteOnTermination: true
      encrypted: true
      volumeSize: 400Gi
      volumeType: gp3

When the nodes are provisioned, we see correct ephemeral storage capacity which is inline with the configs if we run kubectl describe node:

Capacity:
  cpu:                32
  ephemeral-storage:  406250916Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  hugepages-32Mi:     0
  hugepages-64Ki:     0
  memory:             64655124Ki
  pods:               30
Allocatable:
  cpu:                31850m
  ephemeral-storage:  405202340Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  hugepages-32Mi:     0
  hugepages-64Ki:     0
  memory:             62578357018
  pods:               30

However, when checking the same for the NodeClaim (kubectl describe nodeclaim), it shows different values:

Status:
  Allocatable:
    Cpu:                        31850m
    Ephemeral - Storage:        17Gi
    Memory:                     62605389168
    Pods:                       30
    vpc.amazonaws.com/pod-eni:  54
  Capacity:
    Cpu:                        32
    Ephemeral - Storage:        20Gi
    Memory:                     64655116Ki
    Pods:                       30
    vpc.amazonaws.com/pod-eni:  54

Because of this, our pods that declare ephemeral-storage requests/limits in container resources cannot get scheduled

{"error":"incompatible with nodepool \"test\", daemonset overhead={\"cpu\":\"800m\",\"memory\":\"1524435456\",\"pods\":\"7\"}, no instance type satisfied resources {\"cpu\":\"3300m\",\"ephemeral-storage\":\"50Gi\",\"memory\":\"6893144576\",\"pods\":\"8\"} and requirements dedicated In [test], eks.amazonaws.com/capacityType In [ON_DEMAND], karpenter.k8s.aws/ec2nodeclass In [test], karpenter.k8s.aws/instance-cpu In [32], karpenter.k8s.aws/instance-family In [c6g c7g], karpenter.sh/capacity-type In [on-demand], karpenter.sh/nodepool In [test], kubernetes.io/arch In [arm64], topology.kubernetes.io/zone In [us-gov-west-1a us-gov-west-1b us-gov-west-1c] (no instance type has enough resources)"}]}

In comparison, we use EKS official AMI in another cluster with a different block device config:

/dev/xvda=snap-0522165275ca54b4b:20:true:gp2

And it has no such issue. We are able to see consistent ephemeral storage capacity between node and NodeClaim.

The only difference I can tell between the two AMI’s is the block device of EKS official AMI is not encrypted. Not sure if this is relevant though, or how we should modify our configuration to fix the inconsistency if it is.

Expected Behavior: Consistent ephemeral storage capacity between node and NodeClaim for custom AMI.

Reproduction Steps (Please include YAML): See the details above.

Versions:

Chart Version: 1.2.1
Kubernetes Version (kubectl version): 1.30

Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Source: aws/karpenter-provider-aws

View original on GitHub View discussion on GitHub