DRA: no ResourceSliceTemplate definitions for GPU instance types — Karpenter cannot provision nodes for ResourceClaim-only pods
Description
With DRA enabled (IGNORE_DRA_REQUESTS=false), Karpenter schedules DRA pods correctly onto GPU nodes that already exist, but it will never provision a node for a pod whose GPU requirement is expressed only as a ResourceClaim. The pod stays Pending indefinitely and the provisioner logs:
could not schedule pod Pod={"name":"...","namespace":"..."}
error: ... allocating dynamic resources, no instance type can satisfy the allocationThis makes DRA unusable for any elastic GPU workload on AWS: capacity has to be pre-warmed by hand and cannot scale.
Root cause (as I understand it)
kubernetes-sigs/karpenter designs/dra-scheduling.md sources devices from two places:
In-Cluster Devices — built from
ResourceSliceobjects published to the API server. Template devices — "Provided by the cloud provider asResourceSliceTemplateobjects. These represent devices that will exist once an instance type is launched but are not yet published to the API server."
For an in-flight NodeClaim there are no in-cluster slices yet, so provisioning depends entirely on template devices. ResourceSliceTemplate appears in core only in the fake cloudprovider and the kwok driver; I could not find an implementation in this repository, so InstanceType.ResourceSlices() returns nothing for every EC2 instance type and the allocator concludes no instance type can satisfy the claim.
The core tracking issue (kubernetes-sigs/karpenter#3228) marks base DRA support (KEP-4381) Complete and explicitly scopes this half to the cloud provider:
Karpenter will provide primitives for cloudprovider implementations to express which resources will be published for arbitrary instance types […] It is up to individual cloudprovider implementations to provide these definitions for their respective instance type catalog.
I could not find an issue tracking that work here, hence this one.
Steps to reproduce
- EKS 1.36,
karpenter-provider-awsv1.14.0,IGNORE_DRA_REQUESTS=false. nvidia-dra-driver-gpuv25.12.0 installed, classic device plugin off on the target pool.- A NodePool restricted to
g4dnwith a label/taint the DRA kubelet plugin selects on. - Deploy a pod with no
nvidia.com/gpulimit, aResourceClaimTemplateagainstdeviceClassName: gpu.nvidia.com, andresources.claimson the container.
Observed
- Pool empty → pod
Pendingforever, no NodeClaim created,"no instance type can satisfy the allocation". - Seed the pool with an ordinary pod (no ResourceClaim) so a
g4dn.xlargelaunches and the kubelet plugin publishes itsResourceSlice→ the same DRA pod schedules immediately and the claim reachesallocated,reserved.nvidia-smi -Linside the container reports the T4 via CDI injection.
So allocation against published slices works; only the "what will this instance type publish" half is missing.
Expected
Karpenter provisions a GPU node for a pod whose only device requirement is a ResourceClaim, the same way it does today for nvidia.com/gpu extended resources.
Workaround
Keep a "ballast" pod (no ResourceClaim, karpenter.sh/do-not-disrupt: "true") pinned to the DRA NodePool so at least one node exists and publishes slices. Node count then has to be managed by scaling that ballast Deployment; DRA workloads are capped at the pre-warmed GPU count.
Related: kubernetes-sigs/karpenter#3228 (DRA integration tracking issue).
Versions
- karpenter-provider-aws v1.14.0 (core v1.14.0, commit 85eeae8)
- Kubernetes v1.36.2-eks
- nvcr.io/nvidia/k8s-dra-driver-gpu v25.12.0
- Instance type g4dn.xlarge (1× Tesla T4)
Source: aws/karpenter-provider-aws