Baike.dev
Log in
Back to tool/Back to issues
#141290·Kubernetes

[Bug] Device taint NoExecute eviction does not evict pods using DRA-backed extended-resource claims (pod.Status.ExtendedResourceClaimStatus)

Author: yogeshbendreCreated Aug 10, 2026Updated Sep 17, 2026
Labelskind/bugsig/schedulingsig/nodepriority/important-longtermtriage/acceptedwg/device-management

What happened?

TL;DR: the device-taint eviction controller (pkg/controller/devicetainteviction) only discovers which claims a pod is using by reading pod.Spec.ResourceClaims. A pod that got its device through the newer Extended Resource Backed by DRA path (KEP-5004, DRAExtendedResource) has its claim reference in pod.Status.ExtendedResourceClaimStatus instead — a field this controller never looks at. Both DRADeviceTaints and DRAExtendedResource are Beta and default-on since v1.36, so this affects any cluster running a stock 1.36+ apiserver/controller-manager with a DRA driver installed — no unusual configuration required.

Live-reproduced on a kind cluster (full steps and output below): two pods, each holding one device from the same driver, then a single NoExecute DeviceTaintRule applied against that driver:

Pod Requests its device via Claim reference lives in Result after NoExecute taint
pod-to-be-evicted ResourceClaimTemplate pod.spec.resourceClaims Evicted within ~10s (correct)
pod0 plain resources.limits extended-resource request pod.status.extendedResourceClaimStatus Still Running, untouched — the bug

The DeviceTaintRule's own status confirms only one pod was ever considered: "1 pod needs to be evicted in 1 namespace" → "1 pod evicted since starting the controller".

Scope note, to save a maintainer the round-trip: this is specifically about the post-allocation eviction path. Allocation-time taint enforcement (a new pod being blocked from a NoSchedule/NoExecute-tainted device it doesn't tolerate) is unaffected and works correctly for both request styles. The bug is narrowly in the separate, already-allocated-pod eviction controller.

Couple of affected use cases

  • GPU health-triggered auto-remediation. A node-health/GPU-health controller (e.g. something reacting to Xid/ECC errors, similar in spirit to NVIDIA GPU Operator's health checks) detects a failing GPU and applies a NoExecute DeviceTaintRule to force workloads off it before quarantining or repairing the device. Any pod still using the classic nvidia.com/gpu: 1-style request — which is the majority of existing GPU workloads, since KEP-5004's entire purpose is letting them keep working unmodified under DRA — is never evicted, and keeps running on a known-bad GPU instead of being cleanly rescheduled. Worst case: silently corrupted training/inference output rather than a clean failure.
  • Targeted single-device maintenance on a multi-GPU node. An admin wants to service or replace one GPU on a node without draining the whole node (disruptive to the node's other GPUs/workloads), so they taint just that device NoExecute instead of cordoning the node. Any pod holding that device via an extended-resource request keeps running through the "drain," so the device can't actually be safely pulled for maintenance without an admin separately hunting down and manually killing those pods — defeating the point of device-level (vs. node-level) maintenance tooling.
  • Clusters mid-migration from device plugins to DRA. KEP-5004 exists specifically so clusters can adopt DRA drivers without immediately rewriting every pod spec — meaning a long transitional period where legacy extended-resource-requesting Deployments/Jobs and newer ResourceClaim-based workloads coexist on the same devices. During that period, any device-taint-based enforcement (decommissioning a device, isolating a suspect one) only works for the already-migrated minority of workloads, silently failing for everyone else — undermining the mechanism cluster-wide for as long as the migration takes, which for large fleets could be months.

What did you expect to happen?

A NoExecute device taint should evict every pod actually consuming the tainted device, independent of which API surface the pod used to request it. An admin tainting a failing/draining GPU for maintenance has no way to know (and shouldn't need to know) whether a given consumer pod requested it via a ResourceClaim or via a classic resources.limits extended-resource request — both end up running on the same physical device via the same DRA driver and the same underlying ResourceClaim object. Today only one of the two is actually evicted.

How can we reproduce it (as minimally and precisely as possible)?

Reproducible deterministically on a kind cluster with no real GPU hardware, using kubernetes-sigs/dra-example-driver (a maintained, public example DRA driver that simulates GPUs). All manifests below are the driver's own unmodified example manifests (demo/examples/device-taints-tolerations/device-taint-pod-noexecute/ and demo/examples/extended-resource-request/), just applied side by side.

Prerequisites: Docker, kind v0.20+, helm v3.7+, kubectl.

# 1. Create a kind cluster with the required feature gates and API version.
#    (DynamicResourceAllocation and DRAAdminAccess are already GA/locked-on by v1.36 and are
#    omitted; DRADeviceTaints/DRAExtendedResource are Beta+default-on in 1.36 but included
#    explicitly for clarity; DRADeviceTaintRules is Beta+default-OFF in 1.36 and must be set.)
cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  DRADeviceTaints: true
  DRAExtendedResource: true
  DRADeviceTaintRules: true
runtimeConfig:
  resource.k8s.io/v1beta2: "true"
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
nodes:
- role: control-plane
- role: worker
EOF
kind create cluster --name dra-repro --image kindest/node:v1.36.1 --config kind-config.yaml --wait 3m

# 2. Install the example DRA driver (8 simulated GPUs, no real hardware, default values).
git clone --depth 1 --branch v0.4.0 https://github.com/kubernetes-sigs/dra-example-driver.git
helm upgrade -i --create-namespace --namespace dra-example-driver dra-example-driver \
  dra-example-driver/deployments/helm/dra-example-driver
kubectl -n dra-example-driver wait --for=condition=Ready pod -l app.kubernetes.io/component=kubeletplugin --timeout=90s

# 3. Control pod: requests a GPU via a ResourceClaimTemplate (pod.spec.resourceClaims).
kubectl apply -f dra-example-driver/demo/examples/device-taints-tolerations/device-taint-pod-noexecute/1-basic-resourceclaimtemplate.yaml
kubectl apply -f dra-example-driver/demo/examples/device-taints-tolerations/device-taint-pod-noexecute/2-pod-to-be-evicted.yaml
kubectl -n basic-resourceclaimtemplate wait --for=condition=Ready pod/pod-to-be-evicted --timeout=60s

# 4. Treatment pod: requests a GPU via the classic extended-resource field. The scheduler
#    satisfies it via DRA and records the claim in pod.status.extendedResourceClaimStatus instead
#    of pod.spec.resourceClaims (confirmed with the jsonpath check below).
kubectl create namespace extended-resource-request
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  namespace: extended-resource-request
  name: pod0
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      limits:
        deviceclass.resource.kubernetes.io/gpu.example.com: 1
EOF
kubectl -n extended-resource-request wait --for=condition=Ready pod/pod0 --timeout=60s
kubectl -n extended-resource-request get pod pod0 -o jsonpath='{.status.extendedResourceClaimStatus}{"\n"}'
kubectl -n extended-resource-request get pod pod0 -o jsonpath='spec.resourceClaims={.spec.resourceClaims}{"\n"}'

# 5. Taint the whole driver's devices NoExecute — both pods' devices are in scope.
cat <<'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1beta2
kind: DeviceTaintRule
metadata:
  name: example
spec:
  deviceSelector:
    driver: gpu.example.com
  taint:
    key: gpu.example.com/unhealthy
    value: "true"
    effect: NoExecute
EOF

# 6. Observe.
sleep 15
kubectl get devicetaintrule example -o jsonpath='{.status.conditions}{"\n"}'
kubectl -n basic-resourceclaimtemplate get pod pod-to-be-evicted   # expect: NotFound (evicted+deleted)
kubectl -n extended-resource-request get pod pod0 -o wide           # expect: still Running

Actual output, captured just now:

$ kubectl -n extended-resource-request get pod pod0 -o jsonpath='{.status.extendedResourceClaimStatus}'
{"requestMappings":[{"containerName":"ctr0","requestName":"container-0-request-0",
"resourceName":"deviceclass.resource.kubernetes.io/gpu.example.com"}],
"resourceClaimName":"pod0-extended-resources-v77pc"}

$ kubectl -n extended-resource-request get pod pod0 -o jsonpath='spec.resourceClaims={.spec.resourceClaims}'
spec.resourceClaims=

$ kubectl apply -f device-taint-rule.yaml
devicetaintrule.resource.k8s.io/example created

$ kubectl get devicetaintrule example -o jsonpath='{.status.conditions}'
# immediately after creation:
[{"lastTransitionTime":"2026-08-10T07:34:42Z","message":"1 pod needs to be evicted in 1 namespace.",
"observedGeneration":1,"reason":"PodsPendingEviction","status":"True","type":"EvictionInProgress"}]

# ~10s later:
[{"lastTransitionTime":"2026-08-10T07:34:52Z","message":"1 pod evicted since starting the controller.",
"observedGeneration":1,"reason":"Completed","status":"False","type":"EvictionInProgress"}]

$ kubectl -n basic-resourceclaimtemplate get pod pod-to-be-evicted
Error from server (NotFound): pods "pod-to-be-evicted" not found

$ kubectl -n basic-resourceclaimtemplate get events --sort-by=.lastTimestamp | tail -3
28s   Normal   DeviceTaintManagerEviction   pod/pod-to-be-evicted   Marking for deletion
28s   Normal   Killing                      pod/pod-to-be-evicted   Stopping container ctr0

$ kubectl -n extended-resource-request get pod pod0 -o wide
NAME   READY   STATUS    RESTARTS   AGE   NODE
pod0   1/1     Running   0          69s   dra-repro-worker

$ kubectl -n extended-resource-request get events --sort-by=.lastTimestamp
# Scheduled / Pulled / Created / Started only — zero eviction-related events, ever.

$ kubectl get resourceclaim -A
NAMESPACE                   NAME                            STATE
extended-resource-request   pod0-extended-resources-v77pc   allocated,reserved
# (pod-to-be-evicted's generated claim is gone — cleaned up along with the deleted pod)

Only one of the two pods using tainted devices from the same driver was ever evicted. This reproduces every time; there is no timing dependency.

Anything else we need to know?

Where this happens in the code

Traced against tag v1.36.3 and confirmed unchanged at master HEAD 94c136764292cc5fac976c0de6587daaea56410f; live-reproduced above on kindest/node:v1.36.1 (closest publicly available kind node image to v1.36.3 at time of testing — same code, confirmed by reading).

1. The eviction controller enumerates claims from exactly one field.

pkg/controller/devicetainteviction/device_taint_eviction.go, podEvictionTime (current source), is the only place this controller determines which claims a pod uses:

func (tc *Controller) podEvictionTime(pod *v1.Pod) *evictionAndReason {
    if pod.Spec.NodeName == "" {
        return nil
    }
    var eviction *evictionAndReason
    for i := range pod.Spec.ResourceClaims {
        claimName, mustCheckOwner, err := resourceclaim.Name(pod, &pod.Spec.ResourceClaims[i])
        ...
    }
    return eviction
}

Grepping the full 1691-line file for ExtendedResource returns zero matches, on both the tag and master.

2. The sibling resourceclaim controller already had to solve this exact problem.

pkg/controller/resourceclaim/controller.go, enqueuePod:

// Check if pod has any resource claims to process.
// Extended resource claims are stored in pod.Status.ExtendedResourceClaimStatus,
// not in pod.Spec.ResourceClaims, so we need to check both locations.
hasResourceClaims := len(pod.Spec.ResourceClaims) > 0
hasExtendedResourceClaims := pod.Status.ExtendedResourceClaimStatus != nil
...
// Without this, extended resource claims would never be cleaned up when
// pods complete, causing device resources to remain allocated indefinitely.
if hasExtendedResourceClaims {
    claimName := pod.Status.ExtendedResourceClaimStatus.ResourceClaimName
    ...
}

The device-taint eviction controller never received the equivalent treatment.

3. Both contributing feature gates are Beta and default-on as of this release (pkg/features/kube_features.go):

DRADeviceTaints: {
    {Version: version.MustParse("1.33"), Default: false, PreRelease: featuregate.Alpha},
    {Version: version.MustParse("1.36"), Default: true, PreRelease: featuregate.Beta},
},
DRAExtendedResource: {
    {Version: version.MustParse("1.34"), Default: false, PreRelease: featuregate.Alpha},
    {Version: version.MustParse("1.36"), Default: true, PreRelease: featuregate.Beta},
},

So this isn't an edge-case opt-in combination — it's the default state of a stock 1.36+ cluster with any DRA driver and DeviceTaintRule usage.

4. No test coverage exercises this interaction. test/integration/dra/device_taints.go and pkg/controller/devicetainteviction/device_taint_eviction_test.go (2926 lines) both have zero hits for ExtendedResource.

5. Allocation-time taint enforcement is unaffected (see "Scope note" above) — staging/src/k8s.io/dynamic-resource-allocation/structured/internal/stable/allocator_stable.go, taintPreventsAllocation, is reached identically for extended-resource and explicit-claim requests, since extendeddynamicresources.go builds an in-memory claim for the extended-resource path before Filter runs. Included here to preempt the natural "does this mean extended-resource claims ignore taints entirely?" question — they don't; only post-allocation NoExecute eviction is blind.

Suggested fix direction

Mirror the resourceclaim controller's dual-location check into podEvictionTime — i.e. also resolve pod.Status.ExtendedResourceClaimStatus.ResourceClaimName when present, using the same claim lookup/ownership logic already used for pod.Spec.ResourceClaims. This looks like a contained, well-scoped fix; happy to attempt a PR once a maintainer confirms this is the intended approach (rather than, say, deprecating the implicit extended-resource path's exemption from eviction on purpose — which seems unlikely given point 2 above, but worth confirming).

Kubernetes version

Live reproduction (above): `kindest/node:v1.36.1`, vanilla upstream via `kind`. Code paths involved confirmed identical at tag `v1.36.3` (latest stable at time of this report) and master HEAD `94c136764292cc5fac976c0de6587daaea56410f` by direct source comparison.

Cloud provider

None — plain `kind` (Kubernetes-in-Docker) cluster.

OS version

Whatever the `kind` node image ships (containerized, not relevant to this bug).

Install tools

`kind` v0.32.0, Helm v3.7+ (for the example driver chart).

Container runtime (CRI) and version (if applicable)

containerd (as shipped in the `kind` node image), with `enable_cdi = true` (required by the example DRA driver to advertise its simulated devices via CDI).

Related plugins (CNI, CSI, ...) and versions (if applicable)

[kubernetes-sigs/dra-example-driver](https://github.com/kubernetes-sigs/dra-example-driver) v0.4.0 — a maintained, public example/reference DRA driver used purely to simulate GPU devices with no real hardware dependency. Not specific to this driver: the bug is entirely in core `kube-controller-manager` code and would reproduce identically with any DRA driver.

Source: kubernetes/kubernetes

View original on GitHubView discussion on GitHub