DRA: Allocator reads device counter consumption from possibly modified slice
What happened?
/kind bug /sig scheduling /sig node /wg device-management
The structured allocator derives per-pool state about already-allocated devices from the current ResourceSlice contents. Slices are mutable, so a driver can republish a pool without a previously-allocated device, or with different ConsumesCounters for it - and when that happens, the allocated device's contribution silently disappears from the allocator's accounting. This affects two features with different severity:
1. PartitionableDevices: counter availability can be over-committed.
checkAvailableCounters (staging/src/k8s.io/dynamic-resource-allocation/structured/internal/experimental/allocator_experimental.go, mirrored in the incubating and stable allocators) computes per-pool availability by starting from the counter set totals and subtracting the consumption of allocated devices found in the current slices. An allocated device that is no longer published (or whose ConsumesCounters changed) no longer subtracts anything, so the pool appears to have more counters free than it physically does, and new allocations can over-commit the shared counters of a device that is still in use.
2. DRADeviceCompatibilityGroups (https://github.com/kubernetes/kubernetes/pull/139795): the enforcement baseline loses allocated peers.
compatibilityGroupsBaselineForPool deliberately copies the counters pattern (its comment: "exactly like checkAvailableCounters does for counters"): group values come from the allocation-time snapshot in the claim status, but counter-set membership still comes from the current slice. If the allocated device is not in the republished slice, its intersection contribution drops out of the feature-on baseline and a later incompatible device can be admitted onto the counter set. Notably, the feature-off version-skew skip (groupedCounterSetsFromState) is built purely from snapshots and keeps protecting the same counter set - the enforcement path is less conservative than the skip path.
What did you expect to happen?
Accounting and enforcement for already-allocated devices should depend only on allocation-time state, never on the current slice contents, which the compatibility-groups KEP already designates as unsafe for enforcement (that is the snapshot's entire purpose).
How can we reproduce it (as minimally and precisely as possible)?
diff --git a/staging/src/k8s.io/dynamic-resource-allocation/structured/internal/allocatortesting/allocator_testing.go b/staging/src/k8s.io/dynamic-resource-allocation/structured/internal/allocatortesting/allocator_testing.go
index 0812be3023c..505c6fe5df9 100644
--- a/staging/src/k8s.io/dynamic-resource-allocation/structured/internal/allocatortesting/allocator_testing.go
+++ b/staging/src/k8s.io/dynamic-resource-allocation/structured/internal/allocatortesting/allocator_testing.go
@@ -3539,6 +3539,46 @@ func TestAllocator(t *testing.T,
deviceAllocationResult(req1, driverA, pool1, device3, false),
)},
},
+ // device1 is allocated, but the driver has republished the pool
+ // without it. How much it consumed is recorded nowhere: the slice no
+ // longer declares the device, and the allocated state carries only
+ // its ID - there is no allocation-time record of counter
+ // consumption. The premise of this scenario is that device1 consumed
+ // the entire 8Gi counter when it was allocated, so granting device2
+ // (which needs all 8Gi) over-commits the physical counter. The
+ // allocator must not hand out counters it cannot account for; the
+ // allocated DeviceID names this pool, so the unaccounted device is
+ // detectable even without a consumption record.
+ "partitionable-devices-allocated-device-removed-from-slice": {
+ features: Features{
+ PartitionableDevices: true,
+ },
+ claimsToAllocate: objects(claimWithRequest(claim0, req0, classA)),
+ allocatedDevices: []DeviceID{
+ MakeDeviceID(driverA, pool1, device1),
+ },
+ classes: objects(class(classA, driverA)),
+ slices: unwrapResourceSlices(
+ sliceWithDevices(slice1, node1, resourcePool(pool1, 2), driverA,
+ device(device2, fromCounters, nil).withDeviceCounterConsumption(
+ deviceCounterConsumption(counterSet1,
+ map[string]resource.Quantity{
+ "memory": resource.MustParse("8Gi"),
+ },
+ ),
+ ),
+ ),
+ sliceWithCounterSets(slice2, node1, resourcePool(pool1, 2), driverA,
+ counterSet(counterSet1,
+ map[string]resource.Quantity{
+ "memory": resource.MustParse("8Gi"),
+ },
+ ),
+ ),
+ ),
+ node: node(node1, region1),
+ expectResults: nil,
+ },
"partitionable-devices-multiple-capacity-pools": {
features: Features{
PrioritizedList: true,
go test ./staging/src/k8s.io/dynamic-resource-allocation/structured/internal/... -run 'TestAllocator/partitionable-devices-allocated-device-removed-from-slice' -count=1--- FAIL: TestAllocator (0.01s)
FAIL k8s.io/dynamic-resource-allocation/structured/internal/experimental 0.632s
--- FAIL: TestAllocator (0.01s)
FAIL k8s.io/dynamic-resource-allocation/structured/internal/incubating 1.068s
--- FAIL: TestAllocator (0.01s)
FAIL k8s.io/dynamic-resource-allocation/structured/internal/stable 1.562sAnything else we need to know?
This was identified during development of #139795 as a pre-existing pattern, hence this dedicated issue to address it for both PartitionableDevices and CompatibilityGroups, as the latter depends on the former.
This opens the door for possibly additional scenarios the allocator probably does not handle properly, like mutated counters in slice.
Kubernetes version
master (1.37)
Cloud provider
None
OS version
None
Install tools
None
Container runtime (CRI) and version (if applicable)
None
Related plugins (CNI, CSI, ...) and versions (if applicable)
DRA scheduling plugin
Source: kubernetes/kubernetes