[KEP-3521]: Integrate well with out-of-tree resource quota solutions

Context

"KEP-3521: Pod Scheduling Readiness" had graduated to GA in 1.30. It allows gating the scheduling of a pod, hence delaying it until the gate is removed.

This feature is useful for multiple use-cases, one of them being to allow out-of-tree quota management solutions to integrate well with Kubernetes. In fact, in a post in the Kubernetes blog the outlined use-case is the following:

An important use case this feature enables is dynamic quota management. Kubernetes supports ResourceQuota, however the API Server enforces quota at the time you attempt Pod creation. ... Scheduling gates allows an external quota manager to address the above limitation of ResourceQuota.

In addition, in the "What problem does it solve?" section, it writes:

Clearing the gates is the responsibility of external controllers with knowledge of when the Pod should be considered for scheduling (e.g., a quota manager).

As can be seen from the above, the Pod Scheduling Readiness feature was specifically designed to allow out-of-tree quota management solutions to integrate well with Kubernetes.

The problem

The In-place Update of Pod Resources KEP allows to modify pod's resources without re-scheduling them. While this feature is absolutely useful, it removes the assumption in which every pod that uses new resources would be scheduled.

The current behavior contradicts one of the main goals of Pod Scheduling Readiness, which is a GAed feature, and is effectively breaking existing out-of-tree quota solutions.

Possible solutions

Idea 1: Introduce a resize gate, which operates similarly to a scheduling gate, meaning it will gate resizes until removed. This will allow external quota managements to become aware of the resize request, being able to perform custom logic.

Idea 2: Re-use the current scheduling gates as resize gates. In other words, delay resize requests until the scheduling gate is removed. (As far as I remember, scheduling gates are immutable, which could challenge this approach).

Idea 3: ??

Effected projects

I'm aware of two projects that are effected by this, being

• Kueue: https://github.com/kubernetes-sigs/kueue.
• AAQ (Application Aware Quota): https://github.com/kubevirt/application-aware-quota.

Related discussions

Issue opened to Kueue: https://github.com/kubernetes-sigs/kueue/issues/5257. Old issue about the same subject: https://github.com/kubernetes/enhancements/issues/4304.