VirtualMachinePool scaling activity history
Problem
Currently, when a VirtualMachinePool (vmpool) performs scale-out or scale-in operations, there is no built-in mechanism to record and trace the history of each scaling activity. The existing observability tools have limitations:
status.replicas/status.readyReplicas: Only reflect the current state, not the history of how the pool got there.status.conditions: Only show the current condition (e.g., ReplicaFailure), not a complete record of each scaling operation.- Kubernetes Events: Record individual VM-level operations (SuccessfulCreate, FailedDelete, etc.), but lack an activity-level summary.
What we need
We need to record each scaling activity with the following information:
1)Activity type :ScaleOut or ScaleIn 2)Activity status | Succeeded / PartiallySucceeded / Failed 3)Start time | When the scaling activity was triggered 4)End time | When the scaling activity completed 5)Expected instance count | Planned number of VMs to create or delete 6)Actual instance count | Actual number of VMs successfully created or deleted 7)Description | Result details, including failure reason if applicableAdditional considerations:
- A scaling activity may span multiple reconcile loops (due to
burstReplicaslimiting). We need to track the activity across reconciles. - If
spec.replicasis changed mid-activity (e.g., by HPA or user), the current activity should be interrupted and a new one started.
Approaches we've considered
Event-based: Record a single Event per completed activity (not per VM operation). Track the in-progress activity via
status.currentActivity. Pros: minimal code changes, reuses existingEventRecorder.Status field: Store recent activity records in
status.activityHistory(capped at N records). Pros: no extra resources, watchable. Cons: etcd object size limits prevent full retention.ConfigMap-based: Store activity records in ConfigMaps (monthly sharding + daily keys for date filtering). Pros: full retention, date-range querying. Cons: more complex implementation, separate from CRD.
Separate CRD: Create a
VirtualMachinePoolActivityCRD for each activity record. Pros: full retention, standard k8s querying. Cons: many objects for active pools.
Questions for the community
- Has anyone else needed scaling activity history for vmpools?
- Are there any existing plans or discussions around this?
- What approach would the community recommend?
- Would any of the above approaches be acceptable for upstream contribution?
Source: kubevirt/kubevirt