[Bug] 1.19.0: background reconciliation deletes and re-creates all synchronize:true GeneratingPolicy downstreams every hour (fixed by #16920 in 1.19.1, but not mentioned in the release notes)
Kyverno Version
1.19.0 (fixed in 1.19.1)
Kubernetes Version
1.35 and 1.36
Kubernetes Platform
Other (managed Kubernetes)
Kyverno Rule Type
Generate
Description
After upgrading to 1.19.0 we noticed that a GeneratingPolicy with evaluation.synchronize.enabled: true (the usual "copy a registry Secret into every namespace" pattern) had all of its downstream Secrets deleted and re-created once per hour, without anything touching the trigger, the source or the policy. The downstream UIDs changed on every tick, and between the delete and the re-create there was a ~2s window where pods scheduled at that moment failed to pull. We also run a ValidatingPolicy that blocks deletion of these copies, so every tick produced a burst of 30-50 blocked-delete PolicyViolation events, which is how we noticed at all.
What I traced in the 1.19.0 source: the background controller's periodic reconciliation (forceReconciliation, interval from the BACKGROUND_SCAN_INTERVAL env, default 1h, counted from container start) requeues every policy, syncPolicy calls createURForGeneratingPolicy, and the UpdateRequest it creates did not carry the policy's synchronize setting. Processing therefore took the non-sync path and called DeleteDownstreams against the API before regenerating. #16920 (cherry-picked to 1.19.1 as #17446) fixes this by propagating synchronize into the UR. On 1.19.1 the same tick only logs invalidating downstream resources for policy from metadata cache and the downstream UIDs stay the same. We have had 0 re-creations in the ~14h since upgrading, versus one every hour before.
So this is already fixed. I'm filing it anyway for two reasons:
- The v1.19.1 release notes describe the fix only as "propagate synchronize option to UpdateRequests". Nothing tells a 1.19.0 user that their synchronized downstreams are being deleted and re-created every hour. One line in the release notes (or a note on the GeneratingPolicy docs page) along the lines of "1.19.0 periodically deletes and re-creates synchronized downstream resources, upgrade to 1.19.1" would save people a lot of debugging.
- It is the same family as #17452 (no-op UPDATE deletes the downstream), but this variant needs no trigger event at all, so it hits every 1.19.0 installation that uses
synchronize: true.
Steps to reproduce
- Install Kyverno 1.19.0 (chart 3.9.0).
- Apply a GeneratingPolicy with
synchronize.enabled: trueandgenerateExisting: truethat copies a Secret from one namespace into all namespaces. - Record
metadata.uid/creationTimestampof the copies. - Wait for the next reconciliation tick (setting
BACKGROUND_SCAN_INTERVAL=10mon the background controller shortens the wait). - Background controller logs show
Resource deletedfor every copy followed byResource added; the copies now have new UIDs.
Expected behavior
Periodic reconciliation updates downstreams in place (which is what 1.19.1 does), and the release notes call out the 1.19.0 behavior so people know to upgrade.
Kyverno logs
1.19.0, one tick (policy and secret names changed):
deleting downstream resources for policy policyName=sync-registry-secret
Resource deleted logger=WatchManager name=registry-secret (x32, one per namespace)
Resource added logger=WatchManager name=registry-secret (x32)1.19.1, same tick:
invalidating downstream resources for policy from metadata cache policyName=sync-registry-secretTroubleshooting
- I have read and followed the documentation AND troubleshooting guide.
- I have searched other issues in this repository and mine is not recorded.
Source: kyverno/kyverno