Post delete hook health checks

What would you like to be added?

The health checking for resources has been reworked to use kstatus, which is great as it adds wider support for resources. Currently health checking is partially applied to post delete hooks, meaning that it only runs for pods and jobs.

I would like to expand the health checking in hooks to support more kinds of resources.

Why is this needed?

I think my use case may be a bit unique, but I do not think I am the only one who would need this. My project Spegel creates a daemonset and each pod changes the hosts Containerd mirror configuration, so that it works with Spegel. The old configuration is persisted, so that during uninstall I can revert it back. For this to be possible I need to run a pod on every node during the post uninstall hook. Once its done the daemonset should be removed.

https://github.com/spegel-org/spegel/blob/main/charts/spegel/templates/post-delete-hook.yaml

On its own this does not currently work, as Helm considers the daemonset healthy right after apply, causing it to be deleted. Instead of waiting for all pods to be healthy. To solve this currently I have an additional pod which implements the waiting logic that kstatus would otherwise implement. Waiting for all daemonset pods are ready before reporting a healthy state.

I think that there are more projects out there that would want to run a daemonset as part of a uninstall so that they can also revert configuration. The current complexity is probably what is stopping them from implementing it.