Function finalizer never clears when FissionTenant is deleted before its Functions finish tearing down
Summary
Deleting a FissionTenant while its namespace still has Function CRs with a pending fission.io/function-cleanup finalizer can permanently wedge that namespace in Terminating. The finalizer is never removed because the reconciler responsible for removing it stops receiving events for that namespace the moment the tenant is deleted.
Root cause
MembershipPredicate (pkg/controller/membership.go:40) admits an object only when its namespace is in utils.NamespaceResolver's live tenant set. The executor's shared function reconciler ANDs this predicate onto its watch when the CRD cache is cluster-wide (pkg/executor/funcreconciler/reconciler.go:324-327):
funcPredicate := predicate.Or(predicate.GenerationChangedPredicate{}, deletionTimestampPredicate)
if utils.CrdWatchClusterWide() {
funcPredicate = predicate.And(funcPredicate, controller.MembershipPredicate(utils.DefaultNSResolver()))
}Deleting the FissionTenant CR removes the namespace from the resolver's live set essentially immediately. If a Function in that namespace is deleted around the same time (or was already mid-deletion, still holding its finalizer), the delete event that should let the reconciler run RemoveFinalizer and let deletion proceed is dropped by MembershipPredicate before it ever reaches the reconciler. The Function CR — and therefore the namespace — never finishes deleting.
Repro
- Enable dynamic tenancy (
tenancy.mode=dynamic). - Create a
FissionTenantfor namespacedemo, create the namespace, create aFunctionin it. - Delete the
FissionTenant, then delete the namespace (or delete them close together). kubectl get ns demostaysTerminating;kubectl get functions -n demoshows the Function still present withfinalizers: [fission.io/function-cleanup]and adeletionTimestampset, but the finalizer never clears.
Workaround: manually strip the finalizer — kubectl patch functions.fission.io -n --type=merge -p '{"metadata":{"finalizers":[]}}'
Expected
Tenant/namespace teardown should not orphan in-flight finalizer cleanup for CRs still resident in that namespace — either the tenant controller should block/wait on outstanding finalizers before letting the namespace go, or the function reconciler's membership check should still admit delete-events for CRs that already have a deletionTimestamp.
Environment
- Found while investigating fix/3647 (router EndpointSlice tenant rescope) on a kind cluster,
tenancy.mode=dynamic,router.endpointSliceCache.mode=on. - Not related to #3647 itself — a separate, pre-existing bug (an identically-stuck namespace,
manual-reonboard, already existed on the same cluster before this investigation started).
Source: fission/fission