#3660·fission

Function finalizer never clears when FissionTenant is deleted before its Functions finish tearing down

Author: rijojohn85Created Aug 4, 2026Updated Aug 20, 2026
Labelsbugarea-executorarea-multitenancypriority/medium

Summary

Deleting a FissionTenant while its namespace still has Function CRs with a pending fission.io/function-cleanup finalizer can permanently wedge that namespace in Terminating. The finalizer is never removed because the reconciler responsible for removing it stops receiving events for that namespace the moment the tenant is deleted.

Root cause

MembershipPredicate (pkg/controller/membership.go:40) admits an object only when its namespace is in utils.NamespaceResolver's live tenant set. The executor's shared function reconciler ANDs this predicate onto its watch when the CRD cache is cluster-wide (pkg/executor/funcreconciler/reconciler.go:324-327):

funcPredicate := predicate.Or(predicate.GenerationChangedPredicate{}, deletionTimestampPredicate)
if utils.CrdWatchClusterWide() {
    funcPredicate = predicate.And(funcPredicate, controller.MembershipPredicate(utils.DefaultNSResolver()))
}

Deleting the FissionTenant CR removes the namespace from the resolver's live set essentially immediately. If a Function in that namespace is deleted around the same time (or was already mid-deletion, still holding its finalizer), the delete event that should let the reconciler run RemoveFinalizer and let deletion proceed is dropped by MembershipPredicate before it ever reaches the reconciler. The Function CR — and therefore the namespace — never finishes deleting.

Repro

  1. Enable dynamic tenancy (tenancy.mode=dynamic).
  2. Create a FissionTenant for namespace demo, create the namespace, create a Function in it.
  3. Delete the FissionTenant, then delete the namespace (or delete them close together).
  4. kubectl get ns demo stays Terminating; kubectl get functions -n demo shows the Function still present with finalizers: [fission.io/function-cleanup] and a deletionTimestamp set, but the finalizer never clears.

Workaround: manually strip the finalizer — kubectl patch functions.fission.io -n --type=merge -p '{"metadata":{"finalizers":[]}}'

Expected

Tenant/namespace teardown should not orphan in-flight finalizer cleanup for CRs still resident in that namespace — either the tenant controller should block/wait on outstanding finalizers before letting the namespace go, or the function reconciler's membership check should still admit delete-events for CRs that already have a deletionTimestamp.

Environment

  • Found while investigating fix/3647 (router EndpointSlice tenant rescope) on a kind cluster, tenancy.mode=dynamic, router.endpointSliceCache.mode=on.
  • Not related to #3647 itself — a separate, pre-existing bug (an identically-stuck namespace, manual-reonboard, already existed on the same cluster before this investigation started).