Dynamic tenancy: ConfigMap/Secret pod-recycle (RFC-0004) doesn't follow runtime-onboarded namespaces

Summary

In dynamic multi-namespace tenancy mode (tenancy.mode: dynamic), the executor's RFC-0004 "recycle a function's pods when a mounted ConfigMap/Secret changes" behavior does not apply to namespaces onboarded at runtime. Functions in such namespaces still work; only the automatic recycle-on-config-change is missing.

This is a known, deliberately-deferred refinement (Phase 4b of the multi-namespace tenancy effort). Filing to track it — pick up if multiple users hit it.

Scope / impact

• Only tenancy.mode: dynamic. static (the default) and cluster modes are unaffected — see below.
• Functions are not broken. A function in a runtime-onboarded namespace specializes and serves correctly: the fetcher reads its ConfigMaps/Secrets directly, and the recycle action (RefreshFuncPods) reads referenced config via the API. The only gap is the trigger: a change to a referenced ConfigMap/Secret does not auto-recycle the function's pods, so the function keeps serving the old config until it cycles for another reason.
• Workaround: trigger a recycle manually — e.g. fission fn update (or re-create the function), or delete the function's pods — to pick up the new config.

Root cause

The executor's controller-runtime manager cache scopes the Secret/ConfigMap watch to the env-seeded namespace set (defaultNamespace + additionalFissionNamespaces) in dynamic mode:

pkg/executor/start.go — executorCacheOptions(), the DynamicNamespacesEnabled() branch sets byObject[Secret/ConfigMap].Namespaces = nsConfig (the env-seeded set).

This is deliberate and security-load-bearing: a cluster-wide Secret cache would mirror every Secret in the cluster into the executor's memory and require cluster-wide secret-read RBAC — the single biggest isolation regression the tenancy design avoids (PRD §4.1). So Secrets/ConfigMaps stay namespace-scoped.

The pkg/executor/cms ConfigMapReconciler/SecretReconciler watch through that cache, so a Secret/ConfigMap change in a namespace onboarded at runtime (a FissionTenant namespace not in the env-seed) is never observed → no recycle.

• cluster mode does not have this gap (its Secret/ConfigMap cache is cluster-wide by the documented trusted-cluster trade-off).
• static mode does not have this gap (the env-seed is the whole world).

Proposed solution (designed, deferred)

Add a dynamic-mode-only, leader-only "Tier-B watch manager" in the executor that watches Secrets/ConfigMaps in runtime-onboarded namespaces and reuses the existing recycle logic. Additive — the existing manager-cache reconcilers (env-seed / static / cluster) are untouched; the new component covers exactly the delta {FissionTenant namespaces} − {env-seed}.

• A manager.Runnable (leader-only, so recycle writes don't double-fire across replicas), constructed only when tenancy.mode = dynamic.
• Driven by FissionTenant changes; maintains a map namespace → cancelFunc of per-namespace client-go SharedInformerFactory instances (Secrets + ConfigMaps), each started with its own context.WithCancel(rootCtx).
• On onboard → start a namespace-scoped factory; on offboard → cancel() (reflectors stop on ctx.Done() → clean teardown, no goroutine/apiserver-watch leak).
• The Secret/ConfigMap handler fires only on a ResourceVersion-changing Update (mirroring cms.contentChangedPredicate; no Add/Delete — otherwise the initial informer list would recycle every pod on onboard) and calls the existing getConfigmapRelatedFuncs/getSecretRelatedFuncs + refreshPods.

Why client-go informers, not a controller-runtime cluster.Cluster (the PRD's literal wording): controller-runtime (v0.24.x) can add a watch source to a running controller but cannot cleanly remove one, so a per-namespace cluster.Cluster would leak a dead source reference on every offboard. A per-namespace client-go SharedInformerFactory with its own cancellable context has clean, independent teardown and reuses the (already cache-independent) recycle helpers directly.

RBAC: none new — in dynamic mode the executor already holds per-namespace secrets/configmaps get/list/watch in each onboarded namespace (the tenant controller binds fission-executor to fission-executor-tenant-workload, whose executor-kuberules include those verbs).

Key risks & mitigations: informer leak on offboard (per-namespace cancellable context + an active-watch gauge); recycle-storm on onboard (RV-changed UpdateFunc filter only); double-recycle (covered strictly by the tenantNS − envSeed partition); leadership flap (sub-contexts are children of the leader context → cancelled on leadership loss).

Why deferred

It is a refinement, not a blocker — functions work; only convenience auto-recycle lags in runtime-onboarded namespaces, with a simple manual workaround. It is also the highest-risk new code in the design (dynamic informer lifecycle). Worth doing if it's reported by real users; otherwise low priority.