Executor rejects requests with 429 capacity_exceeded far below the function's concurrency limit (stale activeRequests accounting) on v1.27.0

Environment

• Fission: v1.27.0 (helm chart 1.27.0, all components)
• Kubernetes: v1.34.9
• Function runtime: poolmgr executor, Kata Containers + Firecracker VMs (RuntimeClass kata-fc), requestsPerPod: 1
• Affected function config: concurrency: 200, MaxScale: 200

Summary

Under load, the executor rejects requests with 429 {"component":"executor","reason":"capacity_exceeded"} when the actual number of in-flight requests is far below the function's configured concurrency limit (200). Executor logs show hundreds of consecutive function 'b5c3700b...' concurrency '200' limit reached entries. Deleting the executor leader pod fixes it immediately — the new pod takes the lease with fresh state.

Steps to reproduce

1. Create a CPU-bound function (~6s per request) with concurrency: 200, executor type poolmgr, requestsPerPod: 1.
2. Generate load with client timeouts shorter than the specialization queue time (e.g. 200 concurrent clients, 50–70s client timeout). Many clients disconnect while their requests are still queued or being specialized.
3. Within seconds to minutes, all further requests get 429 capacity_exceeded from the executor, even though far fewer than 200 requests are actually in flight.
4. kubectl logs on the executor leader shows repeated: function '<uid>' concurrency '200' limit reached (we observed ~300 such lines while the real concurrency was < 50).
5. Delete the executor leader pod → new leader takes the lease → requests immediately succeed again.

Load test data (single 30s stage, 150 concurrent clients)

• Result: 120 × client-timeout (000), 60 × 429 capacity_exceeded, 0 × 200
• Executor log: 300 × concurrency '200' limit reached
• ResourceQuota at the time: pods 24/300, requests.cpu 8.6/60, limits.cpu 30/120 — quota was not the bottleneck

Root cause analysis

When a client times out at the router and disconnects, the request's in-flight counter for the function is never decremented. We believe this is the same mechanism already identified on main in PR #3596 (fix(crd): drop ResourceVersion from CacheKeyURG, merged 2026-07-22, not contained in any release):

"when the executor writes a function status update (bumping RV) and the router later calls UnTapService with the stale RV from its informer cache, the key doesn't match the entry created by SetSvcValue — UnTapService misses the cache entry, activeRequests is never decremented, and the idle reaper refuses to delete the pod"

Related symptoms we observe on v1.27.0:

• Specialized pods (managed=false) are never drained after client disconnects; after an executor restart the pods are fully orphaned (accounting was in-memory only) and must be cleaned manually.
• Additional unreleased main fixes in the same area: #3560 (saturation/429 fast-fail), #3618 (MarkFuncDeleted multi-generation), #3652/#3658 (warm pod survival/adoption).

Related issues

• #3194 — "Heavy load causes requests to halt … executor believes they still have active requests" (open) — same symptom class
• #2237 — "Inactivity timeout not applied for specific poolmanager pods" (closed 2021-era, root cause identified: router timeout → no untap → count never decremented)
• #3596 — the fix on main (unreleased)

Request

Since v1.27.0 is the latest release and #3596 fixes this exact leak, we would appreciate either a patch release (v1.27.1) with the fix backported or guidance on a recommended interim workaround besides cycling the executor leader.

Workarounds we currently use

1. Delete the executor leader pod when capacity_exceeded storm starts (fresh accounting on lease acquisition).
2. CronJob that deletes specialized pods older than 15 min, gated on CPU pressure (pods pending on Insufficient cpu) to avoid killing pods serving warm traffic.