RFC-0026 P5 benchmark: provisioned-warm burst shows real p95 tail-latency contention on B (3.85x vs 1.84x p50, gated ratio)

Summary

The new provisioned-warm-starvation benchmark scenario (test/benchmark/pkg/scenario/provisionedwarm.go) measures function B's cold-start latency while function A is eagerly warming a provisioned-concurrency burst on the same generic pool. A local run against a kind cluster (poolmgr, EXECUTOR_PROVISIONED_MAX_INFLIGHT_PER_FUNCTION=4, EXECUTOR_PROVISIONED_RECONCILE_INTERVAL=10s, burst target 20, pool size 30, 10 B iterations per phase) produced:

The median ratio comfortably clears the committed 2.0x gate (test/benchmark/config/thresholds.yaml). The p95 ratio does not — B's worst cold starts during A's burst are nearly 4x slower than B's own no-contention baseline, and this is currently invisible to the gate, which only thresholds the p50 ratio.

Suspected mechanism

Traced through pkg/executor/executortype/poolmgr/gp_pod.go:50 (choosePod): every caller wanting a warm pod from a given environment's pool — including function A's up-to-MaxInflightPerFunction concurrent eager specializations and function B's on-demand request — pulls from one shared readyPodQueue per environment. When two callers race for the queue, a loser can draw a key another caller already claimed (skipped via the pod.Labels["managed"] != "true" check) and has to loop back for another Get(). This is a plausible, code-supported explanation for the tail-latency spike: most of B's requests win the queue cleanly, but a request landing during A's active batch can lose one or more races and pay for retries.

This is real interference between two functions sharing a pool's readiness queue — not a benchmark artifact — and is exactly the kind of thing RFC-0026 invariant P5 ("one function's warm-up burst never regresses another function's on-demand cold-start latency beyond the agreed bound," RFC-0026 lines 93/129) is meant to catch. The existing MaxInflightPerFunction limiter bounds how many pods A eagerly consumes at once, but does not appear to prevent queue-level contention with a concurrent on-demand request during that window.

What's NOT done (deliberately, pending this discussion)

Only contended_ratio (p50) is gated, at a fixed 2.0x ceiling. contended_ratio_p95 is emitted as an informational metric only; no p95 gate has been added, pending team discussion on:

• Whether p95 contention is expected/acceptable behavior (occasional queue-race retries under burst load) or worth addressing at the choosePod/readyPodQueue level.
• If worth gating, what ceiling — this run is a single sample on a single kind node, too small/noisy to set a bound from alone.
• Whether the fix (if any) belongs in choosePod's queue-sharing behavior, in a per-function reservation scheme, or elsewhere.

Repro

cd test/benchmark
go build -o fission-benchmark ./cmd/fission-benchmark
PYTHON_RUNTIME_IMAGE=ghcr.io/fission/python-env \
./fission-benchmark run \
  --config config/scenarios.default.yaml \
  --router-url http://127.0.0.1:8888 \
  --namespace default --fission-namespace fission \
  --out results.json \
  --scenarios provisioned-warm-starvation
./fission-benchmark report --in results.json --thresholds config/thresholds.yaml

Requires a cluster with EXECUTOR_PROVISIONED_CONCURRENCY_ENABLED=true and ENABLE_FUNCTION_SERVICES set on the executor (kind/kind-ci skaffold profiles already do).

References

• RFC: docs/rfc/0026-provisioned-concurrency-scheduled-warming.md, invariant P5 (lines 93, 129)
• Scenario: test/benchmark/pkg/scenario/provisionedwarm.go
• Suspected mechanism: pkg/executor/executortype/poolmgr/gp_pod.go:50 (choosePod, readyPodQueue)