The port-forward's 5s startup budget silently drops contexts from a scrape

What

start_svc_port_forward gives kubectl port-forward 5 seconds to print its Forwarding from 127.0.0.1:<port> line:

start_port_forward_timeout = 5

If the line has not appeared by then, the function raises RuntimeError: Port forward failed for service ..., and that context contributes nothing to the scrape. The failure is counted (sky_apiserver_metrics_federation_total{outcome="error"}) but produces no series, so the affected cluster's panels are simply empty for that interval.

Impact

The budget is tighter than it looks. Over 7 days on a production deployment, across 891,993 port-forward attempts:

<= 0.5s    149,937    16.8%
<= 1.0s    276,433    47.8%
<= 2.5s    406,111    93.3%
<= 5.0s     59,354    99.98%   <- the budget
<= 10.0s       157   100.00%   <- over budget, dropped

So ~6.6% of attempts already land in the 2.5–5.0s band, i.e. routine operation sits near the limit rather than comfortably below it, and 157 attempts exceeded it outright and lost their context for that scrape.

Severity is low per occurrence: one context misses one scrape and is back on the next one. It is not a sustained outage. What makes it worth recording is that it is silent in the data — the series just aren't there — and the margin is thinner than the constant suggests.

A kubeconfig using an exec credential plugin makes this much worse, since the plugin's own token fetch happens inside the same 5s window; a cold gke-gcloud-auth-plugin invocation alone was measured at 4.16s. Deployments using static-token kubeconfigs do not pay that, which is why the fleet numbers above are not dominated by it.

Suggested direction

• Make the startup budget a named constant derived from the per-context budget rather than an unrelated literal, so the relationship between the two is visible.
• Consider distinguishing "port-forward never started" from other errors in the outcome label, so this failure mode is separable from an unreachable Prometheus.