A large enough Prometheus cannot be federated inside the 30s per-context budget

What

_PER_CONTEXT_TIMEOUT_SECONDS = 30 bounds one context's whole federation attempt: port-forward, /federate request, transfer, and cluster-label stamping. A source Prometheus holding enough data cannot finish inside it, and the context is then dropped from that scrape with outcome="timeout".

Observed directly: an endpoint answering the production match patterns with ~169 MB compressed (~1 GB of stamped exposition) takes 37–45s end to end, having spent only ~2.3s on the port-forward. Every attempt against it times out at exactly 30s.

Impact

Low today. Over 7 days on a production deployment the outcome split was:

success  971,046
error    520,778
timeout       25

25 timeouts in a week, and they are concentrated on SSH/login-node (Slurm) contexts rather than the Kubernetes path. Per-context p99 payloads there top out around 64 MB, roughly 15x below the size that provokes this.

So this is a scaling cliff rather than a live defect: the budget is comfortable at current payload sizes and has no margin annotation saying where it stops being comfortable.

Worth noting the 30s budget also has to fit inside the scrape timeout the deployment configures for /gpu-metrics (45s in the bundled chart), so raising it is not free.

Suggested direction

Probably not "raise the number". Options worth considering:

• Make the budget derive from, or at least be validated against, the configured scrape timeout rather than being an independent constant.
• Reduce what has to fit inside it — see the streaming issue; a streamed response removes the stamping and join from the critical path.
• Surface the size explicitly: sky_apiserver_metrics_federation_payload_bytes already exists, so a deployment approaching the cliff is measurable before it falls off it.