Federation response is assembled entirely in memory before /gpu-metrics replies

What

/gpu-metrics (and /endpoints-metrics) collect every remote context's /federate body, stamp a cluster label on each, then join them into one string and return it. Three of those steps hold the whole payload in memory at once, and two of them run on the metrics server's event loop:

• response.text in send_metrics_request_with_port_forward decodes the entire body bytes→str, on the loop.
• '\n\n'.join(all_metrics) in the route handler builds one string containing every context's exposition, on the loop.
• the response is then encoded again on the way out.

The CPU-heavy label stamping was already moved off the loop (_stamp_cluster_label runs in a worker thread). The decode and the join were not.

Impact

Latency: small. Measured, join + encode is ~30ms for a 169 MB payload split across 37 contexts, and ~300ms at 1 GB. That is well under the scrape timeout and is not a practical source of loop stalls.

Memory: this is the real cost. A single scrape can hold, concurrently: each context's decoded string, its stamped copy, the joined result, and the encoded bytes of that result. That is several multiples of the total federated payload, allocated and released every scrape interval. On a deployment where the per-context p99 payload is ~64 MB, a handful of contexts puts the peak in the hundreds of MB, once a minute. It grows linearly with both payload size and context count, so the failure mode at scale is the pod hitting its memory limit rather than the loop blocking.

Suggested direction

Stream instead of assembling: write each context's stamped output to the response as it completes, rather than accumulating all_metrics and joining. That removes the join and the second encode, bounds peak memory to roughly one context's payload, and incidentally takes the remaining CPU work off the loop.

Not urgent — no current deployment is near a limit because of this. Filing so the constraint is written down before context counts or payload sizes grow.