Support http_headers in AlertmanagerConfig httpConfig
Component(s)
AlertManagerConfig
What is missing? Please describe.
First of all, thanks for #8357, nice to see http_headers support arriving in
the operator!
Alertmanager >= 0.28 supports custom HTTP headers via http_headers in
http_config (e.g. for API-key-based receiver authentication). Since #8357
(v0.89.0) the operator preserves http_headers when it is set in the raw
Alertmanager configuration secret, but the AlertmanagerConfig CRD still has no
corresponding field: as of v0.93.1, httpConfig only exposes authorization,
basicAuth, oauth2, bearerTokenSecret, tlsConfig, proxy settings,
followRedirects and enableHttp2. Any httpHeaders key in an
AlertmanagerConfig CR is pruned by the API server.
This matters for multi-tenant clusters: tenants define their receivers via
namespaced AlertmanagerConfig CRs and need per-tenant header authentication
(e.g. X-Api-Key), which a single global config secret cannot provide.
Proposal: add httpHeaders to the AlertmanagerConfig httpConfig, supporting
static values and secret-backed values via SecretKeySelector (analogous to
bearerTokenSecret), guarded for Alertmanager >= 0.28 like the raw-config path
from #8357.
Related: #8341 (closed by #8357, which covered the raw config secret only).
I'd be happy to work on a PR for this if the maintainers agree with the direction. Guidance on the preferred API shape would be very welcome.
Describe alternatives you've considered.
http_headersin the raw Alertmanager config secret: works since v0.89.0, but it is cluster-global and platform-owned — it cannot carry per-tenant credentials for receivers defined in namespaced AlertmanagerConfig CRs.httpConfig.authorization/bearerTokenSecret: only produce anAuthorizationheader; many webhook endpoints require custom header names likeX-Api-Key.urlSecret: allows embedding a key as a query parameter, but not all endpoints support that and it leaks the credential into URLs/logs.
Environment Information.
Environment
Prometheus-Operator Version: v0.93.1
Source: prometheus-operator/prometheus-operator