[FEAT] Support custom OTEL exporter headers for OTLP collector
Author: cheriLCreated Sep 14, 2026Updated Sep 18, 2026
Labelsenhancement
Summary
Hatchet's server-side OTLP export only allows a single Authorization header via SERVER_OTEL_COLLECTOR_AUTH. Many self-hosted users need additional headers (e.g.X-API-Key, X-Tenant) when integrating with managed observability backends like Volcengine, or self-hosted Tempo behind a reverse proxy.
This adds support for arbitrary custom headers on both trace and metrics OTLP exporters.
Proposed Solution
New env var SERVER_OTEL_COLLECTOR_HEADERS (YAML key otel.collectorHeadersString), comma-separated key=value pairs, matching the standard OTel SDK's OTEL_EXPORTER_OTLP_HEADERS:
SERVER_OTEL_COLLECTOR_HEADERS="X-API-Key=foo,X-Tenant=bar"
Behavior
- Empty by default → no change for existing deployments.
- Custom headers merged into the existing header map alongside
Authorization. - Backwards compatible:
SERVER_OTEL_COLLECTOR_AUTHstill works; ifAuthorizationis set via the new var it wins.
Scope
Server/Engine only. Worker SDK can be a follow-up.
Implementation Sketch
- Add
loaderutils.ParseHeadersfork=v,k2=v2parsing. - Add
CollectorHeadersString/CollectorHeaderstoOpenTelemetryConfigFile. - Bind
SERVER_OTEL_COLLECTOR_HEADERSand split at loader time. - Add
CollectorHeaderstoTracerOpts; smallbuildHeaders(opts)helper used byInitTracerandInitMeter. - Thread through the four call sites.
- Unit tests for parser + merge helper.
- Update self-hosting docs.
Happy to take this on once accepted and assigned.
Source: hatchet-dev/hatchet