#4963·hatchet

[FEAT] Support custom OTEL exporter headers for OTLP collector

Author: cheriLCreated Sep 14, 2026Updated Sep 18, 2026
Labelsenhancement

Summary

Hatchet's server-side OTLP export only allows a single Authorization header via SERVER_OTEL_COLLECTOR_AUTH. Many self-hosted users need additional headers (e.g.X-API-Key, X-Tenant) when integrating with managed observability backends like Volcengine, or self-hosted Tempo behind a reverse proxy.

This adds support for arbitrary custom headers on both trace and metrics OTLP exporters.

Proposed Solution

New env var SERVER_OTEL_COLLECTOR_HEADERS (YAML key otel.collectorHeadersString), comma-separated key=value pairs, matching the standard OTel SDK's OTEL_EXPORTER_OTLP_HEADERS:

SERVER_OTEL_COLLECTOR_HEADERS="X-API-Key=foo,X-Tenant=bar"

Behavior

  • Empty by default → no change for existing deployments.
  • Custom headers merged into the existing header map alongside Authorization.
  • Backwards compatible: SERVER_OTEL_COLLECTOR_AUTH still works; if Authorization is set via the new var it wins.

Scope

Server/Engine only. Worker SDK can be a follow-up.

Implementation Sketch

  1. Add loaderutils.ParseHeaders for k=v,k2=v2 parsing.
  2. Add CollectorHeadersString / CollectorHeaders to OpenTelemetryConfigFile.
  3. Bind SERVER_OTEL_COLLECTOR_HEADERS and split at loader time.
  4. Add CollectorHeaders to TracerOpts; small buildHeaders(opts) helper used by InitTracer and InitMeter.
  5. Thread through the four call sites.
  6. Unit tests for parser + merge helper.
  7. Update self-hosting docs.

Happy to take this on once accepted and assigned.