Reverse proxy does not emit OpenTelemetry client spans for upstream requests
When using Caddy’s reverse proxy with OpenTelemetry tracing enabled, we’ve noticed that no client spans are created for upstream requests. While the baggage headers are correctly propagated to the upstream servers, the HTTP client calls made by the reverse proxy itself are not instrumented.
This leads to two major issues:
Missing service graph edges Tools like Grafana Tempo’s service graphs rely on both client and server spans to establish relationships. Without client spans from Caddy, the edges between services are missing and Caddy doesn't show up as a distinct entity in service graphs.
Lack of visibility into upstream responses Currently, the only spans available are handler spans that represent requests to Caddy. Any details about the actual upstream request/response are hidden and we only see the handler results.
Expected behavior
Caddy should create OpenTelemetry client spans for each upstream request made by the reverse proxy handler, in addition to propagating baggage headers.
Actual behavior
No client spans are emitted; only the handler span for the inbound request is visible.
Potential solution
The HTTP transport used by the reverse proxy needs to be instrumented to emit client spans. Some open questions:
- Should this be opt-in, e.g. via a property in the reverse proxy directive?
- Or should it be enabled automatically whenever a traced handler performs a reverse proxy?
Assistance Disclosure
AI used
If AI was used, describe the extent to which it was used.
ChatGPT helped me structure my ramblings into a coherent text.
Source: caddyserver/caddy