feat: opentelemetry plugin emits no CLIENT span for the upstream call, breaking APM service maps

Description

The opentelemetry plugin only ever emits a SERVER span for the request. It never emits a CLIENT span for the actual proxy call to the upstream. Confirmed on current master:

$ grep -n "span_kind\." apisix/plugins/opentelemetry.lua
apisix/plugins/opentelemetry.lua:429:        kind = span_kind.server,

span_kind.client isn't used anywhere in the file.

This was originally raised in #13828 by @alice101-dev with a good writeup and screenshots, but never turned into a tracked issue or a PR. I re-checked the root cause against current master myself and found a realistic fix path, so I'm filing it properly.

Evidence

Same app, same collector, same APM backend (HyperDX/ClickStack), only the gateway differs.

Behind APISIX: the service map splits into two disconnected clusters. api-gateway is linked to callers in front of it but not to the services behind it.

Behind Kong (which emits a kong.balancer CLIENT span): one continuous chain end to end.

Reason: APM service maps are built by pairing CLIENT spans with the SERVER spans they call. APISIX's downstream child spans exist and are correctly parented, but there's no CLIENT span for the gateway-to-upstream hop, so there's nothing for tools like Jaeger, Grafana Tempo, or Datadog APM to draw that edge from.

Root cause

apisix/plugins/opentelemetry.lua:428-431 hardcodes kind = span_kind.server for the one span it creates, and there's no second span around the upstream call:

local ctx = tracer:start(upstream_context, span_name, {
    kind = span_kind.server,
    attributes = attributes,
})

The two most recent tracing PRs on this plugin (#12686, #13008) added more phase spans and fixed span nesting, but neither added a CLIENT span for the upstream leg.

apisix/secret.lua:158 already uses tracer.kind.client for outbound secret-manager calls, so the client-kind path through apisix/tracer.lua already works. It's just never used for the upstream proxy hop, which is the one that actually matters for service maps.

Impact

Any APISIX deployment using the opentelemetry plugin with a standard OTel-based APM backend will show the gateway as a disconnected node instead of the hop it actually is. There's no config option to work around it. This directly undercuts one of the main reasons people wire up distributed tracing in front of their services.

Affected components

apisix/plugins/opentelemetry.lua is the scope of this issue.

apisix/plugins/zipkin.lua has the same gap (also only emits a server-kind span) but that's a separate follow-up, not part of this one.

Acceptance criteria

• A CLIENT span is emitted as a child of the existing SERVER span, bounding the upstream call.
• It carries server.address, server.port (from ctx.balancer_ip / ctx.balancer_port in apisix/balancer.lua), and http.response.status_code.
• Span timing reflects real upstream latency, using ngx.var.upstream_connect_time / upstream_response_time (already read elsewhere in apisix/init.lua).
• Existing t/plugin/opentelemetry*.t tests still pass, plus new tests for span presence, parent/child relationship, kind, and attributes.

Solution direction

opentelemetry.lua already has create_child_span(), which takes an arbitrary span.kind plus explicit start_time/end_time, so a CLIENT span can be constructed after the fact from the upstream timing vars instead of needing a live hook inside balancer_by_lua (nginx's actual proxy_pass isn't Lua-driven, so there's no single call to wrap).

Open question for whoever picks this up: should the CLIENT span be created and finished in header_filter or in log. Both have tradeoffs around when the upstream timing vars are fully finalized. That's a design call worth getting maintainer input on before implementing.