Datadog tracer ignores Ingress/Egress — all spans get `span.kind:internal` (client/server metrics gone)

Title: Datadog tracer ignores Ingress/Egress — all spans get span.kind:internal (client/server metrics gone)

Description:

After upgrading from Istio 1.17 (OpenTracing-based Datadog tracer) to Istio 1.30.2 (Envoy 1.38.x / envoy.tracers.datadog via dd-trace-cpp), Datadog APM metrics for Envoy proxy spans stopped splitting on span.kind:client and span.kind:server. All volume moved to span.kind:internal.

Observed (prod, kube_cluster_name=prod-use1-eks1):

Client/server series drop to zero at the cutover boundary; internal appears at the same time. Spans are still emitted correctly (operation_name=envoy.proxy, component=proxy, language=cpp), and direction is still visible via upstream_cluster (inbound|… vs outbound|…). Only the span.kind taxonomy is missing.

Expected: Envoy should set Datadog span.kind from traffic direction:

• Tracing::OperationName::Ingress → span.kind=server
• Tracing::OperationName::Egress → span.kind=client

Suspected root cause: in source/extensions/tracers/datadog/tracer.cc (Tracer::startSpan) and span.cc (Span::spawnChild), the driver explicitly ignores Tracing::Config:

// The OpenTracing implementation ignored the `Tracing::Config` argument,
// so we will as well.

Tracing::Config::operationName() is therefore never mapped onto the dd-trace-cpp span. With no span.kind tag, Datadog treats the span as internal.

Minimal fix — stop ignoring config and, after creating the span, set:

switch (config.operationName()) {
case Tracing::OperationName::Ingress:
  span.set_tag("span.kind", "server");
  break;
case Tracing::OperationName::Egress:
  span.set_tag("span.kind", "client");
  break;
}

(same in spawnChild), plus unit coverage that Ingress/Egress produce the corresponding span.kind meta.

This looks like a regression class from the OpenTracing → dd-trace-cpp rewrite (#26284), similar to the naming fallout fixed in #30235 / #31366. Happy to open a PR if maintainers agree on the approach. Please consider backport to the Envoy 1.38.x line (what Istio 1.30 ships).

cc @xlamorlette-datadog @zacharycmontoya @mattklein123

Repro steps:

1. Run Envoy 1.38.x (or an Istio 1.30.x sidecar) with the Datadog tracer enabled, e.g. via Istio Telemetry:

   apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: mesh-logging namespace: istio-system spec: tracing: - providers: - name: datadog randomSamplingPercentage: 100

   with meshConfig.extensionProviders pointing at a Datadog Agent on port 8126 (or equivalent bare-Envoy envoy.tracers.datadog config).

2. Generate both inbound (ingress) and outbound (egress) HTTP traffic.

3. In Datadog, graph: sum:trace.envoy.proxy.hits{*} by {span.kind}.as_count()

4. Observe only span.kind:internal. Inspect individual operation_name:envoy.proxy spans: no span.kind tag, while upstream_cluster still distinguishes inbound| vs outbound|. Contrast with Envoy ≤1.26, where the same metric split on client/server.

Admin and Stats Output:

Envoy version from live sidecars (Istio 1.30.2):

envoy_34ce4554c88740f88bfc4dc826ae33e1dd21d8fc/1.38.3-dev/clean/release/boringssl

Full /stats and /clusters omitted — not required to observe the tagging bug; can attach sanitized admin output if useful.

Config:

Istio mesh Datadog provider (sanitized):

meshConfig:
  extensionProviders:
    - name: datadog
      datadog:
        service: dd-agent.example.svc.cluster.local
        port: 8126

Tracing driven by the Telemetry API (above). No meshConfig.defaultConfig.tracing set.

Logs:

N/A — spans export to the Datadog Agent successfully; the defect is missing span.kind on exported spans, not delivery failure.

Call Stack:

N/A (not a crash).