[BUG] Tracing: TraceableTcpTransportChannel adds a close listener per inbound request, leaking spans on long-lived transport channels

Describe the bug

With the tracer enabled (telemetry.feature.tracer.enabled=true, telemetry.tracer.enabled=true), every inbound transport request registers a close listener on the underlying TCP channel:

https://github.com/opensearch-project/OpenSearch/blob/0d76e10ad1329f731d1b704fcd4ae7107da850d8/server/src/main/java/org/opensearch/telemetry/tracing/channels/TraceableTcpTransportChannel.java#L54-L74

public static TransportChannel create(TcpTransportChannel delegate, final Span span, final Tracer tracer) {
    if (tracer.isRecording() == true) {
        delegate.getChannel().addCloseListener(new ActionListener<Void>() { ... span.endSpan(); ... });
        return new TraceableTcpTransportChannel(delegate, span, tracer);
    }
    ...

NativeMessageHandler#handleRequest calls create for every request. Inter-node transport connections are long-lived, so the listener never fires and cannot be removed: Netty4TcpChannel#addCloseListener → CompletableContext#addListener → CompletableFuture#whenComplete appends one more node to the callback chain of the channel's close future on every request. Each node retains its Span (with the parent span and the trace/span id strings). Memory grows linearly with the number of requests received over a connection and is only released when the channel closes, which in practice means a node restart.

The leak is independent of sampling: Tracer#isRecording() is true whenever tracing is enabled (DefaultTracer), so non-sampled requests leak a non-recording PropagatedSpan each. The HTTP path (TraceableHttpChannel) does not register a close listener and is not affected, which is why data nodes leak fastest.

Introduced by #10143 (2.11.0); unchanged in every release since, including 3.8.0 and current main (0d76e10ad1329f731d1b704fcd4ae7107da850d8).

Observed

OpenSearch 3.0.0, telemetry.tracer.sampler.probability=0:

• Old generation on data nodes grows linearly with inbound transport traffic (several GB per hour) with zero reclamation, until the heap is nearly full.
• Setting telemetry.tracer.enabled=false dynamically stops the growth immediately; the retained memory is only released by restarting the node.
• Class histogram of a data node (jcmd <pid> GC.class_histogram -all): the per-request leftovers of the close listener all share one instance count, equal to the number of inbound transport requests the node has handled. Sampled spans (SdkSpan) are about 1% of that; the rest are non-recording spans.

  #instances         #bytes  class name
    48401263     1936050520  java.util.concurrent.CompletableFuture$UniWhenComplete
    48398184      774370944  org.opensearch.telemetry.tracing.channels.TraceableTcpTransportChannel$1
    48401263      774420208  org.opensearch.common.concurrent.CompletableContext$$Lambda
    48416188     1161988512  org.opensearch.telemetry.tracing.OTelSpan
    48399691     1161592584  org.opensearch.telemetry.tracing.OTelPropagatedSpan
    96314399     1541030384  io.opentelemetry.api.trace.PropagatedSpan
    96814236     3098055552  io.opentelemetry.api.internal.AutoValue_ImmutableSpanContext
      499838       51983152  io.opentelemetry.sdk.trace.SdkSpan

Related component

Other

To Reproduce

1. Multi-node cluster with tracing enabled (telemetry-otel installed; a sampler probability of 0 is enough).
2. Steady transport traffic, e.g. searches fanned out to data nodes.
3. _nodes/stats/jvm → mem.pools.old.used_in_bytes on data nodes grows without reclamation, and jcmd <pid> GC.class_histogram shows CompletableFuture$UniWhenComplete / TraceableTcpTransportChannel$1 / OTelSpan counts growing with the request count.

Unit level: with a mock TcpChannel and a recording Tracer, calling TraceableTcpTransportChannel.create(...) N times for the same channel invokes addCloseListener N times, so verify(tcpChannel, atMost(1)).addCloseListener(any()) fails on main.

Expected behavior

Per-request state must not be attached to the channel's close future; whatever is registered per channel has to stay bounded regardless of the number of requests served.

Possible fixes, I am happy to submit a PR for whichever maintainers prefer:

1. Drop the per-request close listener. The span is already ended on both response paths (sendResponse(TransportResponse) / sendResponse(Exception)), and when a channel closes mid-request TaskManager#startTrackingCancellableChannelTask cancels cancellable tasks, whose handlers then respond with an exception. Only the "closed without sending the response" event/error mark is lost.
2. One close listener per channel plus an in-flight span registry, following TaskManager#startTrackingCancellableChannelTask: register the listener once per channel, keep the channel's in-flight spans in a set that is cleared on response, and end the remaining spans on close. Keeps the current semantics with a bit more code.

Additional Details

Workaround: PUT _cluster/settings {"persistent":{"telemetry.tracer.enabled":false}} stops the growth; a rolling restart reclaims the heap.

Plugins: telemetry-otel

Host/Environment: OpenSearch 3.0.0 (bundled JDK 21), Linux x86_64