Duplicate spans generated by retries

Author: ctlongCreated Sep 4, 2024Updated Sep 21, 2026

LabelsbugStale

Describe the bug

We noticed that an OTel Collector receiving spans from another OTel Collector ended up with duplicate spans.

The following log lines indicated that it's likely due to retries:

2024-09-04T21:49:00.123Z        info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlp/test", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.18:9100: connect: connection refused\"", "interval": "15.65047055s"}
2024-09-04T21:49:00.214Z        info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlp/test", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.18:9100: connect: connection refused\"", "interval": "7.356982694s"}
2024-09-04T21:49:00.455Z        info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlp/test", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.18:9100: connect: connection refused\"", "interval": "10.380445546s"}
2024-09-04T21:49:01.945Z        info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlp/test", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.18:9100: connect: connection refused\"", "interval": "12.878719348s"}
2024-09-04T21:49:02.465Z        info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "otlp/test", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.18:9100: connect: connection refused\"", "interval": "14.638876911s"}

We also corroborated this by increasing the amount of CPU and memory available to the receiving OTel Collector, which decreased the number of retry warnings and decrease the number of duplicates.

Steps to reproduce

Start an OTel Collector in a VM with a traces pipeline consisting of an otlpreceiver and filexporter.
Start another OTel Collector in another VM with a traces pipeline consisting of an otlpreceiver and otlpexporter, where the otlpexporter is configured to forward to the original OTel Collector.
Siege the second OTel Collector with spans, where each span has a different trace ID.

What did you expect to see?

The file output from the original OTel Collector contains less than or equal to the number of spans sent to the other OTel Collector.

What did you see instead?

The file output from the original OTel Collector contains more spans than were sent to the other OTel Collector. Closer examination shows that there are a number of duplicate trace IDs, where the number of duplicates is the same as the difference in the spans received and spans sent.

What version did you use?

v0.104.0

What config did you use?

yaml

---
exporters:
  file/testtraces:
    path: "/tmp/otel-collector-traces.log"
processors:
  batch:
receivers:
  otlp/cf-internal-local:
    protocols:
      grpc:
        endpoint: 0.0.0.0:9100
        tls:
          cert_file: "/var/vcap/jobs/otel-collector/config/certs/otel-collector.crt"
          key_file: "/var/vcap/jobs/otel-collector/config/certs/otel-collector.key"
          min_version: '1.3'
service:
  pipelines:
    traces:
      exporters:
      - file/testtraces
      processors:
      - batch
      receivers:
      - otlp/cf-internal-local
  telemetry:
    metrics:
      address: 127.0.0.1:14830
      level: basic

yaml

---
exporters:
  otlp/test:
    endpoint: 10.0.1.18:9100
    tls:
      insecure_skip_verify: true
processors:
  batch:
receivers:
  otlp/cf-internal-local:
    protocols:
      grpc:
        endpoint: 127.0.0.1:9100
        tls:
          client_ca_file: "/var/vcap/jobs/otel-collector/config/certs/otel-collector-ca.crt"
          cert_file: "/var/vcap/jobs/otel-collector/config/certs/otel-collector.crt"
          key_file: "/var/vcap/jobs/otel-collector/config/certs/otel-collector.key"
          min_version: '1.3'
service:
  pipelines:
    traces:
      exporters:
      - otlp/test
      processors:
      - batch
      receivers:
      - otlp/cf-internal-local
  telemetry:
    metrics:
      address: 127.0.0.1:14830
      level: basic

Environment

Ubuntu 22.04.4 LTS go1.22.6

Additional context

We were deploying OTel Collectors in Cloud Foundry using BOSH.

Source: open-telemetry/opentelemetry-collector

View original on GitHub View discussion on GitHub