OTLP metrics: start_time and flag partition the series, so a counter restart forks it

What happened

OTLP metrics ingestion writes start_time (from start_time_unix_nano) and flag (from flags) as ordinary columns on every data point, and both columns feed the series hash. Neither is a dimension of the series, so any OTLP producer that legitimately changes them forks one logical series into several.

The common case is a cumulative counter whose process restarts. OpenTelemetry requires a cumulative sum to carry the start time of its aggregation window, so a restart moves start_time for the same stream. OpenObserve then treats the post-restart samples as a different series.

On a production instance collecting YugabyteDB metrics through an OTel collector, one counter had accumulated 45 distinct start_time values:

count(count by (start_time) (count_over_time(rocksdb_number_db_seek[1h])))   # => 11
count(count by (start_time) (count_over_time(rocksdb_number_db_seek[24h])))  # => 45
count(count by (start_time) (count_over_time(rocksdb_number_db_seek[7d])))   # => 45

The same metric scraped from the same database nodes into Prometheus/Thanos stays a single series per stream, because Prometheus' OTLP receiver treats start_time_unix_nano as counter-reset metadata (the created timestamp) and never as an identifying label.

Why it happens

start_time and flag are pushed into the label set that produces the series hash, on both ingestion paths:

• columnar path — otlp.rs#L813-L816, where they are appended to scratch and the result is hashed by signature_of_label_pairs
• JSON path — otlp.rs#L925-L926 for number points, and the same pair again for histogram (L939), exponential histogram (L997) and summary (L1053) points

METRICS_HASH_EXCLUDED_LABELS (promql/mod.rs#L91) already excludes is_monotonic, which is exactly the same class of OTLP metadata, but not start_time or flag.

Being inside the hash is what splits the series rather than merely adding a label. The query path attaches labels once per hash and skips any hash it has already labelled (series_loader/labels.rs#L246-L266), so an excluded column collapses into one series and takes a representative value, while an included one becomes a separate series per distinct value.

Impact

• rate() and increase() silently lose the interval spanning a restart, because the samples either side now sit in different series.
• Series counts inflate over time. Each restart permanently adds a series for the retention window.
• sum by (...) and topk(...) return several entries for what is one real stream, which is how this surfaced: a topk(5, sum(...) by (universe_uuid)) panel showed a universe repeatedly.
• The cost is proportional to restart frequency, so the noisiest workloads are the worst affected.

Expected behaviour

A change in start_time or flag should not change series identity. Both columns can keep being written and stay queryable; they just should not partition the series — the treatment is_monotonic already gets.

Steps to reproduce

1. Send an OTLP cumulative sum for a stream, with some start_time_unix_nano.
2. Send more points for the same stream with the same attributes and a different start_time_unix_nano.
3. Query the metric. Two series come back rather than one, and count by (start_time) shows one group per distinct start time.

Environment

• OpenObserve v0.92.2, commit e5804678dbbfdf34f23b0aa911bea547f96e732c
• Metrics ingested over OTLP HTTP from an OpenTelemetry collector (opentelemetry-collector-contrib 0.145.0)
• Confirmed present on main at 06754ac5b by inspection

Proposed fix

Add start_time and flag to METRICS_HASH_EXCLUDED_LABELS. The columns continue to be stored and remain queryable, and the sidecar metrics index already skips excluded labels when it builds (metrics_index/src/writer.rs#L68), so no schema or index change is needed. Happy to send a PR.