Shared helper for preserving client metadata through delay-based components (aggregation, sampling, batching)
Is your feature request related to a problem?
Components that hold telemetry across the incoming-request boundary, anything that buffers, aggregates, samples with a delay, or batches on a timer, lose the association between the emitted data and the originating request's client.Info (metadata / headers / auth). The emit happens on a timer or a flush, detached from any single request, so downstream extensions that read from context (for example headers_setter) and any per-request auth can no longer route or authenticate the output.
This has been solved a few times in different places:
exporterhelperqueue batcherpartition.metadata_keys(#14795)- persistent queue metadata preservation (#13220)
logdedupprocessormetadata_keys(contrib open-telemetry/opentelemetry-collector-contrib#47581)
There are more components that this would be useful for too.
Describe the solution you'd like
A shared helper, in core, so both core and contrib components can use it, for metadata-partitioned buffering:
- A standard config surface consumers expose, eg
metadata_keysandmetadata_cardinality_limit - The helper owns bucketing the held items by the client-metadata tuple, enforcing the cardinality limit, and reconstructing a context that carries the bucket's
client.Infoon flush - Consumers then only need to apply their hold or aggregation logic, interval last-value, dedup count, sampling decision, etc
This mirrors how processorhelper and exporterhelper already provide shared plumbing rather than each component rolling its own.
Open question
Per-item aggregators like the interval and logdedup processors map cleanly to one metadata bucket per output. Trace-level buffering like tail sampling does not. A trace's spans can arrive on different requests with different metadata, so which context the emitted trace should carry can be ambiguous.
I think we should scope the first version to single-source-context output and multi-source support for later - this would likely need a policy (first-seen, or configurable). Feedback wanted on this.
Describe alternatives you've considered
Continue with per-component implementations. It works, but it is now three copies with more coming. This can lead to divergent config names and cardinality handling.
Additional context
The motivating use case is converting low-volume logs to gauge metrics with signal_to_metrics and interval-aggregating them. The export needs to carry the originating tenant's key via headers_setter from_context, which the interval processor's timer-based emit currently drops.
Source: open-telemetry/opentelemetry-collector