`pyroscope_segment_writer_client_sent_bytes` metric has unbounded cardinality

Describe the bug

The cardinality of the pyroscope_segment_writer_client_sent_bytes metric can grow without bounds. The total number of metrics emitted by the /metrics endpoint requires ever-increasing amounts of memory to process from standard collectors (like otel-collector).

The histogram metric has dimensions on shard, tenant, and addr.

There is no code for clearing / deleting old tracked series from the metric.

As downstream segment-writer Pods restart, they can get a new IP address, creating new series for addr. The old series under the now-unused IP address linger.

If tenants are spread across many shards and/or adaptive placement is making significant changes to things, we can also see orphaned series as a result.

To Reproduce

Default behavior. Restart some segment-writer Pods and watch series count of pyroscope_segment_writer_client_sent_bytes grow.

Expected behavior

Metrics series shouldn't grow without bound.

How you achieve this is up for debate.

But it usually entails a TTL or similar orphan pruning mechanism on metrics series.

For the github.com/prometheus/client_golang/prometheus Go module, there are DeleteLabelValues(), Delete(), DeletePartialMatch(), and Reset() methods to remove series.

There is already precedence for this pattern in the Pyroscope code base. See e.g. https://github.com/grafana/pyroscope/blob/b4d74078918e396eaffa30273b913d18f9c0a42c/pkg/scheduler/scheduler.go#L622

Environment

k8s

Additional Context