Proposal: Metadata manipulation — let users fix a metric's type, unit and help

TL;DR

Prometheus lets operators rewrite labels freely (metric_relabel_configs, write_relabel_configs), but gives them no supported way to rewrite metadata — TYPE, UNIT, HELP. Metadata is whatever the target or sender said it was, forever.

That was tolerable while metadata was decorative. We are slowly in the world where it no longer is: with PROM-39 / --enable-feature=type-and-unit-labels, type and unit are part of series identity, selectable in PromQL, drive rate() counter-ness annotations, and are necessary for downstream systems e.g. OpenTelemetry and Google.

Wrong metadata could be now assumed as actively harmful, and users cannot fix it in Prometheus.

This issue is primarily a collection of use cases to see if we have some consensus to add such mechanism to Prometheus (or other use cases I am missing). Proper proposal should be done, but I wanted to gather feedback first.

Use cases

1. The exporter forgot # TYPE (or says untyped)

A large share of exposition in the wild is untyped: textfile collector output, statsd_exporter, JMX exporter, home-grown /metrics handlers, legacy vendor agents.

# HELP myapp_requests_total Total requests.
myapp_requests_total{route="/api"} 4213

The operator knows perfectly well this is a counter. They cannot patch the exporter (third party, vendored, or 200 teams own it). With type-aware PromQL UX they now get warned — or silently degraded — forever on metrics they don't own.

Wants: "for this job, anything matching myapp_.*_total is a counter."

2. Declaring or correcting a unit

# TYPE http_request_latency_ms gauge
http_request_latency_ms 12.5

No # UNIT (impossible in Prometheus text format anyway), and the name encodes milliseconds. The operator wants __unit__="milliseconds" so unit-aware UI formatting and future unit-aware functions work. The inverse also happens: an exporter declares a misleading unit that the operator wants to strip.

Wants: derive __unit__ from a name suffix, or set/clear it outright.

3. Renaming a metric shouldn't destroy its metadata

Renaming via relabeling is a first-class, extremely common operation:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "old_vendor_name_total"
    target_label: __name__
    replacement: "myorg_requests_total"

Today this silently drops the metadata — the family match is done against the post-relabel name. If we let users rename, we should let them restate the metadata that goes with the new name.

Wants: rename and metadata stay consistent, or can be re-declared in the same block.

4. Dropping huge HELP strings

Real-world exporters ship multi-kilobyte HELP text: whole paragraphs, embedded JSON, even stack traces. That costs head memory, WAL bytes, and Remote Write 2.0 bandwidth — repeated per series.

There is no knob for this today short of all-or-nothing metadata_config.send: false (RW1 only), which also throws away type and unit.

Wants: "drop HELP for this job" / "truncate HELP to N bytes" / "drop HELP only for these metrics."

5. Reinterpreting a metric after transformation

An exporter exposes a monotonically increasing value as a gauge. A pipeline normalizes suffixes. A migration converts classic histograms to NHCB or renames toward OTLP-style naming. In each case the shape of the data changed (or was always mislabelled) and the declared type should be able to follow.

Wants: set the type to match what the data actually is after the pipeline is done with it.

6. OTLP parity — don't make me run a Collector for a one-line fix

OTel users have the Collector transform processor:

transform:
  metric_statements:
    - set(metric.unit, "s") where metric.name == "foo"
    - set(metric.description, "")

Pushing OTLP directly to Prometheus loses this entirely: OTLPConfig exposes nine fixed knobs and no transform hook. Users end up deploying a whole Collector purely to fix a unit string.

Wants: the same metadata fixes available on the OTLP receive path.

7. Remote Write receivers can't sanitize what tenants send

Prometheus as a RW2 receiver (and Thanos/Mimir/Cortex, which follow its behaviour) has no relabeling and no metadata hook at all on ingest. A multi-tenant operator cannot normalize units, strip oversized help text, or reject nonsense types coming from tenants.

Wants: metadata normalization at the receive boundary.

Current state — two unsynchronized metadata channels (click to expand)

There are two parallel metadata channels, and one of them is already user-mutable, by accident:

Findings:

1. Relabeling of __type__/__unit__ already works, undocumented. Parsers inject the labels (promparse.go#L250), relabeling runs after that (scrape_append_v2.go#L224), and nothing strips __-prefixed labels on the sample path — the __meta_* strip at target.go#L691-L697 is target-relabeling only. There is a merged test asserting exactly this, TestTypeUnitReLabel (scrape_test.go#L6937, from #16738/#16743) — and it passes without EnableTypeAndUnitLabels.
2. The docs say the opposite. feature_flags.md#L253: "All the user provided labels with __type__ and __unit__ will be overridden." True only for exposition-provided labels, which the parsers drop; not for relabel-provided ones, which run later and win.
3. The metadata record never sees the relabeling. scrape_append_v2.go#L312-L323 (v1: scrape.go#L2047-L2060) attaches lastMeta straight from the parser. After a relabel, series labels, the metadata WAL record, and /api/v1/metadata can all disagree.
4. Renaming silently drops metadata (use case 3): isSeriesPartOfFamily(lset.Get(__name__), lastMFName, ...) is evaluated on the post-relabel name (scrape_append_v2.go#L315), so the family match fails.
5. Send and receive have opposite precedence. Sender: labels win (queue_manager.go#L1975-L2001). Receiver: the record wins and overwrites incoming labels (write_handler.go#L327-L339). So metadata edits don't reliably round-trip a RW2 hop.
6. HELP has no label form and is explicitly zeroed on the label path (queue_manager.go#L1984). No way to set, rewrite or drop it (use case 4).
7. No hook on OTLP or RW receive (use cases 6, 7): OTLPConfig has fixed knobs only (config.go#L1771-L1789); the RW handler has no relabeling.
8. Unit values are unvalidated and inconsistently normalized between label and record on the OTLP path — UTF-8-aware UnitNamer at helper.go#L165-L173 vs. non-UTF-8 at metrics_to_prw.go#L241. See the TODO at schema/labels.go#L62-L65.

[!IMPORTANT] Findings 1 and 2 mean we already ship an undocumented behaviour that users may depend on, and finding 5 means it doesn't round-trip. Even deciding "we don't want metadata manipulation" requires a spec and doc fix here.

Open questions (click to expand)

• Scope: scrape only, or also the OTLP endpoint and the RW receiver? (Use cases 6 and 7 say all three.)
• Family vs. series semantics: what happens when sibling series of one family get different __type__?
• __help__: write-only relabel target, or a real label dropped before storage? It must never become identifying.
• Precedence, definitively: labels vs. records, on send and receive — today they disagree (finding 5).
• Validation: reject unknown type values at config load? At runtime? With what metric or annotation?
• Flag gating: should metadata manipulation require type-and-unit-labels? Note relabeling __type__ works without it today.
• Interaction with metadata-wal-records, and with the metadata API, which reads the scrape cache rather than TSDB.
• Suffix conventions: if a user sets __unit__=seconds on foo_milliseconds, do we warn? Rename? Neither?
• Removal semantics: is __type__=unknown / __unit__="" the way to delete metadata?
• Agent mode, which currently drops metadata entirely (tsdb/agent/db.go#L1093-L1094).

Related

• https://github.com/prometheus/prometheus/pull/17864 and https://github.com/prometheus/prometheus/pull/19555 for recording rules.

• #16610 — [meta] PROM-39 type-and-unit-labels stability (open) — natural parent for this work.
• #16738 / #16743 — PROM-39: Add tests for relabeling of type and unit labels (closed) — closest prior art: behaviour tested, never specified or documented.
• #17900 — textparse: Move scrape.isSeriesPartOfFamily to text parser and optimize (open) — overlaps the family-matching fragility above.
• PROM-39 proposal, including its open "More strict unit and type value definition" question.