remote-write: a series with a duplicate label name is answered 204 and silently dropped
What happens
A remote-write WriteRequest containing a series with the same label name twice
is answered 204 No Content. Prometheus logs the problem, increments a counter,
and stores nothing. The sender is told the write succeeded.
Remote-write 1.0 has no partial-success field, so the status code is the only signal a sender has, and here it says the opposite of what happened.
Reproduction
Prometheus 3.14.0 started with the receiver enabled:
docker run -d -p 9090:9090 prom/prometheus:v3.14.0 \
--config.file=/etc/prometheus/prometheus.yml --web.enable-remote-write-receiver
The body below is a snappy-compressed WriteRequest holding one series,
dup_demo{zone="a", zone="b"}, with a single sample. (Its timestamp is fixed,
so if it falls outside the ingest window, regenerate rather than reusing it.)
echo 'QIgKPgoUCghfX25hbWVfXxIIZHVwX2RlbW8KCQoEem9uZRIBYRkLSGISEAkAAAAAAADwPxCMtI+XiDQ=' | base64 -d > dup.pb.snappy
curl -i -X POST http://localhost:9090/api/v1/write \
-H 'Content-Type: application/x-protobuf' \
-H 'Content-Encoding: snappy' \
-H 'X-Prometheus-Remote-Write-Version: 0.1.0' \
--data-binary @dup.pb.snappy
Observed:
HTTP/1.1 204 No Content
$ curl -sG localhost:9090/api/v1/series --data-urlencode 'match[]=dup_demo'
{"status":"success","data":[]}
$ curl -s localhost:9090/metrics | grep invalid_labels
prometheus_api_remote_write_invalid_labels_samples_total 1
and in the log:
level=WARN source=write_handler.go:181 msg="Invalid labels for series."
labels="{__name__=\"dup_demo\", zone=\"a\", zone=\"b\"}" duplicated_label=zone
Is this intended?
I ask because it looks deliberate rather than accidental, and because #14716
already moved the TSDB append duplicate-label path from 5xx to 4xx. This is a
different path — validation in write_handler.go, which counts the sample and
continues — so a request whose every series is invalid still gets a 204.
I can see the argument for it: failing the whole request over one bad series would cost every good series batched alongside it, and a sender that retries a 5xx forever is worse than one that loses a sample. If that is the reasoning, is the intended answer for a 1.0 sender simply that it cannot know, and that the remote-write 2.0 written-samples response headers are where this gets fixed?
If so I'd suggest saying it in the remote-write receiver documentation, since "2xx means stored" is the natural reading and it is not true here.
How this was found
Sending deliberately unusual but conformant payloads to several stores and comparing what comes back. On the same request:
| Prometheus 3.14.0 | 204, stores nothing |
| Grafana Mimir 3.2.0 | 400, "received a series with duplicate label name, label: 'zone'" |
| VictoriaMetrics 1.151.0 | 204, keeps zone="b" |
| GreptimeDB 1.2.0 | 204, keeps zone="b" |
Mimir refuses; two stores take the later label; Prometheus keeps neither. Only the Mimir answer is distinguishable by a sender.
Source: prometheus/prometheus