#13785·prometheus

promtool test with `/dev/stdin` fails for multiple test cases

Author: TribuneXCreated Mar 18, 2024Updated Sep 27, 2026

What did you do?

I use promtool test with the following rules and tests:

rules

yaml
# prometheus-rules.yml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
  groups:
    - name: myalerts
      rules:
        - alert: ErrorInLog
          expr: changes(log_messages_total{log_level="ERROR"}[1m]) > 0
          annotations:
            summary: 'New error(s) appeared in the log of container {{ $labels.container }}'
        - alert: NoNewMessages
          expr: sum(increase(messages_sent_total[3m])) == 0
          annotations:
            summary: 'My component did not send any messages for 3 minutes'

tests

yaml
# test-alerts.yml
rule_files:
  - /dev/stdin

tests:
  - name: 'Should trigger if no messages were sent for more than 3 minutes'
    interval: 1m
    input_series:
      - series: 'messages_sent_total'
        values: '0 0 0 0'
    alert_rule_test:
      - eval_time: 4m
        alertname: NoNewMessages
        exp_alerts:
          - exp_annotations:
              summary: 'My component did not send any messages for 3 minutes'
  - name: 'should trigger in case an error is detected'
    interval: 1m
    input_series:
      - series: 'log_messages_total{container="mycontainer", log_level="ERROR"}'
        values: '0 1 2'
    alert_rule_test:
      - eval_time: 2m
        alertname: ErrorInLog
        exp_alerts:
          - exp_labels:
              log_level: "ERROR"
              container: "mycontainer"
            exp_annotations:
              summary: 'New error(s) appeared in the log of container mycontainer'

It seems that there is a bug with the way promtool reads from /dev/stdin compared to executing the tests with a static file.

What did you expect to see?

The same tests with a static input file (alerts.yaml) pass:

bash
 yq .spec prometheus-rules.yml > alerts.yaml && promtool test  rules test-alerts.yml && rm alerts.yaml
Unit Testing:  test-alerts.yml
  SUCCESS

What did you see instead? Under which circumstances?

The tests fail for /dev/stdin:

bash
 yq .spec prometheus-rules.yml | promtool test  rules test-alerts.yml
Unit Testing:  test-alerts.yml
  FAILED:
    name: should trigger in case an error is detected,
    alertname: ErrorInLog, time: 2m, 
        exp:[
            0:
              Labels:{alertname="ErrorInLog", container="mycontainer", log_level="ERROR"}
              Annotations:{summary="New error(s) appeared in the log of container mycontainer"}
            ], 
        got:[]

System information

macOS 14.4

Prometheus version

promtool, version 2.50.1 (branch: non-git, revision: non-git)
  build user:       [email protected]
  build date:       20240226-11:31:35
  go version:       go1.22.0
  platform:         darwin/arm64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response