Scrape timeout defaults and validation ignore per-target __scrape_interval__ from SD/relabeling

What did you do?

Context

I am moving scrape jobs to service discovery (http_sd_configs) where the SD endpoint provides the scrape interval per target via the __scrape_interval__ label, so intervals are managed centrally instead of in each job. Some exporters compute metrics dynamically at scrape time and need several minutes to respond, so they use a long interval (5m) and a long timeout (4m).

I first reported this against vmagent (VictoriaMetrics/VictoriaMetrics#11597). While investigating, the VictoriaMetrics maintainer compared behavior with Prometheus and found that Prometheus has the same underlying issue: the scrape timeout is resolved and validated at job level, before per-target __scrape_interval__ is known.

Case A: explicit job-level timeout, interval provided by SD

global:
  scrape_interval: 1m
  scrape_timeout: 1m

scrape_configs:
  - job_name: example
    scrape_timeout: 4m
    # no scrape_interval: provided per target by SD
    http_sd_configs:
      - url: http://sd.example/targets

SD response:

[
  {
    "targets": ["target.example:9100"],
    "labels": { "__scrape_interval__": "5m" }
  }
]

Case B: timeout inherited from global, job interval overridden by SD

global:
  scrape_interval: 1m
  scrape_timeout: 1m

scrape_configs:
  - job_name: example
    scrape_interval: 40s
    http_sd_configs:
      - url: http://sd.example/targets

Same SD response as above (__scrape_interval__: 5m).

What did you expect to see?

The timeout should be resolved and checked against the final per-target scrape interval, after SD labels and relabeling are applied:

• Case A: the target is accepted with __scrape_interval__="5m" and __scrape_timeout__="4m", since 4m < 5m.
• Case B: __scrape_interval__="5m" and __scrape_timeout__="1m". The only timeout setting is the global 1m, which is valid against the 5m per-target interval.

What did you see instead? Under which circumstances?

• Case A: Prometheus refuses to load the configuration: scrape timeout greater than scrape interval for scrape config with job name "example" The job's scrape_timeout is compared with the global interval (1m) inherited by the job, even though every target from SD has a 5m interval.

• Case B: the target gets __scrape_interval__="5m" and __scrape_timeout__="40s". The default timeout is capped to the job-level interval (40s) at config load time, and this capped value is then used for targets whose SD-provided interval is 5m.

Root cause

1. ScrapeConfig.Validate fills in the job interval from global, then rejects an explicit timeout that exceeds it, or caps the inherited timeout to it (min(global timeout, job interval)): https://github.com/prometheus/prometheus/blob/50461b669f0be3a9f0b31a94f72cde8e98f2779f/config/config.go#L919-L934
2. PopulateLabels sets __scrape_timeout__ from this job-level value as a default target label: https://github.com/prometheus/prometheus/blob/50461b669f0be3a9f0b31a94f72cde8e98f2779f/scrape/target.go#L620-L625
3. After relabeling, the per-target interval and timeout are checked against each other, but the timeout is not re-derived from the final interval: https://github.com/prometheus/prometheus/blob/50461b669f0be3a9f0b31a94f72cde8e98f2779f/scrape/target.go#L666-L686

Possible approach

Keep the per-target validation in PopulateLabels (step 3) as the source of truth, and resolve the default timeout per target from the final __scrape_interval__ instead of at config load. The load-time check could remain for jobs where the interval cannot be overridden, or be turned into a warning.

Workaround: set __scrape_timeout__ explicitly in the SD response or via relabel_configs.

System information

No response

Prometheus version

Prometheus configuration file

Alertmanager version

Alertmanager configuration file

Logs