vmagent: scrape_timeout capped by global scrape_interval, ignoring __scrape_interval__ from SD

Describe the bug

Hi,

To begin with some context, I am migrating a set of scrape jobs from file_sd_configs to http_sd_configs.

With file_sd_configs, scrape_interval and scrape_timeout were set manually in each job, so the timeout was always checked against the correct interval. With http_sd_configs, the SD endpoint now provides the scrape interval per target via __scrape_interval__. The goal is to manage intervals centrally and remove them from the vmagent configuration, keeping only a job-level scrape_timeout.

Some of the scraped exporters compute certain metrics dynamically at scrape time, so they can take several minutes to respond. This is why they use a long scrape interval (5m) and need a matching long timeout (4m). With the behavior described below, those scrapes are cut off after 1m (the global interval), even though the effective per-target interval is 5m.

This setup worked with file_sd_configs and broke silently after the migration, with no configuration error and no warning.

When a job defines scrape_timeout but not scrape_interval, and the scrape interval is provided per target via the __scrape_interval__ label (e.g. from http_sd_configs), vmagent silently caps scrape_timeout to the global scrape_interval, ignoring the per-target interval.

The capped value is then injected as __scrape_timeout__ into every target's labels, so the configured job-level scrape_timeout is never applied. There is no warning in the logs, and /config still shows the configured (uncapped) value, which makes this hard to diagnose.

Expected behavior: either the cap is applied after per-target __scrape_interval__ / __scrape_timeout__ labels are resolved (so a 4m timeout with a 5m per-target interval is kept), or vmagent reports an explicit error/warning, as Prometheus does.

Root cause

In lib/promscrape/config.go:

1. getScrapeWorkConfig() resolves scrape_interval from the job or global config and caps scrape_timeout to it: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/af8394c5180ec43c047d6eb36741fba55a39eb77/lib/promscrape/config.go#L916-L935
2. mergeLabels() adds __scrape_timeout__ with this already-capped value to every target: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/af8394c5180ec43c047d6eb36741fba55a39eb77/lib/promscrape/config.go#L1397-L1408
3. getScrapeWork() then reads __scrape_interval__ and __scrape_timeout__ from target labels, but does not re-check the timeout against the resolved per-target interval: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/af8394c5180ec43c047d6eb36741fba55a39eb77/lib/promscrape/config.go#L1275-L1291

A side effect of (3): a __scrape_timeout__ greater than __scrape_interval__ set via SD or relabeling is accepted as-is, with no cap and no error.

Comparison with Prometheus

Prometheus validates the same situation explicitly instead of silently changing the value:

• At config load, a job whose scrape_timeout exceeds the (inherited) scrape_interval is rejected with scrape timeout greater than scrape interval for scrape config with job name "...": https://github.com/prometheus/prometheus/blob/50461b669f0be3a9f0b31a94f72cde8e98f2779f/config/config.go#L919-L934
• After relabeling, each target is re-validated and dropped if __scrape_timeout__ > __scrape_interval__: https://github.com/prometheus/prometheus/blob/50461b669f0be3a9f0b31a94f72cde8e98f2779f/scrape/target.go#L666-L686

Possible fixes

• Move the timeout/interval check to getScrapeWork(), after per-target labels are resolved.
• At minimum, log a warning when scrape_timeout is capped, and apply the same check to label-provided values.

To Reproduce

1. -promscrape.config:

global:
  scrape_interval: 1m
  scrape_timeout: 1m

scrape_configs:
  - job_name: example
    scrape_timeout: 4m
    # no scrape_interval here: it is provided per target by the SD
    http_sd_configs:
      - url: http://sd.example/targets

1. HTTP SD response:

[
  {
    "targets": ["target.example:9100"],
    "labels": {
      "__scrape_interval__": "5m"
    }
  }
]

1. Open the target relabel debug page in the vmagent web UI.

2. Observed: __scrape_interval__="5m" and __scrape_timeout__="1m0s". With exporters that need more than 1m to respond, every scrape fails with a timeout error.

3. Expected: __scrape_timeout__="4m0s", since 4m < 5m, or an explicit error/warning.

Workarounds: set scrape_interval: 5m at job level, or set __scrape_timeout__ via the SD payload or relabel_configs.

Version

vmagent-20250912-132832-tags-v1.126.0-0-gaa429631a6

Logs

No warnings or errors are logged when scrape_timeout is capped.

Screenshots

No response

Used command-line flags

No response

Additional information

No response