query-frontend/query: penalty HA deduplication produces incorrect rate() values at split boundaries
Thanos, Prometheus and Golang version used: Thanos 0.42.4 (go1.26.5), Prometheus 3.5.4 (go1.25.11)
Object Storage Provider: None involved.
What happened:
Range queries using rate() return a large false drop at every Thanos Query Frontend split boundary when HA deduplication uses the default penalty algorithm.
With a 30m split interval, the false value appears at 23:30 and 00:00. With the production 24h interval, it appears at 00:00. Disabling splitting or using irate() instead of rate() removes the false values.
Our topology is:
Tier A Query Frontend + Query
-> Tier B Query
-> Tier C Query
-> 2x Prometheus HA replicas via SidecarThe source data from both Prometheus replicas is valid, and the false value does not appear when either replica is queried independently, or when querying any of the Query tiers (bypassing Query Frontend).
What you expected to happen:
Splitting a range query should not change its result. The response through Query Frontend should match the same query sent directly to Query.
The genuine CPU idle rate is approximately 0.97. Query Frontend should not introduce values of approximately 0.74 at split boundaries.
How to reproduce it (as minimally and precisely as possible):
Configure two Prometheus HA replicas scraping the same target every 30s. Give them different values for the external label prometheus_replica.
Configure Thanos Query with:
--query.replica-label=<replica label>
--deduplication.func=penalty- Configure Query Frontend with:
--query-frontend.downstream-url=http://query:10902
--query-range.split-interval=30m (or 24h)- Run a one-hour range query with a 30s step:
avg(
rate(
node_cpu_seconds_total{
mode="idle",
instance="<instance>"
}[1m]
)
)Observe false values at 23:30 and 00:00.
Repeat with:
--query-range.split-interval=0sThe interior false values disappear.
Changing the split interval changes where the false values appear:
- 24h: false value at 00:00
- 30m: false values at 23:30 and 00:00
The same erroneous value is also visible at the beginning of the overall range
Full logs to relevant components:
No errors or warnings are emitted when this occurs. All requests complete successfully; the problem is an incorrect value in the returned data.
Anything else we need to know: The following control tests were performed:
| Test | Result |
|---|---|
| Each Prometheus replica queried directly | No false value |
| Queried through Tier A with deduplication disabled | No false value |
| Only one Prometheus HA replica available | No false value |
| --query-range.split-interval=0s | No false value |
| --deduplication.func=penalty | Issue present |
| --deduplication.func=chain on all Query tiers | No false value |
| --query-range.max-query-parallelism=1 | Issue present |
| Step alignment enabled or disabled | Issue present |
| Partial responses enabled or disabled | Issue present |
| Downsample fallback enabled or disabled | Issue present |
| Prometheus and Thanos PromQL engines | Issue present |
Switch to irate() instead of rate() |
No false value |
Example with query splitting set to 30m - issue present:
Example with query splitting disabled - no false value:
Example with query splitting set to 30m, deduplication disabled and splitting avg by replica label - no false value (only drops to 0.96, not 0.75):
Source: thanos-io/thanos