quantile_over_time returns max_over_time's value regardless of requested quantile
Loki version: 3.6.7 (all components — distributor, ingester, querier, query-frontend, compactor — consistently on this version, running since 2026-03-23)
Deployment mode: microservices (separate distributor/ingester/querier/query-frontend/compactor Deployments/StatefulSets, single shared config.yaml via -target=<component>)
Relevant config:
limits_config:
split_queries_by_interval: 6h
query_range:
align_queries_with_step: true
cache_results: true
results_cache:
cache:
embedded_cache:
enabled: true
ttl: 24hlimits_config.shard_aggregations is not set (so the experimental sketch-based quantile sharding path, e.g. #20326, should not be in play).
Bug
quantile_over_time(q, ...) returns the exact same value as max_over_time(...) on the same stream/window, for every value of q I tried (0, 0.1, 0.5, 0.99) — instead of the correct rank-interpolated value.
Example, same log stream, same 1h window, instant queries via the HTTP API:
max(max_over_time({...} | unwrap file_size_bytes [1h])) -> 24835635
max(quantile_over_time(0, {...} | unwrap file_size_bytes [1h])) -> 24835635
max(quantile_over_time(0.5, {...} | unwrap file_size_bytes [1h])) -> 24835635min_over_time(...) on the identical stream/window correctly returns a very different, much lower value (e.g. ~1.1M vs ~29M over a 24h window), confirming the underlying samples have real variance and the log/unwrap pipeline itself is fine — the bug is specific to quantile_over_time's own computation.
Ruled out:
- Not a sort-direction bug:
quantile_over_time(0, ...)(which should equalmin_over_time) instead equalsmax_over_time, notmin_over_time. A reversed sort would swap low/high quantiles, not collapse all of them to max. - Not query splitting/sharding: reproduces identically at both 1h and 24h windows;
split_queries_by_intervalis 6h, so the 1h window should never be split, yet still exhibits the bug. - Not stale/cached results: re-querying with the quantile argument varied (0 vs 0.5) and the query time shifted by 90s both still return the same value as
max_over_timefor that specific window — and the value correctly moves when the window itself moves (re-tested minutes later, all three functions shifted together to a new but still-identical value), so it's not a frozen cache entry either. limits_config.shard_aggregationsis unset, so the known immature sketch-quantile-sharding codepath (#20326) shouldn't be reachable.
Expected
quantile_over_time(0.5, ...) should return the median of the unwrapped values in the window, not the max.
Impact
This makes quantile_over_time silently useless for detecting "is this metric's distribution flat/pinned near its max" type checks (e.g. quantile_over_time(0.99,...)/max_over_time(...) > threshold) — the ratio always evaluates to 1.0 regardless of the actual distribution shape, since both sides compute the same value.
Happy to provide more detail/help reproduce against a minimal fixture if useful — this was found via a real alerting rule that turned out to be silently non-functional in its core statistic.
Source: grafana/loki