Time-shifted query-builder queries return and cache wrong-hour data on a partial bucket-cache hit
Bug description
For a time-shifted (timeShift) trace, log, or metric query-builder query, when the bucket
cache has a partial hit and needs to fetch only the missing sub-range, querier
applies the time-shift adjustment to that sub-range a second time before executing it —
so the executed query covers the wrong time window, and the wrong data is both returned to
the caller and durably stored in the cache under the correct range's key.
This is not a crash or an error: the query still succeeds, just with data from the wrong hour, silently.
Expected behavior
When executeWithCache re-queries a missing sub-range of an already-shifted query window
(querier.go:786-799), the sub-range should be executed as-is — it was already computed
from the shifted window, so it does not need shifting again.
How to reproduce
Root cause, traced end-to-end:
buildQueries(pkg/querier/querier.go:249,255,272) callsadjustTimeRangeForShiftonce on the raw request range (req.Start/req.End) and constructs thebuilderQuerywith that shifted range. From this point on,query.Window()(pkg/querier/builder_query.go:194-196) returns the already-shifted bounds.executeWithCache(pkg/querier/querier.go:750-853) asks the bucket cache for missing ranges viaGetMissRanges, which readsq.Window()(pkg/querier/bucket_cache.go:56) — i.e. the missing ranges it computes are sub-ranges of the already-shifted window.- For each missing sub-range,
executeWithCachecallscreateRangedQuery(orgID, query, *tr)(pkg/querier/querier.go:795). createRangedQuery(pkg/querier/querier.go:856-903), for thebuilderQuery[TraceAggregation],builderQuery[LogAggregation], andbuilderQuery[MetricAggregation]branches (lines 869-892), callsadjustTimeRangeForShift(specCopy, timeRange, qt.kind)again ontimeRange— which is already a shifted sub-range from step 2/3. The shift is applied twice.
Minimal reproduction (Go, run inside pkg/querier so it can call the unexported
functions directly — not committed, for illustration):
package querier
import "testing"
import "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
func TestDoubleShiftBug(t *testing.T) {
spec := querybuildertypesv5.QueryBuilderQuery[querybuildertypesv5.MetricAggregation]{
Functions: []querybuildertypesv5.Function{
{Name: querybuildertypesv5.FunctionNameTimeShift,
Args: []querybuildertypesv5.FunctionArg{{Value: float64(3600)}}},
},
}
rawReq := querybuildertypesv5.TimeRange{From: 10_000_000, To: 20_000_000}
shiftedOnce := adjustTimeRangeForShift(spec, rawReq, querybuildertypesv5.RequestTypeTimeSeries)
// shiftedOnce == {6_400_000, 16_400_000} -- this is what query.Window() returns
missingSubRange := querybuildertypesv5.TimeRange{From: shiftedOnce.From + 5_000_000, To: shiftedOnce.To}
// missingSubRange == {11_400_000, 16_400_000} -- e.g. the bucket cache's "missing" tail
specCopy := spec.Copy()
specCopy.ShiftBy = extractShiftFromBuilderQuery(specCopy)
doubleShifted := adjustTimeRangeForShift(specCopy, missingSubRange, querybuildertypesv5.RequestTypeTimeSeries)
// BUG: doubleShifted == {7_800_000, 12_800_000}, not {11_400_000, 16_400_000}
t.Logf("expected %v, got %v", missingSubRange, doubleShifted)
}Output:
=== RUN TestDoubleShiftBug
querier_test.go:24: expected {11400000 16400000}, got {7800000 12800000}
--- PASS: TestDoubleShiftBug (0.00s)7,800,000 = 11,400,000 − 3,600,000 and 12,800,000 = 16,400,000 − 3,600,000 — confirming
the 3600s shift is applied a second time on top of the already-shifted sub-range that
createRangedQuery receives.
Impact
Any dashboard panel or alert using a timeShift function on a trace/log/metric
query-builder query is affected whenever the bucket cache produces a partial hit — a very
common case in practice (e.g. a live dashboard where the cached window plus a small
uncached tail is the normal steady state). The result: the tail of the series is silently
populated with data from a different, incorrect hour, and that wrong data is written back
into the cache under the correct range's key (querier.go:850, Put uses the original,
correctly-shifted query, not the ranged copy) — so it persists for subsequent
requests/viewers until cache TTL expiry.
Version information
- Signoz version: commit
30dcb944eb4f788cff2c67366d983cdccce47211(main) - Component: backend,
pkg/querier(v5 query engine)
Additional context
This is unrelated to the earlier timeshift/cache bug fixed in #5646 — that fix was in the
legacy pkg/query-service/app/querier (v2) package and addressed a cache-key/fingerprint
collision (the shift amount wasn't part of the cache key). That fix is already reflected in
the current v5 fingerprinting (pkg/querier/builder_query.go:170-172 includes
shiftby=%d). This issue is a distinct bug in the current v5 pkg/querier package's
partial-cache-hit path, which I did not find covered by any existing issue or PR.
I'd be happy to open a PR — the fix looks contained to createRangedQuery not re-deriving
the shift for the three affected branches, plus a regression test.
Source: SigNoz/signoz