Thanos querier rate/increase function creating huge spikes/incorrect results when deduplication is enabled

What happened: Thanos querier rate/increase function creating huge spikes/incorrect results when deduplication is enabled with value as 790.2 but when deduplication is disabled rate function just displays values as 0.31 for prometheus replica 0, 0.31 for replica 1 and 0.3 for replica 2. We are currently using 3 prometheus replicas

What you expected to happen: Thanos querier rate/increase function results to match with one of the prometheus replicas

How to reproduce it (as minimally and precisely as possible):

Query used below rate(http_server_requests_seconds_count{service='xxx',namespace='xxx', pod=~"xxx",uri="/actuator/health"}[2m])

We captured the time series in the 2m interval as shown below and analysed deduplication was not performed on the first 3 samples from replicas on same interval when the counter value is lesser for the recent sample compared to two other samples for the prometheus replicas for the same time frame

Deduplication samples, 8 results

78988 @1719976786.6 --- replica 0, dedup not performed 78989 @1719976787.413 --- replica 2, dedup not performed 78988 @1719976787.488 --- replica 1, dedup not performed, recent sample with lesser counter value 78995 @1719976806.6 79001 @1719976826.6 79007 @1719976846.6 79013 @1719976866.6 79019 @1719976886.6

Without deduplication -- prometheus replica 0 -- 6 results

78988 @1719976786.6 --- sample to consider 78995 @1719976806.6 79001 @1719976826.6 79007 @1719976846.6 79013 @1719976866.6 79019 @1719976886.6

Without deduplication -- prometheus replica 1 -- 6 results

78988 @1719976787.488 --- sample to consider 78995 @1719976807.488 79001 @1719976827.488 79007 @1719976847.488 79013 @1719976867.488 79019 @1719976887.488

without deduplication -- prometheus replica 2 -- 6 results

78989 @1719976787.413 --- sample to consider 78995 @1719976807.413 79001 @1719976827.413 79007 @1719976847.413 79013 @1719976867.413 79019 @1719976887.413

However if the time frame @171997678x.xxx is the second value in the results samples then dedup is performed successfully as below

Query below

rate(http_server_requests_seconds_count{service='xxx',namespace='xxx', pod=~"xxx",uri="/actuator/health"}[2m20s])

Deduplication samples

78982 @1719976766.6 78988 @1719976786.6 -- dedup working fine when the sample is second value 78995 @1719976806.6 79001 @1719976826.6 79007 @1719976846.6 79013 @1719976866.6 79019 @1719976886.6

without deduplication, prometheus replica 0

78982 @1719976766.6 78988 @1719976786.6 -- sample to consider 78995 @1719976806.6 79001 @1719976826.6 79007 @1719976846.6 79013 @1719976866.6 79019 @1719976886.6

without deduplication, prometheus replica 1

78982 @1719976767.488 78988 @1719976787.488 -- sample to consider 78995 @1719976807.488 79001 @1719976827.488 79007 @1719976847.488 79013 @1719976867.488 79019 @1719976887.488

without deduplication, prometheus replica 2

78982 @1719976767.413 78989 @1719976787.413 -- sample to consider 78995 @1719976807.413 79001 @1719976827.413 79007 @1719976847.413 79013 @1719976867.413 79019 @1719976887.413

Thanos, Prometheus and Golang version used: Thanos: 0.35.1 prometheus: 2.52.0 golang: 1.22.5