Inspector.GetQueueInfo is O(#groups) and blocks Redis for seconds when a queue has many aggregation groups
Inspector.GetQueueInfo runs two Lua scripts that iterate every aggregation group in the queue:
currentStatsCmd:SMEMBERSon the groups set, thenZCARDper group — https://github.com/hibiken/asynq/blob/v0.25.1/internal/rdb/inspect.go#L128-L137memoryUsageCmd: sameSMEMBERS+ZCARDloop again. ThegroupSampleSize(5) only limits theMEMORY USAGEsampling; theZCARDstill runs for all groups — https://github.com/hibiken/asynq/blob/v0.25.1/internal/rdb/inspect.go#L295-L314
Since Lua scripts execute atomically, each script blocks the entire Redis instance for the duration. With fine-grained group keys (e.g. one group per user/entity), a queue can easily reach millions of groups. And because x/metrics.QueueMetricsCollector calls GetQueueInfo for every queue on every Prometheus scrape — and Inspector.Queues() includes queues no consumer is draining — this runs continuously in production. We hit this as a serious production incident: Redis CPU pinned by monitoring alone.
Still present on master.
Repro (1M groups)
package main
import (
"fmt"
"sync"
"time"
"github.com/hibiken/asynq"
)
func main() {
// 1M tasks, each in its own group; no server running, so groups accumulate
// (same shape as a slow or paused consumer).
var wg sync.WaitGroup
for w := 0; w < 16; w++ {
wg.Add(1)
go func(w int) {
defer wg.Done()
c := asynq.NewClient(asynq.RedisClientOpt{Addr: "127.0.0.1:6379"})
defer c.Close()
for i := w * 62500; i < (w+1)*62500; i++ {
_, err := c.Enqueue(asynq.NewTask("noop", []byte(`{}`)),
asynq.Queue("repro"), asynq.Group(fmt.Sprintf("g:%d", i)))
if err != nil {
panic(err)
}
}
}(w)
}
wg.Wait()
insp := asynq.NewInspector(asynq.RedisClientOpt{Addr: "127.0.0.1:6379"})
t0 := time.Now()
info, err := insp.GetQueueInfo("repro")
if err != nil {
panic(err)
}
fmt.Printf("GetQueueInfo: %s (groups=%d)\n", time.Since(t0), info.Groups)
}Output on a laptop (asynq v0.25.1, Redis 7, localhost — a concurrent client measured PING latency during the call):
GetQueueInfo: 4.947s (groups=1000000 aggregating=1000000)
max PING latency from another client while GetQueueInfo ran: 2.546sOne call = ~5s of Redis CPU and two multi-second full-server stalls. Multiply by number of queues × scrape interval × processes running the metrics collector.
Suggested fixes
- Bound the
ZCARDloop the way memory usage is already sampled:SRANDMEMBER groups N+ZCARDon the sample, extrapolate viaSCARD(exactGroupscount stays O(1) viaSCARD). - Or add an option to skip aggregating/memory stats in
GetQueueInfoso the metrics collector can opt out.
Happy to send a PR if maintainers have a preference between the two.
Source: hibiken/asynq