[BUG] XADD IDMP entries never expire once the stream's last ID is ahead of the clock
Describe the bug
stream-idmp-duration is documented in redis.conf as how long IDMP
identifiers are remembered: "After this duration expires, old identifiers are
automatically removed. This prevents unbounded memory growth".
handleExpiredIdmpEntries() (src/t_stream.c) decides whether an entry has
expired by comparing the stream ID it was recorded under with the clock:
uint64_t expire_time = server.mstime - (s->idmp_duration * 1000);
...
while (producer->idmp_head != NULL) {
idmpEntry *entry = producer->idmp_head;
if (entry->id.ms <= expire_time) {
... free ...
} else {
break;
}
}
/* If this producer has no entries left, remove it from the rax tree */and entry->id is the ID XADD produced (idmpInsertEntry(): entry->id = *id;). Stream IDs are monotonic and can be set explicitly, so after a single
ordinary XADD key <future-ms>-0 ... (no IDMP needed on that command) every
later auto-generated * ID carries that future ms. From then on the head
entry of every producer fails the test, the loop breaks immediately, producers
are never removed from s->idmp_producers, and the key is never removed from
db->stream_idmp_keys. IDMP tracking for that stream grows without bound and
is rescanned by the cron forever.
Two visible effects:
- Memory: every distinct producer id stays tracked for the life of the key.
- Semantics: an
(pid, iid)pair keeps being reported as a duplicate long afterstream-idmp-durationhas passed.
XIDMPRECORD key pid iid <id> reaches the same state directly, since it is an
ordinary write command that takes the ID from the caller.
To reproduce
redis-cli CONFIG SET stream-idmp-duration 1
redis-cli XADD future 9999999999999-0 f v # one ordinary XADD, no IDMP
for p in p1 p2 p3; do
redis-cli XADD normal IDMP $p i1 '*' f v
redis-cli XADD future IDMP $p i1 '*' f v # 9999999999999-1, -2, -3
done
sleep 3 # 3x the 1 s window
redis-cli XINFO STREAM normal # pids-tracked 0, iids-tracked 0
redis-cli XINFO STREAM future # pids-tracked 3, iids-tracked 3
redis-cli XADD normal IDMP p1 i1 '*' f v # new entry (window expired)
redis-cli XADD future IDMP p1 i1 '*' f v # 9999999999999-1 -> still treated as a duplicateScaled up, with stream-idmp-duration 1 and 8000 distinct producer ids per
stream, measured 6 seconds later on unstable @ f12f2a1e:
pids-tracked iids-tracked MEMORY USAGE
auto IDs 0 0 94935
one XADD 9999999999999-0 first 8000 8000 807806It is linear in the number of producers (32000 producers: ~3.4 MB pinned vs
~376 KB) and is never reclaimed. stream-idmp-maxsize does not bound it: it
caps entries per producer, and the unbounded dimension here is the number of
producers.
The future ID does not need to be extreme: any explicit ID ahead of the clock
(for example IDs generated by a client whose clock runs ahead of the server)
suspends expiry for as long as the stream's last ID stays ahead of
now - duration.
Expected behavior
IDMP entries expire stream-idmp-duration seconds after they were recorded,
regardless of the stream's ID values, as described in redis.conf and in
#14615 ("each iid expires automatically after duration seconds from
insertion"). In the example above, future should end with pids-tracked 0 /
iids-tracked 0 just like normal, and the second XADD future IDMP p1 i1
should create a new entry.
A possible fix is to record the wall clock at insertion and expire on it:
typedef struct idmpEntry {
...
streamID id;
+ mstime_t recorded_ms; /* server.mstime in idmpInsertEntry() */
} idmpEntry;- if (entry->id.ms <= expire_time) {
+ if (entry->recorded_ms <= expire_time) {The list is ordered by insertion, so the head-and-break scan stays correct with
a monotonic insertion stamp. The same comparison exists in the RDB save/load
filtering added in #14862, which would need to persist/use that stamp as well.
If adding a field is undesirable, a cheaper mitigation is to treat an entry
whose id.ms is greater than server.mstime as recorded now.
Additional information
- Reproduced on
unstable@f12f2a1e0f9ac47c1bbd356a95ac49b59d28012e(redis-server v=8.9.241, 2026-09-23), onc3215f01(2026-09-22), and on 8.10.2. - Ubuntu 22.04, x86_64, default
make(jemalloc).
Source: redis/redis