#15836·Redis

[BUG] XADD IDMP entries never expire once the stream's last ID is ahead of the clock

Author: hanke580Created Sep 23, 2026Updated Sep 27, 2026

Describe the bug

stream-idmp-duration is documented in redis.conf as how long IDMP identifiers are remembered: "After this duration expires, old identifiers are automatically removed. This prevents unbounded memory growth".

handleExpiredIdmpEntries() (src/t_stream.c) decides whether an entry has expired by comparing the stream ID it was recorded under with the clock:

c
    uint64_t expire_time = server.mstime - (s->idmp_duration * 1000);
    ...
    while (producer->idmp_head != NULL) {
        idmpEntry *entry = producer->idmp_head;
        if (entry->id.ms <= expire_time) {
            ... free ...
        } else {
            break;
        }
    }
    /* If this producer has no entries left, remove it from the rax tree */

and entry->id is the ID XADD produced (idmpInsertEntry(): entry->id = *id;). Stream IDs are monotonic and can be set explicitly, so after a single ordinary XADD key <future-ms>-0 ... (no IDMP needed on that command) every later auto-generated * ID carries that future ms. From then on the head entry of every producer fails the test, the loop breaks immediately, producers are never removed from s->idmp_producers, and the key is never removed from db->stream_idmp_keys. IDMP tracking for that stream grows without bound and is rescanned by the cron forever.

Two visible effects:

  1. Memory: every distinct producer id stays tracked for the life of the key.
  2. Semantics: an (pid, iid) pair keeps being reported as a duplicate long after stream-idmp-duration has passed.

XIDMPRECORD key pid iid <id> reaches the same state directly, since it is an ordinary write command that takes the ID from the caller.

To reproduce

redis-cli CONFIG SET stream-idmp-duration 1

redis-cli XADD future 9999999999999-0 f v          # one ordinary XADD, no IDMP
for p in p1 p2 p3; do
  redis-cli XADD normal IDMP $p i1 '*' f v
  redis-cli XADD future IDMP $p i1 '*' f v         # 9999999999999-1, -2, -3
done

sleep 3                                            # 3x the 1 s window

redis-cli XINFO STREAM normal                      # pids-tracked 0, iids-tracked 0
redis-cli XINFO STREAM future                      # pids-tracked 3, iids-tracked 3

redis-cli XADD normal IDMP p1 i1 '*' f v           # new entry (window expired)
redis-cli XADD future IDMP p1 i1 '*' f v           # 9999999999999-1  -> still treated as a duplicate

Scaled up, with stream-idmp-duration 1 and 8000 distinct producer ids per stream, measured 6 seconds later on unstable @ f12f2a1e:

                                    pids-tracked  iids-tracked  MEMORY USAGE
auto IDs                                       0             0         94935
one XADD 9999999999999-0 first              8000          8000        807806

It is linear in the number of producers (32000 producers: ~3.4 MB pinned vs ~376 KB) and is never reclaimed. stream-idmp-maxsize does not bound it: it caps entries per producer, and the unbounded dimension here is the number of producers.

The future ID does not need to be extreme: any explicit ID ahead of the clock (for example IDs generated by a client whose clock runs ahead of the server) suspends expiry for as long as the stream's last ID stays ahead of now - duration.

Expected behavior

IDMP entries expire stream-idmp-duration seconds after they were recorded, regardless of the stream's ID values, as described in redis.conf and in #14615 ("each iid expires automatically after duration seconds from insertion"). In the example above, future should end with pids-tracked 0 / iids-tracked 0 just like normal, and the second XADD future IDMP p1 i1 should create a new entry.

A possible fix is to record the wall clock at insertion and expire on it:

c
 typedef struct idmpEntry {
     ...
     streamID id;
+    mstime_t recorded_ms;   /* server.mstime in idmpInsertEntry() */
 } idmpEntry;
c
-    if (entry->id.ms <= expire_time) {
+    if (entry->recorded_ms <= expire_time) {

The list is ordered by insertion, so the head-and-break scan stays correct with a monotonic insertion stamp. The same comparison exists in the RDB save/load filtering added in #14862, which would need to persist/use that stamp as well. If adding a field is undesirable, a cheaper mitigation is to treat an entry whose id.ms is greater than server.mstime as recorded now.

Additional information

  • Reproduced on unstable @ f12f2a1e0f9ac47c1bbd356a95ac49b59d28012e (redis-server v=8.9.241, 2026-09-23), on c3215f01 (2026-09-22), and on 8.10.2.
  • Ubuntu 22.04, x86_64, default make (jemalloc).