[Bug] TimelineRollService may repeatedly roll the same timer messages
Before Creating the Bug Report
I found a bug, not just asking a question, which should be created in GitHub Discussions.
I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.
I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
Runtime platform environment
all environment
RocketMQ version
develop
JDK Version
1.8
Describe the Bug
TimelineRollService used a fixed sleep plus a scan window of [now + rollRange, now + rollRange + timerMaxDelaySec].
Two issues follow from that:
- The window is much larger than the interval, so the same not-yet-expired timer message stays in range and is rolled back to
TIMER_TOPICmany times. - The next window is computed from
System.currentTimeMillis()after sleep. If a scan is delayed, the next round can skip or rescan the same delayTime range. The progress was also only in memory, so a restart could roll the same messages again.
Steps to Reproduce
Enable the rocksdb scheduled messages, send a scheduled message, and you can change the roll interval to execute every minute. This way, you can verify more quickly.
What Did You Expect to See?
Set a reasonable roll interval and scan range, and simultaneously record checkpoints to ensure that messages are not lost and no invalid duplicate deliveries occur.
What Did You See Instead?
Drive roll by a persisted RocksDB checkpoint (timeline_roll_checkpoint) instead of a fixed sleep:
- Each round scans
[checkpoint, checkpoint + interval). - After a successful scan, advance and persist the checkpoint.
- The next due time is
checkpoint + interval - timerMaxDelaySec. Trigger 1s early; if it is not due yet, poll with at most 1s wait. - If the service is behind, scan the next window immediately so delayed work does not leave a gap.
Test plan
-
MessageRocksDBStorageTest#testWriteAndGetRollCheckpoint -
MessageRocksDBStorageTest#testScanAdjacentWindowsNoOverlap - Send a long-delay timer message (e.g. 3h,
timerMaxDelaySeca bit larger) and confirm it is rolled at most once - Slow down or restart broker during roll and confirm the next scan continues from the checkpoint without skipping or duplicating the previous window
Additional Context
No response
Source: apache/rocketmq