[Bug] Controller: a stale channel-close event evicts a re-registered broker and triggers a spurious master election
Before Creating the Bug Report
- I found a bug, not just a question.
- I searched open GitHub Issues and pull requests and found no duplicate.
- I confirmed that this bug belongs to Apache RocketMQ.
Runtime platform environment
All platforms; reproduced with a deterministic unit test on the current develop branch.
RocketMQ version
develop at ff8f6f74c
JDK Version
All
Describe the Bug
DefaultBrokerHeartbeatManager#onBrokerHeartbeat never rebinds the channel of an existing BrokerLiveInfo (the prev != null branch only refreshes the timestamp/timeout/priority/epoch), so the live entry stays pinned to the broker's first-ever channel. When that old channel eventually fires channelInactive / onChannelException / onChannelIdle, onBrokerChannelClose matches it against the stale stored channel, removes the live entry of a broker that is healthy on a new channel, and fires the broker-inactive lifecycle event, which makes ControllerManager#onBrokerInactive trigger a master election for a perfectly healthy master.
This is realistic in at least two ordinary situations:
- The broker-to-controller connection dies half-open on the controller side; the broker client detects it first, reconnects, and keeps heartbeating on the new channel. Minutes later the controller's TCP stack learns the old channel is dead and fires the close event.
- The controller-side idle/close event for the old channel is only delivered after the broker has already re-registered on a new channel.
Steps to Reproduce
- Broker B (master of its broker-set) sends a heartbeat to the controller on channel A →
BrokerLiveInfoholds channel A. - The broker reconnects and keeps heartbeating on channel B (every heartbeat hits the
prev != nullbranch, channel stays A). - Channel A fires
onBrokerChannelClose(A)on the controller. - Observe that B's live entry is removed and
notifyBrokerInActivefires although B is actively heartbeating on channel B;ControllerManager#onBrokerInactivethen callstriggerElectMasterbecause the evicted broker is the current master.
What Did You Expect to See?
A close event for a channel that the broker no longer uses must not evict the live entry: the heartbeat path should rebind BrokerLiveInfo to the current channel, so onBrokerChannelClose only removes entries whose stored channel is the one that actually closed.
What Did You See Instead?
The live entry of the re-registered master is removed, a broker-inactive notification is fired, and the controller bumps the master epoch / runs a failover for a healthy broker-set.
Additional Context
The sibling NameServer implementation (RouteInfoManager) updates the broker's channel on every heartbeat, so the controller behavior is inconsistent with the nameserver. The fix is to rebind the channel in the prev != null branch of onBrokerHeartbeat when a non-null channel arrives; a regression test in DefaultBrokerHeartbeatManagerTest reproduces the eviction deterministically.
Corresponding PR
- PR apache/rocketmq#11046
Source: apache/rocketmq