[Bug] [Registry] HA server can adopt a predecessor's selector and retain ACTIVE after losing election

Search before asking

• I searched existing issues; related reports and the distinct remaining defect are described below.

This is a focused follow-up to the stale-identity trigger already described by @yaodongen in #18624. That issue was closed by #18623, which fixes serial coordinator restartability. This report concerns the remaining shared AbstractHAServer ownership and role-reconciliation logic. Related etcd adapter issue #18640 and PR #18641 remain separate.

What happened

A restarted HA server can enter ACTIVE while its predecessor's ephemeral selector still exists. serverIdentify is the stable service address, and participateElection() accepts an existing value equal to that address without creating a selector owned by the new process.

After the predecessor's key disappears, the new process can lose another election but retain ACTIVE: the non-self REMOVE branch applies ACTIVE on success and does not apply STAND_BY on failure. Startup election and status application also run independently of watch callbacks, allowing local results and transitions to be applied out of order.

We reproduced the persistent dual-coordinator consequence on DS 3.4.2 with etcd using an ordinary StatefulSet restart. The production store was etcd 3.6.14; a separate isolated probe also reproduced the DS path with 3.5.21. Both versions define the preceding value in prev_kv, so the evidence does not point to a version-specific etcd defect. The store had a single selector owner; two DS serial coordinators continued scanning. The etcd adapter's empty REMOVE value contributes to this particular path. No broken etcd lock or namespace split was required.

We also reproduced the common ownership defect first-hand against all three Registry implementations using independent predecessor and successor clients:

These results concern the earlier UUID-based candidate (a6ac4168ac27b62b8260ae2d8dbee5db8c3b481b), not the subsequently revised timestamp identity or the minimal candidate described below. All eight original/fixed cases passed their respective assertions. In both JDBC databases, deleting the predecessor selector through its Registry client subsequently delivered the real change event: the fixed successor became ACTIVE and created a selector owned by its own client ID. These two recovery checks also passed.

These fixtures deliberately retain the predecessor session/lease/client and seed a legacy bare-address selector. They establish the shared identity error; they do not reproduce a killed JVM, natural expiration, complete Master startup or duplicate workflows on every backend. The ZooKeeper/etcd probes use frozen HA/plugin sources and mixed cached supporting dependencies, not an official release distribution. JDBC uses the compiled dev classes, native database schemas and real Spring/MyBatis Registry wiring. The predecessor-identity trigger was already reported for JDBC in #18624; this report credits that analysis and adds cross-backend verification.

What you expected to happen

• A fresh HA instance must not adopt a predecessor's selector merely because its address is unchanged.
• Current ownership should determine the local role after an election; losing or failing to establish ownership must not leave the server ACTIVE.
• A delayed notification must not overwrite a newer local role decision or force a needless restart when the current selector is still owned by this instance.

How to reproduce

Common scenario reproduced against each real backend:

1. Keep a predecessor's ephemeral selector with value master-a:5678 in the Registry.
2. Create a new AbstractHAServer instance with the same address and call start().
3. Observe ACTIVE without a new put(); the predecessor still owns the node.

Additional etcd consequence, rather than a claim about every backend:

1. Delete the predecessor's selector and allow another HA server to create its selector first.
2. Deliver the removal to the first server with empty event data, as the current etcd adapter does.
3. It loses the election but stays ACTIVE, while the peer is also ACTIVE.

Deployment observation, DS 3.4.2 / etcd, times UTC+8:

15:08:47.609 restarted master-a reports ACTIVE
15:08:47.610 master-a serial coordinator started
15:08:48.xxx predecessor lease expires; selector removed
15:08:48.281 master-b reports ACTIVE after writing the new selector
15:08:48.282 master-b serial coordinator started
Both coordinators keep scanning for more than seven minutes.

This captured window proves concurrent coordinators, not duplicate creation of a new command: only master-a emitted new serial launch records in that window. Complete business logs contain private parameters and are not attached.

Proposed fix / discussion

PR #18645 is under review. Following review and controlled validation, the candidate retains:

1. Distinguish successive instances at the same address using serverIdentify + "#" + System.currentTimeMillis(), as discussed in the PR. This distinguishes ordinary restarts; it is not a strict uniqueness guarantee for construction at the same address within one millisecond or after clock rollback.
2. Reconcile the current selector ownership on REMOVE events and publish STAND_BY when another instance owns it.
3. Serialize startup election, Registry callbacks and role publication so an older local decision cannot overwrite a newer one.
4. Protect both Master coordinators during stop/restart: an old run must finish its business work before its replacement starts.

Immediate demotion on election exceptions, terminal closed handling and related Alert shutdown changes have been removed from PR #18645 in 3687df9. Existing exception-retry and consumer shutdown behavior is preserved.

The coordinator close/start overlap found during regression analysis is an existing defect, not the established cause of the original cross-process incident. However, normal loss and reacquisition of leadership reproduces it even without exception-driven demotion. The coordinator handoff protection from 48f3e2a is therefore retained so corrected role transitions can stop and restart those services safely.

Validation update (2026-09-16): the candidate without coordinator handoff protection failed deterministic lifecycle checks. The reduced combination with that protection passed 26 targeted JDK 8 tests and was backported to DS 3.4.2 for controlled leader-restart and scale-down validation with our etcd Registry. Selector ownership and coordinator thread snapshots matched the expected roles, and the natural workflow executions checked completed successfully. The original image and single-Master topology were restored. These observations do not rule out all transient overlap or establish complete business-data acceptance. The earlier cross-backend table remains evidence for the original ownership defect and its earlier UUID-based fix, not fresh acceptance results for this candidate. See validation results for selected evidence and the current PR description for local upstream build results.

This is not a fencing protocol and does not promise zero overlap during watch delays, network partitions or downstream work already in progress. Those require separate guarantees.

Version

Reproduced on 3.4.2. The relevant shared HA implementation is unchanged in 3.4.3 and dev at 9839c418c1a6d7f2a8c3395552edcd99c2de1b37.

中文说明

这是对 #18624 已有身份复用分析的后续修复，不是重复报告其已经修好的串行线程重启问题。新进程仅因地址相同就继承旧进程的协调器身份；删除事件触发再次参选后，即使落选，也可能保留 ACTIVE。我们在 etcd 上复现了持续双协调器，本地真实 etcd、ZK、JDBC（MySQL 和 PostgreSQL）新旧客户端对照均确认了身份误认这一层，早期 UUID 版补丁均拒绝误认；两个 JDBC 后端还验证了旧节点删除后可以正常接管。这些结果不能替代当前时间戳版或精简候选的验证。不能据此声称所有 Registry 都已复现相同双活后果。

建议在共享 HA 层区分进程实例、统一按当前所有权更新角色，并增加可确定重现竞态的通用测试。现有 #18641 仍负责正确传递 etcd 删除事件旧值；两项修复应独立审阅。补丁不包含分布式 fencing，不将修复范围夸大为消除所有分区或通知延迟下的重叠执行。

按评审意见，最终保留实例身份、所有权与角色校正、紧邻并发顺序修复，以及两个协调器的线程交接保护。移除异常立即降级、closed 和 Alert 关闭扩展。仅前三项的候选未通过协调器恢复反例；补回交接保护的组合通过定向测试及 3.4.2 回移版本的重启、缩容验证，证据和边界见 PR。

Contribution

The fix and regression tests were submitted in #18645 with Codex assistance. The latest revision narrows the implementation to the four items above; its tests and the separate 3.4.2 backport validation are described in the PR. We welcome review of the ownership reconciliation and the smallest safe fix.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow the project Code of Conduct.