[Bug] [Registry] Etcd DELETE events lose previous values and leave stale members

Search before asking

• I searched the existing issues and pull requests and did not find a duplicate report for the etcd-specific DELETE conversion described below.

Related: #18296 fixes a similar missing-value problem in the JDBC registry. The etcd implementation still reads the current DELETE KV rather than its previous value.

What happened

With DolphinScheduler 3.4.2 and the etcd registry, a removed Master can remain in another Master's in-memory membership list. After reducing a two-Master cluster to one Master, the remaining process can continue using two command slots, leaving commands assigned to the departed Master unconsumed.

The registry contains only the live Master, but the subscriber rejects the removal event because its heartbeat data is empty. A representative, sanitized message is:

Unknown cluster change event: Event(..., eventData=, type=REMOVE)

Root cause

EtcdRegistry.subscribe() already requests withPrevKV(true). However, EtcdRegistry.toEvent() maps DELETE to REMOVE and then reads eventData from watchEvent.getKeyValue().getValue() for every event type.

For an etcd DELETE, the current KV identifies the deleted key and deletion revision; the deleted value is available in prev_kv. Consequently, the converted REMOVE has an empty value even though the previous value was provided by the watch.

AbstractClusterSubscribeListener.notify() parses the heartbeat before dispatching REMOVE and returns when parsing yields null. The departed Master therefore remains in MasterClusters. Updates for the surviving Master do not remove that entry. With both cached members NORMAL, command partitioning can remain at two slots.

• etcd event semantics
• EtcdRegistry conversion, inspected dev
• Cluster subscriber, inspected dev

What you expected to happen

An etcd REMOVE event should carry the deleted node's previous value and the deleted key's path. Subscribers should be able to remove the departed member and recalculate the surviving Master's command slot.

ADD and UPDATE should continue carrying the current value.

How to reproduce

Minimal event reproduction

Use an isolated local etcd instance. In one terminal, start the watch:

etcdctl --endpoints=http://127.0.0.1:2379 \
  watch /ds-repro/member --prev-kv --write-out=json

In another terminal:

etcdctl --endpoints=http://127.0.0.1:2379 \
  put /ds-repro/member heartbeat-before-delete
etcdctl --endpoints=http://127.0.0.1:2379 \
  del /ds-repro/member

The DELETE event's current kv has no value, while prev_kv contains heartbeat-before-delete. This was reproduced with local etcd 3.5.21.

Constructing the corresponding jetcd 0.5.11 WatchEvent and passing it to the deployed 3.4.2 converter yields REMOVE with empty eventData. Using the deployed MasterClusters, an empty-data REMOVE leaves both members present; supplying the deleted member's previous heartbeat removes it. The local regression tests accompanying the proposed fix cover this event-conversion contract without requiring Docker. The focused test is EtcdRegistryEventTest.testDeleteUsesPreviousValue; testAddUsesCurrentValue, testUpdateUsesCurrentValue, and testDeleteWithoutPreviousValuePreservesPath cover the corresponding controls. With the regression test added and the production converter left unchanged, run the focused reactor test with JDK 8:

./mvnw -pl dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-etcd \
  -am test -Dtest=EtcdRegistryEventTest \
  -Dsurefire.failIfNoSpecifiedTests=false

The original converter fails testDeleteUsesPreviousValue: expected previous-heartbeat, actual empty string. The three control tests pass.

Master-level reproduction conditions

In an isolated DS deployment:

1. Use the etcd registry and two NORMAL Masters. Confirm that the surviving Master has cached both members.
2. Stop the other Master and let its membership key be deleted. Keep the surviving JVM running.
3. Check the surviving Master's log for the empty REMOVE above.
4. With idStep=1, observe that commands assigned to the departed slot can remain pending while commands assigned to the live slot are consumed.

Restarting the surviving JVM rebuilds membership and can clear this symptom; it does not fix the conversion defect. If the departed cached member was already BUSY, or is replaced at the same address, the slot symptom may differ.

Anything else

Affected versions checked

• Deployed and locally tested: 3.4.2.
• Same event conversion in 3.4.3, commit a190201acffa03d199d4ca216288734a6513de3d.
• Same event conversion in dev, commit 9839c418c1a6d7f2a8c3395552edcd99c2de1b37.

Proposed fix

Keep eventPath derived from the current KV, and use prevKV.value for DELETE eventData. Keep the current value for PUT events. The watch already requests previous KVs, so this needs no configuration change or dependency.

Add focused tests for DELETE previous-value delivery, current-key retention, and unchanged ADD/UPDATE behavior. This issue concerns event data and stale membership. It does not claim to resolve every coordinator failover or duplicate workflow execution problem.

Patch validation

The proposed patch passes all 4 event-conversion tests with JDK 8. A clean reactor run including the existing etcd registry and lease tests passes 18 tests with no failures, errors, or skipped tests. Spotless checks also pass. This is module-level validation, not a patched multi-Master end-to-end or production deployment test.

中文说明

在使用 etcd 的 DS 3.4.2 中，退出 Master 的 REMOVE 事件可能被存活 Master 忽略，导致内存成员表仍保留旧节点，部分命令因分片残留无法消费。

根因是 watch 已开启 prevKV，但 DELETE 转换仍从当前 KV 读取 value。 etcd 的 DELETE 当前 KV 不含旧值，旧值在 prev_kv；因此事件数据为空， 成员监听器解析心跳后提前返回，未执行成员删除。

该行为已通过真实本地 etcd 事件和部署的 3.4.2 类验证。3.4.3 与上述固定 dev 提交仍有同样代码。建议仅在 DELETE 时使用旧值，并保留当前 key，增加回归测试。 修复后的 JDK 8 clean 构建共 18 项测试通过，格式检查通过；尚未部署或进行多 Master 端到端验收。 已有 #18296 修复的是 JDBC 注册中心，不能据此认定 etcd 已修复。 复现成员残留时需要两台 NORMAL Master，退出其中一台，保持另一台 JVM 不重启； idStep=1 时可观察到对应分片命令积压。最小 etcd 命令见上方复现步骤。 本 issue 不将其它双协调器或重复执行问题一并声明为已定位或已解决。

AI assistance

Codex generated the initial proposed patch and regression tests, assisted with source analysis and this report, and ran local reproduction and test commands. Reproduction evidence and investigation limits are described above.

Codex 生成了初始修复与回归测试，协助源码分析和本报告，并执行了本地复现及测试命令。 复现证据和调查边界见上文。

Version

3.4.2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.