[Bug](meta-service) Node status flip in decommission is not atomic, readers may observe phantom node removal
Search before asking
- I had searched in the issues and found no similar issues.
Version
master, branch-3.1 (cloud mode)
What's Wrong?
In cloud mode meta-service, handle_decommission_node and handle_notify_decommissioned
(cloud/src/meta-service/meta_service_resource.cpp) flip a node's status by calling
ResourceManager::modify_nodes twice in two separate FDB transactions: first deleting
the node from InstanceInfoPB.nodes[], then re-adding it with the new status
(DECOMMISSIONING / DECOMMISSIONED).
Between the two committed transactions, the node is genuinely absent from the cluster
PB. Any concurrent reader of get_cluster/get_instance (e.g. FE CloudClusterChecker,
which polls every 10s) that lands in this window observes a phantom node removal.
Consequences observed in production during a 28-node scale-in:
- FE dropped the backend from memory, then re-registered it as a brand-new backend (new backend id) on the next poll;
- The tablet rebalancer treated it as a fresh idle BE and moved ~120k tablets back onto it;
- The subsequent DROP_NODE failed, leaving the decommission workflow wedged (compute group stuck in "changing" state).
What You Expected?
Node status transitions should be atomic: a reader sees either the old status or the new status, never "node missing".
How to Reproduce?
Inject a delay (e.g. sleep a few seconds) between the two modify_nodes calls in
handle_notify_decommissioned, then concurrently call get_cluster — the node
disappears from the response during the window.
Anything Else?
Proposed fix: replace the two modify_nodes calls (delete + re-add) with a single
ResourceManager::update_cluster call that mutates the node's status in place within
one FDB transaction — the same pattern already used by handle_rename_cluster /
handle_set_cluster_status. PR: #68119
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Source: apache/doris