#68120·doris

[Bug](meta-service) Node status flip in decommission is not atomic, readers may observe phantom node removal

Author: LemonCLCreated Sep 17, 2026Updated Sep 17, 2026

Search before asking

  • I had searched in the issues and found no similar issues.

Version

master, branch-3.1 (cloud mode)

What's Wrong?

In cloud mode meta-service, handle_decommission_node and handle_notify_decommissioned (cloud/src/meta-service/meta_service_resource.cpp) flip a node's status by calling ResourceManager::modify_nodes twice in two separate FDB transactions: first deleting the node from InstanceInfoPB.nodes[], then re-adding it with the new status (DECOMMISSIONING / DECOMMISSIONED).

Between the two committed transactions, the node is genuinely absent from the cluster PB. Any concurrent reader of get_cluster/get_instance (e.g. FE CloudClusterChecker, which polls every 10s) that lands in this window observes a phantom node removal.

Consequences observed in production during a 28-node scale-in:

  • FE dropped the backend from memory, then re-registered it as a brand-new backend (new backend id) on the next poll;
  • The tablet rebalancer treated it as a fresh idle BE and moved ~120k tablets back onto it;
  • The subsequent DROP_NODE failed, leaving the decommission workflow wedged (compute group stuck in "changing" state).

What You Expected?

Node status transitions should be atomic: a reader sees either the old status or the new status, never "node missing".

How to Reproduce?

Inject a delay (e.g. sleep a few seconds) between the two modify_nodes calls in handle_notify_decommissioned, then concurrently call get_cluster — the node disappears from the response during the window.

Anything Else?

Proposed fix: replace the two modify_nodes calls (delete + re-add) with a single ResourceManager::update_cluster call that mutates the node's status in place within one FDB transaction — the same pattern already used by handle_rename_cluster / handle_set_cluster_status. PR: #68119

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct