Cluster operations partially succeed without rollback or success reporting

Description

Operations distributed by the API across all cluster nodes are not atomic and have no rollback. When they fail on a subset of nodes, the API returns an error, but the nodes where the operation succeeded keep the applied state, and the response does not indicate which nodes those were.

The most visible case is PUT /agents/upgrade_custom. The operator receives a Bad Request response and reasonably concludes that the upgrade was never scheduled. In reality, upgrade tasks remain active on the nodes where the operation succeeded, and the agent will later receive and execute them.

An agent upgrade reinstalls software on the endpoint and may restart it. Triggering an upgrade that the operator believes was cancelled is a change-management issue, not just a usability issue.

What happens

flowchart TD

    OP["Operator:<br/>PUT /agents/upgrade_custom"] --> API["API on the master"]

    API -->|"DAPI distributes to all nodes"| M["master"]
    API --> W1["worker1"]
    API --> W2["worker2"]

    M --> MC{"WPK present on disk?"}
    W1 --> W1C{"WPK present on disk?"}
    W2 --> W2C{"WPK present on disk?"}

    MC -->|yes| T1["create upgrade task"]
    W1C -->|yes| T2["create upgrade task"]
    W2C -->|no| E["error 1824"]

    E --> R["API returns Bad Request<br/>mentioning only worker2"]

    R --> OP2["Operator concludes:<br/>nothing happened"]

    T1 & T2 -.->|"no rollback"| AG["Agent receives the task during the next<br/>/control notify<br/>and executes the upgrade"]

    style R fill:#fdd
    style AG fill:#fdd

Distribution does not stop at the first failure. Each node is evaluated independently, and the API aggregates the errors only after all nodes have been processed.

Reproduction

Cluster with one master and two workers. Agent enrolled and active.

1. Place the WPK on every node except one

for n in master worker1 worker2; do
  ssh $n "mkdir -p /var/wazuh-manager/var/upgrade && \
          echo dummy > /var/wazuh-manager/var/upgrade/t.wpk && \
          chown -R wazuh-manager:wazuh-manager /var/wazuh-manager/var/upgrade"
done

ssh worker2 rm -f /var/wazuh-manager/var/upgrade/t.wpk

2. Count pending tasks on each node

for n in master worker1 worker2; do
  ssh $n "/var/wazuh-manager/framework/python/bin/python3 -c \"
import sqlite3
c=sqlite3.connect('file:/var/wazuh-manager/queue/tasks/tasks.db?mode=ro',uri=True)
print(list(c.execute('select count(*) from TASKS'))[0][0])\""
done

3. Trigger the upgrade

curl -sk -H "Authorization: Bearer $TOKEN" -X PUT -G \
  "https://<master>:55000/agents/upgrade_custom" \
  --data-urlencode "agents_list=001" \
  --data-urlencode "file_path=/var/wazuh-manager/var/upgrade/t.wpk"

4. Count the tasks again

Observed result

The API returns an error mentioning only the node that failed:

{
  "title": "Bad Request",
  "detail": "The WPK file does not exist",
  "dapi_errors": {
    "wazuh-worker2": {
      "error": "The WPK file does not exist"
    }
  },
  "error": 1824
}

However:

Two new active upgrade tasks exist after an error response.

The master failing first does not stop distribution

Repeating the test with the WPK missing only on the master (the first node in the distribution path):

Both workers still created the upgrade task. There is no short-circuit, no privileged execution order, and no transaction.

Expected behavior

Either of these, in order of preference:

1. Rollback. If the operation cannot be applied successfully to every node, undo it on the nodes where it already succeeded, so the observable cluster state matches the API response.

2. At minimum, report it. The response should include which nodes successfully applied the operation, so the operator knows where manual cleanup is required and understands that the agent may still receive the upgrade command.

Currently, neither happens.

Scope

The reproduced case is upgrade_custom, but the issue is not specific to the task manager. It is a property of the cluster distribution pattern. Any API operation distributed to all nodes that produces persistent per-node state shares the same behavior. The full set of distributed operations should be reviewed before deciding where to implement a fix.

Detected while validating a cluster behind a load balancer using the laboratory under:

src/remoted/remoted_module/tools/load_balancer/