Cluster readiness does not detect degraded agent API dependencies

Description

A node can lose an internal dependency, stop serving part of the agent API, and continue receiving traffic indefinitely. There is no signal that reflects this state: not the load balancer health check, not the cluster status, and not the server API.

In a load-balanced deployment, this means that a fraction of the fleet proportional to the number of nodes loses group synchronization, configuration hashes, and task delivery, while continuing to send events normally. From the outside, the cluster appears healthy.

What happens

flowchart TD

    LB["Load balancer<br/>health check"] --> R["GET /"]

    R --> OK["200 OK"]

    OK --> KEEP["Node stays in rotation"]

    KEEP --> TRAF["One-third of traffic reaches this node"]

    TRAF --> EV["POST /stateless<br/>202, events continue flowing"]

    TRAF --> CT["POST /control<br/>500 database_error"]

    CT --> LOSS["That third of the fleet loses<br/>groups, config_hash, and tasks"]

    style LOSS fill:#fdd

GET / is a liveness indicator, not a readiness indicator. Its handler always returns 200 and does not check any internal dependencies.

Reproduction

Three-node cluster behind a load balancer. Agent enrolled and active.

# Simulate the loss of an internal dependency on one node.
ssh worker2 pkill -f wazuh-manager-db

# Query that node directly.
curl --cacert root-ca.pem https://worker2:1517/wazuh-manager/   # liveness

# Then, with an authenticated agent:
# POST /stateless
# POST /control

Observed result

The node succeeds on some endpoints and fails on others

This state persists indefinitely. remoted continuously logs database connection failures while continuing to serve requests.

No layer reports the node as degraded

The degraded node cannot be identified from any centralized status endpoint. It is only detectable by querying each node individually.

Why the usual mitigation is not enough

The natural mitigation is to configure the load balancer to remove nodes that consistently return overload errors. That does not work here.

flowchart LR

    subgraph N["Degraded node responses"]
        A["GET / → 200"]
        B["POST /stateless → 202"]
        C["POST /control → 500"]
    end

    A --> R1["503-based rule<br/>does not trigger"]
    B --> R2["503-based rule<br/>does not trigger"]
    C --> R3["500 is not 503"]

    R1 & R2 & R3 --> OUT["Node remains in rotation"]

    style OUT fill:#fdd

The broken capability returns 500, not 503, while healthy event ingestion still returns 202. A load balancer that relies on HTTP status codes alone cannot distinguish this degraded state.

What exists today and why it is not sufficient

There is a readiness endpoint, GET /status, but it does not solve this problem. Its response, captured from the same node while /control was returning 500, was:

{
  "ready": true,
  "enrollment_password": {
    "ready": true
  },
  "keystore": {
    "readable": true,
    "agents_loaded": 36,
    "entries_skipped": 0
  },
  "enrollment_tokens": {
    "loaded": 8,
    "last_reload_ok": true
  }
}

There are two issues:

• It lives on the administration UDS socket. On the agent listener, the same path returns 404, so a load balancer cannot query it.
• It reports ready: true while the node is degraded. It validates enrollment password, keystore, and enrollment tokens, but it does not validate the dependency required by /control, which is exactly the dependency that has failed.

Expected behavior

Expose a dependency-aware readiness signal that is reachable from the agent listener and distinguishes "the process is alive" from "the node can serve the agent API."

flowchart LR

    LB["Load balancer"] --> RD["Readiness endpoint<br/>on the agent listener"]

    RD --> CH{"Required dependencies<br/>available?"}

    CH -->|yes| UP["200 — keep node in rotation"]

    CH -->|no| DOWN["503 — remove node from rotation"]

    style UP fill:#dfd
    style DOWN fill:#ffd

Design considerations:

1. Where it lives. It should be exposed on the agent listener so load balancers can query it. It should also be a different endpoint from GET /, preserving the current liveness endpoint for existing deployments.
2. What it checks. At minimum, the dependencies required for /control to function correctly. It is also worth deciding whether event ingestion should remain available when the control plane is degraded, since that is the current behavior.
3. Whether the cluster should expose it as well. Having cluster_control and the server API report degraded nodes would make the condition visible to operators, not just to load balancers.

Additional note

During the same validation, after stopping and restarting a node's container, startup logged:

Started wazuh-manager-db...
Completed.

even though wazuh-manager-db was not running afterward. It was not possible to determine whether this is specific to the container environment, so it is not reported as part of this issue, but it is worth investigating: a startup sequence that reports success without verifying the daemon state is one possible path into the degraded state described above.

Detected while validating a cluster behind a load balancer using the laboratory under:

src/remoted/remoted_module/tools/load_balancer/