Validate the Agent-to-Manager HTTPS Channel in a Multi-Node Cluster Behind Load Balancers

Description

In 5.0.0, 5.x agents enroll, report, and receive work exclusively through remoted's HTTPS agent API on port 1517 (agent-api.yaml).

So far, this channel has primarily been validated against a single manager:

agent -- TLS 1.3 --> wazuh-manager-remoted

A production deployment is different: a cluster with one master and multiple workers behind a load balancer exposing a single address. This is the topology recommended by the cluster documentation for large fleets (cluster/README.md).

The protocol itself has already been validated behind load balancers. #38156 produced the remoted load-balancer guides and a reproducible lab (src/remoted/remoted_module/tools/load_balancer/) with 57 checks under NGINX and HAProxy.

However, that work explicitly excludes a real cluster. The existing lab:

• Copies client.keys between nodes instead of synchronizing it through wazuh-clusterd.
• Does not run the complete manager stack.
• Uses a Python probe instead of a real agent.
• Exercises only POST /stateless out of remoted's nine agent-facing routes.

Therefore, what is currently proven is:

The HTTPS protocol survives a proxy.

What is not yet proven is:

A fleet of real agents can enroll, report, receive configuration, synchronize state, perform VD scans, and receive upgrades correctly when requests from the same agent are distributed across the nodes of a real Wazuh cluster.

The corresponding production architecture is also not fully documented.

This issue closes that gap by:

1. Building a representative multi-node cluster behind a load balancer.
2. Validating the complete agent-to-manager HTTPS surface with real agents.
3. Identifying and fixing product defects exposed by per-request load balancing.
4. Measuring cluster propagation and other timing-sensitive behavior.
5. Turning the critical scenarios into repeatable automated tests.
6. Completing the documentation for a supported cluster-behind-a-load-balancer deployment.

Scope

In Scope

Agent HTTPS API

Validate the complete remoted HTTPS surface:

• GET /
• GET /cacerts
• POST /enroll
• POST /control/startup
• POST /control/notify
• POST /stateless
• POST /stateful
• POST /download
• POST /stats
• POST /config
• POST /scan/vd

Load Balancers

Validate both deployment models documented in load-balancers/README.md:

• TLS passthrough (L4).
• TLS termination with re-encryption (L7).

Test both:

• NGINX.
• HAProxy.

TLS and Trust

Validate:

• Listener certificates on every manager node.

• The CA trusted by agents.

• subjectAltName requirements for:

  • The load-balancer address.
  • Individual backend manager addresses/names.

• <remote><https><verification_mode> behavior in both LB models.

• The behavior of GET /cacerts when a load balancer is between the agent and manager.

Enrollment

Validate all supported enrollment credential modes:

• No credential.
• authd.pass.
• Enrollment token.
• Re-enrollment bearer.

The enrolling request must be able to land on any appropriate node, while identity creation remains master-controlled.

Cluster Data Propagation

Validate the behavior agents observe while cluster data propagates:

• client.keys
• authd.pass
• enrollment_tokens.json
• etc/shared/
• var/multigroups/merged.mg

Reference: cluster.json.

The propagation window and its effect on agent behavior must be measured rather than inferred.

Per-Node State

Validate state that can differ between nodes when requests from the same agent are load-balanced independently:

• AgentRegistry
• config_token
• config_hash
• settings_hash
• vd_feed_offset
• Task polling and delivery state

Legacy and Mixed Fleets

Validate the legacy channel (1514/1515) at the same time as the HTTPS channel (1517).

A real migration requires 4.x and 5.x agents to coexist behind the same load balancer.

Documentation and Automation

Produce:

• A reference architecture.
• Configuration and certificate procedures.
• Troubleshooting guidance.
• Production checklists.
• Automated regression coverage for the behaviors that cannot be covered by existing single-node tests.

Out of Scope

• Cluster-internal synchronization performance.
• The wazuh-clusterd protocol itself.
• Server API behind a load balancer (55000), which has a different API and authentication model.
• Indexer and Dashboard clustering.
• The 4.x → 5.0.0 data migration covered by #39109.

Current Gaps and Findings to Validate

The following findings come from reviewing the 5.0.0 branch. They must be confirmed or refuted in the reference environment.

1. Existing Load-Balancer Lab Is Not a Real Cluster

The existing add_second_manager.sh creates a second bare remoted under /var/wazuh-manager-2 with no engine.

This is intentional: the lab uses the second instance as an authentication oracle:

• 503 → signature accepted.
• 401 → signature rejected.

However:

• client.keys is copied rather than synchronized.
• wazuh-clusterd is not running.
• wazuh-db shared state is not represented.
• There is no downstream event pipeline.
• run_issue_checks.sh exercises only POST /stateless.

Consequently, everything an agent performs after authentication is currently untested through a load balancer, including:

• Configuration download.
• Inventory/state sessions.
• Task polling.
• VD scan requests.
• Agent upgrades.

2. Cluster Documentation Does Not Describe Port 1517

cluster/lb.md documents balancing for the legacy 1514/1515 channels and redirects 5.x users to the remoted load-balancer documentation.

This leaves the documentation fragmented:

• No complete reference architecture for cluster + load balancer + agents.
• No consolidated port map.
• No explanation of which traffic must reach the master.
• No certificate topology.
• No explanation of how legacy and HTTPS channels coexist during migration.

architecture.md does not fill this gap.

3. Per-Request Balancing Interacts With Per-Node State

The load-balancer guide states that, with TLS termination, any manager can receive any request from any agent.

The critical missing piece is the state that is not shared between nodes.

/download authorization

POST /download authorizes against remoted's in-memory AgentRegistry, which is populated by /control on that specific node.

Therefore:

/control on worker A
        ↓
/download on worker B
        ↓
403

This has already been reported in #39147. Behind a non-sticky load balancer, this becomes a normal request-routing scenario rather than a rare edge case.

config_hash

config_hash is the SHA-256 of that node's var/multigroups/merged.mg.

If the file differs between nodes, the agent can observe different hashes and repeatedly re-download its configuration.

settings_hash

settings_hash is derived from limits and cluster.name and cached process-wide.

A divergent limits configuration on one node can therefore make the value change depending on which node handles the request.

vd_feed_offset

vd_feed_offset is read from the local vulnerability-detection module.

Nodes at different feed offsets can therefore expose different values to the same agent.

Task Delivery

/control/notify marks pending tasks as delivered locally without a cluster broadcast.

Custom WPKs must exist on every node under var/upgrade/ (agent-upgrades.md), and var/upgrade/ is not included in cluster.json.

Agent Database State

#38422 reports agent databases remaining on a worker after an agent reconnects elsewhere.

With non-sticky balancing, this should be treated as a normal routing scenario and explicitly validated.

Required Outcome

Every confirmed per-node-state dependency must be:

• Fixed.
• Documented as an explicit architectural constraint.
• Or tracked in a dedicated issue.

4. Cluster Key Propagation Window Is Not Measured

cluster.json configures worker sync_integrity to 9 seconds.

A freshly enrolled agent may therefore be routed to a worker that has not received its key yet and receive 401:

enrollment succeeds
        ↓
agent immediately reports
        ↓
request reaches worker without the key
        ↓
401

The same propagation problem applies to:

• authd.pass — enrollment fails closed with 401 while unsynchronized.
• enrollment_tokens.json — an unknown kid causes a rate-limited re-read before TokenUnknown.

The spike must measure:

• Actual propagation time.
• Agent retry behavior during the window.
• Whether the agent eventually recovers without intervention.
• The operator-visible behavior.
• The documentation and recommended mitigation.

5. Certificates Are Not Cluster-Synchronized

cluster.json synchronizes selected files under etc/, including:

• client.keys
• authd.pass
• enrollment_tokens.json

It does not synchronize etc/certs/.

Therefore, every node requires its own listener certificate issued externally.

The certificate setup must account for:

TLS Termination

The balancer presents the certificate validated by the agent.

TLS Passthrough

The agent connects through the balancer to individual manager nodes and therefore must be able to validate the certificate presented by each backend.

The spike must establish and document:

• Certificate issuance using wazuh-certs-tool.

• Required SANs.

• Certificate layout per node.

• Balancer certificate requirements.

• Rotation without dropping the fleet.

• Failure behavior for:

  • Expired certificate.
  • Wrong CA.
  • SAN mismatch.
  • Replaced node certificate.

The current lab uses only valid certificates and does not validate these failure modes.

6. verification_mode Documentation Contradicts the Implementation

The current documentation is inconsistent.

The lab README states that a third mode:

existed while this spike ran

and was later removed.

However:

• wazuh-manager.schema.json still accepts none | certificate | full.
• Remoted still implements full.
• full includes the 403 behavior and startup warning requiring an appropriate IP: SAN.
• load-balancers/README.md contains a dedicated section describing full.

This must be resolved before the deployment can be documented as supported.

Required outcome: determine whether full is supported in 5.0.0 and make the schema, implementation, lab, and documentation consistent.

7. global_prefix Has No Cluster Validation Story

The global endpoint prefix must be identical:

• On every manager node.
• On every agent.
• Through the load balancer.

The load balancer must not rewrite the prefix.

A mismatch results in 404, not an authentication error.

Fresh configurations use /wazuh-manager/, but there is currently no validation preventing a later-added node or agent from using a different value.

The spike must validate:

• Correct configuration.
• Node-to-node consistency.
• Agent consistency.
• Failure behavior.
• Diagnostic behavior.

8. GET / Is a Liveness Check, Not a Readiness Check

GET / returns 200 even when the engine is unavailable and the node rejects event processing with 503.

A load balancer can therefore continue routing traffic to a node that is alive but unable to process agent traffic.

The spike must either:

1. Define and implement a readiness endpoint, or
2. Document a supported balancer-side mitigation, such as treating sustained 503 responses as unhealthy.

9. Mixed 4.x/5.x Fleets Are Not Documented as a Single Architecture

During migration, the same load balancer must support:

4.x agents ──> 1514 / 1515
5.x agents ──> 1517

Enrollment on 1515 must reach the master.

The current documentation describes these channels separately but does not explain the combined front end required during migration.

This must be covered by both testing and documentation.

10. There Is No Automated Regression Coverage

The existing integration suites contain:

• test_remoted
• test_enrollment
• test_agentd

but none are cluster- or proxy-aware.

The load-balancer validation is currently implemented as manually executed shell scripts under src/remoted/remoted_module/tools/.

Without automated coverage, a future change to remoted, cluster.json, or the manager schema could silently reintroduce these failures.

Tasks

Phase 1 — Build the Reference Environment

• Build the reference environment:

  • One master.
  • Two workers.
  • Real wazuh-clusterd synchronization.
  • Indexer.
  • Dashboard.
  • Load balancer in front of 1517.

• Support both:

  • NGINX.
  • HAProxy.

• Support both TLS models:

  • Passthrough.
  • Termination with re-encryption.

• Issue the PKI using wazuh-certs-tool:

  • CA trusted by agents.
  • One listener certificate per manager node.
  • Required SAN for each backend.
  • Load-balancer certificate with the SAN used by agents.

• Record the exact certificate-generation and deployment commands.

• Enroll real Linux and Windows agents through the load balancer.

• Use several agent groups, including at least one multigroup agent.

• Test all enrollment modes:

  • No credential.
  • authd.pass.
  • Enrollment token.
  • Re-enrollment.

• Add the legacy front end:

  • 1514 → all nodes.
  • 1515 → master.

• Enable <remote><legacy> and <auth><legacy_enrollment>.

• Connect 4.x agents through the same load balancer.

Phase 2 — Validate the Architecture

HTTPS Route Matrix

• Run the complete agent-api.yaml route matrix through the load balancer using real agents.

• Test both TLS models.

• Test both proxies.

• Record for every request:

  • Route.
  • HTTP status.
  • Node that handled the request.
  • Agent action following the response.

Cluster Propagation

• Measure the key-propagation window:

  • Enroll an agent.
  • Immediately begin reporting.
  • Count 401 responses.
  • Measure how long failures persist.
  • Confirm automatic recovery.

• Repeat for:

  • authd.pass.
  • Enrollment token minted on the master and immediately used against a worker that has not yet synchronized.

Per-Node State

• Deliberately spread requests from the same agent across different nodes.

• Validate:

  • /control → /download across nodes.
  • config_hash stability.
  • config_token behavior.
  • settings_hash stability.
  • vd_feed_offset stability.
  • Task polling when the task was created on a different node.

• Create one dedicated issue for every confirmed product defect and link it from this issue.

Certificates

• Validate:

  • Expired leaf.
  • Wrong CA.
  • SAN mismatch.
  • Node certificate replacement while agents are reporting.
  • Full CA rotation.

• Confirm that agent-side errors are diagnosable from agent logs.

verification_mode

• Determine whether verification_mode: full is supported in 5.0.0.

• Align:

  • Schema.
  • Code.
  • Lab.
  • Remoted load-balancer documentation.

global_prefix

• Validate identical prefixes across:

  • All manager nodes.
  • Agents.
  • Load balancer.

• Deliberately introduce a mismatch on one node.

• Confirm the resulting failure is 404.

• Confirm the failure is diagnosable.

Agent Upgrade

• Execute an agent upgrade through the load balancer.
• Create the upgrade task on one node and retrieve the WPK from another.
• Test both TLS models.
• Test a custom WPK.
• Verify the constraint that upgrades to >= 5.0.0 are rejected unless verification_mode is none.
• Determine whether this constraint remains appropriate behind a load balancer.

Inventory and VD

• Exercise /stateful at fleet scale.

• Exercise /scan/vd at fleet scale.

• Validate:

  • 20 MB request-body limit.
  • 20-second /stateful deadline.
  • Proxy limits from sections 4.4 and 4.5 of the load-balancer guide.
  • 503 back-pressure reaching the agent unchanged.

Failure Drills

• Kill a worker while agents are reporting.

• Kill the master and determine:

  • What stops.
  • What continues.
  • Whether enrollment stops.
  • Whether group changes stop.

• Restart the load balancer.

• Remove a worker and add it back.

• Confirm agent recovery in every case.

Mixed Fleet

• Run 4.x and 5.x agents simultaneously through the same load balancer.
• Upgrade a 4.x agent in place.
• Verify its transition from 1514 to 1517 without re-enrollment.

Phase 3 — Automated Tests

• Decide where the automated cluster + load-balancer suite will live.

• Extend the existing load-balancer lab into a real cluster:

  • Replace copied client.keys with wazuh-clusterd synchronization.
  • Add missing routes to run_issue_checks.sh.
  • Preserve its existing PASS/FAIL contract.

• Add automated checks for:

  • Certificate failures.
  • Propagation behavior.
  • Critical cross-node request flows.

• Ensure the documented measurements are generated and/or validated by the automated suite where practical.

Phase 4 — Documentation

• Create a reference architecture page for a Wazuh cluster behind a load balancer covering:

  • Topology.

  • Architecture diagram.

  • Port map.

  • Traffic that must reach the master.

  • TLS termination vs. passthrough.

  • Certificate layout.

  • Per-node data not synchronized by the cluster:

    • etc/certs/
    • var/upgrade/

• Rewrite cluster/lb.md around the 5.x architecture:

  • 1517 as the primary 5.x channel.
  • Legacy 1514/1515 as the migration section.
  • Combined mixed-fleet architecture.

• Extend load-balancers/README.md §7 ("In a cluster") with:

  • Measured cluster constraints.
  • Propagation window.
  • Per-node state requirements.
  • Certificate requirements.

• Document the certificate procedure:

  • Node certificate generation.
  • Load-balancer certificate generation.
  • Required SANs.
  • Deployment.
  • Rotation.
  • Failure modes.
  • GET /cacerts behavior in both TLS models.

• Add troubleshooting guidance for:

  • Intermittent 401.
  • 404 on every request.
  • 403 only on /download.
  • 413.
  • 502.
  • 503.
  • TLS failures that do not produce an HTTP status.

• Update docs/SUMMARY.md.

• Resolve every documentation contradiction found during validation.

• Add a production checklist specifically for the cluster-behind-a-load-balancer deployment.

• Rebuild the environment from scratch using only the new documentation and repeat the Phase 1/2 validation.

• Attach the resulting evidence to this issue.

Automation Strategy

The validation is only valuable if the cri