Gateway hot-reload has no candidate validation or rollback — a runtime-invalid config can take down a running gateway

Summary

The gateway applies a config hot-reload / structural restart by draining the live channels first and only then trying to bring the new config up. Config that passes the Pydantic schema (extra="forbid") but fails at runtime — an unreachable model, a plugin/channel adapter that throws on load, a bad route/tool wiring — is not caught before cutover, and there is no rollback to the last-known-good running config. The result is that a single bad edit to gateway.yaml/bot.yaml can take a healthy, always-on gateway offline, which is the worst-case failure for a production control plane.

The gateway is otherwise excellent here: reload is diff-driven, SIGHUP/drain-coordinated, and observable via health() (Issues #2105, #2533, #3378 all landed). The missing piece is the safety of the cutover itself: validate the candidate before withdrawing the live one, and restore the live one if the candidate cannot serve.

Current behaviour

The structural-reload path stops the running channels before the new config is proven to work, and on failure it records failed and re-raises with the old channels already gone — no restore.

src/praisonai-bot/praisonai_bot/gateway/server.py — _reload_config_locked():

# server.py ~9192: schema-only validation (runtime validity not checked)
try:
    new_cfg = self.load_gateway_config(config_path)
except (FileNotFoundError, ValueError) as e:
    logger.error(f"Reload failed — config invalid: {e}")
    self._record_reload_status("failed", error=str(e))
    return                       # good: schema errors keep old config

...

# server.py ~9267: structural change → tear the live gateway down FIRST
if plan.full_restart:
    logger.info("Performing full restart due to structural changes")
    await self.stop_channels(drain_timeout=self._reload_drain_timeout)   # live channels gone
    # recreate agents from new_cfg ...
    # await self.start_channels(new_cfg["channels"])  ← if THIS raises, there is
    #   no path back to the previous working config; the gateway is now down.

The first-load branch has the same shape (stop_channels() → _create_agents_from_config() → start_channels() inside a try/except that only records failed and re-raises, server.py:9207-9244).

Schema validation is strict (GatewayServerSchema(extra="forbid"), bots/_config_schema.py:805), so typos are rejected safely — but schema validity is not runtime validity. gateway doctor --turn/--dry-run (cli/commands/gateway.py:891) can validate a candidate offline, but it is a manual, separate step that the reload/restart path never invokes, and it does not perform an atomic validated cutover with rollback.

Core already owns the reload-classification contract that this should build on: praisonaiagents.gateway.config exposes classify_reload, ReloadScope, HOT_APPLIABLE_KEYS, ReloadStatus, and compute_config_revision (consumed at server.py:8781-8808). There is simply no candidate-validation + rollback contract alongside them.

Desired behaviour

A hot-reload or structural restart should be an atomic, validated cutover:

1. Build the candidate runtime (agents/channels/hooks) from the new config without stopping the live one — or in an isolated pre-flight — and run the existing readiness/turn pre-flight against it.
2. Only when the candidate is proven healthy, drain the live channels and swap in the candidate.
3. If the candidate fails to build/start/pass pre-flight, keep the previous config serving (or restore it), and surface a failed reload with the exact reason via health()/ReloadStatus — never leave the gateway down.

Net effect: a bad gateway.yaml edit becomes a rejected reload with the old gateway still up, not an outage.

Layer placement

• Primary layer: core (praisonaiagents)
• Why not core: — it is core: the reload lifecycle contract already lives in praisonaiagents.gateway.config (classify_reload/ReloadScope/ReloadStatus). The validate-then-swap-or-rollback semantics belong next to them as a protocol, keeping heavy runtime work out of core.
• Why not wrapper: the wrapper/CLI should surface the safe reload (praisonai gateway reload/restart), but the invariant ("never cut over to an unvalidated candidate") must be owned by the runtime contract so every gateway build honours it, not re-implemented per entry point.
• Why not tools: not agent-callable; this is control-plane lifecycle, not a task-time integration.
• Why not plugins: this is the gateway's own reload safety (an owner responsibility), not a cross-cutting lifecycle guardrail that wraps user runs.
• Secondary touch: wrapper (praisonai gateway reload --canary/--no-canary, restart pre-flight) + gateway runtime impl in the bot package consuming the core contract.
• 3-way surface (CLI + YAML + Python): yes — gateway.reload.validate: true|false (YAML) + praisonai gateway reload/restart flags (CLI) + runtime reload_config() honouring the contract (Python).

Proposed approach

• Extension point: a core protocol + status enum, e.g. ReloadValidationProtocol producing a CandidateReloadPlan, plus a ReloadOutcome that distinguishes applied | rejected_validation | rolled_back.
• Minimal API sketch (core, protocol-only — no heavy imports):

# praisonaiagents/gateway/config.py (or a sibling reload.py)
class ReloadValidationProtocol(Protocol):
    async def validate_candidate(self, new_cfg: dict) -> "CandidateReport": ...
    # builds/boots the candidate in isolation and runs readiness/turn preflight

@dataclass
class CandidateReport:
    ok: bool
    failures: list[str]           # feeds ReloadStatus.error / health()

# reload_config() contract: never drain the live runtime until validate_candidate().ok

Resolution sketch

# Before (today) — server.py _reload_config_locked(), full_restart branch
await self.stop_channels(drain_timeout=self._reload_drain_timeout)  # live gone first
self._create_agents_from_config(new_cfg["agents"], ...)
await self.start_channels(new_cfg["channels"])                      # may raise → outage, no restore

# After (proposed) — validate candidate, then atomic swap or keep old serving
report = await self._validate_candidate(new_cfg)   # build agents + preflight channels/model/turn
if not report.ok:
    self._record_reload_status("failed", error="; ".join(report.failures))
    return                                          # OLD config still serving — no outage

await self.stop_channels(drain_timeout=self._reload_drain_timeout)
try:
    self._activate_candidate(report.candidate)      # swap in the already-proven runtime
    self._record_reload_status("ok")
except Exception as e:                               # last-resort safety net
    await self._restore_previous(self._loaded_config)   # bring the known-good config back up
    self._record_reload_status("rolled_back", error=str(e))

Severity

High — a reload/restart is a routine operator action (edit YAML → SIGHUP, or gateway restart), and today a runtime-invalid-but-schema-valid change on a structural key takes a healthy always-on gateway offline with no automatic recovery. High blast radius, common trigger, and it directly undercuts the "robust, world-class, always-on" goal for the gateway.

Validation

• Traced the reload path in src/praisonai-bot/praisonai_bot/gateway/server.py: reload_config() (9145), _reload_config_locked() (9187) — schema-only guard at 9192, first-load stop_channels → create_agents → start_channels under a record-and-raise try/except (9207-9244), and the plan.full_restart branch draining live channels at 9270 before rebuilding, with no restore of the previous config on a start_channels/agent-build failure.
• Confirmed strict schema validation exists but is not runtime validation: GatewayServerSchema(extra="forbid") in bots/_config_schema.py:805.
• Confirmed the reload classification contract lives in core (praisonaiagents.gateway.config: classify_reload, ReloadScope, HOT_APPLIABLE_KEYS, ReloadStatus, compute_config_revision) and is consumed at server.py:8781-8808 — but there is no candidate-validation/rollback contract beside it.
• Confirmed gateway doctor --turn/--dry-run (cli/commands/gateway.py:891) can validate a candidate offline, but is a separate manual step not wired into reload_config()/restart, and does not perform an atomic validated cutover.
• Confirmed prior reload issues (#2105 diff-driven, #2533 SIGHUP+drain, #3378 apply) landed the mechanics of reload but none address candidate validation or rollback safety.