`hermes update` leaves a permanent "did not restart running gateways" warning when the restart is deferred

Summary

When hermes update is invoked from inside the gateway's own process tree, the gateway defers its restart until in-flight work units finish (correct — #77184), but the updater verifies the fleet immediately and records state: stale. The run then ends partial and leaves fleet_restart_pending behind. Every later CLI start prints:

⚠ A previous `hermes update` pulled new code but did not restart running gateways.
  Gateways may still be serving pre-update modules (mixed sys.modules).
  Run `hermes update` or `hermes gateway restart`.

…including after the gateway has genuinely restarted onto the new SHA. In my case the warning persisted for ~9 minutes of a fully-current fleet, and the receipt-based reconciliation path that is supposed to discharge it can never fire for this run shape (see Root cause 2).

A second, independent defect also shows up: the warning names hermes gateway restart as a remedy, but that command never clears the obligation.

Environment

• Hermes Agent v0.21.1 (main), Linux, systemd --user supervised gateway
• 6 profiles (default + 5 named), 1 gateway, 1 managed dashboard service
• pre-update fef0e16fe19b79ded929209f87c7434270b03825 → post-update 67764dc0863349a384c16425e73ee8571f3a94b7

Reproduction

1. Have an active gateway with a work unit in flight (an interactive/agent session is enough).
2. Run hermes update from inside that gateway's process tree (e.g. an agent or cron job invoking it).
3. Observe the updater report success, then fail its own fleet check.

Observed output:

✓ Restarted hermes-gateway
⟲ Restarting managed dashboard service (the running backend no longer matches the updated frontend)
    ✓ restarted hermes-dashboard.service
Fleet version check:
  ✗ default (pid 1321969) @ fef0e16f — STALE (pre-update code)

The gateway log at the same moment:

13:46:28 gateway.run: Restart requested with 1 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)
13:46:28 gateway.run: Restart deferred: waiting on 1 active work unit(s) (0 wedged and excluded; 1800s remaining before force drain)

The gateway actually restarted at 13:48:25 (once the work unit finished), i.e. ~2 minutes after the receipt was written. The receipt therefore records a 2-minute-old truth forever:

{
  "started_at": "2026-09-10T13:45:05Z", "finished_at": "2026-09-10T13:46:33Z",
  "outcome": "partial",
  "gateway_restart": {"restarted_services": ["hermes-gateway"], "incomplete": false},
  "runtime_outcomes": [{"kind": "gateway", "profile": "default", "outcome": "restarted"}, ...],
  "fleet": [{"profile": "default", "pid": 1321969, "code_sha": "fef0e16f…", "state": "stale"}]
}

Note gateway_restart.incomplete: false and runtime_outcomes.outcome: "restarted" — the updater believes it restarted the gateway successfully, and it did signal it. Only the verification sample raced the deferral.

Root cause

1. The fleet snapshot races the deferred restart

_run_pending_fleet_restart() treats "signalled" as "restarted" and the caller samples the fleet immediately afterwards. With a drained-then-restarting gateway the sample inevitably shows the old PID and old code_sha, which is then frozen into the receipt.

2. _live_fleet_covers_receipt() can never discharge a receipt that includes a non-gateway runtime

hermes_cli/update_cmd_fleet.py:164-171:

entries: list[tuple[object, str | None]] = [(entry, None) for entry in runtimes]
entries.extend((entry, "gateway") for entry in recorded_fleet)
for entry, default_kind in entries:
    if not isinstance(entry, dict):
        return False
    kind = entry.get("kind", default_kind)
    profile = entry.get("profile")
    if kind != "gateway" or not profile or profile == "unknown":
        return False
    owed.add((kind, profile))

The updater now restarts the managed dashboard in the same run, and that entry lands in receipt["plan"]["runtimes"] with "kind": "dashboard", so this loop returns False on every evaluation. Observed live on the same receipt, using the module's own functions while collect_fleet_versions() reported the gateway as current:

marker path     : …/fleet_restart_pending False
current sha     : 67764dc0
receipt stale?  : True
covers receipt? : False      <-- unconditionally False; plan.runtimes = ['gateway', 'dashboard']
pending needed? : True

So _pending_fleet_restart_needed() stays True on stale-receipt alone, and the warning is pinned independently of the marker. It only cleared later by luck: the next update run wrote a receipt whose fleet array was empty, so _receipt_reports_stale_runtime() returned False and short-circuited before _live_fleet_covers_receipt() was ever consulted.

3. The warning's stated remedy does not work

fleet_restart_pending is cleared only inside the update flow (hermes_cli/update_cmd_fleet.py:360 and :1402). A plain hermes gateway restart — which the warning text explicitly suggests — does not clear it.

Impact

A correct, fully-updated fleet reports itself as potentially running mixed pre/post-update modules. Agents reading this warning (or the receipt) will conclude the runtime is stale and take unnecessary action — in my case this was surfaced while diagnosing why subagents were behaving conservatively. Because covers receipt? is permanently False for any update that also touched the dashboard, the false state can only be cleared by an unrelated future update that happens to sample an empty fleet.

Workaround

Run hermes update again from outside the gateway's process tree with no active work units, e.g. a detached systemd unit that waits for active_agents == 0 in gateway_state.json before invoking it. That run writes a clean receipt and the warning disappears:

✓ Already up to date! [main @ 67764dc0]
→ Running the pending fleet restart...
  ✓ Pending fleet restart completed.
marker absent / gateway pid 1333682 67764dc0 / warning gone

Suggested fixes

1. Don't verify before the restart lands. After signalling a drain, poll the target gateway's code_sha (gateway state file or the control socket) with a bounded timeout and record the fleet snapshot only then — or record the deferral explicitly so the receipt isn't read as a stale runtime.
2. Model non-gateway runtimes in _live_fleet_covers_receipt() instead of returning False on them. The obligation being discharged is a gateway restart; a dashboard entry should be checked against its own restart outcome (or ignored), not treated as unprovable.
3. Fix the warning text or make hermes gateway restart clear the marker when the fleet is verified current.