Sandbox stuck in `Phase: Error` despite healthy container and working gateway RPC channel

User Story

As an operator running a production NemoClaw/OpenShell sandbox with a live messaging channel (Telegram) attached, after the gateway's log-push RPC connection to a sandbox is disrupted and the underlying container later recovers on its own, I need the gateway to correctly re-recognize the sandbox as Ready, so that the web dashboard and CLI-based recovery commands remain usable for that sandbox without requiring a destructive destroy + re-onboard.

Problem Statement

A sandbox (harmony) is permanently stuck in Phase: Error as reported by openshell sandbox list and nemoclaw <name> status, even though:

• The underlying Docker container is running and healthy (docker ps shows Up ... (healthy))
• The sandbox is actively serving real traffic (Telegram bot polling getUpdates every ~30s and successfully processing real conversations via GPT-5.4 inference)
• The gateway's gRPC channel to the sandbox (127.0.0.1:8080) shows healthy ESTABLISHED connections with no drops or timeouts
• nemoclaw doctor reports every other check as ok (Docker daemon, OpenShell CLI, gateway binding, gateway connection, inference route)

Both nemoclaw <name> start and nemoclaw <name> gateway restart fail to clear the Error phase, and no combination of these commands re-triggers a successful readiness check against the already-running, already-healthy container.

Suspected root cause / trigger event:

The sandbox was healthy prior to a host-level event: .wslconfig was edited (setting a memory cap) and wsl --shutdown was run to apply it. This interrupted the sandbox↔gateway internal log-push RPC connection:

log push RPC failed ... peer closed connection without sending TLS close_notify

This connection never re-established on its own, and the sandbox's underlying Docker container eventually crashed (exit 255). After a later docker start <container> brought the container back up (Docker-level health: healthy, service resumed normally), the OpenShell/NemoClaw state layer never re-synced — it appears the gateway's own reconciliation/recovery scan may have run at a moment when the container was down, latched the sandbox into Error, and does not re-check sandboxes already marked Error.

This looks similar in nature to #1765 ("fix: resume unspecified sandbox phase") — a gateway-restart-time reconciliation scan misclassifying a sandbox state if it runs during a window when the container isn't up yet, and not re-scanning sandboxes already in Error.

Impact / Why This Matters

• The messaging service itself is unaffected — the sandbox works, real customer conversations are being handled correctly throughout.
• However, the OpenClaw web dashboard is completely unusable for this sandbox, because it depends on the sandbox being recognized as Ready. Any dashboard-dependent workflow (monitoring, manual intervention, session management via the web UI) is blocked.
• CLI-based recovery paths (start, gateway restart, recover) do not clear the stuck state, and there is no documented CLI command to force a phase re-check or manually clear Error short of destroying and rebuilding the sandbox (destroy + re-onboard), which is high-risk for a production sandbox and was avoided.
• There is currently no accessible log output for the gateway's internal decision process (stdout/stderr for the gateway process point to an internal socket, not a file; no systemd journal entries; no *.log/*.jsonl files found under ~/.local/state/nemoclaw), making this hard to self-diagnose further.
• Because the sandbox is marked Error, downstream doctor checks that depend on reaching it through the named gateway are also skipped (Inference route (gateway), Runtime channel registry), reducing the diagnostic visibility into the very state that needs debugging.

Acceptance Criteria

• A sandbox whose container has recovered to Up (healthy) after an interruption is correctly re-recognized as Ready without requiring destroy + re-onboard.
• A CLI command (or flag on start/recover) exists to force the gateway to re-run its readiness/reconciliation check against a sandbox currently in Error, independent of whatever cached decision put it there.
• The gateway's internal readiness/reconciliation decision (the one that currently causes nemoclaw start / gateway restart to wait ~41–59s and then re-latch to Error) is surfaced in --json/verbose output, so the failing check is identifiable without needing gateway-internal log access.

Reproduction Steps

1. Have a sandbox running normally under NemoClaw/OpenShell with a live messaging channel (Telegram) attached.
2. Cause an abrupt disruption to the gateway↔sandbox log-push RPC connection (in our case: wsl --shutdown while the sandbox was active).
3. Underlying container eventually exits.
4. docker start <container> brings the container back to Up (healthy), and the in-container service resumes working correctly (verified via real Telegram conversations, real GPT-5.4 inference calls in logs).
5. openshell sandbox list continues to report Phase: Error for this sandbox indefinitely.

Note: exact trigger not confirmed reproducible on demand — observed once via the above sequence.

Diagnostics performed (ruling out other causes):

• docker ps -a: Up ... (healthy), confirmed via repeated checks over time
• docker inspect (RestartCount, StartedAt/FinishedAt): manually restarted via docker start; no crash loop
• nemoclaw <name> doctor --json: all checks ok except Sandbox / Live sandbox = fail, detail: "harmony present (Error)"
• Docker daemon health (docker version, docker info): fully healthy
• Gateway process (ps aux): single long-running background process (not systemd-managed), stable, no crash/restart
• lsof -p <gateway pid>: only holds fds on its own SQLite state db; fuser on that db shows no other process holding a lock
• nemoclaw <name> gateway restart: reports gateway health passed but fails with Warning: OpenShell ForwardTcp 18789 for harmony did not start: OpenShell forward service exited before binding 0.0.0.0:18789 (status 1) — Failure layer: forward recovery failure
• Port 18789: confirmed free via lsof -i :18789 and ss -tlnp — not a port conflict
• gRPC channel to sandbox, port 8080: ss -tlnp/ss -tnp show healthy ESTABLISHED connections from the gateway process to itself via 127.0.0.1:8080 (routed into sandbox netns) — no drops, no timeouts
• nemoclaw <name> start: detects container is already running, attempts to "restore sandbox startup state," waits ~41–59s, then reports: Sandbox 'harmony' entered 'Error' state
• In-sandbox logs (nemoclaw <name> logs --follow): show completely normal operation — policy load, network namespace setup, Landlock rules applied, live Telegram getUpdates polling every ~30s, successful GPT-5.4 inference calls, successful sendMessage

Environment

• nemoclaw: v0.0.124
• openshell CLI: 0.0.116
• OpenClaw (in-sandbox agent): 2026.7.1 (base image), upgraded to 2026.9.4 via plugin work in a separate sandbox
• Docker Desktop: 29.8.0 (server), Docker Desktop 4.91.0 (239619)
• Host: WSL2 Ubuntu, NVIDIA GeForce RTX 5060 Ti
• Gateway process: /home/abt/.local/bin/openshell-gateway, run as a plain background process (not systemd-managed)

Logs

log push RPC failed ... peer closed connection without sending TLS close_notify

Warning: OpenShell ForwardTcp 18789 for harmony did not start: OpenShell forward
service exited before binding 0.0.0.0:18789 (status 1)
Failure layer: forward recovery failure

{
  "schemaVersion": 1,
  "sandbox": "harmony",
  "status": "fail",
  "failed": 1,
  "warnings": 0,
  "checks": [
    { "group": "Host", "label": "CLI build", "status": "ok", "detail": "dist/nemoclaw.js present" },
    { "group": "Host", "label": "Docker daemon", "status": "ok", "detail": "server 29.8.0" },
    { "group": "Host", "label": "OpenShell CLI", "status": "ok", "detail": "/home/abt/.local/bin/openshell" },
    { "group": "Gateway", "label": "Registered gateway binding", "status": "ok", "detail": "resolved to 'nemoclaw'" },
    { "group": "Gateway", "label": "OpenShell status", "status": "ok", "detail": "connected to nemoclaw" },
    { "group": "Sandbox", "label": "Live sandbox", "status": "fail", "detail": "harmony present (Error)", "hint": "run `nemoclaw harmony status` or `nemoclaw harmony logs --follow`" },
    { "group": "Inference", "label": "Route", "status": "ok", "detail": "openai-api / gpt-5.4" },
    { "group": "Inference", "label": "Inference route (gateway)", "status": "info", "detail": "skipped because the sandbox is not reachable through its named gateway" },
    { "group": "Inference", "label": "Provider health (upstream)", "status": "info", "detail": "OpenAI health requires OPENAI_API_KEY; skipping model-invocation probe instead of reporting endpoint reachability as healthy." },
    { "group": "Inference", "label": "Serving process", "status": "info", "detail": "not checked — serving-process probing is not implemented" },
    { "group": "Sandbox", "label": "Agent version", "status": "ok", "detail": "OpenClaw v2026.7.1" },
    { "group": "Sandbox", "label": "Lifecycle registration", "status": "ok", "detail": "registry entry has lifecycle metadata for snapshot, rebuild, upgrade, recovery, and reboot" },
    { "group": "Sandbox", "label": "Config permissions", "status": "ok", "detail": "runtime config permission contract verified" },
    { "group": "Messaging", "label": "Channels", "status": "info", "detail": "telegram enabled; telegram inbound delivery is not inferred from conflict signatures", "hint": "run `nemoclaw harmony channels status --channel telegram` to probe inbound delivery" },
    { "group": "Messaging", "label": "Runtime channel registry", "status": "info", "detail": "skipped because the sandbox is not reachable through its named gateway", "hint": "fix the gateway and live sandbox checks, then rerun `nemoclaw harmony doctor`" },
    { "group": "Local services", "label": "Ollama", "status": "info", "detail": "not reachable or invalid response at http://127.0.0.1:11434/api/tags" },
    { "group": "Local services", "label": "cloudflared", "status": "info", "detail": "stopped", "hint": "no cloudflared process; run `nemoclaw tunnel start` to start it" }
  ]
}