[All Platforms][Onboard] forward-ownership check refuses every port allocation client-side, blocking onboarding (dashboard port and Hermes API port)
Description
On DGX Spark, onboarding aborts before creating the sandbox:
Error: Cannot allocate dashboard port: NemoClaw could not prove OpenShell forward ownership.
Nothing about the host explains the refusal. The documented dashboard range has nine of its eleven ports free, the gateway port is one that has never been used on the host and has no state directory, and the sandbox name is new. The message names no port, no competing owner, and no action the operator can take.
The failure is not specific to a profile or an agent: it reproduces with a plain onboard using the default agent and cloud inference, with no managed local-inference profile involved.
Scope observed so far: both DGX Spark hosts available for this pass reproduce it. The same build on three non-Spark Linux hosts onboarded successfully many times during the same session, so the failure appears specific to these Spark hosts rather than to the build in general. What distinguishes them is that they carry other users' sandboxes that already hold some dashboard ports, but free ports remain in the range.
- Platform scope: Reproduced on two DGX Spark (GB10) hosts running Ubuntu 24.04 aarch64; non-Spark Linux hosts on the same build did not reproduce it this pass.
- Regression: Unknown — earlier builds not tested for this path this pass.
- OpenShell issue: Unclear from the outside. The message attributes the failure to an OpenShell forward-ownership proof, but it is NemoClaw's onboarding that refuses; the OpenShell gateway itself is healthy and
openshell forward listreports no active forwards.
Environment
Device: NVIDIA DGX Spark (GB10), two separate hosts
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Docker: Docker version 29.2.1
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.126
Agent: OpenClaw
Provider: NVIDIA cloud inference
Steps to Reproduce
On a DGX Spark that already hosts other sandboxes, record which ports in the documented dashboard range are in use, and confirm several are free:
for p in $(seq 18789 18799); do ss -ltnH | awk '{print $4}' | grep -qw $p && echo "$p busy" || echo "$p free" doneChoose a gateway port that has never been used on this host and confirm it has no state directory under the NemoClaw state root.
Onboard a brand-new sandbox name with the default agent and cloud inference:
NEMOCLAW_GATEWAY_PORT={unused port} nemoclaw onboard --fresh --name {new name} \ --non-interactive --yes --yes-i-accept-third-party-softwareRead the tail of the output and the exit code.
Run
openshell forward listandopenshell gateway list.
Expected Result
Onboarding allocates one of the free dashboard ports and continues.
If it genuinely cannot prove ownership of a candidate port, the error should name the port it tried, say who it believes owns it, and tell the operator what to do — the same way the port-conflict message for the managed local-inference port does, which names both the port and the holding process.
Actual Result
dashboard range 18789 free, 18790 free, 18791 busy, 18792 busy,
18793-18799 all free
gateway port chosen previously unused, no state directory
sandbox name new
onboard exit code 1
Error: Cannot allocate dashboard port: NemoClaw could not prove OpenShell forward ownership.
Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume
No port number appears anywhere in the message. Afterwards:
openshell forward list No active forwards.
openshell gateway list the gateway is registered and healthy
Reproduced on a second DGX Spark host with the same build, a different unused gateway port and a different new sandbox name.
Logs
onboard (plain, default agent) exit 1, "could not prove OpenShell forward ownership"
onboard (managed llama.cpp profile) exit 1, same message
free ports in the dashboard range 9 of 11
gateway state directory before run absent
openshell forward list after No active forwards.
non-Spark hosts, same build onboarding succeeded repeatedly
Related issues / not duplicate of
No existing report covers this refusal.
Closest, and not duplicates:
- #8798 — the dashboard port was baked into the sandbox image before being bound, so onboard destroyed the freshly built sandbox when the port turned out to be in use. That one is about a port that is genuinely occupied and is already fixed; here the range has free ports and no port is named at all.
- #4084 — the dashboard bound an occupied 18789 instead of moving to the next free port in the range. That is the inverse: it bound something it should not have, where this refuses to bind anything.
- #11963 — dashboard forward times out on macOS/WSL2. Different symptom (a timeout after the port is chosen) on different platforms.
Source: NVIDIA/NemoClaw