[Audit] Current onboarding failures that v1 must prevent
Decision requested
Use the failures found in the current onboarding flow as acceptance criteria for v1. The default implementation target is v1; this issue does not request a parallel rewrite of the legacy flow on main.
Consider a main backport separately only when a maintainer identifies a release-blocking, security, or immediate data-loss risk.
Result
The audit found six source-level failure modes and 26 related reports across installation, image selection, sandbox creation, readiness, and retry. Together, they define the minimum recovery and diagnostic contract that v1 should satisfy.
The reported user failure is one chain:
- The managed image is unavailable.
- NemoClaw starts a slow local build.
- The target storage fills.
- The sandbox reaches
NotReady. - Recreation requires an SSH backup from a sandbox that cannot accept SSH.
- The backup path removes the useful SSH error.
OpenShell owns the live sandbox lifecycle. NemoClaw owns desired state, installation intent, resource identity, credentials, recovery decisions, and the meaning of a successful operation.
What v1 already improves
The v1 branch replaces the imperative onboarding sequence with one desired-state plan/apply path and durable resource bindings. It also checks the exact Docker engine storage root for managed-model work when the required download and preparation sizes are known.
Evidence:
- v1 operation result and deployment state
- Docker-root capacity observation
- Known-size model capacity check
These foundations address part of the problem. They do not yet provide a complete storage destination inventory, structured failure context, native-data recovery procedure, or verified inventory of everything retained after failure or destroy.
v1 acceptance requirements
1. Record mutation intent and exact resource identity
Before or immediately after an external mutation, retain enough durable identity to adopt, retry, compensate, or remove only that resource after interruption. A retry must not create a duplicate because the previous command stopped between mutation and receipt publication.
Cover providers, sandboxes, package installations, volumes, routes, and other owned resources.
2. Make data preservation possible, explicit, and bounded
Do not require a transport such as SSH when the resource state already proves that transport may be unavailable. Plan/apply should distinguish:
- backup succeeded;
- no recoverable data was found;
- backup is impossible for a stated reason;
- the user explicitly authorized replacement without a backup.
Preserve a short, redacted cause, exit status, and recovery action. v1 currently documents native-data backup, restore, and portable snapshots as TBD: state and recovery documentation.
3. Never destroy a working resource before its replacement is verified
Use build, verify, and swap semantics for future replacement support. Until v1 implements that contract, reject an unsafe in-place replacement and preserve the existing deployment.
Track the existing atomic-replacement requirement in #5801.
4. Inventory storage targets and diagnose failures where they occur
Do not add a generic preflight probe for every possible disk failure. That creates a slow, incomplete game of whack-a-mole.
Instead, plan/apply should associate known large operations with their storage target and report:
- resource address and operation;
- selected engine or host identity;
- storage class, such as Docker root, deployment state, bundle, managed volume, or backup destination;
- available and required capacity when they are reliably observable;
- whether partial artifacts were retained;
- a safe retry or cleanup action.
Keep proactive capacity checks for deterministic large operations whose remaining size and target are known. If an operation still fails with ENOSPC, classify it and return the target-aware context above.
The current v1 result contains resource actions, deferred checks, retained addresses, and health, but no storage-target inventory: operation result. Its documentation also states that the retained list is not a complete inventory: retention contract.
5. Define the success boundary
A successful operation must state what was proved. Configuration, container readiness, inference readiness, and a working agent request are different claims. If a check is unsupported or deferred, report that explicitly rather than implying end-to-end success.
6. Return structured progress and failures
Failures should identify the resource, lifecycle phase, retained state, cause class, and next safe action. Preserve provider and backend diagnostics after redaction. Do not replace them with an internal token or a generic NotReady message.
The current v1 CLI gives special structure only to selected health failures; most other errors become plain strings: CLI error rendering.
Evidence from the current flow
These are failures in the audited main implementation. They are evidence for the v1 requirements above, not a request to reproduce the same architecture in v1.
E-01 · NotReady recreation can require an impossible backup
Explicit recreation can require SSH backup when the sandbox cannot accept SSH. Automatic repair uses a different rule, and the test mocks backup success instead of exercising an unreachable sandbox.
Evidence: backup condition, recreation decision, and mocked backup test.
E-02 · Backup errors lose the useful cause
getSshConfig converts every nonzero result to null. Backup discovery can then report unreachable without the underlying error, and recreation prints a reason only when backup.error exists.
Evidence: getSshConfig, backup discovery, and user-facing error.
E-03 · Provider creation has a receipt gap
Normal apply failures remove a new provider. Process interruption after OpenShell accepts the provider but before NemoClaw saves its receipt can leave an untracked provider.
Evidence: apply cleanup and mutation and receipt order.
E-04 · Sandbox creation can leave an unregistered sandbox
Restore and DCode validation fail closed before the registry row is published. The replacement can therefore exist without a durable ID receipt.
Evidence: sandbox finalization order.
E-05 · Rebuild deletes the old sandbox too early
Rebuild deletes the old sandbox before the replacement is healthy. The defect is non-atomic replacement, not loss of the source registry row.
Evidence: rebuild destroy phase.
E-06 · Storage exhaustion lacks target-aware diagnosis
The current flow checks CPU and memory but does not identify and report capacity for the image build, Docker data root, NemoClaw state, or backup destination. The swap and Model Router checks do not protect general sandbox onboarding.
Evidence: CPU and memory preflight, swap-only disk check, and Model Router storage check.
Related reports
Keep platform-specific reproduction and ownership in the existing issues. Use them as v1 scenarios and regression tests.
Required v1 tests
- Interrupt each create operation after the external mutation and before receipt publication; retry must adopt or compensate the exact resource without duplication.
- Exercise recovery from a real SSH-unreachable or otherwise inaccessible sandbox; do not mock backup success.
- Inject
ENOSPCindependently at the Docker root, deployment-state directory, managed volume, and backup target; report the correct target and retained state. - Fail replacement health verification; the previous healthy resource must remain available.
- Retry an incomplete apply; unchanged verified resources must be no-ops.
- Verify every failure reports the resource, phase, retained state, and safe next action without leaking credentials.
- Verify success and deferred checks match the documented success boundary.
Work routing
- Use declarative onboarding (#10904) as the parent direction.
- Carry the receipt and compensation requirements from #7702 into v1 rather than duplicating the legacy orchestration.
- Carry atomic replacement from #5801 into v1 before replacement support is enabled.
- Apply the storage inventory and failure contract to future package installation work such as #11928.
Audit scope
The current-flow source scan used main at 987086b and the proposed state after PR #11909 at 52984bb. The v1 review used a83b5b17.
The audit covered installer and onboarding entry points, desired-state planning, image selection, sandbox creation, readiness, retry, backup, restore, receipts, storage observation, and success reporting. It did not rerun every platform reproduction.
Source: NVIDIA/NemoClaw