> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemoclaw/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemoclaw/_mcp/server. # Recover and Rebuild Sandboxes > Recover a stopped agent runtime or rebuild a sandbox while preserving supported state. Use the lightest recovery operation that repairs the sandbox while preserving its supported state. ## Restart a Stopped Sandbox Container If NemoClaw reports that a Docker-driver sandbox is stopped, restart the existing container: ```bash nemo-deepagents start ``` This path preserves the sandbox workspace and repairs the agent runtime and host-side forwards after the container starts. If the container is paused, follow the printed `docker unpause` guidance instead. If Docker no longer has the container, destroy the stale NemoClaw entry and onboard a clean replacement. Restore a separately created snapshot afterward if one is available; rebuild cannot recover a workspace or live OpenShell policy that no longer exists. ## Recover the Agent Runtime Deep Agents sandboxes are terminal runtimes and do not expose an OpenClaw or Hermes in-sandbox gateway. Use `nemo-deepagents status`, `logs`, `connect`, and `rebuild` for recovery. If the terminal runtime reports degraded health, rebuild the sandbox instead of using `recover` or `gateway restart`. ### Understand Launch Readiness Leases A successful complete preflight for `nemo-deepagents launch ` can publish a credential-free launch-readiness lease on Linux. Linux infrastructure can publish the same evidence with `nemo-deepagents connect --probe-only`. The lease has a fixed 24-hour lifetime that repeated launches do not extend. Leaving the agent with `/exit` does not revoke it, and users do not refresh it manually. During the lease, `launch` still verifies the owning OpenShell gateway, exact live sandbox identity, registry and agent configuration, effective policy, inference route, required forwards, and semantic runtime health. Configured inference must return HTTP 2xx from the semantic `inference.local` probe for every provider except `openrouter-api`. The OpenRouter adapter returns HTTP 404 for `GET /v1/models`. NemoClaw accepts that result only after a bounded inference request for the recorded model succeeds. A failed request rejects launch readiness and identifies the inference request as the failed check. Hermes and LangChain Deep Agents Code retain their existing session setup on the lease-accepted path. When those checks pass, it can skip duplicate recovery, readiness polling, and inference-route repair. The lease is not a health guarantee or repair authority. For missing, unsafe, malformed, expired, mismatched, changed, or unhealthy evidence, NemoClaw fences any prior acceptable evidence before it runs the complete preflight. Ordinary launch continues only when NemoClaw proves that no old authority or evidence can exist, or durably rotates the runtime epoch. If an old epoch might exist and cannot be durably rotated, `launch` and `connect --probe-only` stop before complete preflight or recovery. Their redacted guidance asks you to repair the current user's secure OS runtime authority and NemoClaw state permissions, then retry. If NemoClaw securely proves that both the authority and receipt are absent but cannot create new authority, ordinary `launch` can run the complete preflight without optimization; on Linux, `connect --probe-only` exits nonzero because it could not publish evidence. If that preflight succeeds before the lease expires, replacement evidence keeps the original start and expiry time. After expiry, a successful complete preflight starts a new 24-hour lease only when publication succeeds. Before the first mutation in the complete preflight, the producer revalidates its sandbox-global runtime epoch under the sandbox lifecycle lock followed by the owning gateway lock. It holds both locks through all mutations in the complete preflight, final state capture, and publication. A stale producer makes no changes and re-inspects the newer lease. If unsafe or malformed authority history makes the prior lease timeline untrustworthy, NemoClaw durably invalidates the old epoch and starts one conservative 24-hour quarantine. Publication remains disabled until both wall time and monotonic uptime span the full quarantine. Repeated attempts do not extend it. After it elapses, the next successful complete preflight can publish a new fixed 24-hour lease. Lease acceptance and publication are currently Linux-only and require a secure, independently writable OS per-user runtime authority under `/run/user/`. It never uses caller-provided environment variables to select this authority. On macOS, `launch` runs the complete preflight every time and does not publish a launch-readiness lease. `connect --probe-only` also runs the complete preflight, including recovery and probes. After a successful probe and recovery, it prints a note that launch-readiness evidence is unavailable on this platform and exits zero. On Linux, the publication-failure diagnostic is redacted and does not print filesystem paths or environment values. Infrastructure must run `connect --probe-only` as the same final numeric user that later runs `launch`. Run it after the final durable home and state volume is mounted and after policy and network provisioning is complete. On completion, `connect --probe-only` writes one credential-free `Probe timing:` line with elapsed milliseconds for readiness, authority, lifecycle, gateway, processes, forward, inference, pairing, and publication stages. The line also reports the lifecycle and forward actions, the result, and the failed stage when the probe fails. Timing output is diagnostic only and does not change probe success or failure. NemoClaw rejects evidence after a bound sandbox, configuration, policy, or network identity changes. Deployment ordering remains responsible for external changes that OpenShell and NemoClaw cannot observe. ## Rebuild While Preserving State If you changed the underlying Dockerfile, upgraded Deep Agents Code, enabled Tavily Search, or want to pick up a new base image without losing manifest-defined Deep Agents state, use `rebuild` instead of destroying and recreating. ```bash nemo-deepagents rebuild ``` On WSL with Docker Desktop, a generated replacement image build uses a temporary credential-free Docker configuration when the configured Docker Desktop credential helper is unavailable. NemoClaw removes the temporary configuration after the build and does not modify your Docker configuration. An explicit custom Dockerfile continues to use your configured Docker credentials because its base image or build steps might require a private registry. If that custom rebuild cannot reach the credential helper, restore the Docker Desktop session or credential-helper access before retrying. ### Resolve Rebuild Preflight Stops Before it backs up or deletes the existing sandbox, `rebuild` validates the recorded sandbox, gateway, inference route, policy, MCP, agent, and operation-lock state. When one of these checks fails, NemoClaw prints `Rebuild preflight failed`, explains how to recover, and ends with `Aborting rebuild`. At this boundary, the existing sandbox is unchanged and no sandbox data has been removed. Use the recovery guidance that matches the reported check: * Verify the sandbox name when its registry entry is missing. * Follow the printed OpenShell gateway recovery steps when the gateway schema is incompatible. * Restore access to the current OpenShell policy when rebuild reports that the live policy cannot be read, then rerun `rebuild`. NemoClaw does not reconstruct policy from registry state. * Replace literal policy credentials with supported OpenShell credential bindings or resolver placeholders when rebuild refuses the policy handoff. NemoClaw stops before it creates the handoff or changes sandbox resources. * Resolve an incomplete MCP destroy transaction before retrying. * Back up the sandbox state and recreate it with `nemo-deepagents onboard` when the record contains multiple agents. Transactional multi-agent rebuild is not supported. * Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually. * Set the live OpenShell inference route to the sandbox's recorded provider and model when rebuild reports route drift. A gateway that reports no live inference route does not stop the rebuild. Replacement onboarding configures and verifies the recorded route before it recreates the sandbox. The rebuild command preserves manifest-defined Deep Agents state, regenerates `config.toml`, reconstructs managed MCP projection state, and passes the complete current OpenShell policy to replacement creation. ### Continue an Interrupted Replacement Before `rebuild` deletes the existing sandbox, NemoClaw records a replacement journal in the onboarding session. The journal binds the operation to the sandbox name, recorded OpenShell gateway, source identity, and replacement settings. It stores fingerprints instead of credential values or raw OpenShell sandbox IDs. If `rebuild` stops after recording the journal, rerun the command with the same replacement settings. The rerun takes one of these actions: * It continues deletion when the live sandbox still has the journaled source identity. * It continues creation when the recorded OpenShell gateway explicitly reports the source sandbox as absent. * It accepts an existing replacement only when its live identity and sandbox registry generation match the journal. * It starts a new replacement when the journaled replacement provably never took effect. NemoClaw treats a journaled replacement as void when the sandbox registry row and the live same-name sandbox report the same OpenShell identity on the journaled gateway, and that identity is not the journaled replacement. A registry row or a live sandbox on any other gateway leaves the journal in place, because that journal may still own an unregistered replacement on its own gateway. The rerun atomically replaces the void journal with a fresh journal against that live source, without leaving an unjournaled interval. It then reports `Replaced the void replacement journal for ''; its source sandbox is registered and live.` A journal whose replacement never took therefore stops blocking `rebuild` for the rest of the session. The journal binds the durable source registry row, so it does not resume after a command that rewrites that row. Messaging channel state is exempt: `channels add`, `channels stop`, `channels start`, and `channels remove` rewrite the recorded messaging plan, and the journal ignores that field. #### Recover an Unsafe Retained Policy Handoff Do not retry rebuild when a retained policy handoff contains a literal credential. Recover any required data from the backup before deletion, and keep the backup and handoff until that recovery is complete. Restore or select the recorded gateway and confirm `openshell status` is healthy. Only then use the ordinary fail-closed destroy flow and confirm OpenShell reports the sandbox deleted: ```bash openshell gateway select openshell status nemo-deepagents destroy --yes ``` Do not use `--force` for this recovery. If deletion is unconfirmed, preserve the local recovery state and restore gateway access. After confirmed deletion, run `nemo-deepagents onboard --name ` and recover any required data from the retained backup. Use a new sandbox name: recovery retirement refuses while the failed sandbox name is still present on its recorded gateway. Only after the old sandbox is confirmed absent and required data recovery is complete, retire the exact failed transaction printed by rebuild: ```bash nemo-deepagents rebuild --retire-recovery --yes ``` `--yes` confirms that required data recovery is complete. The command rechecks the old sandbox on the transaction's recorded gateway, binds cleanup to the exact sandbox and transaction, and removes that transaction's retained policy handoff and recovery marker. It cannot retire another sandbox or transaction. On failure, it prints the exact backup path that remains retained; preserve that path and retry only after correcting the reported condition. Keep the remaining backup until it is no longer needed. If retirement reports that the recovery marker is invalid or unreadable, preserve the printed backup path, marker, and policy handoff. Do not edit the marker JSON or delete either recovery file. If only the marker's mode or ownership changed, restore mode `0600` and ownership by the current NemoClaw user. Otherwise, restore the exact marker from a trusted backup of the same backup directory. Then rerun the same retirement command; it rechecks deletion on the recorded gateway before removing the handoff. If no trusted marker is available, preserve the backup and ask a NemoClaw maintainer to inspect it. A mount-free journal written before NemoClaw bound host-mount identity remains resumable. An older journal for a rebuild with one or more host mounts stops as incompatible, even when the visible mount settings are unchanged, because it cannot prove the original host source identity. Preserve the live sandbox, onboarding session, printed backup, exact error, and the sandbox name, gateway, and journal phase from the `Journaled replacement` diagnostic. Do not change the target settings, edit the session, or delete the same-name sandbox. Ask a NemoClaw maintainer to review that retained recovery state before taking another recovery action. An accepted replacement is not deleted again. The command reports `Sandbox '' already holds the replacement from the interrupted rebuild.` and preserves the state backup path when one exists. Pass `--verbose` to include the replacement identifier, OpenShell gateway, and journal phase in rebuild diagnostics. After the sandbox registry proves the journaled replacement identity and generation, NemoClaw removes an obsolete source image that it owns. It retains the image when the source is shared or the registered replacement reuses it. If image removal fails, NemoClaw keeps the accepted replacement and tells you to run `nemo-deepagents gc` for cleanup. NemoClaw fails closed when the selected gateway, replacement settings, durable source registry fields, or live source or target identity no longer matches the journal. The error names the sandbox and the mismatch that stopped recovery. Do not delete a same-name sandbox to bypass this check. Inspect the named OpenShell gateway and sandbox, correct the reported drift, and rerun the original command. Visible settings cannot correct the legacy host-mount journal case described above. A same-name recreation started by `nemo-deepagents onboard` uses the same replacement journal. If that recreation is interrupted after the `Journaled replacement` message, rerun the original onboarding command with the same target settings. The active replacement can continue without adding `--resume`. Use `--resume` for interrupted onboarding steps that occur before a replacement journal exists. If any manifest-declared state directory or state file cannot be archived, NemoClaw reports each failed path and stops before deleting the original sandbox. This fail-closed behavior also applies to `rebuild --force`, even when the backup contains other usable entries. When an incomplete snapshot contains captured entries, NemoClaw retains it at the printed path but excludes it from `snapshot list` and snapshot restore selection. The retained snapshot may contain unsanitized credentials. Do not restore, copy, or share it. Do not edit its manifest or try to make it selectable. Repair access to the original sandbox state, then rerun the rebuild. Remove the incomplete host-side snapshot only after you verify that the original sandbox or a later complete snapshot contains every required state item. #### Forced Rebuild Scope `rebuild --force` skips the confirmation prompt. When a sandbox with managed MCP servers cannot run a pre-mutation no-op, NemoClaw can use bounded host-side recovery when you pass `--force`. This path requires complete bridge entries plus exact provider and target identities, and it does not scrub the unreachable in-sandbox adapter. Every bridge entry must record the adapter for the sandbox's recorded agent. NemoClaw rechecks that read-only bridge snapshot immediately before deletion and stops if the target, registry, provider, or recorded gateway changed. Policy is not part of that ownership proof. The independently captured live OpenShell policy is handed to replacement creation unchanged. NemoClaw sends the delete request and every deletion-confirmation lookup to the sandbox's exact recorded gateway. Across every rebuild path, NemoClaw does not attempt to stop the local NVIDIA NIM through the delete attempt. Cleanup is attempted on a best-effort basis only after deletion is positively confirmed. After a nonzero delete, an explicit missing result converges as deleted. A `Ready` or `Running` result triggers an attempt to restore prepared MCP state and any lockdown that rebuild temporarily relaxed. NemoClaw reports any MCP or lockdown restoration failure and does not present the operation as a successful rollback. Any partial or unreachable result remains ambiguous. NemoClaw preserves the MCP ownership and rebuild-recovery records, does not attempt to stop NIM, skips the rebuild process's immediate lockdown restoration, and does not claim that the original sandbox is intact. Inspect the live sandbox and gateway state before retrying recovery. This recovery also stops for incomplete MCP adds or ambiguous ownership. An error after a successful no-op does not fall back to the host-side path. When rebuild starts with shields up, NemoClaw opens a 30-minute shields-down window for backup and recreation. A detached auto-lock timer remains the recovery authority until NemoClaw commits a successful shields-up state, including when the host rebuild process exits unexpectedly. Refer to [`nemo-deepagents rebuild`](../../reference/commands#nemo-deepagents-name-rebuild) for flag details. ## Use the Canonical Configuration Workflows * Use [Switch Inference Providers](../../inference/manage-inference/switch-providers) to change a model or provider. ## Related Topics * [Create and Restore Snapshots](../state-and-backups/create-and-restore-snapshots) for the state-preservation contract. * [Troubleshooting](../../reference/troubleshooting) for `privileged control unavailable`, stopped sandboxes, and failed rebuilds.