Recover and Rebuild Sandboxes

Use the lightest recovery operation that repairs the sandbox while preserving its supported state.

Upgrade from a Release That Used Shields

Shields has been retired from NemoClaw. The current CLI has no Shields commands, status, timers, or supported locked posture. After an upgrade, NemoClaw prints a retirement notice while it still finds legacy Shields state and identifies the affected sandbox records when possible.

If the sandbox has only an inert legacy state record, create a trusted snapshot or backup. Then rebuild the affected sandbox before running further mutations.

If NemoClaw blocks an operation because it found an old timer, transition, policy snapshot, authority-bearing runtime-provider recovery artifact, or ambiguous recovery state, do not onboard or mutate any sandbox yet. These files can represent detached mutation authority that the current CLI deliberately does not interpret. A different requested sandbox name does not make onboarding safe.

A notice-only provider intent recorded before mutation authority was granted is preserved for review but does not by itself block lifecycle operations. The retirement notice prints each exact absolute notice-only path and labels it nonblocking. A notice-only path is not part of the quarantine inventory unless a later blocking error explicitly lists that same path.

Use the active state directory and exact blocking paths printed in the error as the complete quarantine inventory. A provider path can name a whole lifecycle transaction directory; treat the whole printed path as one unit.

1. Record the active state directory and every printed blocking path.
2. Reboot the host. Before continuing, ensure that no older NemoClaw binary restarts. Stopping selected processes without a reboot is not sufficient authority-stop proof.
3. Create trusted backup storage and a quarantine directory outside the printed active state directory. Back up each listed path, then move the original whole path into that outside quarantine. Do not delete, edit, or interpret any artifact, and do not move only selected files from a printed transaction directory.
4. Confirm that none of the printed paths remains in the active state directory. Only then retry the current-version rebuild or recreate flow. For recreation, use the manual transfer workflow to copy only trusted user data.

If you cannot complete the reboot, backup, or full quarantine, leave every artifact untouched. NemoClaw continues to block ordinary mutations while any listed path remains in active state.

Do not use cross-sandbox snapshot restore from the legacy sandbox because that flow preserves the source image. The current CLI deliberately does not interpret, resume, or delete ambiguous legacy recovery state.

Restart a Stopped Sandbox Container

If NemoClaw reports that a Docker-driver sandbox is stopped, restart the existing container:

$
nemoclaw <sandbox-name> start

This path preserves the sandbox workspace after the container starts. If the container is paused, follow the printed docker unpause guidance instead. If Docker no longer has the container, destroy the stale NemoClaw entry and onboard a clean replacement. Restore a separately created snapshot afterward if one is available; rebuild cannot recover a workspace or live OpenShell policy that no longer exists.

The start command repairs the agent runtime and host-side port forwards. It returns success only after it authenticates the recovered agent runtime, OpenShell reports the sandbox ready, and host-side port forwards pass their checks. If a check fails, the command exits nonzero, identifies the failure, and prints recovery guidance before you retry start.

Recover the Agent Runtime

recover can start an existing stopped Docker-driver container before it repairs the agent runtime. It starts only a non-paused container that Docker still associates with the registered sandbox. It leaves a running or paused container unchanged. If Docker cannot start the container, recovery continues to the OpenShell readiness check and reports the resulting failure.

If nemoclaw <name> status reports the sandbox container or gateway is not running, run the recover command instead of opening a shell.

$
nemoclaw <sandbox-name> recover

The command repairs a stopped in-sandbox gateway and re-establishes the dashboard port-forward in one step. It is idempotent and safe to script. If the gateway is already healthy, recover exits after the probe and does not restart it. If the host forward is already active, recovery accepts it only after OpenShell ownership is reconciled and the local endpoint is reachable.

Use gateway restart when you intentionally need a supported OpenClaw gateway to reload runtime configuration or plugins.

$
nemoclaw <sandbox-name> gateway restart

The restart command asks the topology-specific controller to stop the tracked gateway child, wait for the entrypoint to launch a replacement, and prove listener and HTTP health. The host then checks or recovers host-side dashboard, messaging, and agent forwards. Refer to nemoclaw <name> recover and nemoclaw <name> gateway restart for details.

Recovery uses registry-scoped privileged direct-container control and does not fall back to ordinary openshell sandbox exec or a manual in-sandbox relaunch. For a local Docker-driver sandbox whose container still uses the legacy keepalive startup, recover can transactionally recreate the registered container with a credential-free managed startup command.

NemoClaw keeps the previous container available throughout the replacement health, OpenShell readiness, state restoration, gateway restart, and settle checks. Before recreation, NemoClaw backs up the state directories and files declared by the agent manifest. NemoClaw waits for OpenShell to re-register the exact replacement before it restores state. After state restoration, it restarts the gateway in that replacement and reruns the managed health and settle checks. It commits only after the replacement identity, state restoration, gateway restart, and settle check pass. NemoClaw removes the temporary state backup after a successful restore or rollback. If state restoration and rollback both fail, it retains the backup and prints host recovery guidance. Mounted state remains available, but a committed swap does not retain other writable-layer changes. After a transactional recreation, NemoClaw waits 120 seconds for OpenShell to re-register the sandbox before state restoration and replacement commit. Set NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS before the recovery command to change this budget. A definitive managed-health failure still stops immediately. If re-registration, state restoration, or a later gateway check fails, NemoClaw attempts to roll back the replacement and leaves the primary dashboard or API host forward stopped. If NemoClaw cannot confirm rollback to the previous container, inspect Docker state before you retry recovery.

For the controller topology, trust boundary, and fail-closed conditions, refer to Understand Gateway Lifecycle Control. If recovery cannot repair a sandbox that needs credentials or a current controller contract, rebuild it.

Understand Launch Readiness Leases

A successful complete preflight for nemoclaw launch <sandbox-name> can publish a credential-free launch-readiness lease on Linux. Linux infrastructure can publish the same evidence with nemoclaw <sandbox-name> connect --probe-only. The lease has a fixed 24-hour lifetime that repeated launches do not extend. Leaving the agent with /exit does not revoke it, and users do not refresh it manually.

During the lease, launch still verifies the owning OpenShell gateway, exact live sandbox identity, registry and agent configuration, effective policy, inference route, required forwards, and semantic runtime health.

Configured inference must return HTTP 2xx from the semantic inference.local probe, which is stricter than the HTTP 200–499 reachability diagnostic used by ordinary connect.

For OpenClaw, connect --probe-only settles the existing allowlisted pairing flow before it publishes a credential-free pairing qualification with the lease. The readiness evidence binds the OpenClaw version and trusted registry and agent manifest configuration. Its pairing qualification binds the canonical CLI client, exact paired device identity, required operator role and scopes, owning OpenShell gateway, sandbox lifecycle identity, and fixed lease epoch. Before lease acceptance, launch makes a bounded, read-only observation of the current OpenClaw-owned pairing state through the owning OpenShell gateway. It skips the complete pairing approval pass only when the qualification still matches exactly and no relevant allowlisted request is pending. Any missing, unreadable, malformed, ambiguous, or changed observation runs the complete pairing approval pass. A relevant allowlisted pending request also runs that complete path.

For a current Portable OpenClaw lifecycle receipt, NemoClaw also requires a finalized onboarding policy step and strictly settled local CLI operator pairing. If only the paired device exists and no request is pending, recovery runs the canonical OpenClaw request producer once and makes at most one approval attempt. An ambiguous approval result receives one final observation and no approval retry. NemoClaw publishes no lease when the policy step is incomplete or the receipt, runtime identity, or pairing state is invalid or ambiguous. The command exits nonzero with an incomplete-onboarding diagnostic and tells you to resume or rerun onboarding.

Hermes and LangChain Deep Agents Code retain their existing session setup on the lease-accepted path. When those checks pass, it can skip duplicate recovery, readiness polling, and inference-route repair. The lease is not a health guarantee or repair authority.

For missing, unsafe, malformed, expired, mismatched, changed, or unhealthy evidence, NemoClaw fences any prior acceptable evidence before it runs the complete preflight. Ordinary launch continues only when NemoClaw proves that no old authority or evidence can exist, or durably rotates the runtime epoch. If an old epoch might exist and cannot be durably rotated, launch and connect --probe-only stop before complete preflight or recovery. Their redacted guidance asks you to repair the current user’s secure OS runtime authority and NemoClaw state permissions, then retry. If NemoClaw securely proves that both the authority and receipt are absent but cannot create new authority, ordinary launch can run the complete preflight without optimization; on Linux, connect --probe-only exits nonzero because it could not publish evidence. If that preflight succeeds before the lease expires, replacement evidence keeps the original start and expiry time. After expiry, a successful complete preflight starts a new 24-hour lease only when publication succeeds.

Before the first mutation in the complete preflight, the producer revalidates its sandbox-global runtime epoch under the sandbox lifecycle lock followed by the owning gateway lock. It holds both locks through all mutations in the complete preflight, final state capture, and publication. A stale producer makes no changes and re-inspects the newer lease.

If unsafe or malformed authority history makes the prior lease timeline untrustworthy, NemoClaw durably invalidates the old epoch and starts one conservative 24-hour quarantine. Publication remains disabled until both wall time and monotonic uptime span the full quarantine. Repeated attempts do not extend it. After it elapses, the next successful complete preflight can publish a new fixed 24-hour lease.

Lease acceptance and publication are currently Linux-only and require a secure, independently writable OS per-user runtime authority under /run/user/<numeric-uid>. It never uses caller-provided environment variables to select this authority.

On macOS, launch runs the complete preflight every time and does not publish a launch-readiness lease. connect --probe-only also runs the complete preflight, including recovery and probes. After a successful probe and recovery, it prints a note that launch-readiness evidence is unavailable on this platform and exits zero. On Linux, the publication-failure diagnostic is redacted and does not print filesystem paths or environment values.

Infrastructure must run connect --probe-only as the same final numeric user that later runs launch. Run it after the final durable home and state volume is mounted and after policy and network provisioning is complete. On completion, connect --probe-only writes one credential-free Probe timing: line with elapsed milliseconds for readiness, authority, lifecycle, gateway, processes, forward, inference, pairing, and publication stages. The line also reports the lifecycle and forward actions, the result, and the failed stage when the probe fails. Timing output is diagnostic only and does not change probe success or failure. NemoClaw rejects evidence after a bound sandbox, configuration, policy, or network identity changes. Deployment ordering remains responsible for external changes that OpenShell and NemoClaw cannot observe.

Host OpenShell Gateway Versus In-Sandbox Agent Recovery

recover and start repair the in-sandbox agent gateway and the host-side port forwards for one named sandbox. connect --probe-only waits for that sandbox to become ready, rechecks it on its recorded gateway, and then verifies or repairs the same sandbox-scoped processes and forwards. These commands do not restart, replace, or reap the shared host OpenShell gateway process. If the host gateway RPC returns an error while these sandbox-scoped commands run, they surface the error with explicit next-step guidance and exit rather than swapping the shared gateway out from under other sandboxes.

The OpenShell CLI has no command that starts a gateway, and openshell status only reports the gateway state. If NemoClaw starts the gateway on your host, run nemoclaw onboard again to repair the host gateway itself. If a deployment outside NemoClaw owns the gateway process, start the gateway with that deployment, then run openshell gateway select <gateway>. NemoClaw prints the applicable recovery guidance when a sandbox-scoped command reports that the host gateway is down. Other workflows, including onboarding, rebuild, and doctor --fix, can explicitly recover the named host gateway when their operation requires it. nemoclaw <name> gateway restart instead restarts only the supported agent gateway inside the named sandbox.

Recover Portable Local Ollama

For a portable experimental-profile sandbox with the recorded ollama-local provider, connect --probe-only and recover also verify the host-side inference chain. Before it decides whether to start Ollama, the command probes http://127.0.0.1:11434/api/tags and leaves a healthy daemon unchanged. When that API is unhealthy, it starts the fixed user-local executable only if NemoClaw has a valid ownership receipt. NemoClaw releases that predate this receipt do not claim an existing executable after an upgrade. To authorize recovery for a previous NemoClaw user-local install, first verify that ${HOME}/.local/bin/ollama is the executable you want NemoClaw to manage. Then run NEMOCLAW_PORTABLE_OLLAMA_REENROLL=1 nemoclaw <name> recover once. The command rejects a symbolic link or non-executable file before it records ownership. It refuses to launch a duplicate when another ollama process exists but the API remains unhealthy. After a launch, recovery waits up to 30 seconds for /api/tags to return valid JSON with a models array. It does not take over a system service or an unrelated user-managed daemon. It refuses a symbolic link, non-regular file, or non-executable file at the receipt-bound executable path.

On every ollama-local completion path, the command verifies the authenticated proxy on port 11435. It also requires HTTP 2xx from https://inference.local/v1/models before it reports success. If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its serve argument, then tells you to retry recovery. An Ollama startup or route failure exits non-zero and prints the available recovery guidance.

Rebuild While Preserving State

If you changed the underlying Dockerfile, upgraded OpenClaw, or want to pick up a new base image without losing your sandbox’s workspace files, use rebuild instead of destroying and recreating.

$
nemoclaw <sandbox-name> rebuild

On WSL with Docker Desktop, a generated replacement image build uses a temporary credential-free Docker configuration when the configured Docker Desktop credential helper is unavailable. NemoClaw removes the temporary configuration after the build and does not modify your Docker configuration. An explicit custom Dockerfile continues to use your configured Docker credentials because its base image or build steps might require a private registry. If that custom rebuild cannot reach the credential helper, restore the Docker Desktop session or credential-helper access before retrying.

Rebuild creates the replacement from the base image selected during preflight, even when the source sandbox records an older base-image hint.

If the sandbox’s container is stopped (for example, an out-of-band container kill left the sandbox in Phase: Error), rebuild starts the container for the duration of the pre-recreate state backup and returns it to its stopped state afterward, the same recovery nemoclaw backup-all uses for a stopped container. If no stopped container can be found for the sandbox, or the retried backup still fails, rebuild aborts without recreating the sandbox; it reports an incomplete-snapshot notice only when a partial snapshot was retained. If the backup itself succeeds but the started container cannot be returned to its stopped state, rebuild also aborts, so a sandbox that was stopped before rebuild started is never left running without a warning.

When the target manifest declares an agent version, NemoClaw runs a fresh live version probe after state restoration. It records the replacement version and reports success only when the observed version matches the rebuild target. An unavailable or different version ends the rebuild with a nonzero status instead of recording the target version as observed.

After post-restore writes, rebuild verifies that the final openclaw.json and .config-hash pair match. If verification fails, NemoClaw exits nonzero and does not report a successful rebuild.

Resolve Rebuild Preflight Stops

Before it backs up or deletes the existing sandbox, rebuild validates the recorded sandbox, gateway, inference route, policy, MCP, agent, and operation-lock state. When one of these checks fails, NemoClaw prints Rebuild preflight failed, explains how to recover, and ends with Aborting rebuild. At this boundary, the existing sandbox is unchanged and no sandbox data has been removed.

Use the recovery guidance that matches the reported check:

• Verify the sandbox name when its registry entry is missing.
• Follow the printed OpenShell gateway recovery steps when the gateway schema is incompatible.
• Restore access to the current OpenShell policy when rebuild reports that the live policy cannot be read, then rerun rebuild. NemoClaw does not reconstruct policy from registry state.
• Replace literal policy credentials with supported OpenShell credential bindings or resolver placeholders when rebuild refuses the policy handoff. NemoClaw stops before it creates the handoff or changes sandbox resources.
• Resolve an incomplete MCP destroy transaction before retrying.
• Back up the sandbox state and recreate it with nemoclaw onboard when the record contains multiple agents. Transactional multi-agent rebuild is not supported.
• Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually.
• Set the live OpenShell inference route to the sandbox’s recorded provider and model when rebuild reports route drift.

A gateway that reports no live inference route does not stop the rebuild. Replacement onboarding configures and verifies the recorded route before it recreates the sandbox.

The rebuild command preserves the mounted workspace and carries the complete current OpenShell policy into sandbox recreation. For that one replacement create, NemoClaw adds baseline fields required by the current replacement image when they are absent from the live policy. Existing values, network keys, and same-name live network entries win, so host changes are not overwritten. NemoClaw does not save a desired-policy copy or enforce that handoff after OpenShell creates the sandbox. When no host web-search key is staged, rebuild preflight reuses an existing Brave or Tavily credential only when the provider name, type, and credential key match the sandbox’s binding on its recorded OpenShell gateway. A missing or mismatched binding stops before recreation and requires the matching host environment variable before you retry. After state restoration, NemoClaw clears a session’s stale model and provider pin when it still targets the managed inference provider but no longer matches the configured default model. The session then follows the current default selected through inference set, while sessions pinned to another provider remain unchanged.

Continue an Interrupted Replacement

Before rebuild deletes the existing sandbox, NemoClaw records a replacement journal in the onboarding session. The journal binds the operation to the sandbox name, recorded OpenShell gateway, source identity, and replacement settings. It stores fingerprints instead of credential values or raw OpenShell sandbox IDs.

If rebuild stops after recording the journal, rerun the command with the same replacement settings. The rerun takes one of these actions:

• It continues deletion when the live sandbox still has the journaled source identity.
• It continues creation when the recorded OpenShell gateway explicitly reports the source sandbox as absent.
• It accepts an existing replacement only when its live identity and sandbox registry generation match the journal.
• It starts a new replacement when the journaled replacement provably never took effect.

NemoClaw treats a journaled replacement as void when the sandbox registry row and the live same-name sandbox report the same OpenShell identity on the journaled gateway, and that identity is not the journaled replacement. A registry row or a live sandbox on any other gateway leaves the journal in place, because that journal may still own an unregistered replacement on its own gateway. The rerun atomically replaces the void journal with a fresh journal against that live source, without leaving an unjournaled interval. It then reports Replaced the void replacement journal for '<name>'; its source sandbox is registered and live. A journal whose replacement never took therefore stops blocking rebuild for the rest of the session.

The journal binds the durable source registry row, so it does not resume after a command that rewrites that row. Messaging channel state is exempt: channels add, channels stop, channels start, and channels remove rewrite the recorded messaging plan, and the journal ignores that field.

$
openshell gateway select <recorded-gateway>
$
openshell status
$
nemoclaw <sandbox-name> destroy --yes

$
nemoclaw <failed-sandbox-name> rebuild --retire-recovery <transaction-id> --yes

A mount-free journal written before NemoClaw bound host-mount identity remains resumable. An older journal for a rebuild with one or more host mounts stops as incompatible, even when the visible mount settings are unchanged, because it cannot prove the original host source identity. Preserve the live sandbox, onboarding session, printed backup, exact error, and the sandbox name, gateway, and journal phase from the Journaled replacement diagnostic. Do not change the target settings, edit the session, or delete the same-name sandbox. Ask a NemoClaw maintainer to review that retained recovery state before taking another recovery action.

An accepted replacement is not deleted again.

The command reports Sandbox '<name>' already holds the replacement from the interrupted rebuild. and preserves the state backup path when one exists. Pass --verbose to include the replacement identifier, OpenShell gateway, and journal phase in rebuild diagnostics.

After the sandbox registry proves the journaled replacement identity and generation, NemoClaw removes an obsolete source image that it owns. It retains the image when the source is shared or the registered replacement reuses it. If image removal fails, NemoClaw keeps the accepted replacement and tells you to run nemoclaw gc for cleanup.

NemoClaw fails closed when the selected gateway, replacement settings, durable source registry fields, or live source or target identity no longer matches the journal. The error names the sandbox and the mismatch that stopped recovery. Do not delete a same-name sandbox to bypass this check. Inspect the named OpenShell gateway and sandbox, correct the reported drift, and rerun the original command. Visible settings cannot correct the legacy host-mount journal case described above.

A same-name recreation started by nemoclaw onboard uses the same replacement journal. If that recreation is interrupted after the Journaled replacement message, rerun the original onboarding command with the same target settings. The active replacement can continue without adding --resume. Use --resume for interrupted onboarding steps that occur before a replacement journal exists.

If any manifest-declared state directory or state file cannot be archived, NemoClaw reports each failed path and stops before deleting the original sandbox. This fail-closed behavior also applies to rebuild --force, even when the backup contains other usable entries. When an incomplete snapshot contains captured entries, NemoClaw retains it at the printed path but excludes it from snapshot list and snapshot restore selection. The retained snapshot may contain unsanitized credentials. Do not restore, copy, or share it. Do not edit its manifest or try to make it selectable. Repair access to the original sandbox state, then rerun the rebuild. Remove the incomplete host-side snapshot only after you verify that the original sandbox or a later complete snapshot contains every required state item.

Refer to nemoclaw <name> rebuild for flag details.

Use the Canonical Configuration Workflows

• Use Switch Inference Providers to change a model or provider.
• Use Credential Rotation to reset or replace a stored provider credential.
• Use Customize the Network Policy to add or remove policy presets.

Related Topics

• Create and Restore Snapshots for the state-preservation contract.
• Troubleshooting for privileged control unavailable, stopped sandboxes, and failed rebuilds.