Recover and Rebuild Sandboxes
Use the lightest recovery operation that repairs the sandbox while preserving its supported state.
Restart a Stopped Sandbox Container
If NemoClaw reports that a Docker-driver sandbox is stopped, restart the existing container:
This path preserves the sandbox workspace and repairs the agent runtime and host-side forwards after the container starts.
If the container is paused, follow the printed docker unpause guidance instead.
If Docker no longer has the container, follow the printed rebuild --yes guidance so NemoClaw can recreate the sandbox from its recorded metadata.
Recover the Agent Runtime
Deep Agents sandboxes are terminal runtimes and do not expose an OpenClaw or Hermes in-sandbox gateway.
Use nemo-deepagents <sandbox-name> status, logs, connect, and rebuild for recovery.
If the terminal runtime reports degraded health, rebuild the sandbox instead of using recover or gateway restart.
Understand Launch Readiness Leases
A successful complete preflight for nemo-deepagents launch <sandbox-name> can publish a credential-free launch-readiness lease on Linux.
Linux infrastructure can publish the same evidence with nemo-deepagents <sandbox-name> connect --probe-only.
The lease has a fixed 24-hour lifetime that repeated launches do not extend.
Leaving the agent with /exit does not revoke it, and users do not refresh it manually.
During the lease, launch still verifies the owning OpenShell gateway, exact live sandbox identity, registry and agent configuration, effective policy, inference route, required forwards, and semantic runtime health.
Configured inference must return HTTP 2xx from the semantic inference.local probe for every provider except openrouter-api.
The OpenRouter adapter returns HTTP 404 for GET /v1/models.
NemoClaw accepts that result only after a bounded inference request for the recorded model succeeds.
A failed request rejects launch readiness and identifies the inference request as the failed check.
Hermes and LangChain Deep Agents Code retain their existing session setup on the lease-accepted path.
When those checks pass, it can skip duplicate recovery, readiness polling, and inference-route repair.
The lease is not a health guarantee or repair authority.
For missing, unsafe, malformed, expired, mismatched, changed, or unhealthy evidence, NemoClaw fences any prior acceptable evidence before it runs the complete preflight.
Ordinary launch continues only when NemoClaw proves that no old authority or evidence can exist, or durably rotates the runtime epoch.
If an old epoch might exist and cannot be durably rotated, launch and connect --probe-only stop before complete preflight or recovery.
Their redacted guidance asks you to repair the current user’s secure OS runtime authority and NemoClaw state permissions, then retry.
If NemoClaw securely proves that both the authority and receipt are absent but cannot create new authority, ordinary launch can run the complete preflight without optimization; on Linux, connect --probe-only exits nonzero because it could not publish evidence.
If that preflight succeeds before the lease expires, replacement evidence keeps the original start and expiry time.
After expiry, a successful complete preflight starts a new 24-hour lease only when publication succeeds.
Before the first mutation in the complete preflight, the producer revalidates its sandbox-global runtime epoch under the sandbox lifecycle lock followed by the owning gateway lock. It holds both locks through all mutations in the complete preflight, final state capture, and publication. A stale producer makes no changes and re-inspects the newer lease.
If unsafe or malformed authority history makes the prior lease timeline untrustworthy, NemoClaw durably invalidates the old epoch and starts one conservative 24-hour quarantine. Publication remains disabled until both wall time and monotonic uptime span the full quarantine. Repeated attempts do not extend it. After it elapses, the next successful complete preflight can publish a new fixed 24-hour lease.
Lease acceptance and publication are currently Linux-only and require a secure, independently writable OS per-user runtime authority under /run/user/<numeric-uid>.
It never uses caller-provided environment variables to select this authority.
On macOS, launch runs the complete preflight every time and does not publish a launch-readiness lease.
connect --probe-only also runs the complete preflight, including recovery and probes.
After a successful probe and recovery, it prints a note that launch-readiness evidence is unavailable on this platform and exits zero.
On Linux, the publication-failure diagnostic is redacted and does not print filesystem paths or environment values.
Infrastructure must run connect --probe-only as the same final numeric user that later runs launch.
Run it after the final durable home and state volume is mounted and after policy and network provisioning is complete.
On completion, connect --probe-only writes one credential-free Probe timing: line with elapsed milliseconds for readiness, authority, lifecycle, gateway, processes, forward, inference, pairing, and publication stages.
The line also reports the lifecycle and forward actions, the result, and the failed stage when the probe fails.
Timing output is diagnostic only and does not change probe success or failure.
NemoClaw rejects evidence after a bound sandbox, configuration, policy, or network identity changes.
Deployment ordering remains responsible for external changes that OpenShell and NemoClaw cannot observe.
Rebuild While Preserving State
If you changed the underlying Dockerfile, upgraded Deep Agents Code, enabled Tavily Search, or want to pick up a new base image without losing manifest-defined Deep Agents state, use rebuild instead of destroying and recreating.
On WSL with Docker Desktop, a generated replacement image build uses a temporary credential-free Docker configuration when the configured Docker Desktop credential helper is unavailable. NemoClaw removes the temporary configuration after the build and does not modify your Docker configuration. An explicit custom Dockerfile continues to use your configured Docker credentials because its base image or build steps might require a private registry. If that custom rebuild cannot reach the credential helper, restore the Docker Desktop session or credential-helper access before retrying.
Resolve Rebuild Preflight Stops
Before it backs up or deletes the existing sandbox, rebuild validates the recorded sandbox, gateway, inference route, policy, MCP, agent, and operation-lock state.
When one of these checks fails, NemoClaw prints Rebuild preflight failed, explains how to recover, and ends with Aborting rebuild.
At this boundary, the existing sandbox is unchanged and no sandbox data has been removed.
Use the recovery guidance that matches the reported check:
- Verify the sandbox name when its registry entry is missing.
- Follow the printed OpenShell gateway recovery steps when the gateway schema is incompatible.
- Repair the named pending baseline policy transition, then rerun
rebuild. - Resolve an incomplete MCP destroy transaction before retrying.
- Back up the sandbox state and recreate it with
nemo-deepagents onboardwhen the record contains multiple agents. Transactional multi-agent rebuild is not supported. - Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually.
- Set the live OpenShell inference route to the sandbox’s recorded provider and model when rebuild reports route drift.
A gateway that reports no live inference route does not stop the rebuild. Replacement onboarding configures and verifies the recorded route before it recreates the sandbox.
The rebuild command preserves manifest-defined Deep Agents state, regenerates config.toml, reconstructs managed MCP projection state, and reapplies registered policies while recreating the container.
Continue an Interrupted Replacement
Before rebuild deletes the existing sandbox, NemoClaw records a replacement journal in the onboarding session.
The journal binds the operation to the sandbox name, recorded OpenShell gateway, source identity, and replacement settings.
It stores fingerprints instead of credential values or raw OpenShell sandbox IDs.
If rebuild stops after recording the journal, rerun the command with the same replacement settings.
The rerun takes one of these actions:
- It continues deletion when the live sandbox still has the journaled source identity.
- It continues creation when the recorded OpenShell gateway explicitly reports the source sandbox as absent.
- It accepts an existing replacement only when its live identity and sandbox registry generation match the journal.
A mount-free journal written before NemoClaw bound host-mount identity remains resumable.
An older journal for a rebuild with one or more host mounts stops as incompatible, even when the visible mount settings are unchanged, because it cannot prove the original host source identity.
Preserve the live sandbox, onboarding session, printed backup, exact error, and the sandbox name, gateway, and journal phase from the Journaled replacement diagnostic.
Do not change the target settings, edit the session, or delete the same-name sandbox.
Ask a NemoClaw maintainer to review that retained recovery state before taking another recovery action.
An accepted replacement is not deleted again.
The command reports Sandbox '<name>' already holds the replacement from the interrupted rebuild. and preserves the state backup path when one exists.
Pass --verbose to include the replacement identifier, OpenShell gateway, and journal phase in rebuild diagnostics.
After the sandbox registry proves the journaled replacement identity and generation, NemoClaw removes an obsolete source image that it owns.
It retains the image when the source is shared or the registered replacement reuses it.
If image removal fails, NemoClaw keeps the accepted replacement and tells you to run nemo-deepagents gc for cleanup.
NemoClaw fails closed when the selected gateway, replacement settings, durable source registry fields, or live source or target identity no longer matches the journal. The error names the sandbox and the mismatch that stopped recovery. Do not delete a same-name sandbox to bypass this check. Inspect the named OpenShell gateway and sandbox, correct the reported drift, and rerun the original command. Visible settings cannot correct the legacy host-mount journal case described above.
A same-name recreation started by nemo-deepagents onboard uses the same replacement journal.
If that recreation is interrupted after the Journaled replacement message, rerun the original onboarding command with the same target settings.
The active replacement can continue without adding --resume.
Use --resume for interrupted onboarding steps that occur before a replacement journal exists.
If an archive command preserves at least one state directory, NemoClaw keeps the usable entries and reports the manifest-defined paths that could not be archived.
If a manifest-declared state file fails, NemoClaw stops before deleting the original sandbox even when it preserved state directories, unless you explicitly pass --force.
If every state directory fails, NemoClaw stops before deleting the original sandbox even when it captured loose files, unless you explicitly pass --force.
rebuild --force can continue when no state directory was preserved or a manifest-declared state file failed.
NemoClaw restores any entries captured in the partial backup; if nothing usable was captured, it recreates the sandbox from recorded registry metadata without restoring prior sandbox state.
Use this recovery path only when losing the state that could not be backed up is acceptable.
When a sandbox with managed MCP servers cannot run a pre-mutation no-op, explicit --force uses its complete registry entries plus the exact live generated policies and provider identities to preserve MCP intent without scrubbing the unreachable in-sandbox adapter.
Every bridge entry must record the adapter for the sandbox’s recorded agent.
The registered policy must match the policy NemoClaw generates for that adapter, server name, endpoint URL, and resolved addresses.
NemoClaw rechecks that read-only snapshot immediately before deletion and stops if the target, registry, policy, provider, or recorded gateway changed.
NemoClaw sends the delete request and every deletion-confirmation lookup to the sandbox’s exact recorded gateway.
Across every rebuild path, NemoClaw does not attempt to stop the local NIM through the delete attempt, and cleanup is attempted on a best-effort basis only after deletion is positively confirmed.
After a nonzero delete, an explicit missing result converges as deleted.
A Ready or Running result triggers an attempt to restore prepared MCP state and any shields lockdown that rebuild temporarily opened.
NemoClaw reports any MCP or shields restoration failure and does not present the operation as a successful rollback.
Any partial or unreachable result remains ambiguous.
NemoClaw preserves the MCP ownership and rebuild-recovery records, does not attempt to stop NIM, skips the rebuild process’s immediate shields relock, and does not claim that the original sandbox is intact.
Inspect the live sandbox and gateway state before retrying recovery.
This recovery also stops for incomplete MCP adds or ambiguous ownership; an error after a successful no-op does not fall back to the host-side path.
When rebuild starts with shields up, NemoClaw opens a 30-minute shields-down window for backup and recreation. A detached auto-lock timer remains the recovery authority until NemoClaw commits a successful shields-up state, including when the host rebuild process exits unexpectedly.
Refer to nemo-deepagents <name> rebuild for flag details.
Use the Canonical Configuration Workflows
- Use Switch Inference Providers to change a model or provider.
Related Topics
- Create and Restore Snapshots for the state-preservation contract.
- Troubleshooting for
privileged control unavailable, stopped sandboxes, and failed rebuilds.