Recover and Rebuild Sandboxes
Use the lightest recovery operation that repairs the sandbox while preserving its supported state.
Restart a Stopped Sandbox Container
If status reports Phase: Error and confirms that the sandbox container exists but is stopped, restart the existing container:
This path preserves the sandbox workspace and repairs the agent runtime and host-side forwards after the container starts.
If the container is paused, follow the printed docker unpause guidance instead.
If the container is missing or OpenShell reports another terminal phase such as Failed, follow the printed rebuild --yes guidance so NemoClaw can recreate the sandbox from its recorded metadata.
Recover the Agent Runtime
If nemohermes <name> status reports the sandbox is alive but the Hermes gateway is not running, run the recover command instead of opening a shell.
The command repairs a stopped in-sandbox gateway and re-establishes the dashboard port-forward in one step.
It is idempotent and safe to script.
If the gateway is already healthy, recover does not restart it.
If the host forward is already active, recovery accepts it only after OpenShell ownership is reconciled and the local endpoint is reachable.
After it checks gateway health and host forwards, recover checks for a NemoClaw cron restore gate left by an interrupted rebuild.
The gate continues to block new Hermes turns and cron dispatch across gateway and container restarts in the same sandbox.
When the gate exists, recover waits for active agent work to finish and validates the restored cron jobs and scripts.
It clears only the NemoClaw gate after validation succeeds.
If no independent operator drain exists, successful recovery prints Hermes cron dispatch resumed after restored jobs and scripts were validated.
If an operator drain exists, recovery prints Hermes cron restore gate cleared; the independent operator drain remains active.
The command does not own or clear the Hermes operator drain, so new Hermes turns and cron dispatch remain blocked while that drain is active.
If cron validation fails, recover exits nonzero and retains the NemoClaw gate.
Use gateway restart when you intentionally need a supported Hermes gateway to reload runtime configuration or plugins.
The restart command asks the topology-specific controller to stop the tracked gateway child, wait for the entrypoint to launch a replacement, and prove listener and HTTP health. The host then checks or recovers host-side dashboard, messaging, and agent forwards.
For Hermes, the entrypoint supervisor owns the gateway, dashboard process, internal API relay, dashboard relay, and gateway log stream. The nonroot managed supervisor repairs those processes continuously, stops an alive but deaf gateway after four consecutive failed health checks, and quarantines relaunch after five exits within 60 seconds until sandbox recreation.
The host does not start the in-sandbox processes independently.
Refer to nemohermes <name> recover and nemohermes <name> gateway restart for details.
Recovery uses registry-scoped privileged direct-container control and does not fall back to ordinary openshell sandbox exec or a manual in-sandbox relaunch.
For a local Docker-driver sandbox whose container still uses the legacy keepalive startup, recover can transactionally recreate the registered container with a credential-free managed startup command.
NemoClaw keeps the previous container available throughout the replacement health, OpenShell readiness, state restoration, gateway restart, and settle checks.
Before recreation, NemoClaw backs up the state directories and files declared by the agent manifest.
NemoClaw waits for OpenShell to re-register the exact replacement before it restores state.
After state restoration, it restarts the gateway in that replacement and reruns the managed health and settle checks.
It commits only after the replacement identity, state restoration, gateway restart, and settle check pass.
NemoClaw removes the temporary state backup after a successful restore or rollback.
If state restoration and rollback both fail, it retains the backup and prints host recovery guidance.
Mounted state remains available, but a committed swap does not retain other writable-layer changes.
After a transactional recreation, NemoClaw waits 120 seconds for OpenShell to re-register the sandbox before state restoration and replacement commit.
Set NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS before the recovery command to change this budget.
A definitive managed-health failure still stops immediately.
If re-registration, state restoration, or a later gateway check fails, NemoClaw attempts to roll back the replacement and leaves the primary dashboard or API host forward stopped.
For the controller topology, trust boundary, and fail-closed conditions, refer to Understand Gateway Lifecycle Control. If recovery cannot repair a sandbox that needs credentials or a current controller contract, rebuild it.
Host OpenShell Gateway Versus In-Sandbox Agent Recovery
recover and start repair the in-sandbox agent gateway and the host-side port forwards for one named sandbox.
connect --probe-only waits for that sandbox to become ready, rechecks it on its recorded gateway, and then verifies or repairs the same sandbox-scoped processes and forwards.
These commands do not restart, replace, or reap the shared host OpenShell gateway process.
If the host gateway RPC returns an error while these sandbox-scoped commands run, they surface the error with explicit next-step guidance and exit rather than swapping the shared gateway out from under other sandboxes.
To repair the host gateway itself, use openshell status, openshell gateway start --name <gateway>, or run nemohermes onboard again.
Other workflows, including onboarding, rebuild, and doctor --fix, can explicitly recover the named host gateway when their operation requires it.
nemohermes <name> gateway restart instead restarts only the supported agent gateway inside the named sandbox.
Rebuild While Preserving State
If you changed the underlying Dockerfile, upgraded Hermes, or want to pick up a new base image without losing your sandbox’s state files, use rebuild instead of destroying and recreating.
Legacy Upgrade Recovery Scope
When the installer offers prepared backup recovery for a legacy sandbox, the recreate restores only the managed state directory recorded in the validated backup manifest, such as /sandbox/.openclaw or /sandbox/.hermes.
It does not preserve files outside that recorded path, including /sandbox/user-data.
Back up those paths outside the sandbox before you approve legacy recovery.
Resolve Rebuild Preflight Stops
Before it backs up or deletes the existing sandbox, rebuild validates the recorded sandbox, gateway, policy, MCP, agent, and operation-lock state.
When one of these checks fails, NemoClaw prints Rebuild preflight failed, explains how to recover, and ends with Aborting rebuild.
At this boundary, the existing sandbox is unchanged and no sandbox data has been removed.
Use the recovery guidance that matches the reported check:
- Verify the sandbox name when its registry entry is missing.
- Follow the printed OpenShell gateway recovery steps when the gateway schema is incompatible.
- Repair the named pending baseline policy transition, then rerun
rebuild. - Resolve an incomplete MCP destroy transaction before retrying.
- Back up the sandbox state and recreate it with
nemohermes onboardwhen the record contains multiple agents. Transactional multi-agent rebuild is not supported. - Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually.
The rebuild command preserves Hermes state, registered policies, and managed MCP configuration while recreating the container. A rebuild creates a new sandbox home and a new Hermes API bearer token. After the rebuild succeeds, retrieve the replacement token before reconnecting API clients:
Before post-restore repairs, NemoClaw verifies that the recreated sandbox still identifies as Hermes and exits nonzero if its identity does not match the rebuild target.
After state restore, NemoClaw restores managed MCP configuration through the normal lifecycle, then re-proves or recovers gateway health and performs final MCP reconciliation.
rebuild exits nonzero instead of reporting success when it cannot verify final gateway health or managed MCP state.
Follow the printed recovery guidance, using nemohermes <sandbox-name> recover for gateway health and nemohermes <sandbox-name> mcp restart for incomplete managed MCP restoration.
When the rebuild backup contains active Hermes cron jobs that reference scripts, NemoClaw validates those script references before it deletes the existing sandbox.
The check covers the default profile and named profiles.
Each referenced script must exist, be readable, be a regular file, and remain inside its profile’s scripts directory.
Disabled and paused jobs do not require their referenced scripts.
If this validation fails, the rebuild keeps the existing sandbox and reports the preserved backup path.
After NemoClaw creates the replacement, it acquires an independent root-owned gate that blocks new Hermes turns and cron dispatch.
The gate remains active across gateway and container restarts in the replacement sandbox.
NemoClaw waits for active agent work to finish before restoring state.
It validates the restored jobs and scripts against the same running gateway before it clears its gate.
If an operator already drained the gateway, NemoClaw clears only its gate and leaves the operator drain active.
If state restore or cron validation fails after gate acquisition, the command exits nonzero, preserves the backup, and retains the NemoClaw gate.
Failures before gate acquisition do not create a new gate.
Do not manually remove the root-owned cron restore marker because removal bypasses restored cron validation.
After you correct the reported restore problem, run nemohermes <sandbox-name> recover to validate the restored cron tree and clear the NemoClaw gate.
Continue an Interrupted Replacement
Before rebuild deletes the existing sandbox, NemoClaw records a replacement journal in the onboarding session.
The journal binds the operation to the sandbox name, recorded OpenShell gateway, source identity, and replacement settings.
It stores fingerprints instead of credential values or raw OpenShell sandbox IDs.
If rebuild stops after recording the journal, rerun the command with the same replacement settings.
The rerun takes one of these actions:
- It continues deletion when the live sandbox still has the journaled source identity.
- It continues creation when the recorded OpenShell gateway explicitly reports the source sandbox as absent.
- It accepts an existing replacement only when its live identity and sandbox registry generation match the journal.
An accepted replacement is not deleted again.
A rerun that accepts a journaled Hermes replacement checks for any retained NemoClaw gate before it retires the replacement journal.
When the gate exists, the rerun validates the restored cron tree and releases the gate first.
If validation or gate release fails, NemoClaw preserves the accepted replacement, state backup, and replacement journal.
New Hermes turns and cron dispatch remain blocked.
Correct the reported restore problem, then run nemohermes <name> recover.
After recovery succeeds, rerun rebuild with the same replacement settings so NemoClaw can retire the journal.
The command reports Sandbox '<name>' already holds the replacement from the interrupted rebuild. and preserves the state backup path when one exists.
Pass --verbose to include the replacement identifier, OpenShell gateway, and journal phase in rebuild diagnostics.
After the sandbox registry proves the journaled replacement identity and generation, NemoClaw removes an obsolete source image that it owns.
It retains the image when the source is shared or the registered replacement reuses it.
If image removal fails, NemoClaw keeps the accepted replacement and tells you to run nemohermes gc for cleanup.
NemoClaw fails closed when the selected gateway, replacement settings, durable source registry fields, or live source or target identity no longer matches the journal. The error names the sandbox and the mismatch that stopped recovery. Do not delete a same-name sandbox to bypass this check. Inspect the named OpenShell gateway and sandbox, correct the reported drift, and rerun the original command.
A same-name recreation started by nemohermes onboard uses the same replacement journal.
If that recreation is interrupted after the Journaled replacement message, rerun the original onboarding command with the same target settings.
The active replacement can continue without adding --resume.
Use --resume for interrupted onboarding steps that occur before a replacement journal exists.
If an archive command preserves at least one state directory, NemoClaw keeps the usable entries and reports the manifest-defined paths that could not be archived.
If a manifest-declared state file fails, NemoClaw stops before deleting the original sandbox even when it preserved state directories, unless you explicitly pass --force.
If every state directory fails, NemoClaw stops before deleting the original sandbox even when it captured loose files, unless you explicitly pass --force.
rebuild --force can continue when no state directory was preserved or a manifest-declared state file failed.
NemoClaw restores any entries captured in the partial backup; if nothing usable was captured, it recreates the sandbox from recorded registry metadata without restoring prior sandbox state.
Use this recovery path only when losing the state that could not be backed up is acceptable.
When a sandbox with managed MCP servers cannot run a pre-mutation no-op, explicit --force uses its complete registry entries plus the exact live generated policies and provider identities to preserve MCP intent without scrubbing the unreachable in-sandbox adapter.
Every bridge entry must record the adapter for the sandbox’s recorded agent.
The registered policy must match the policy NemoClaw generates for that adapter, server name, endpoint URL, and resolved addresses.
NemoClaw rechecks that read-only snapshot immediately before deletion and stops if the target, registry, policy, provider, or recorded gateway changed.
NemoClaw sends the delete request and every deletion-confirmation lookup to the sandbox’s exact recorded gateway.
Across every rebuild path, NemoClaw does not attempt to stop the local NIM through the delete attempt, and cleanup is attempted on a best-effort basis only after deletion is positively confirmed.
After a nonzero delete, an explicit missing result converges as deleted.
A Ready or Running result triggers an attempt to restore prepared MCP state and any shields lockdown that rebuild temporarily opened.
NemoClaw reports any MCP or shields restoration failure and does not present the operation as a successful rollback.
Any partial or unreachable result remains ambiguous.
NemoClaw preserves the MCP ownership and rebuild-recovery records, does not attempt to stop NIM, skips the rebuild process’s immediate shields relock, and does not claim that the original sandbox is intact.
Inspect the live sandbox and gateway state before retrying recovery.
This recovery also stops for incomplete MCP adds or ambiguous ownership; an error after a successful no-op does not fall back to the host-side path.
When rebuild starts with shields up, NemoClaw opens a 30-minute shields-down window for backup and recreation. A detached auto-lock timer remains the recovery authority until NemoClaw commits a successful shields-up state, including when the host rebuild process exits unexpectedly.
For an older Hermes image that predates sealed shields transitions, only the rebuild workflow may use the descriptor-safe compatibility transition needed to archive and replace the sandbox.
That transition verifies the strict and compatibility hashes and publishes fresh config inodes before changing their lock posture, while ordinary shields up and shields down commands continue to refuse the older protocol.
Refer to nemohermes <name> rebuild for flag details.
Use the Canonical Configuration Workflows
- Use Switch Inference Providers to change a model or provider.
- Use Credential Rotation to reset or replace a stored provider credential.
- Use Customize the Network Policy to add or remove policy presets.
Related Topics
- Create and Restore Snapshots for the state-preservation contract.
- Troubleshooting for
privileged control unavailable, stopped sandboxes, and failed rebuilds.