Troubleshooting
This page covers common installation, onboarding, and runtime issues, along with resolution steps.
The diagnostic commands on this page assume nemo-deepagents is on your PATH (re-source your shell profile after an nvm- or fnm-managed install) and that your user can reach the Docker socket — either as a member of the docker group or by running the Docker commands with sudo.
Get Help
If your issue is not listed here, join the NemoClaw Discord channel to ask questions and get help from the community. You can also file an issue on GitHub.
Installation
nemo-deepagents not found after install
If you use nvm or fnm to manage Node.js, the installer may not update your current shell’s PATH.
The nemo-deepagents binary is installed but the shell session does not know where to find it.
Run source ~/.bashrc (or source ~/.zshrc for zsh), or open a new terminal window.
When installing from a source checkout with npm install, NemoClaw first tries npm link.
If the global npm prefix is not writable, it writes a managed shim to ~/.local/bin/nemoclaw instead.
Add ~/.local/bin to your PATH if the command is still not found.
Source-checkout installs also bootstrap OpenShell when it is missing before running preflight.
If a source install still reports that openshell is not available, re-run the installer from the repository root and check that ~/.local/bin is on your PATH.
Installer fails on unsupported platform
The installer checks for a supported OS and architecture before proceeding. If you see an unsupported platform error, verify that you are running on a tested platform listed in the Container Runtimes table in the quickstart guide.
Node.js version is too old
NemoClaw requires Node.js 22.19 or later. If the installer exits with a Node.js version error, check your current version:
If the version is below 22.19, install a supported release. If you use nvm, run:
Then re-run the installer.
Contributor Setup Fails with a JavaScript Heap Out-of-Memory Error
This applies to a source checkout, not to an installed release.
Node.js derives its default old-space limit from host memory.
On a host with 8 GB of RAM, that limit is about 2.2 GB.
The CLI type check needs more heap than that limit, so ./scripts/dev-setup.sh stops at the type-check step and Node.js reports JavaScript heap out of memory.
Raise the limit, then run setup again:
Keep that variable set for later type-check, build, and test commands.
Image push fails with out-of-memory errors
The sandbox image is approximately 2.4 GB compressed. During image push, the Docker daemon, k3s, and the OpenShell gateway run alongside the export pipeline, which buffers decompressed layers in memory. On machines with less than 8 GB of RAM, this combined usage can trigger the OOM killer.
If you cannot add memory, configure at least 8 GB of swap to work around the issue at the cost of slower performance.
Docker is not running
Check the host before onboarding:
The command does not start Docker or apply a repair.
A host.docker.daemon_unreachable finding means Docker is installed but NemoClaw cannot reach the daemon.
For JSON output and exit-code details, refer to System Readiness.
The installer and onboard wizard require Docker to be running. If you see a Docker connection error, start the Docker daemon:
On macOS with Docker Desktop, open the Docker Desktop application and wait for it to finish starting before retrying.
Docker permission denied on Linux
On Linux, if the Docker daemon is running but you see “permission denied” errors, your user may not be in the docker group.
The installer can add your user to the group, but Linux does not activate that membership in the current shell automatically.
Add your user and activate the group in the current shell:
Docker group access
NemoClaw needs Docker access.
On personal Linux development machines, adding your user to the docker group is the standard way to run Docker without sudo.
Members of the docker group can control the daemon with root-level impact, so grant this access only to trusted local accounts; on shared or managed systems, use your organization’s approved Docker access path.
For background, review Docker’s daemon attack surface guidance.
Then retry nemo-deepagents onboard.
If the installer stopped after printing newgrp docker, run that command and then re-run the installer:
Installer reports Docker access outside the docker group
On Linux, the installer may report that Docker is reachable even though your user is not in the docker group.
This means the host grants Docker daemon access through another path, such as a custom DOCKER_HOST, socket ACL, or managed runtime policy.
NemoClaw can continue when docker info works, but the diagnostic explains why a negative Docker-permission test will not reproduce on that host.
Check the Docker access path before relying on the host as a clean permission baseline:
The managed default gateway service accepts DOCKER_HOST only as an absolute local unix:// socket path.
It rejects remote endpoints and relative socket paths before service startup.
Onboarding Reports an Invalid Docker Host
The invalid_docker_host advisory means that DOCKER_HOST is not an absolute local unix:// socket path that NemoClaw can write to the managed OpenShell gateway service environment.
NemoClaw does not use the standalone gateway fallback when this validation fails.
Onboarding prints the advisory identifier in parentheses after each action title in the Suggested fix list.
The terminal output names invalid_docker_host when this validation fails, so you can match the message to this section.
Remove the override to use Docker’s default local socket:
If your Docker daemon uses another local socket, set an absolute unix:// path before you retry:
NemoClaw rejects TCP and SSH endpoints, relative socket paths, values that contain single quotes, and values that contain line breaks. Do not wrap the socket path in single quotes inside the variable value.
Onboarding Warns About the Docker Desktop Credential Store in a Headless Session
The docker_desktop_credential_store_headless advisory means that the Docker client config sets credsStore to desktop (macOS) or desktop.exe (WSL2) and the session looks headless, for example an SSH session without a GUI.
The Docker Desktop credential helper needs an interactive GUI session.
Without one, the helper can fail and block every image pull, even for public images.
A common failure message from Docker in this state is A specified logon session does not exist.
Onboarding preflight prints this warning before the first image pull and then continues, on both fresh and resumed onboarding.
NemoClaw reads the config from $DOCKER_CONFIG/config.json when DOCKER_CONFIG is set, and from ~/.docker/config.json otherwise.
On WSL, NemoClaw probes the credential helper with a read-only list call instead of relying on session markers, because WSLg can set DISPLAY in every WSL shell.
To work around a failed pull, resume onboarding with an isolated Docker config so every remaining image pull bypasses the unavailable helper:
Alternatively, temporarily remove the credsStore entry from the Docker client config named above, then rerun nemo-deepagents onboard.
Restore the entry afterward if you use registries that need stored credentials in GUI sessions.
macOS first-run failures
The two most common first-run failures on macOS are missing developer tools and Docker connection errors.
To avoid these issues, install the prerequisites in the following order before running the NemoClaw installer:
- Install Xcode Command Line Tools (
xcode-select --install). These are needed by the installer and Node.js toolchain. - Install and start a supported container runtime (Docker Desktop or Colima). Without a running runtime, the installer cannot connect to Docker.
docker is missing after installing Colima
Homebrew Colima does not install the Docker CLI binary.
If you install only Colima, colima start can succeed while later docker commands fail with command not found.
Install both packages, start Colima with enough resources for the sandbox image build, and verify Docker before onboarding:
Permission errors during installation
The NemoClaw installer does not require sudo or root.
It installs Node.js via nvm and NemoClaw via npm, both into user-local directories.
The installer also handles OpenShell installation automatically using a pinned release.
If you see permission errors during installation, they typically come from Docker, not the NemoClaw installer itself. Docker must be installed and running before you run the installer, and installing Docker may require elevated privileges on Linux.
npm install fails with permission errors
If npm install fails with an EACCES permission error, do not run npm with sudo.
Instead, configure npm to use a directory you own:
Add the export line to your ~/.bashrc or ~/.zshrc to make it permanent, then re-run the installer.
Installer fails on NVIDIA Jetson
The installer auto-detects NVIDIA Jetson devices (Orin and Thor) and applies required host configuration before the normal install flow.
If the Jetson setup step fails, verify that you have sudo access and that Docker is installed and running.
For JetPack 6 (L4T 36.x), the setup switches iptables to legacy mode and adjusts the Docker daemon configuration.
For JetPack 7 (L4T 38.x / Thor), only bridge netfilter and sysctl settings are applied.
For JetPack 7 (L4T 39.x), bridge netfilter is loaded only when the host is missing it.
Some R39 images already ship with br_netfilter configured and are left untouched.
On affected R39 hosts, the installer prints loading br_netfilter (required by k3s inside the OpenShell gateway).
Without this fix, sandbox pods fail DNS resolution against the in-cluster service and the onboard Setting up OpenClaw inside sandbox step times out.
If the L4T version is not recognized, the setup step is skipped and the installer continues normally.
DNS resolution from inside docker fails (corporate firewall)
Some corporate networks block outbound UDP port 53 to public DNS servers and force all host name resolution through DNS over TLS on TCP port 853. Containers do not inherit the host’s DNS-over-TLS configuration, so the sandbox build’s npm ci step times out trying to resolve registry.npmjs.org against 1.1.1.1 or 8.8.8.8.
NemoClaw’s preflight runs a short docker run --rm busybox nslookup nemoclaw-dns-probe-<random>.invalid probe before starting the sandbox build. The fresh .invalid name should return NXDOMAIN through a working resolver, so cached answers cannot hide blocked DNS egress. When the probe confirms a DNS failure, onboarding stops with platform-specific remediation instead of hanging for ~15 minutes and printing a cryptic Exit handler never called.
Use the preflight headline to choose the recovery path:
- If no DNS servers could be reached, Docker could not reach its configured resolver. Follow the platform-specific UDP port 53 and Docker DNS steps below.
- If the DNS server was reachable but rejected the query with
NXDOMAINorREFUSED, the resolver answered, so the UDP port 53 fix is not relevant. Check the resolver used by Docker, such as dnsmasq, Pi-hole, unbound, or systemd-resolved, and remove any forwarding rule, blocklist entry, or ACL that rejectsregistry.npmjs.org. If needed, configure Docker to use an organization-approved resolver that can resolve public names, restart Docker, and retry onboarding.
For an unreachable resolver, pick the matching platform path below, apply it, then re-run nemo-deepagents onboard.
- Linux with systemd-resolved. Add a
DNSStubListenerExtradrop-in pointing at the docker bridge gateway IP (the preflight prints the detected IP), then add the same IP to/etc/docker/daemon.jsonunderdns. Restartsystemd-resolvedanddocker. - macOS with Colima. Restart Colima with the corporate DNS address, for example
colima stop && colima start --dns <corp-dns-ip>. - macOS with Docker Desktop. Add the corporate DNS address to
~/.docker/daemon.jsonunderdns, then restart Docker Desktop. - Windows or WSL. Configure DNS in the Docker Desktop settings GUI, or apply the Linux fix above when running native docker inside WSL.
Verify the fix worked:
When the lookup returns an answer, retry onboarding.
Direct DNS lookups fail in a Docker-driver GPU sandbox
Egress covered by OpenShell network policies resolves destinations through the gateway. A direct DNS lookup inside the agent network namespace can fail on Docker-driver GPU hosts such as DGX Spark even when policy-covered inference, messaging, and search work normally. Direct in-sandbox DNS depends on Docker and the host resolver and is not a supported NemoClaw network-policy path, so NemoClaw does not use it as a sandbox health check.
Run a manual lookup only when you are diagnosing a custom tool that performs its own DNS resolution:
If the host alias resolves but the external name does not, Docker’s embedded resolver may be forwarding to an upstream DNS server that the sandbox bridge cannot use.
Do not treat this result as evidence that a policy-covered feature is unhealthy.
Test the affected inference, messaging, or search request through its normal policy path and inspect denied requests with openshell term.
If a custom tool requires direct DNS, configure the Docker daemon to use a resolver that containers can reach. On hosts with VPN or split-DNS software, use an upstream resolver that remains reachable from the Docker bridge, then recreate or rebuild the sandbox. Keep bridge networking enabled so the sandbox retains its normal Docker network isolation.
Host DNS resolution is blocked before provider validation
NemoClaw also checks that the host process can resolve the provider host before it starts NVIDIA provider validation.
A firewall rule that blocks host DNS traffic on port 53 can make later validation fail with curl: (6) Could not resolve host: integrate.api.nvidia.com even when container DNS probes look healthy.
Current onboarding stops earlier with a host DNS diagnostic and remediation hints.
Verify host DNS outside NemoClaw:
Fix the host firewall, VPN, or DNS policy so the host can resolve the provider endpoint, then rerun onboarding.
If you intentionally use a non-NVIDIA provider and need to bypass only this preflight, set NEMOCLAW_SKIP_HOST_DNS_PREFLIGHT=1.
Older-glibc gateway compatibility container
OpenShell 0.0.106 directly supports Linux hosts with glibc 2.39 or newer.
On an older trusted host, NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1 explicitly opts into NemoClaw’s compatibility container.
Leave it unset on supported hosts.
The compatibility container uses host networking and mounts the host Docker socket read-only. A read-only socket mount still permits privileged Docker API operations and can control the host, so do not enable this mode on an untrusted or shared host. The gateway remains loopback-bound, and startup fails closed unless the configured Unix socket answers as a Docker daemon. See Gateway Compatibility Container for the container boundary and removal conditions.
Refer to Environment Variables for the full list of port overrides.
A Gateway Port Stays Bound After Uninstall or Re-Onboard
A gateway port that keeps listening after you uninstall, or that keeps serving after you re-onboard without NEMOCLAW_GATEWAY_PORT set, can belong to a second environment.
Onboarding under a non-default NEMOCLAW_GATEWAY_PORT registers the sandbox on gateway nemoclaw-<port> and stores its registry and state under ~/.nemoclaw/gateways/<port>/.
Every later command reads the state root that its own NEMOCLAW_GATEWAY_PORT selects, so a command run without that variable operates on port 8080 and neither reuses nor stops the other environment.
Clearing the variable does not move an existing sandbox back to the default port.
List the gateways the host still has:
If the gateway name and its port-scoped state remain, treat it as a second environment and select that port for cleanup.
If the gateway is absent but the port still listens, cleanup did not stop the listener; follow the process or service remediation printed by uninstall before you retry.
If uninstall reported that it kept an openshell-gateway process owned by another user running, that process still holds the port.
This can happen after uninstall exits successfully because NemoClaw does not treat another user’s process as a cleanup failure.
Ask that user to stop the process, or onboard under a different NEMOCLAW_GATEWAY_PORT.
Remove one environment by selecting its port:
Remove every gateway port in one run:
After you confirm uninstall and it exits with status 0, run openshell gateway list again.
For a NemoClaw-managed gateway without --keep-openshell, the gateway name that uninstall removed must be absent.
An externally supervised gateway or a run with --keep-openshell preserves the gateway process and its resources.
The same scoping applies to nemo-deepagents <name> stop.
When stop reports that no valid gateway binding is registered for a sandbox, the sandbox can be registered under a different gateway port.
Rerun stop with that NEMOCLAW_GATEWAY_PORT value set.
If that does not find the sandbox, resolve the missing, invalid, or unreadable registry entry that the command reports.
Refer to Uninstall NemoClaw for the full sweep contract.
A shared inference route conflicts with another sandbox
Refer to Use Shared Gateway Routes for route time-sharing, provider-global compatibility, and status drift fields.
If inference set reports a valid shared-route conflict, align the named sandbox records or remove a sandbox you no longer need.
If onboarding or connect reports a provider-global identity conflict, align the same-name provider’s custom endpoint, API family, and credential environment-variable name across the named sandboxes, or remove a conflicting sandbox you no longer need.
If the error names incomplete legacy custom-route metadata, back up and remove the affected sandbox, then re-onboard it with an explicit custom endpoint and API family. For an OpenAI-compatible route, replace the example endpoint, model, and sandbox name in this recovery sequence:
If the error names an invalid gateway binding, restore the affected row’s known-good gatewayName and gatewayPort metadata from a trusted backup; otherwise back up and remove the sandbox, then re-onboard it.
Do not guess or copy a binding from another sandbox because lifecycle commands use it to select the gateway.
Onboarding
Cgroup v2 errors during onboard
Older NemoClaw releases relied on a Docker cgroup workaround on Ubuntu 24.04, DGX Spark, and WSL2. Current OpenShell releases handle that behavior themselves, so NemoClaw no longer requires a Spark-specific setup step.
If onboarding reports that Docker is missing or unreachable, fix Docker first and retry onboarding:
Podman is not a tested runtime. If onboarding or sandbox lifecycle fails, switch to a tested runtime (Docker Desktop, Colima, or Docker Engine) and rerun onboarding.
Cluster fails with overlayfs snapshotter cannot be enabled on Docker 26+
Docker Engine 26 and later default fresh installations to the containerd image store, which exposes its layers via the overlayfs snapshotter rather than the legacy overlay2 graph driver.
The k3s server inside the OpenShell cluster image needs to mount its own overlay filesystem on top, and the kernel rejects nesting two non-trivial overlay mounts.
The cluster container then loops with:
This is a Docker default-driver change, not a NemoClaw or OpenShell regression.
The same hardware uses the legacy overlay2 driver and is unaffected when it runs Docker 25 or earlier, or any Docker version with the containerd image store disabled.
NemoClaw detects the Docker 26+ containerd-snapshotter overlayfs configuration during onboarding and transparently builds a small drop-in replacement for the cluster image on the local Docker engine.
The patched image installs fuse-overlayfs and selects it as the k3s snapshotter, bypassing the kernel-level nested-overlay limitation.
No host configuration changes, sudo, or Docker restart required.
The auto-fix runs once per OpenShell version on the affected host.
Subsequent onboarding runs reuse the cached patched image.
Hosts without the conflict (Driver: overlay2 in docker info, macOS Docker Desktop, or Linux installations that disable the containerd image store) see no change in behavior.
Override knobs:
NEMOCLAW_DISABLE_OVERLAY_FIX=1: skip the auto-fix and run against the unmodified upstream cluster image. Useful for diagnosis or when you have already applied the manual workaround below.NEMOCLAW_OVERLAY_SNAPSHOTTER=native: build the patched image with k3s’snativesnapshotter instead offuse-overlayfs. Thenativesnapshotter copies image layers instead of overlaying them, so it uses more disk but does not depend on FUSE. Default isfuse-overlayfs.
If you prefer to disable the new Docker storage driver instead of running the patched image, edit /etc/docker/daemon.json:
Then restart Docker (sudo systemctl restart docker) and re-run nemo-deepagents onboard.
This restores the legacy overlay2 driver host-wide, which kills any other running containers.
Prefer the auto-fix unless you need the change for unrelated reasons.
Switching storage drivers also rebuilds the entire local image graph: previously-pulled images become unusable and Docker re-pulls them on first reference, so expect a cold cache and additional disk usage right after the restart.
OpenShell version above maximum
Each NemoClaw release validates against a range of tested OpenShell versions.
If the installed OpenShell version exceeds the configured maximum, nemo-deepagents onboard exits with an error:
Upgrade NemoClaw to a version that supports your OpenShell release, or install a supported OpenShell version from the OpenShell releases page.
For fresh installs, NemoClaw passes the blueprint range to install-openshell.sh and resolves a compatible published OpenShell release before downloading.
If GitHub release metadata is unavailable, the script uses its bundled fallback pin and the post-install gate still enforces the configured range.
Installer Reports an OpenShell Gateway Version Mismatch
On Linux, an existing OpenShell package can provide a systemd user service that starts a different gateway version from the user-local version that NemoClaw installs. The installer stops before onboarding instead of using the two versions together. The error reports both gateway versions and binary paths.
Do not remove the existing OpenShell package if its gateway manages resources outside NemoClaw. Package removal can stop that gateway. Align the package with the version in the installer error, or plan the migration of those resources first.
If you no longer need the APT-installed OpenShell package, remove it and rerun the installer:
The next installer run must continue past the OpenShell installation step without reporting a version mismatch.
Docker Driver Gateway Reports an Incompatible Migration
Onboarding can stop when the Docker driver gateway log contains both parts of either error:
The first error means the installed OpenShell migration set does not contain migration N.
The second error means that migration set defines migration N with different contents.
NemoClaw identifies <selected-state-dir>/openshell.db as incompatible with the installed OpenShell migration set for both errors.
This failure can happen after an OpenShell downgrade.
Installing a NemoClaw release that is older than the installed one performs that downgrade, because each release pins one OpenShell version and the installer reinstalls OpenShell at the pin.
You reach that state in one of three ways:
- You select an older release with
NEMOCLAW_INSTALL_TAGorNEMOCLAW_INSTALL_REF. - The default
lkgrelease is older than the NemoClaw release already on the host. - You install an older OpenShell yourself.
The diagnosis always prints the database path. When an unused archive path is available, it also prints that archive path beside the selected state directory and the profile-specific onboarding command. When no unused archive path is available, it asks you to keep the gateway stopped and inspect the state directory instead.
The selected state directory contains the gateway database, mutual TLS private keys, JSON Web Token signing material, and every sandbox and provider registration on the selected gateway. Moving it makes those registrations and credentials unavailable to the fresh gateway. Other sandboxes on the selected gateway can require re-onboarding and credential entry.
When a service manager owns the gateway, the printed recovery stops it in the same command chain that moves the state directory. NemoClaw cannot establish in advance that the directory stays free, because the managed service restarts the gateway on failure and a replacement can start at any point before the move. Running the stop inside the chain removes that gap.
NemoClaw resolves the owning service before it prints, so the stop names the unit that runs on this host: the upstream OpenShell package unit, the NemoClaw user service, or the Homebrew formula. When no service manager owns the gateway, NemoClaw runs it standalone. Before it offers the state move in that case, the gateway runtime checks the recorded process and scans current gateway process identities for the runtime namespace tied to the selected state directory. It withholds the move unless that scan establishes that the standalone gateway state is unused.
When NemoClaw prints the state move, run the exact commands it prints:
- Stop the owning gateway service, when the printed chain includes that step.
- Create the printed
.incompatiblearchive with owner-only access. If that path exists, NemoClaw adds a numeric suffix instead of nesting or replacing an earlier archive. - Move the selected state directory into the archive as
gateway-state. - Run the printed onboarding command only after the stop, the archive, and the move succeed.
Standard onboarding prints
nemo-deepagents onboard --resume. The portable experimental profile prints its required fresh-onboarding command for this gateway-state recovery.
The archive remains beside the selected state directory and retains the previous gateway records and credentials. Keep it owner-only until onboarding completes and every required sandbox and provider registration is restored. Delete the archive only after you no longer need its gateway records or credentials for recovery.
Installer Reports That the Systemd User Manager Is Unavailable
On Linux, an OpenShell package can install /usr/lib/systemd/user/openshell-gateway.service on a host without a reachable systemd user manager.
The service query can then return this diagnostic:
The installer accepts only recognized user-manager-unavailable diagnostics for the standalone gateway fallback.
It checks .wants, .requires, and .upholds links in the standard systemd user unit paths.
The installer keeps the standalone lifecycle only when neither gateway service has an activation path that can later claim port 8080.
The installer also stops when SYSTEMD_UNIT_PATH overrides the standard paths.
The installer does not parse, modify, or remove a package or foreign unit to make this decision.
If an activation path exists, the installer stops because the service can start later and compete for port 8080.
Restore the systemd user manager, then inspect both possible services:
Resolve the competing service through its package or platform owner.
Do not delete an activation link or edit a unit file by hand.
Rerun the installer only after the owner confirms that no enabled user service can claim port 8080.
Unknown service query errors remain fatal. The installer also stops for malformed effective metadata, an untrusted unit or executable path, an executable failure, or a gateway version mismatch. Follow the reported condition instead of forcing the standalone fallback.
Sandbox containers cannot reach the gateway
On native Linux Docker-driver hosts, nemo-deepagents onboard verifies the route that sandbox containers use to reach the OpenShell gateway.
If a host firewall blocks that path, onboarding exits with output like:
Apply the ufw command printed by onboarding, then rerun onboarding.
If the message does not include a subnet, derive it from the OpenShell Docker network:
This reachability check uses a disposable Docker probe and does not create or replace a sandbox. If Docker GPU compatibility recreation fails later, follow GPU routing or compatibility patch failed. That path can restore the pre-patch sandbox. If its diagnostics report manual cleanup, use only the printed exact-container command. That command targets the failed replacement and preserves the restored sandbox.
Invalid sandbox name
Sandbox names must contain 1 to 19 characters.
They must be lowercase, start with a letter, contain only letters, numbers, and single internal hyphens, and end with a letter or number.
Consecutive hyphens (--) are not allowed.
The CLI rejects names that do not match these rules.
It prints a Try: <suggested-slug> recovery line whenever it can derive a valid lowercase, hyphen-separated form from the input, so passing --name MyAssistant reports Try: myassistant and you can rerun with the suggested slug.
The CLI writes the rejected value as a quoted preview instead of raw input.
The preview reads at most the first 80 UTF-16 code units from the input and escapes each code unit outside printable ASCII as \uXXXX.
Escaping can make the preview longer than 80 output characters.
This prevents a rejected name from injecting control sequences into terminal or CI output.
Names that collide with global CLI commands are also rejected.
Reserved names include onboard, list, deploy, setup, start, stop, status, debug, uninstall, credentials, and help.
Using a reserved name would cause the CLI to route to the global command instead of the sandbox.
If the name does not match these rules or is reserved, the wizard exits with an error.
Choose a name such as my-assistant or dev1.
Sandbox creation fails on DGX
On DGX machines, sandbox creation can fail if the gateway’s DNS has not finished propagating or if a stale port forward from a previous onboard run is still active.
Run nemo-deepagents onboard to retry.
The wizard cleans up stale port forwards and waits for gateway readiness automatically.
GPU Setup Fails with a Placeholder GPU Name
On Windows, WSL, and native Linux ARM64 hosts, some systems report a placeholder display adapter name even when no NVIDIA GPU firmware is present.
This section also applies when preflight reports no GPU on an ARM64 Linux host whose nvidia-smi shows a non-placeholder GPU name.
NVIDIA NIM and GPU-backed sandbox setup require a real NVIDIA GPU.
When the primary memory-query probe reports exactly one placeholder-named GPU row on a native or Docker Desktop-backed WSL ARM64 Linux host without firmware-confirmed NVIDIA platform metadata, onboarding runs one bounded Docker CUDA workload.
When that probe reports a single non-placeholder NVIDIA GPU name on such a host and the NVIDIA kernel-driver interface (/proc/driver/nvidia) is absent, onboarding runs the same workload.
A WSL2 host never provides that interface because Windows paravirtualizes the GPU through /dev/dxg.
A non-placeholder name that does not identify an NVIDIA GPU or product family does not start the workload.
NemoClaw treats a recognized NVIDIA product model from /sys/class/dmi/id/product_name or /sys/firmware/devicetree/base/model, or a known Tegra device node, as authoritative platform identity.
Docker may pull the CUDA sample image from nvcr.io and keeps the image in the local cache after the container exits.
The workload uses this command:
The run is bounded to 3 minutes.
Set NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS to a positive millisecond value to change that bound.
Increase the value only when the image pull or GPU workload needs more than 3 minutes.
A passing workload lets onboarding treat the detected GPU as eligible for GPU passthrough during that run.
For Windows-on-Arm, this proof is a technical detection check and does not change the Unsupported product status or establish platform qualification.
Refer to Platform Support and Launch Claims for the current support boundary.
A failed or timed-out workload leaves the GPU unproven and does not enable GPU passthrough.
The names-only unified-memory fallback does not run this workload and rejects denylisted names.
WSL hosts that are not Docker Desktop-backed do not run the workload and continue to report the GPU as unavailable.
When GPU detection rejects the nvidia-smi report, preflight prints the failed check under the Local NIM unavailable — no GPU detected line, for example an absent /proc/driver/nvidia interface or a failed bounded CUDA proof.
If NemoClaw rejects the detected GPU name during preflight, select a CPU or remote inference provider, or move the setup to a host with a supported NVIDIA GPU and current drivers.
Jetson/Tegra hosts support sandbox GPU passthrough through the compatibility route.
Onboarding detects those hosts separately and propagates eligible host group IDs for selected /dev/nvmap, /dev/nvhost-*, and /dev/nvgpu/igpu0/* nodes plus real /dev/dri/renderD* character devices.
If that path fails, follow the Jetson/Tegra compatibility guidance below instead of treating a missing nvidia-smi result as a placeholder adapter.
Colima socket not detected (macOS)
Newer Colima versions use the XDG base directory (~/.config/colima/default/docker.sock) instead of the legacy path (~/.colima/default/docker.sock).
Some installations expose a top-level Colima socket at ~/.colima/docker.sock.
NemoClaw checks all three paths.
If neither is found, verify that Colima is running:
Sandbox build is slow or hangs (under-provisioned container runtime)
Default Colima ships with 2 vCPU and 2 GiB of memory, which is not enough headroom for the BuildKit-driven sandbox image build. On macOS Apple Silicon, the build can stall part-way through with no progress and no error, leaving the wizard waiting indefinitely.
Preflight inspects docker info for NCPU and MemTotal and prints a warning when the runtime falls below 4 vCPU or 8 GiB.
In interactive onboarding, the warning prompt defaults to abort, so pressing Enter stops the run before the sandbox build reaches the likely stall point.
Type y only when you intentionally want to continue on the smaller runtime.
Non-interactive onboarding prints the warning and continues.
On Colima, raise the resources before re-running onboard:
On Docker Desktop, raise CPU and memory limits in Settings → Resources, then apply and restart.
To silence the warning when the host is intentionally small, set NEMOCLAW_IGNORE_RUNTIME_RESOURCES=1 before running nemo-deepagents onboard.
Sandbox creation killed by OOM (exit 137)
On systems with 8 GB RAM or less and no swap configured, the sandbox image push can exhaust available memory and get killed by the Linux OOM killer (exit code 137).
NemoClaw automatically detects low memory during onboarding and prompts to create a 4 GB swap file.
If this automatic step fails or you are using a custom setup flow, create swap manually before running nemo-deepagents onboard:
Onboarding Reports a Rejected or Unconfirmed Policy Update
Onboarding can submit several policy mutations in sequence when you deselect policy presets and select others. It submits deselection mutations before selection mutations. Each successful mutation updates the live OpenShell gateway policy and the sandbox registry before the next mutation starts. If a later mutation fails, the earlier successful mutations remain applied and recorded.
Each mutation uses a temporary policy.yaml file in a nemoclaw-policy-* directory.
When the submission finishes, NemoClaw removes that directory, whether or not the gateway accepted the mutation.
If cleanup fails, NemoClaw reports the directory that still holds the policy instead of reporting the submission result.
When NemoClaw reports this directory, do not retry the policy operation. Use Bash to enter and validate the exact path from the error before you remove its contents:
The procedure opens the validated directory without following a symbolic link and removes policy.yaml relative to that open directory.
It fails if the directory contains anything else.
It intentionally leaves the empty directory because deleting it later by pathname would reintroduce a directory-replacement race.
Continue only after the procedure reports that the retained policy material was removed.
Do not remove the empty directory by pathname.
If validation or cleanup fails, preserve the exact path and complete error message for support.
Do not use another removal command on that path.
After temporary-directory cleanup, treat the gateway state as unknown because the cleanup error replaced the submission result. Restore access to the OpenShell gateway, then read the sandbox policy:
If policy list reports ⚠ Could not query gateway — showing local state only., stop because the command did not read the gateway policy.
If it reports container-runtime recovery guidance, restore that runtime and run policy list again.
Do not resume or start fresh onboarding until policy list reports the live gateway policy.
When NemoClaw reports a rejected or unconfirmed policy mutation, onboarding stops and:
- leaves earlier successful mutations applied and recorded;
- does not record that mutation in the sandbox registry;
- marks the session failed and resumable;
- reports one of the two results below.
After either result, run policy list before you resume or start fresh onboarding.
The gateway read the policy and refused it:
An OpenShell refusal means this mutation did not change the live policy.
Earlier successful mutations in the same onboarding step remain applied and recorded.
Use the quoted OpenShell diagnostic to identify what the gateway refused.
After you inspect policy list, follow Previous onboarding session failed to start fresh onboarding and choose a different preset selection.
NemoClaw could not confirm the result. This covers a connection that ended before the result arrived, an unreachable gateway, an elapsed deadline, a rejected credential, and a refusal the gateway reported with a status NemoClaw does not recognize as final. NemoClaw reports this whenever the gateway did not return an explicit refusal, because only an explicit refusal proves the policy was not applied:
The gateway state is unknown, so read the sandbox policy with policy list before you retry.
policy list compares the sandbox registry with the live gateway policy and flags a preset that is applied in one place but not the other.
The unconfirmed mutation does not update the sandbox registry.
The same connection problem that made the result unconfirmed can also stop policy list from reaching the gateway.
When that happens, policy list prints ⚠ Could not query gateway — showing local state only. and still exits 0.
If the container runtime is down, it prints that runtime’s recovery guidance instead.
Stop while policy list reports local state only because that output does not show whether the gateway applied the mutation.
Restore gateway access and run policy list again before you resume or start fresh onboarding.
After policy list reads the live gateway policy, follow the result below.
The failed-session recovery steps provide the resume and fresh onboarding commands.
- If an affected preset reports
active on gateway, missing from local state, resume the failed session. The gateway completed the addition, and resume can record the applied preset locally without submitting that policy change again. - If an affected preset reports
recorded locally, not active on gateway, do not resume or start fresh onboarding. The gateway may have completed the removal while the sandbox registry retained the preset. Preserve the original unconfirmed error and the completepolicy listoutput for support. - If
policy listreports no disagreement for the unconfirmed mutation, compare the live preset set with the selection in the failed session. Resume only if the live policy still requires the unconfirmed addition or removal to match that selection. Otherwise, start fresh onboarding and select the exact preset set thatpolicy listreports as active.
Previous onboarding session failed
If a previous nemo-deepagents onboard attempt fails partway through (for example, a provider or inference-setup step reporting an error), NemoClaw records the failure in ~/.nemoclaw/onboard-session.json.
When you re-run the installer, it detects the failed session and does not silently retry it. Silent retry would loop on the same failure if your original choice, such as an unreachable provider, was the cause.
- In an interactive terminal, the installer prompts whether to resume the failed session or start fresh.
Press
R(or Enter) to retry the same session, orfto discard it and make fresh choices. - In non-interactive mode (piped
curl | bashwithNEMOCLAW_NON_INTERACTIVE=1, CI, scripts), the installer refuses and exits with a non-zero status so a scripted re-run cannot loop. You must opt in to one of two paths explicitly:
Start over with new choices to discard the recorded session and provider/model selection.
Or use environment variables instead.
Set them on the bash side of the pipe because only the right-hand process inherits them.
Retry the same session.
This is only useful if the original failure was transient, for example a network blip or a stopped Docker daemon, and not a wrong provider choice:
For a checkpoint schema 4 portable session, the plain command restores the portable profile from the checkpoint. You can also state the matching profile explicitly:
Portable resume does not trust ambient Docker or Podman runtime selectors.
It derives and verifies the recorded current-user rootless Podman authority before it continues onboarding.
If NemoClaw reports unsafe ownership, type, or mode, correct that filesystem condition and retry.
Portable onboarding always uses the .config directory beneath the canonical home reported by the operating system; changing HOME or XDG_CONFIG_HOME does not select another location.
For a recorded alternate configuration root or other user ID, home, runtime root, endpoint, runtime kind, or ownership drift, do not edit the checkpoint; run fresh onboarding.
If NemoClaw reports that an active checkpoint uses schema 1, 2, or 3, the older checkpoint did not record enough profile and runtime authority for resume. NemoClaw preserves the session and exits before portable configuration, socket activation, or resource changes. Discard that active session and start fresh onboarding:
If you intend to use the portable experimental profile, select it again for fresh onboarding:
As a last resort, you can also delete the session file directly and re-run the installer:
Kubernetes namespace not ready
If onboarding fails with Kubernetes namespace not ready, a previous failed or interrupted setup may have left stale OpenShell or NemoClaw state behind.
Clean up the failed installation before re-running the installer:
The normal uninstall path keeps user data under ~/.nemoclaw/, including sandbox registry metadata, backups, and saved credentials unless you explicitly remove them.
If nemo-deepagents uninstall reports that the local uninstall script is missing, follow the CLI’s security boundary: download the versioned NVIDIA/NemoClaw tag URL that it prints, inspect the script locally, run that local copy, and then retry the installer.
Runtime
Reconnect after a host reboot
After a host reboot, the container runtime, OpenShell gateway, and sandbox may not be running. Follow these steps to reconnect.
-
Start the container runtime.
- Linux: start Docker if it is not already running (
sudo systemctl start docker) - macOS: open Docker Desktop or start Colima (
colima start)
- Linux: start Docker if it is not already running (
-
Check the managed OpenShell gateway service.
If a custom-port gateway is NemoClaw-managed, skip this service check and continue with the NemoClaw recovery step using the same environment value. Only the default port
8080uses a NemoClaw-managed service. If the gateway is externally supervised, inspect and restart it through the supervisor declared byNEMOCLAW_GATEWAY_MANAGEMENT, regardless of port.On Apple Silicon macOS with Homebrew, let NemoClaw inspect and restart the official formula service. NemoClaw runs each Homebrew operation inside the checksum-verified temporary trust boundary. Continue to the NemoClaw recovery step below instead of running
brew servicesdirectly.NemoClaw verifies the staged formula checksum and temporarily trusts only
nvidia/openshell/openshellaround each Homebrew inspection, start, or stop operation. If formula verification fails or Homebrew cannot grant or remove temporary trust, rerun the standard NemoClaw installer and then rerun onboarding:The standalone gateway is selected only when Homebrew or both the staged formula and installed keg are absent. When the formula or keg exists, Homebrew remains the lifecycle authority. NemoClaw does not switch to the standalone gateway after a Homebrew inspection, start, or stop failure. Follow the reported repair guidance instead of changing service ownership manually. If the installed service fails inspection, startup, or its health check, NemoClaw prints this log command:
Rerun the NemoClaw installer to restore the managed service for later onboarding runs.
On Linux package installs, inspect and restart the upstream service.
If the service fails inspection, startup, or its health check, NemoClaw prints this log command:
On Linux tarball installs, inspect and restart the marked NemoClaw service.
If the service fails inspection, startup, or its health check, NemoClaw prints this log command:
The tarball unit is under
$XDG_CONFIG_HOME/systemd/user, or~/.config/systemd/userwhenXDG_CONFIG_HOMEis not absolute. It starts with your user session; NemoClaw does not enable lingering. On Linux, NemoClaw attempts the standalone fallback when a managed service fails inspection, startup, or its health check. The standalone gateway starts only after NemoClaw verifies exclusive ownership of the gateway port. The fallback does not bypass managed-service trust validation or unsafe environment configuration. These conditions remain hard failures:- Homebrew formula identity query, metadata, or official-tap validation errors
- Foreign or symlinked systemd units, or an untrusted systemd executable identity
- An invalid
DOCKER_HOSTor a symlinked service environment file
Without Homebrew on macOS, or without a reachable systemd user manager on Linux, continue with the standalone recovery step below.
-
Check sandbox state.
If the sandbox shows
Ready, skip to step 5. -
Recover the managed gateway (if needed).
If the sandbox is not listed after the service restart, or
systemctl --useris unavailable, first ask NemoClaw to reconnect through the recorded sandbox:If that cannot restore the gateway registration, resume onboarding to recreate the managed gateway metadata:
Wait a few seconds, then re-check with
openshell sandbox list. On Docker-driver hosts, NemoClaw also looks for OpenShell-labeled sandbox containers when the gateway is healthy but reports the sandbox as missing. It can start a stopped labeled container, or restore the latest GPU-backup sibling container name and start it. -
Reconnect.
The gateway usually rotates its SSH host keys across a reboot.
connectdetects the resulting identity drift, prunes the staleopenshell-*entries from~/.ssh/known_hosts, and retries automatically. You do not need to editknown_hostsby hand or re-runnemo-deepagents onboardin this case.
If the sandbox does not recover
If the sandbox remains missing after restarting the gateway, run nemo-deepagents <name> rebuild --yes while the local registry entry still exists.
The rebuild path uses the recorded sandbox metadata and the snapshot flow to preserve supported workspace and agent state.
If the sandbox was intentionally deleted and you want a clean setup instead, run nemo-deepagents <name> destroy to remove the stale local entry, then run nemo-deepagents onboard.
Create a snapshot first when the sandbox is reachable enough to back up state.
For details, refer to Create and Restore Snapshots.
Gateway Port Stays Bound After Destroying the Last Sandbox
Destroying the final sandbox with --cleanup-gateway stops the packaged OpenShell gateway service before it reaps host gateway processes, so the gateway port is released.
The service is stopped, not disabled or removed, and the next onboarding run starts it again.
If the service cannot be stopped, destroy exits non-zero and prints the status command for the service.
On Apple Silicon macOS with Homebrew, rerun the standard NemoClaw installer to restore the pinned formula and temporary trust contract:
Then rerun nemo-deepagents <name> destroy --cleanup-gateway instead of stopping the Homebrew service directly.
On Linux, stop the service yourself, then rerun destroy.
Use the service name that matches the install.
For package installs:
For tarball installs:
Sandbox is running an outdated agent version
After upgrading NemoClaw, nemo-deepagents <name> connect and nemo-deepagents <name> status warn if the sandbox is running an older agent version than the current image.
To upgrade the sandbox while preserving workspace state, run:
The rebuild command backs up state, destroys the old sandbox, recreates it with the current image, and restores state. Create a snapshot before rebuilding if you want an additional safety net:
Snapshot Sanitization Requires a Verified Python Interpreter
The CLI operations listed below stop when NemoClaw cannot resolve a verified interpreter. The error begins with this text:
NemoClaw removes credentials from copied state with an isolated python3 helper.
The operation fails closed when that helper cannot run.
These operations can report the error:
nemo-deepagents <name> snapshot createnemo-deepagents <name> rebuild, including rebuilds started bynemo-deepagents upgrade-sandboxesnemo-deepagents backup-all, including the installer’s pre-upgrade backup and an eligible stopped Docker-driver sandbox
A host that already has python3 can still report this message.
NemoClaw does not search PATH for this credential-bearing helper.
NemoClaw accepts python3 only at these locations:
/usr/bin/python3/usr/local/bin/python3/opt/homebrew/bin/python3/opt/local/bin/python3- A
python3executable beside the canonical Node.js executable
NemoClaw rejects a candidate that fails its ownership, permission, or executable checks. For more information about the interpreter requirement, refer to Prerequisites.
If NemoClaw reports that it removed the incomplete snapshot, install or repair python3 at an accepted location.
Then rerun the complete command.
If cleanup fails, treat the exact reported directory as retained until you confirm that it is absent.
Retained Incomplete Snapshot
A retained incomplete snapshot can contain unsanitized credentials. The directory must remain owner-only. You must not restore, copy, or share the directory. You must remove the directory only by the exact path that NemoClaw reports. You must confirm that the exact path no longer exists before you rerun the complete command.
Sandbox shows as stopped
When status reports sandbox_container_stopped, Docker still has a container for the sandbox, but the container is not running.
Use the lightest recovery path first instead of rebuilding immediately.
-
Confirm Docker can still see the labeled container.
-
Run recovery from the host.
Deep Agents Code does not provide the
recovercommand. Usenemo-deepagents <name> startto start its existing container. -
Check status from the host.
On Docker-driver hosts, status also attempts non-destructive recovery when OpenShell reports the sandbox as missing but Docker still has a stopped
openshell.ai/sandbox-name=<name>container or the latest GPU-backup sibling. A successful recovery prints that the sandbox was recovered from Docker and then shows the refreshed OpenShell state. -
Rebuild only if the sandbox cannot be restarted or status still cannot recover it while the local registry entry exists:
Rebuild recreates the sandbox from recorded metadata and preserves supported workspace and agent state. If the sandbox was intentionally deleted and you want a clean setup, run
nemo-deepagents <name> destroyto remove the stale local entry, then runnemo-deepagents onboard.
Sandbox is registered locally but missing from the gateway
After a gateway restart, host reboot, or manual OpenShell cleanup, NemoClaw may still have a local registry entry for a sandbox that the live gateway no longer lists.
nemo-deepagents <name> status and nemo-deepagents <name> connect preserve that local registry entry and print recovery guidance instead of deleting it automatically.
Run nemo-deepagents <name> rebuild --yes when you want NemoClaw to recreate the sandbox from the recorded metadata, or run nemo-deepagents <name> destroy when you intentionally want to remove the stale entry.
A command reports that the registry file is not valid JSON
Registry operations that require complete sandbox records, such as nemo-deepagents list and nemo-deepagents onboard, stop with Configuration file is present but is not valid JSON, followed by the path to sandboxes.json and the recovery commands.
These operations stop instead of reading the file as an empty registry, so they cannot replace your sandbox records with empty state. Optional messaging health checks omit registry-derived information when they cannot read the registry. It does not rename, move, or rewrite the file.
Follow Malformed Registry File to keep a copy and remove it.
Status shows “not running” inside the sandbox
This is expected behavior. When checking status inside an active sandbox, host-side sandbox state and inference configuration are not inspectable. The status command detects the sandbox context and reports “active (inside sandbox)” instead.
Run openshell sandbox list on the host to check the underlying sandbox state.
Deep Agents
Deep Agents Config Lock Failure Recovery
A CRITICAL Deep Agents config-lock diagnostic can report fail-closed containment=, rollback failed, or that the lock rollback could not restore the trusted posture.
A containment result identifies one of two confirmed postures or an incomplete containment attempt.
A rollback failed result or lock-rollback diagnostic does not confirm containment.
Both rollback diagnostics mean NemoClaw could not restore or confirm the original trusted posture.
- Config-root posture (
fail-closed containment=config-root) means NemoClaw installed fresh0444 root:rootconfig and hash inodes. NemoClaw also confirmed0500 root:rooton/sandbox/.deepagentsand1775 root:sandboxon/sandbox. - Sandbox-parent posture (
fail-closed containment=sandbox-parent) means NemoClaw confirmed0700 root:rooton/sandbox. NemoClaw uses this posture when it cannot confirm the complete config-root posture. fail-closed containment=incompletemeans NemoClaw could not confirm either complete posture.
Preserve the complete CRITICAL diagnostic.
Do not retry shields up.
Do not run chmod, chown, or another repair inside the sandbox.
A confirmed containment posture removes the sandbox identity’s access to the Deep Agents configuration.
An incomplete containment result, a rollback failed result, or a lock-rollback diagnostic does not establish a trustworthy boundary from which to accept the current bytes.
An ordinary rebuild cannot turn the current state into a trustworthy snapshot.
If you have a trusted host-side snapshot from before the failure, list the snapshots and record its selector:
Destroying the sandbox permanently discards state newer than the selected snapshot. Confirm that the trusted host-side snapshot exists before you destroy the sandbox.
Destroy the sandbox, re-onboard the same name from trusted host configuration, and restore the snapshot:
For snapshot contents and selector rules, refer to Create and Restore Snapshots.
If no trusted snapshot exists and you do not need to preserve the current state, recreate the sandbox from host-side onboarding configuration.
This recreation permanently discards the current sandbox state. Continue only if you accept that loss.
After either recovery path, verify the recreated sandbox from the host:
Continue only when status identifies the expected Deep Agents sandbox and shields status returns without a CRITICAL or corrupt-state diagnostic.
Then retry the original shields up operation.
dcode status reports a stale inference route
The managed dcode runtime reads provider and model settings from /sandbox/.deepagents/config.toml.
NemoClaw generates that file during onboarding and rebuilds, and Deep Agents provider or model changes require fresh named recreation rather than inference set.
Check the host-recorded route first:
Then connect and compare the in-sandbox identity:
If the provider or model does not match the host route, rebuild the sandbox to regenerate config.toml from recorded metadata:
To intentionally switch provider or model, recreate the named sandbox with fresh onboarding:
Trusted route-probe helper is missing
Deep Agents sandboxes created by older NemoClaw images may not contain the image-owned /usr/local/lib/nemoclaw/dcode-managed-exec helper.
Current connect, status, and doctor fail closed when that helper is missing because NemoClaw cannot run the authoritative inference.local route probe safely.
If the output says the trusted Deep Agents Code route-probe helper is missing, rebuild the sandbox with the current image and retry the command:
dcode refuses to start because upstream auth state exists
The managed launchers refuse upstream credential state inside /sandbox/.deepagents/.state/auth.json and /sandbox/.deepagents/.state/chatgpt-auth.json.
Those files can contain provider credentials or OAuth state that bypasses NemoClaw’s host-owned credential boundary.
Remove the upstream auth state from the sandbox, then start dcode again:
Do not put provider credentials in /sandbox/.deepagents/.env, project .env files, or Deep Agents config files.
Register credentials with NemoClaw or OpenShell on the host so the gateway can inject them at egress.
Managed MCP commands report an older Deep Agents runtime
For the current rebuild and retry workflow, refer to Agent MCP Capability Is Missing.
Tavily remains blocked after opt-in
Deep Agents does not have a NemoClaw-managed web-search feature. The Tavily flow only opens Python egress for project code or manually configured tools that call Tavily.
Confirm that the target sandbox has the tavily preset applied:
If it is missing, apply the preset, register the host-side credential, and rebuild so the provider attaches:
If Tavily is still blocked after rebuild, inspect recent policy denials:
The tavily preset is a managed-Python opt-in.
It is process-wide for sandbox Python and is not a dcode-only boundary.
Deep Agents read-only path checks fail
The Deep Agents image keeps the managed Python environment under /opt/venv read-only and leaves /sandbox writable for project and agent state.
Writes under /usr, /etc, or /opt/venv should fail, while writes under /sandbox and /tmp should work.
If a read-only path probe reports that protected paths are writable, rebuild with the current NemoClaw image:
If startup reports that Landlock enforcement is unavailable, the Deep Agents sandbox fails closed instead of running with reduced filesystem enforcement.
Deep Agents uses compatibility: strict for its managed filesystem policy, so kernels older than 5.13 or VM-backed Docker runtimes without Landlock support can block sandbox creation.
Move the sandbox to a Linux kernel and container runtime that support Landlock, then rerun onboarding or rebuild the sandbox.
Git clone fails with a certificate verification error
In networks that inspect TLS, OpenShell injects a proxy CA bundle into the sandbox.
Current NemoClaw exports that bundle as GIT_SSL_CAINFO during sandbox startup and persists it for nemo-deepagents <name> connect sessions, so Git can trust the proxy CA.
It also forwards standard CA bundle variables for subprocesses, including GIT_SSL_CAPATH, CURL_CA_BUNDLE, and REQUESTS_CA_BUNDLE.
If Git still reports server certificate verification failed, reconnect to the sandbox and check that the CA variables are present:
grep exits non-zero when it finds no matches, so empty output (with the trailing || true) simply means none of these CA variables are set in the current shell.
If they are missing on an older sandbox, upgrade NemoClaw and run:
A request inside the sandbox fails with CONNECT tunnel failed, response 403
Sandbox outbound network access is denied by default and enforced by the OpenShell proxy. When a request targets a host that no applied policy preset allows, the proxy refuses the tunnel and tools surface only the protocol-level error:
This is a network-policy denial, not a tool or certificate problem.
When you run a command through nemo-deepagents <name> exec -- ... and it exits non-zero, NemoClaw checks the sandbox audit log for a policy denial recorded after the command started.
If it finds one, it appends a short breadcrumb to stderr after the tool’s own output, naming the denied host:port when it can be extracted safely and showing the commands below:
If the log probe fails, no breadcrumb is added. Unsafe endpoint data is omitted from the breadcrumb, and an invalid sandbox name is shown as <name>.
An IPv6 target is named in its RFC 3986 bracketed form:
The tool’s own stdout/stderr bytes and its exit code are left unchanged. The breadcrumb is printed by the host CLI after the command finishes, and only for a genuine failure with a fresh denial. A command that succeeds, or one that fails for an unrelated reason, prints no breadcrumb. Set NEMOCLAW_NO_POLICY_HINT to any non-empty value other than 0 or case-insensitive false (for example, 1, true, TRUE, yes, or YES) to suppress it entirely.
The first interactive nemo-deepagents <name> connect shell also prints a one-line reminder of this denial signature and the logs command below.
The reminder is shown once per top-level interactive session, and only when all of these hold: an egress proxy is configured, the shell is interactive with a terminal attached to stderr, and it is a top-level shell (not a nested subshell or pane).
Suppress it with NEMOCLAW_NO_POLICY_HINT=1.
The reminder names the sandbox when NemoClaw receives a valid sandbox name during sandbox creation.
If no valid name is available, it shows <name>; run nemo-deepagents list to see your sandbox names.
If the reported sandbox name contains characters that are not valid in a sandbox name (uppercase letters, underscores, control characters, and similar) or exceeds 19 characters, the reminder shows the <name> placeholder for safety rather than echoing the untrusted value.
The reminder is intentionally proactive: the denial itself is surfaced by the OpenShell proxy, so the curl/git error text is left unchanged and the reminder points you to the logs instead.
To see which rule denied the request, read the merged logs from the host:
If the host should be reachable, allow it with a built-in preset or apply a reviewed custom preset file from the host:
Replace <preset> with a real preset name such as github, pypi, or npm.
Run nemo-deepagents <name> policy add with no preset to list the available presets.
Sandbox creation reports a TLS certificate mismatch
If sandbox creation reports a TLS or certificate mismatch, the OpenShell gateway certificate may have changed since the CLI last registered it. Remove the stale local gateway registration and then resume onboarding so NemoClaw refreshes the registration:
Inference requests time out
Verify that the inference provider endpoint is reachable from the host. Check the active provider and endpoint:
The main Inference line probes https://inference.local/v1/models from inside the sandbox and then sends one inference request over the same route, so it reflects the route the agent uses.
If that line shows unauthorized, unhealthy, unreachable, or not probed, inspect the labeled diagnostic lines to identify the failing hop.
An unauthorized line means the route answered but rejected the request, so refresh the provider credential rather than the route.
For local Ollama and local vLLM, Inference (ollama backend) or the corresponding local-backend line reports the host-side service separately.
For Local Ollama, current releases can also print Inference (auth proxy) when a proxy token is available.
If a local backend or auth-proxy diagnostic fails, start the backend or re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
For Ollama-backed OpenClaw sandboxes, agent passthrough uses the registered host route to warm an unloaded model after an Ollama daemon restart.
If that bounded warm-up fails or times out, NemoClaw reports the result and continues so OpenClaw can emit its canonical backend error.
If the endpoint is correct but requests still fail, check for network policy rules that may block the connection. Then verify the credential and base URL for the provider you selected during onboarding.
If you entered an AWS Bedrock Runtime URL such as https://bedrock-runtime.us-east-1.amazonaws.com in the Other Anthropic-compatible endpoint flow, NemoClaw auto-detects it and routes sandbox traffic through a host-local adapter.
Use the raw Bedrock Runtime host, not an Anthropic /v1/messages path, and verify that the model ID or inference profile ID is valid for that region.
For auth, export AWS_BEARER_TOKEN_BEDROCK, AWS_PROFILE, or standard IAM environment credentials before onboarding; if you paste a key at the COMPATIBLE_ANTHROPIC_API_KEY prompt, NemoClaw uses it only as the adapter’s Bedrock bearer token.
Region errors usually mean the pasted endpoint region, AWS_REGION, AWS_DEFAULT_REGION, or the model/inference profile ID do not match.
For Ollama, vLLM, NIM, and compatible-endpoint inference validation, the default timeout is 180 seconds.
The managed NIM startup health wait uses a separate 15-minute (900-second) default and still exits early if the container stops before it becomes healthy.
On Docker 29.x or hosts using the containerd image store, managed NIM onboarding resolves and pulls the host-platform image digest when NGC exposes a multi-architecture image index.
If you still see NGC repository-format or attestation errors, confirm Docker can run docker manifest inspect for the selected image and that you are logged in to nvcr.io.
If large prompts still cause timeouts, increase it with NEMOCLAW_LOCAL_INFERENCE_TIMEOUT before re-running onboard:
For local Ollama and vLLM, onboarding retries the container reachability check and can fall back to the host-side health check when the local backend is healthy. If Ollama times out during a cold model load, NemoClaw retries once with a 300-second probe budget before failing. If all attempts fail, the error includes container reachability diagnostics such as HTTP status and host gateway resolution.
NEMOCLAW_LOCAL_INFERENCE_TIMEOUT only covers the inference-server validation probe.
The post-create readiness wait has its own budget (NEMOCLAW_SANDBOX_READY_TIMEOUT); refer to Sandbox onboard times out with “did not become ready within Ns” for the readiness path.
Sandbox onboard times out with “did not become ready within Ns”
Onboarding ends with:
This is a separate budget from NEMOCLAW_LOCAL_INFERENCE_TIMEOUT.
It covers the readiness wait that follows sandbox creation, including in-sandbox boot, OpenClaw start, and policy load.
It does not cover the inference probe.
The 180-second default fits typical workstations but can be exceeded when:
- The host is building or uploading the sandbox image for the first time (cold caches, slow link).
- The selected model is large (70B+ parameters or 4-bit/8-bit quantisations that take time to memory-map).
- Onboarding runs on a remote VM where image upload to the gateway streams over the network (for example DGX Station first-run installer).
Raise the budget before re-running onboard:
The variable accepts seconds and applies to the readiness wait only.
When the ordinary create deadline expires, NemoClaw tries to delete the partially created sandbox.
After successful cleanup, the output ends with Retry: nemo-deepagents onboard.
If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints Manual cleanup: openshell sandbox delete "<name>".
The failure path also differs when NemoClaw recreates an OpenShell-managed Docker runtime immediately before this wait.
NemoClaw pins the exact OpenShell sandbox ID before recreation.
Within the same deadline, NemoClaw requires two consecutive Ready observations that each confirm the exact ID and successful command execution.
It retries only OpenShell’s exact sandbox is not ready response.
If the deadline expires, the ID changes, or another probe fails, NemoClaw preserves diagnostics and attempts to restore the pre-recreation Docker container.
If restoration fails, NemoClaw reports that the sandbox and container state is uncertain.
NemoClaw does not start dashboard or other host forwarding, and it does not delete a sandbox by its mutable name.
It leaves the sandbox in place for inspection and recovery.
If readiness still fails after the extended budget, inspect the gateway and sandbox status:
If onboarding instead reports that the sandbox “did not re-register with OpenShell after policy application,” the same timeout controls that post-policy command-readiness probe. Raise the budget before retrying, then inspect the same gateway and sandbox status if re-registration still fails.
Sandbox onboard fails with “entered Error phase before it became ready”
Onboarding ends with:
On a fresh onboard the OpenShell gateway can (re)start its supervisor session and re-register the just-created sandbox.
During that window openshell sandbox list briefly reports the sandbox in the transient Error phase before it flips to Ready, as seen on DGX Spark when supervisor restart races the sandbox bootstrap.
NemoClaw polls immediately, starts retrying after 250ms, and backs off to a 2-second cap.
It tolerates 30 consecutive Error observations by default so this transient recovers on its own.
Only Error that persists through the debounce count is terminal, unless the overall NEMOCLAW_SANDBOX_READY_TIMEOUT deadline expires first.
Failed and CrashLoopBackOff are always terminal and fail immediately.
If your host needs more observations for slower re-registration, raise the debounce.
Raise NEMOCLAW_SANDBOX_READY_TIMEOUT too if the overall deadline is too short.
To fail fast on the first Error poll, set the debounce to 1:
If the failure persists after the debounce, the sandbox is stuck. Inspect the retained diagnostics and gateway state:
A Shields command reports corrupt persisted state
If shields up, shields down, or shields status reports corrupt persisted state, NemoClaw refuses to infer or change the Shields posture.
It preserves the Shields state file, timer marker, and transition record so an active recovery authority remains intact.
While the state remains corrupt, do not use shields up or an ordinary rebuild to replace it.
Inspect the reported state-file error and restore the state file from a trusted host backup before retrying.
Landlock filesystem policy blocks sandbox startup
Deep Agents uses strict Landlock compatibility. If the host kernel, Docker VM, or sandbox filesystem mount cannot enforce the managed read-only policy, OpenShell refuses to start the sandbox instead of silently degrading.
Run Deep Agents on a Linux kernel 5.13 or later with a container runtime that exposes Landlock to the sandbox. After moving to a compatible host or runtime, rerun onboarding or rebuild the sandbox:
Sandbox lost after gateway restart
Sandboxes created with OpenShell versions older than 0.0.24 can become unreachable after a gateway restart because SSH secrets were not persisted.
Running nemo-deepagents onboard automatically upgrades OpenShell to 0.0.24 or later during the preflight check.
After the upgrade, recreate the sandbox with nemo-deepagents onboard.
DNS-backed HTTPS endpoint is not supported
NemoClaw rejects an explicit custom endpoint when it resolves a public HTTPS hostname but cannot pin the same peer address across the downstream OpenShell runtime boundary while preserving TLS SNI and host validation.
This can appear during a direct blueprint run, custom-endpoint onboarding, or a host-side config set write.
Use an HTTPS IP-literal endpoint whose certificate is valid for that address. If your deployment permits non-TLS provider traffic, you can instead use a public HTTP endpoint that NemoClaw can rewrite to a DNS-pinned address. Do not bypass the check with a private or internal address or by editing the persisted sandbox config directly. For the full endpoint rules, refer to Meet Custom Endpoint Security Requirements.
Agent cannot reach external hosts through a proxy
NemoClaw uses a default proxy address of 10.200.0.1:3128 (the OpenShell-injected gateway).
If your environment uses a different proxy, set NEMOCLAW_PROXY_HOST and NEMOCLAW_PROXY_PORT before onboarding:
These are build-time settings baked into the sandbox image.
Changing them after onboarding requires re-running nemo-deepagents onboard to rebuild the image.
When HTTP_PROXY or HTTPS_PROXY is set on the host, NemoClaw adds localhost, 127.0.0.1, ::1, 0.0.0.0, the container-host aliases host.docker.internal and host.containers.internal, and the managed inference hostname inference.local to NO_PROXY for host-side subprocesses and for the env forwarded into openshell sandbox create.
This keeps local Ollama health checks, model pulls, and managed inference traffic from being chained through a corporate or desktop proxy at the sandbox-create boundary, while preserving the proxy for external hosts.
For the local provider validation probe, NemoClaw removes HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY from the probe process and sets NO_PROXY=* instead.
A host proxy therefore cannot answer for the local endpoint, including the host.docker.internal alias used for Windows-host Ollama.
Inside the running sandbox, processes continue to use the OpenShell L7 proxy for inference.local so OpenShell’s internal routing, DNS, and audit boundaries stay intact.
Agent cannot reach a host-side HTTP service
When a sandbox needs to call an HTTP service running on the host, use the normal OpenShell network policy path.
Expose the service on a host IP address that the OpenShell gateway can reach, create a custom NemoClaw policy preset for that IP and port, and apply it with nemo-deepagents <sandbox> policy add --from-file.
The sandbox request then flows through the OpenShell proxy while NemoClaw preserves the existing live policy entries.
Do not rely on host.docker.internal or host.openshell.internal as a general-purpose host-service path.
Those names may appear in the sandbox’s /etc/hosts, but in OpenShell’s sandbox network they are not guaranteed to point at a reachable host gateway.
Bypassing the proxy with --noproxy '*' also bypasses network policy enforcement and audit.
First, make sure the host-side service listens on a non-loopback address.
For example, a health endpoint on port 50001 should be reachable from the host IP, not only from 127.0.0.1:
Expected output:
Then create a custom NemoClaw preset for the host-side service.
Replace 10.0.0.5, 50001, paths, methods, and binaries with the service you want the sandbox to reach:
Apply the preset to the running sandbox with the NemoClaw CLI:
After you apply the policy, retry the request from inside the sandbox without disabling the proxy:
Expected output:
If the request is still denied, check the blocked request in openshell term.
The policy binaries list must include the executable path that actually made the request.
If the response changes from policy_denied to upstream_unreachable, the policy matched, but the OpenShell gateway could not reach the host IP and port.
Agent cannot reach an external host
OpenShell blocks outbound connections to hosts not listed in the network policy. Open the TUI to see blocked requests and approve them:
To permanently allow an endpoint, add it to the network policy. For Deep Agents, follow Customize the Network Policy to choose between built-in presets, reviewed custom presets, baseline edits, and live-policy replacement.
Ollama auth proxy did not start
NemoClaw keeps Ollama bound to 127.0.0.1:11434 and starts a token-gated
reverse proxy on 0.0.0.0:11435 so the sandbox can reach Ollama without
exposing it to the local network.
If the proxy fails to start, onboarding exits before configuring inference.
Check whether the proxy port is occupied by another process:
Stop the conflicting process and re-run nemo-deepagents onboard.
The wizard cleans up stale proxy processes from previous runs automatically,
so most failures resolve by retrying.
The proxy token is persisted to ~/.nemoclaw/ollama-proxy-token with 0600
permissions.
If the file is missing or unreadable after a host reboot, re-running
nemo-deepagents onboard regenerates it.
Ollama auth proxy is unreachable from the sandbox
On native Linux Docker-driver hosts, a host firewall can allow the host proxy check but block sandbox traffic to the Ollama auth proxy. When that happens, onboarding exits before it saves the inference route and prints output like:
Apply the ufw command printed by onboarding, then rerun onboarding.
If the message does not include a subnet, derive it from the OpenShell Docker network:
Docker Desktop, WSL, and hosts without the OpenShell Docker network use different routing models. In those cases NemoClaw treats an unavailable sandbox-side probe as non-blocking and relies on the regular proxy health check.
host.docker.internal does not reliably reach the host from the sandbox
Configuring an inference provider with a base URL like http://host.docker.internal:11434/v1 does not reliably reach a host Ollama service from inside the OpenShell sandbox.
OpenShell runs sandboxes inside a k3s network, where host.docker.internal is not a portable host-service route.
Depending on the platform, it may fail DNS resolution or resolve to an internal gateway/bridge address where the host’s port 11434 is not forwarded.
The sandbox then sees a DNS failure or connection refused:
Expected output:
Expected output:
For local Ollama, use the auth-proxy URL that NemoClaw’s “Local Ollama” onboard option configures automatically:
host.openshell.internal resolves to the same gateway IP, and the
token-gated Ollama auth proxy binds port
11435 there and forwards requests to 127.0.0.1:11434 on the host.
If you need a different host service exposed to the sandbox, route it through
the OpenShell gateway rather than relying on host.docker.internal.
Refer to issue #3136.
Local inference health check resolves to IPv6
Local inference health checks now use 127.0.0.1 instead of localhost.
On systems where localhost resolves to ::1 first, older NemoClaw releases
could probe the wrong address and report the local backend as unreachable
even when it was running.
If you see this on a current NemoClaw release, verify that the local backend
binds an IPv4 address and not only ::1.
Blueprint run failed
View the error output for the failed blueprint run:
Use --follow to stream logs in real time while debugging.
DGX Spark
For an end-to-end walkthrough with local inference on DGX Spark, refer to the NVIDIA Spark playbook.
Host freezes or logs NVRM NV_ERR_NO_MEMORY under local vLLM load
Treat a full host freeze separately from an agent tool-call hang.
If the Spark stops responding to SSH and ping, and the journal contains NVRM NV_ERR_NO_MEMORY or no software-side crash record, first isolate the local inference server before changing MCP or network policy configuration.
For onboarding-time context, refer to Use an Existing Server.
Check whether vLLM is a bring-your-own server or the NemoClaw managed Spark profile:
Inspect the running containers, current memory, and kernel evidence:
For a NemoClaw-managed Spark profile, derive the host port from the managed container’s fixed 8000/tcp mapping.
Bearer-protected profiles publish two bindings with the same host port, while bearerless profiles publish one all-interface binding.
For an existing vLLM server, set NEMOCLAW_VLLM_PORT to its host port in the current shell:
Both checks use /health because a managed profile can require bearer authentication for /v1/models.
For an existing vLLM server, inspect its launch arguments:
Large checkpoints without explicit quantization, very long --max-model-len values, high --gpu-memory-utilization, and multiple concurrent sequences all consume the Spark’s shared CPU/GPU memory pool.
Before reintroducing agent tools, restart vLLM with a smaller envelope, for example:
If the host still logs NVRM NV_ERR_NO_MEMORY while loading the model, switch to a smaller or quantized checkpoint.
For managed setup, prefer NEMOCLAW_PROVIDER=install-vllm, which selects the Spark profile and its registered model-specific serve arguments.
After standalone vLLM is stable, re-run onboarding and add MCP servers back one group at a time.
CoreDNS CrashLoop after onboarding
If CoreDNS in the embedded k3s cluster crashes shortly after setup, it is usually because it resolves against 127.0.0.11, which does not route inside the gateway container.
Run fix-coredns.sh to point CoreDNS at the container gateway IP instead, then recreate the sandbox.
k3s cannot find a freshly built image
After building a new sandbox image, k3s inside the gateway container sometimes fails to pull it even though the image exists on the host.
Remove the gateway registration, then resume onboarding.
If a privileged host gateway remains, do not use a host-wide process match.
Verify its live owner, exact gateway name and port, command line, PID file, runtime marker, and loaded sandbox namespace immediately before you stop it.
GPU passthrough on Spark
GPU passthrough is not CI-tested on DGX Spark.
It is expected to work when you pass --gpu and the NVIDIA Container Toolkit is configured.
Verify the toolkit is configured by running docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi from the host.
If nvidia-smi works on the host but onboarding says GPU passthrough was not enabled, install or repair the NVIDIA Container Toolkit, then run sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker.
If a reusable gateway was previously started without GPU passthrough, NemoClaw replaces it automatically only when no other registered sandboxes depend on it, or when --recreate-sandbox is recreating the only registered sandbox with the same name.
When shared gateway cleanup would be unsafe, follow the targeted destroy or gateway-removal commands printed by onboarding.
unresolvable CDI devices nvidia.com/gpu=all during gateway start
Recent NVIDIA Container Toolkit installs configure the Docker daemon for Container Device Interface (CDI) device injection, which a GPU-enabled gateway start then auto-selects.
If no nvidia.com/gpu CDI spec has been generated on the host yet, gateway start fails with Docker responded with status code 500: CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all.
Outside Station Express, the standard NemoClaw installer detects this gap before onboarding, first tries to enable the NVIDIA CDI refresh systemd units, and can fall back to generating the spec directly with nvidia-ctk.
Station Express never falls back to direct CDI generation.
The generic Ubuntu, Colossus BaseOS, and exact AI Developer Tools paths require the packaged refresh lifecycle to work; if it fails or omits nvidia.com/gpu=all, inspect nvidia-cdi-refresh.service, repair it, and rerun the printed exact-commit install command.
Other factory-runtime profiles stop when the CDI device is missing without enabling or restarting the refresh units.
If you run nemo-deepagents onboard directly, preflight prints the manual remediation instead.
The native Linux fix is the same on Docker hosts whose docker info advertises a non-empty CDISpecDirs.
On WSL with Docker Desktop, Docker may advertise CDI directories even though --device nvidia.com/gpu=all is not usable from the WSL distro.
For that runtime, NemoClaw skips Linux CDI repair and uses Docker’s --gpus compatibility path for sandbox GPU access.
This compatibility path can be retired once Docker Desktop exposes usable nvidia.com/gpu CDI specs inside WSL, or once OpenShell no longer requires host-visible CDI specs for Docker Desktop WSL GPU passthrough.
Enable the refresh units, verify they list nvidia.com/gpu entries, then rerun onboarding:
For other native Linux installations, if the refresh units are unavailable or do not generate CDI devices, generate the spec directly:
On WSL with Docker Desktop, confirm Docker Desktop WSL integration is enabled for your distro and verify Docker GPU access from WSL:
If GPU passthrough is not required on this host, rerun onboarding with --no-gpu instead.
GPU routing or compatibility patch failed
The route depends on the host environment and the operator control. Identify the matching path before applying the recovery guidance.
Do not apply this compatibility guidance to portable onboarding.
Portable onboarding requires native OpenShell GPU injection for every agent and does not use NEMOCLAW_DOCKER_GPU_PATCH.
Do not set fallback, 1, or another legacy nonzero value for a portable run.
For bridge-networked compatibility recreation without an explicit container DNS setting, NemoClaw selects a usable IPv4 upstream from systemd-resolved and probes that exact --dns path before it stops the original container.
If the probe confirms that the resolver is unreachable, recreation stops and leaves the original container in place.
An IPv6-only upstream list does not become a compatibility override; NemoClaw preserves Docker’s default resolver path instead.
Containers with explicit DNS settings or host networking keep their existing DNS path and do not use the fallback probe.
Ordinary native Linux bounded fallback
Ordinary Linux GPU onboarding uses native OpenShell GPU injection and stops on failure by default.
Unset, auto, and 0 all preserve this native-only confinement boundary.
NEMOCLAW_DOCKER_GPU_PATCH=fallback is the explicit operator authorization for one bounded retry.
With that control set, if sandbox creation rejects the native GPU flag before progress, the exact OpenShell-managed container labeled for that sandbox records a host runtime GPU-injection error, or an explicit nvidia-smi driver proof fails while that container’s immutable host configuration confirms that no GPU was attached, NemoClaw captures redacted diagnostics, deletes the incomplete sandbox, verifies that no OpenShell-managed Docker container labeled for that sandbox remains, and retries exactly once through the compatibility path.
Free-form build/list text and sandbox-reported CUDA output never independently authorize the broader retry.
Without corroborating host evidence, onboarding fails closed even when fallback is set and directs the operator to clean up and explicitly select compatibility with NEMOCLAW_DOCKER_GPU_PATCH=1 if desired.
Before the authorized retry, NemoClaw warns that the legacy GPU compatibility envelope recreates the OpenShell-managed Docker container and may relax container confinement compared with native injection.
Specifically, compatibility recreation adds SYS_PTRACE, adds apparmor=unconfined when the original container has no AppArmor option, and uses a compatibility policy that makes /proc writable for the NVIDIA runtime’s process-name initialization.
These broader settings are why onboarding warns before the swap and retains a native-only opt-out.
NemoClaw verifies cleanup with two stable checks (that sandbox is absent from the gateway list and no OpenShell-managed Docker containers labeled for that sandbox remain) before retrying through the compatibility path.
Cleanup is polled at most five times, one second apart, and both conditions must pass twice consecutively; otherwise onboarding stops before the retry.
These fail-closed safety limits are the internal constants STABLE_ABSENCE_CHECKS (2), MAX_CLEANUP_ATTEMPTS (5), and CLEANUP_POLL_INTERVAL_MS (1,000 ms); they are not configurable through environment variables.
The first observation is immediate, so the default bound performs at most four one-second sleeps plus the five gateway/container queries.
The bounds are intentionally fixed.
Allowing environment input to weaken or extend the cleanup proof would make a security gate deployment-dependent.
On a host that cannot prove absence within the bound, onboarding fails closed; select compatibility from the outset with NEMOCLAW_DOCKER_GPU_PATCH=1 instead of weakening the handoff proof.
If deletion or container cleanup cannot be proven safe, onboarding stops before the retry and prints manual cleanup guidance.
Image build, upload, TLS, provider, policy, dashboard, and inference failures stay on their existing error paths and do not trigger the GPU compatibility fallback.
Set NEMOCLAW_DOCKER_GPU_PATCH=1 to use only the compatibility path for diagnostics or older host compatibility.
Other legacy nonzero values keep that behavior through the v0.0.x release line and will be removed in v0.1.0.
Docker Desktop WSL compatibility route
Automatic GPU onboarding uses the compatibility path directly; it does not make a native attempt first.
The path creates the sandbox and then recreates the OpenShell-managed Docker container with NVIDIA GPU flags.
NEMOCLAW_DOCKER_GPU_PATCH=0 is ignored because this runtime requires the compatibility patch for GPU passthrough, and onboarding logs a warning when it is set.
To skip GPU passthrough entirely, rerun with --no-gpu or set NEMOCLAW_SANDBOX_GPU=0.
Jetson and Tegra compatibility default
Automatic GPU onboarding uses the compatibility path directly; it does not make a native attempt first.
The path recreates the OpenShell-managed Docker container with NVIDIA GPU flags and propagates eligible host group IDs for the supported Jetson GPU device nodes.
Use NEMOCLAW_DOCKER_GPU_PATCH=0 only for troubleshooting because it bypasses that group propagation and CUDA may not initialize.
Common compatibility-path recovery
After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, direct GPU, and applicable local-inference checks.
If a later check fails, onboarding prints failure diagnostics and attempts to restore the pre-patch container before it exits.
When rollback succeeds, the pre-patch sandbox remains available.
If the failed replacement may remain and NemoClaw retains its validated exact container ID, it prints only an exact-container docker rm -f command.
When a replacement is not stably running, the failure diagnostic includes its exact runtime ID, inspected state, and a bounded redacted log tail when available.
If replacement cleanup cannot be confirmed without a validated exact ID, onboarding reports cleanup as unknown and prints no deletion command.
When rollback fails, onboarding reports that the sandbox and container state is uncertain and prints no deletion command.
A diagnostic bundle captured before rollback records cleanup as pending and contains no deletion command.
Inspect the diagnostics, the sandbox, and its labeled Docker containers before removing anything.
Starting with NemoClaw v0.0.43, the standard installer handles the /proc/<pid>/task/<tid>/comm permission case during this patch path.
If an older release fails direct GPU proof with that path and Permission denied, upgrade NemoClaw and rerun onboarding.
When inspection confirms that the failed sandbox remains, delete it with a command such as:
Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics.
Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding.
If you do not need GPU access inside the sandbox, rerun with --no-sandbox-gpu.
If sandbox creation fails with CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all, the OpenShell gateway tried docker create --device nvidia.com/gpu=all and Docker could not resolve the CDI spec.
This injection happens inside the gateway, so NEMOCLAW_DOCKER_GPU_PATCH=0 does not bypass it.
Rerun with --no-gpu, or set NEMOCLAW_SANDBOX_GPU=0 and resume onboarding.
If onboarding reports OpenShell supervisor did not reconnect to the GPU-enabled container. even though the diagnostic bundle shows the patched container is running and healthy, the supervisor-reconnect wait is treating a transient Error phase (reported while the OpenShell host re-registers the new container) as fatal.
The reconnect wait debounces consecutive Error-phase polls before fast-failing, defaulting to 60 consecutive polls of about 120 seconds in total.
Increase the debounce window with NEMOCLAW_DOCKER_GPU_SUPERVISOR_RECONNECT_ERROR_DEBOUNCE if your host needs more time to re-register the patched container, for example slow WSL2 + Docker Desktop setups.
Set it to an integer above the default of 60, such as 120 (about 240 seconds), and rerun onboarding; the value is clamped to a minimum of 1.
If reconnect still fails after the GPU patch, NemoClaw attempts to restore the pre-patch CPU container before exiting.
When rollback succeeds, the output says the pre-patch sandbox was restored.
When rollback fails, the error says rollback failed and the pre-patch container was not restored, so inspect Docker state before retrying.
pip install fails with a system-packages error
Recent Ubuntu releases (including DGX Spark’s Ubuntu 24.04) mark the system Python install as externally managed, so pip install without a virtual environment fails.
Use a venv instead.
Avoid --break-system-packages unless you understand the risk, since it can break host tooling.
Port 3000 conflict with AI Workbench
NVIDIA AI Workbench’s Traefik proxy binds ports 3000 and 10000. If you run other services on Spark that expect port 3000, bind them to a different port.
Windows Subsystem for Linux
For environment setup steps, refer to Windows Prerequisites.
wsl --install --no-distribution returns Forbidden (403)
Check your network connectivity.
If you are behind a VPN, try reconnecting or switching to a different network.
If your network or Windows image blocks the online WSL installer, install WSL manually with Microsoft’s offline install guidance.
Download the latest official WSL .msi package from the Microsoft WSL releases page, choose the matching .x64.msi or .arm64.msi, install it, reboot if Windows requests it, then rerun wsl --status.
Bootstrap says “Windows Subsystem for Linux is not fully installed”
The bootstrap script checks wsl --status before it installs or opens Ubuntu.
If Windows reports that the WSL runtime is not installed, the script attempts wsl --install --no-distribution automatically.
If the repair command succeeds and WSL reports that the changes require a reboot, reboot and let the bootstrap resume after sign-in.
If the repair command succeeds but WSL still cannot be verified and the output does not request a reboot, follow the printed repair guidance instead of rebooting by default.
If the repair command fails, follow the printed repair steps.
If the repair command returns Forbidden (403) or remains blocked, install WSL manually with Microsoft’s offline install guidance.
Download the latest official WSL .msi package from the Microsoft WSL releases page, choose the matching .x64.msi or .arm64.msi, install it, reboot if Windows requests it, then rerun the bootstrap script.
Use the same repair flow if the bootstrap says “Windows Subsystem for Linux could not be verified” and reports a nonzero wsl --status exit code.
Bootstrap says “Windows reports that WSL 2 cannot start yet”
The bootstrap script attempts wsl --install --no-distribution automatically when wsl --status reports that WSL 2 cannot start.
If the repair command succeeds, reboot when prompted and let the bootstrap resume after sign-in.
If the message persists after repair and reboot, enable virtualization in firmware and confirm that the Virtual Machine Platform optional component is enabled.
Manual WSL installation only helps when the WSL runtime is missing or the online installer is blocked.
wsl -d Ubuntu says “There is no distribution with the supplied name”
The Ubuntu package was installed with --no-launch but never registered, or Windows finished the install command before the distribution appeared in wsl -l.
When this happens during the NemoClaw bootstrap, the script prints a sanitized WSL install output block.
PowerShell transcript headers, footers, temporary transcript paths, and status-file paths are redacted before display so you can paste the useful WSL output into a bug report with less local machine metadata.
If the sanitized output says a reboot is required, reboot and rerun the bootstrap.
If it does not request a reboot, register the distro manually or reinstall without --no-launch:
Bootstrap says a Docker executable “is not signed by a trusted publisher”
The bootstrap script runs elevated, so before it launches Docker Desktop.exe or uses docker.exe, it checks the resolved executable’s Authenticode signature and refuses to run one that is not validly signed by Docker.
The script accepts Docker Inc as the certificate subject common name.
If Docker changes the signer identity, the script refuses the executable until maintainers verify the signer on an official Docker download and update the allowlist.
For a current-user installation, the administrator child completes the system changes and returns to the original non-elevated PowerShell process before the script starts Docker Desktop or uses its CLI.
If you started the script from an elevated PowerShell window, rerun it from a normal PowerShell window so it can use the current-user installation without administrator privileges.
Reinstall Docker Desktop from docker.com or winget install --id Docker.DockerDesktop, then rerun the bootstrap script.
If reinstalling does not clear the warning, treat the existing executable as untrusted and do not run it manually either.
The script continues after this warning instead of stopping, so the docker info fails inside WSL symptom below can appear a few minutes later even though the real cause is the untrusted executable, not WSL integration.
docker info fails inside WSL
Confirm that Docker Desktop is running and that WSL integration is enabled for Ubuntu (Settings > Resources > WSL integration). Then restart WSL:
Windows-host Ollama is installed but not shown during onboarding
When NemoClaw runs inside WSL, it checks both the Windows-host Ollama HTTP endpoint and the Windows ollama.exe process.
If Ollama is installed but the daemon is not reachable through host.docker.internal:11434, the wizard should still offer a start or restart action.
If the Windows-host option does not appear, confirm that PowerShell interop is enabled in WSL and that Windows can locate Ollama:
If the process is missing, start Ollama from Windows and rerun onboarding.
If the process exists but the endpoint is unreachable, use the restart action when the wizard offers it, or restart Ollama from Windows with OLLAMA_HOST=0.0.0.0:11434.
Ollama inference fails or hangs in WSL
Ollama configures context length based on your hardware.
Verify that Ollama inference works:
Replace <model-id> with the model you selected during onboarding (for example qwen3.5:4b).
If ollama serve fails with Error: listen tcp 127.0.0.1:11434: bind: address already in use, check whether Ollama is configured for automatic startup:
If it is active, stop it first, then start with the custom context length:
For additional troubleshooting, refer to the Windows Setup page.
Podman
Podman is not a tested runtime. OpenShell officially documents Docker-based runtimes only. If you encounter issues with Podman, switch to a tested runtime (Docker Engine, Docker Desktop, or Colima) and rerun onboarding.
The portable experimental profile uses the docker command to drive rootless Podman.
Before you run this profile, make sure a Docker-compatible CLI is available on PATH.
On a Podman-only host, install the podman-docker shim for your distribution:
Then rerun portable onboarding:
Portable CPU Delegation Preflight Fails
The portable experimental profile requires the current user’s systemd hierarchy to expose the cpu cgroup controller to app.slice. NemoClaw checks this requirement before it writes portable configuration, activates services, starts the registry, builds an image, or creates a sandbox. The credential-free preflight reads only cgroup.controllers files under /sys/fs/cgroup.
The preflight distinguishes a missing controller file from a read failure. Classify the file state before you select a recovery action:
An access policy can prevent the current user from testing whether a path exists. For each missing or read failed result, ask an administrator to run sudo stat -- <reported-path>. If stat finds the file but the current-user read failed, classify the file as unreadable. If stat reports that the path does not exist, classify the file as missing. An empty readable file is valid evidence that exposes no controllers. A malformed result means the successful read did not contain the bounded, space-separated controller names supplied by the kernel.
Do not print malformed bytes directly to a terminal. Ask an administrator to inspect only the reported path and its cgroups v2 mount:
The administrator must correct the cgroups v2 mount or kernel-provided evidence before you rerun the classification command. Do not change systemd delegation, stop or start the user manager, or reboot the host to repair malformed evidence.
The error and file state identify the required recovery:
Do not use a boot, delegation, or service lifecycle action to correct an unreadable or malformed file. Those actions do not restore read access or valid kernel evidence.
The systemd changes require administrator access. NemoClaw does not edit /etc/systemd, invoke sudo, remove the sandbox CPU limit, or continue with weaker resource isolation.
The user@.service template applies to every user manager on the host. The per-UID slice drop-in
applies to the affected user’s ancestor, and the app.slice drop-in applies to every user manager
on the host. Applying or removing any of the three drop-ins requires the administrator to stop the
affected user’s manager, reload systemd, and start the manager. The administrator can reboot the
host instead of running that sequence. Stopping the manager stops that user’s systemd services,
including rootless Podman and other user services. Plan each interruption with the affected user
and host administrator.
Record the affected user’s numeric ID:
Use the three dedicated NemoClaw drop-in paths below. The service drop-in delegates cpu to user managers. The per-UID slice and app.slice drop-ins each request the kernel default CPU weight of 100. Those explicit settings activate the controller at both slice boundaries. If any file exists, stop and ask the administrator to inspect its ownership and content. Do not replace any file.
If a creation command fails, do not reload systemd. Ask the administrator to inspect all three drop-in paths and their directories. Use the final value printed for every *_created record and its matching device:inode record. The initial 0 records describe only the state before creation starts; always route recovery from the final printed values. Do not search for staging names. Use only an exact *_staging_dir_path printed before its atomic mkdir attempt.
Choose the cleanup route that matches the final records:
- If any final
*_createdvalue is0but the same command printed its matching*_idafterward, that file or directory receipt was interrupted between its identity and final commit records. Retain that identity and replace the matching final*_createdvalue with1. The general cleanup will require the path type and identity to match that record before removal. - If any final
*_drop_in_dir_createdor*_staging_dir_createdvalue isunrecorded, do not treat a missing identity as proof of absence and do not enter the general partial cleanup. Complete Recover an Unrecorded Drop-In Directory for each exact recorded intent, replace each resolved final value with0, and leave its identity empty. Enter the general partial cleanup only after every final*_createdvalue is0or1. - Otherwise, when every final
*_createdvalue is0or1and every final0has an empty identity, go directly to Clean Up a Partial Drop-In Creation.
Recover an Unrecorded Drop-In Directory
If the final directory record is created=unrecorded, the procedure recorded an exact creation intent but did not commit a creation identity. The interruption can occur before mkdir, during it, or before its identity record. Do not treat the initial created=0 record as proof that the directory was pre-existing. Do not use the general partial-creation cleanup until this state is resolved.
Establish an exclusive host-configuration maintenance window with the administrator. Pause package operations and all other changes under /etc/systemd/system and /etc/systemd/user for the command’s duration. Set unrecorded_directory to the exact recorded drop-in-directory path or unpredictable staging intent. The command accepts only those six exact paths, validates each staging name without scanning for basenames, requires the expected type and metadata, requires an empty directory, and binds removal to two matching identity reads.
If the command refuses the path, preserve it and inspect the reported condition in the same maintenance window. After a successful removal or an already-absent result, replace the final created=unrecorded value with created=0 and leave its identity empty before running the general cleanup.
Clean Up a Partial Drop-In Creation
Run this command only when creation failed before any systemctl daemon-reload. Enter this procedure only when every final *_created value is 0 or 1. Copy the final creation records into every placeholder. The command validates the complete record before it removes anything. It removes only objects recorded as created by this procedure and bound to their creation-time identity. It preserves pre-existing objects, accepts recorded objects that are already absent, and refuses identity or type drift.
If the command exits nonzero, inspect every refusal and preserve the affected path. After correcting a transient cleanup failure, rerun the complete command with the same final creation records. Do not reload systemd after partial-creation cleanup.
Verify that all three persistent paths are root-owned regular files with mode 0644:
Continue only when the command exits with status 0, all three device:inode values match the creation command’s rollback records, the service file contains the service settings below, and both slice files contain the slice settings below:
Record all rollback lines from the creation command in the administrator’s change record, including each file’s last created value, each published file’s device:inode value, and all three directory creation records. They identify the exact files and any directories created by this procedure and contain no credentials. Rollback must use those recorded values. If a required value is lost, do not use the removal command below; ask the administrator to inspect and remove the configuration through the host’s normal change-management process.
The administrator’s sudo policy can request authentication. NemoClaw never receives that credential.
Save the affected user’s work before running the next command. Stopping and starting the user manager interrupts that user’s systemd services, including rootless Podman and other user services. Run the command from an administrator session that does not depend on the affected user’s manager.
Run the stop, reload, and start sequence:
After the affected user signs in again, start app.slice in that user’s session:
The stop must complete before daemon-reload runs. Reloading while the instantiated unit is inactive lets systemd recalculate ancestor controller masks for the new delegation. The final start makes the user manager read the app.slice CPU weight and enable the delegated controller. Do not replace this sequence with systemctl restart, which can fail with 219/CGROUP before systemd recalculates the masks. The sequence stops at the first failed command. If a command fails after the manager stops, correct the failure and rerun the failed command and each remaining command.
On Ubuntu 22.04, the immediate systemctl start user@${uid}.service can still fail with status=219/CGROUP even after the inactive reload. That result leaves the current user manager stopped, but a later login can create it under the corrected cgroup hierarchy. Do not remove any of the three drop-ins or repeat the creation command. Save any remaining work in the affected user’s sessions, sign out of all those sessions, and sign in again. Then start app.slice and run the verification below. For any other start failure, inspect sudo systemctl status "user@${uid}.service" --no-pager and sudo journalctl -u "user@${uid}.service" --no-pager before retrying the failed start.
The administrator can reboot the host instead of running the stop, reload, and start sequence.
Save work for every host user before rebooting. A reboot interrupts all user services and host workloads, not only the affected user’s services.
Reboot only after the drop-in verification succeeds:
Verify the root hierarchy, current user manager, and app.slice: this command also verifies the per-user slice, for four boundaries in total.
Continue only when all four lines end in : cpu. This verification reads no credentials. Then rerun portable onboarding:
Remove the CPU Controller Drop-Ins
Remove only the three named files and any drop-in directory that the creation record marks as created by this procedure. Do not use systemctl revert. Do not remove another user@.service, per-user-slice, or app.slice drop-in or a pre-existing drop-in directory.
Retrieve each file’s final created value and every applicable device:inode value from the administrator’s creation-time change record. Inspect only paths that the record marks as created:
For each path marked as created, continue only when its content and current device:inode value match the creation-time record. Use an empty identity value for a file or directory whose final created value is 0. The removal command validates the complete record before it stops the manager or removes a path. An already absent recorded path is accepted on retry, an identity mismatch preserves every remaining recorded path, and a path marked as not created is never removed.
Save the affected user’s work before running the removal command. Stopping and starting the user manager interrupts that user’s systemd services, including rootless Podman and other user services. Run the command from an administrator session that does not depend on the affected user’s manager.
The inactive reload removes all three NemoClaw drop-ins from the instantiated units and recalculates ancestor controller masks before the manager starts. Do not replace this sequence with systemctl restart. If the command exits nonzero, inspect every reported failure. After correcting it, rerun the complete removal command with the same creation-time identities; paths already removed by the earlier attempt are accepted.
If the immediate start reports 219/CGROUP, the recorded drop-ins have already been removed, but the current user manager remains stopped. Save any remaining work in the affected user’s sessions, sign out of all those sessions, and sign in again so systemd creates the user manager under the restored hierarchy. Do not restore the drop-ins only because the immediate start returned 219/CGROUP. Then run the removal verification below. For another start failure, use the reported systemctl status output and sudo journalctl -u "user@${uid}.service" --no-pager to correct the service failure before retrying the removal command.
After the affected user signs in again, verify that systemd no longer loads any of the three drop-ins:
The output must not list any NemoClaw CPU-controller drop-in. Another administrator-owned drop-in can still configure cpu; do not remove it as part of this rollback. Removing the NemoClaw drop-ins can make the portable CPU delegation preflight fail again.
Portable Podman Readiness Fails
Portable commands use the current user’s rootless Podman socket authority recorded in NemoClaw state.
They ignore ambient Docker and Podman runtime selectors, including named connections.
Do not export another DOCKER_HOST, DOCKER_CONTEXT, CONTAINER_HOST, or CONTAINER_CONNECTION to bypass a readiness failure.
When podman.service reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority.
A valid server version classifies the endpoint as warm and avoids starting another socket service.
A missing socket or a response without a valid server version enters bounded cold activation.
Any socket authority change during this precheck fails at the socket authority stage and is not eligible for inode requalification.
During cold activation, the first API probe can cause systemd to replace the socket inode. NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged. Any other authority change or a second inode replacement fails at the socket authority stage.
A portable readiness failure identifies the stage that did not complete:
The remaining service and API inspection applies only when the failure reports a recorded socket path. If the failure does not report one, follow the recovery in the table and do not try another endpoint.
When a recorded socket path is reported, inspect the current user’s units without changing them:
If the units need activation, restart the active service when present and start the socket for the current user session:
These commands do not enable the socket for later user sessions.
Then use the exact socket path from the NemoClaw failure to require a real server response:
Continue only when the command exits with status 0 and prints a nonempty server version.
The request and the readiness report contain no credentials.
Rerun the original NemoClaw command without exporting a Docker or Podman runtime selector.
If valid cold activation needs a larger budget, set an integer from 15000 through 300000 milliseconds:
The default cold-start budget is 60,000 ms.
The later steady-state API deadline is fixed at 10,000 ms and does not use this setting.
A successful cold path uses the cold timing label and reports activation, API, and total time in milliseconds.
A successful warm path uses the warm timing label and reports steady-state API and total time in milliseconds.
Portable Host Gateway Is Unreachable
The portable experimental profile maps host.openshell.internal to the OpenShell Podman host gateway.
Do not apply the Docker bridge UFW command when portable onboarding reports this route as unreachable.
Use the same procedure when onboarding reports that the Podman service is unreachable for the portable gateway probe.
Portable onboarding reports output like this:
If podman.service is active, restart it.
Then start the user-scoped Podman socket for the current user session:
The first command does not start an inactive service. The second command starts the current user’s Podman API socket without enabling it for later user sessions. These commands affect only the current user’s Podman units. They do not read or write credentials.
Verify that the socket is active:
Expected output:
An active socket alone does not establish API health. Run the explicit Podman API request from Portable Podman Readiness Fails before you rerun onboarding.
Then rerun portable onboarding:
If the failed run has a checkpoint schema 4 resumable session, resume it without exporting Docker or Podman runtime selectors:
Continue only when onboarding no longer reports that the Podman service or OpenShell Podman host gateway is unreachable.