InferenceLocal Inference

Set Up Ollama

View as Markdown

Use Ollama when you want the default local inference setup path. NemoClaw detects Ollama on the host and can install, start, or upgrade it on supported systems.

Prerequisites

  • Install NemoClaw by following the Quickstart.
  • Use a host where Ollama is running or where the onboard wizard can install or start it.

Install or Upgrade Ollama

During onboarding, if Ollama is installed but not started, NemoClaw starts it for you. On macOS and Linux, the wizard can offer to install Ollama when it is not present.

When either the Ollama CLI or running daemon is below the minimum version for the starter models, currently 0.7.0, the wizard displays an explicit Upgrade Ollama entry. The wizard checks ollama --version and /api/version on port 11434 independently, so the entry appears when either side is stale.

On macOS, the wizard uses brew upgrade ollama for the platform upgrade path. On Linux, the wizard uses the official https://ollama.com/install.sh path. Linux upgrades use the sudo-driven system path because a user-local fallback would leave an existing system daemon serving the stale binary. If sudo is unavailable in a non-interactive run, rerun interactively or upgrade Ollama manually.

After an upgrade, NemoClaw probes the running daemon again and stops if its reported version remains below the minimum. Fresh installs skip this second probe because the bundled installers provide a daemon at or above the minimum.

The version gate does not apply to Windows-host Ollama reached from WSL through host.docker.internal. The Windows-host menu entries perform their own actions on the Windows side.

Choose a Linux Install Mode

On native Linux, NemoClaw chooses between a system install under /usr/local and a sudo-free user-local install under ${HOME}/.local.

  • Running as root or with passwordless sudo selects the system install.
  • A non-interactive run, including NEMOCLAW_NON_INTERACTIVE=1 or a run without a TTY on stdin, without passwordless sudo selects the user-local install.
  • An interactive shell without passwordless sudo selects the system install and lets the official installer prompt for a password.

Set NEMOCLAW_OLLAMA_INSTALL_MODE=system or NEMOCLAW_OLLAMA_INSTALL_MODE=user to override this detection.

The user-local mode downloads and extracts the Ollama release to ${HOME}/.local, then starts ${HOME}/.local/bin/ollama serve once. It does not configure a systemd service, create the ollama system user, or install CUDA drivers. For ordinary user-local installs, restart the daemon manually after a reboot.

For a sandbox created with the portable experimental profile, nemoclaw <name> connect --probe-only and nemoclaw <name> recover can restart the user-local daemon that NemoClaw installed. Before it decides whether to start Ollama, recovery probes http://127.0.0.1:11434/api/tags. If the API is unhealthy, recovery starts Ollama only when the sandbox records ollama-local and a valid ownership receipt names the fixed regular executable. NemoClaw releases that predate this receipt do not claim an existing executable after an upgrade. To authorize recovery for a previous NemoClaw user-local install, first verify that ${HOME}/.local/bin/ollama is the executable you want NemoClaw to manage, then run NEMOCLAW_PORTABLE_OLLAMA_REENROLL=1 nemoclaw <name> recover once. The command rejects a symlink or non-executable file before it records ownership. It refuses to launch a duplicate when another ollama process exists but the API is unhealthy. After it launches the daemon, it waits up to 30 seconds for /api/tags to return valid JSON with a models array. Recovery then verifies the authenticated proxy and requires HTTP 2xx from the sandbox inference.local/v1/models route. If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its serve argument, then tells you to retry recovery. The command exits non-zero with recovery guidance when startup or route validation fails. It does not start, stop, or replace a system service or another user-managed Ollama daemon.

NemoClaw prints a PATH hint when ${HOME}/.local/bin is missing from your PATH. Add export PATH="${HOME}/.local/bin:$PATH" to your shell profile if you want to invoke ollama directly.

Both Linux install modes require zstd for archive extraction. On Debian and Ubuntu, the system path can install zstd with sudo apt-get after explaining the prompt. The user-local path cannot install system packages without elevation. Install zstd manually if NemoClaw prints a distro-specific hint and exits.

Run Onboarding

Run the onboard wizard.

$nemoclaw onboard

Select the Ollama entry for your host. The menu identifies a reachable daemon as running and labels an installed but stopped daemon Start local Ollama. NemoClaw lists installed models or offers starter models when none are installed.

For registry-known models, each menu entry shows the download size and approximate VRAM requirement. The menu shows currently available GPU memory when the host reports it, or total GPU memory when available memory is unknown.

The starter list includes qwen3.6:35b and selects it by default when current GPU memory can accommodate it. When another GPU workload consumes most of the available memory, NemoClaw selects the largest starter model that still fits. It pulls the selected model, loads it into memory, and validates it before continuing.

When Ollama reports a loaded-model context length, NemoClaw uses it for the contextWindow written to openclaw.json unless you set NEMOCLAW_CONTEXT_WINDOW. Onboarding stops when the selected model does not declare tool support or returns tool-call JSON as plain message text instead of structured tool calls.

If the validation probe times out, NemoClaw retries with a larger timeout before failing. Each Ollama-backed OpenClaw passthrough checks whether the selected model is still loaded and sends a bounded warm-up request when necessary.

Use Windows-Host Ollama from WSL

When NemoClaw runs in WSL, the provider menu can offer these Windows-host actions:

  • Use Ollama when the Windows daemon is reachable.
  • Restart Ollama when it is installed but bound only to Windows loopback.
  • Start Ollama when it is installed but not running.
  • Install Ollama when it is not installed on Windows.

The install and restart paths set OLLAMA_HOST=0.0.0.0:11434 on Windows so Docker and WSL can reach the daemon through host.docker.internal. NemoClaw relaunches Ollama from the detected Windows tray application or verified ollama.exe path and waits for the endpoint to respond. It pulls missing models through the Ollama HTTP API without requiring an Ollama CLI inside WSL.

Windows-host Ollama does not require authentication on port 11434. Keep the Windows network profile private, block inbound access to port 11434 on public interfaces, and allow only the local WSL/Docker network path that needs host.docker.internal. Do not expose this port to your LAN or the internet.

If the endpoint is not reachable, NemoClaw also checks the Windows ollama.exe process through PowerShell interop. When the daemon does not become reachable, onboarding prints PowerShell commands for inspecting the Windows process and port state. Run only one Ollama instance on port 11434 at a time.

Windows-host Ollama requires Docker Desktop WSL integration. When NemoClaw detects native Docker Engine inside WSL, it labels the Windows-host actions as requiring Docker Desktop integration; selecting one prints remediation guidance and returns to the menu, or exits when onboarding runs non-interactively.

WSL Ollama paths do not use the authenticated reverse proxy described below.

Understand the Authenticated Proxy

On non-WSL hosts, NemoClaw keeps Ollama bound to 127.0.0.1:11434 and starts a token-gated reverse proxy on 0.0.0.0:11435. The native install and start paths reset NemoClaw-managed systemd launches to the loopback binding.

When non-interactive Linux onboarding finds an existing systemd Ollama service but cannot use passwordless sudo, it verifies that the service is active and that every listener reported for port 11434 is loopback-only. Wildcard or non-loopback listeners and missing or unreadable evidence cause onboarding to stop. Rerun from a terminal with NEMOCLAW_NON_INTERACTIVE_SUDO_MODE=prompt or configure passwordless sudo when this check fails.

Containers and other local network hosts reach Ollama only through the proxy, which requires a Bearer token. The agent inside the sandbox never receives the token directly because the OpenShell L7 proxy injects it at egress.

The wizard manages the proxy lifecycle:

  • It generates a random 24-byte token and stores it in ~/.nemoclaw/ollama-proxy-token with 0600 permissions.
  • It starts and verifies the proxy after Ollama.
  • It removes stale matching proxy processes from previous runs.
  • It probes the sandbox Docker network path before saving the inference route.
  • It stops matching proxy processes during uninstall.
  • It reuses the persisted token after a host reboot and during re-onboarding for the committed local Ollama route.

All proxy endpoints require the token, including GET /api/tags. The host-side proxy liveness check treats any HTTP response, including 401, as evidence that the proxy answered. The authoritative status and doctor route probe runs inside the sandbox and reports HTTP 200 through 499 as healthy, HTTP 500 through 599 as unhealthy, and transport failures or unavailable probes as unreachable or not probed.

If Ollama is already bound to a non-loopback address, onboarding restarts it on 127.0.0.1:11434 so the proxy becomes the only network path.

Allow the Docker Bridge Through a Firewall

On native Linux, a firewall can allow the host proxy health check while blocking sandbox containers on the OpenShell Docker bridge. When the sandbox-side proxy probe fails with a TCP error, onboarding exits before saving the route and prints a command like this one.

$SUBNET=$(docker network inspect openshell-docker --format '{{(index .IPAM.Config 0).Subnet}}')
$sudo ufw allow from "$SUBNET" to any port 11435 proto tcp
$nemoclaw onboard

If the probe cannot run because the host uses another routing model, onboarding continues and relies on the regular proxy health check.

Point Memory Search at a Host Ollama Container

OpenClaw can use a separate host Ollama container for memory-search embeddings. Follow Configure Memory Search to permit the bridge route, configure the provider, and rebuild an existing index.

Run Non-Interactive Onboarding

Use this configuration to onboard with Ollama without prompts.

$NEMOCLAW_PROVIDER=ollama \
>  NEMOCLAW_MODEL=qwen3.5:9b \
>  nemoclaw onboard --non-interactive --yes --yes-i-accept-third-party-software

Non-interactive runs cannot display the third-party software notice. Include --yes-i-accept-third-party-software or set NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE=1 whenever the selected path might install or start Ollama.

The following settings control non-interactive selection:

VariablePurpose
NEMOCLAW_PROVIDERSet to ollama.
NEMOCLAW_MODELOptional Ollama model tag.
NEMOCLAW_PROVIDER_MODELCompatibility fallback for the external NVIDIA QA non-interactive Ollama invocation recorded in #6869, the only identified consumer. Its configuration is not stored in this repository, so changes here cannot migrate or verify the invocation. NEMOCLAW_MODEL takes precedence, and non-interactive re-onboarding ignores this fallback when the requested provider differs from the recorded provider. Remove the fallback after the QA owner confirms that the invocation uses NEMOCLAW_MODEL or is retired.
NEMOCLAW_YESOptional model download authorization when set to 1.

Under --non-interactive, include --yes or set NEMOCLAW_YES=1 to authorize a model download. Onboarding exits when a download requires confirmation and the run cannot prompt.

Understand Model Selection

When NEMOCLAW_MODEL and the NEMOCLAW_PROVIDER_MODEL compatibility fallback are unset, NemoClaw selects a starter model based on currently available memory. During interactive onboarding, if the requested model env contains a safe model tag that appears in the current Ollama menu, NemoClaw uses that menu entry as the default. If the tag is absent from the rendered installed or starter list, the normal memory-based default remains.

If a known bootstrap model does not fit, NemoClaw warns and falls back to the largest known model that does fit. Unknown or custom tags pass through to the Ollama runner for validation.

OpenShell currently uses one inference route per gateway. If another sandbox on the same gateway already uses an Ollama model, onboarding names that required model instead of silently presenting it as a new choice. Use a separate gateway on an unused port to select a different model for the new agent. The following command uses 8990 as an example; choose another unused port when necessary:

$NEMOCLAW_GATEWAY_PORT=8990 NEMOCLAW_MODEL=qwen3.5:9b nemoclaw onboard

Interactive onboarding filters installed registry-known tags that do not fit current GPU memory. If no installed known tag fits, NemoClaw displays starter choices and warns when even the smallest tag might not fit. After a model fails validation, NemoClaw excludes it from the next installed-model menu.

On Windows on Arm N1X systems with a Snapdragon X processor, automatic selection omits the 30B and 35B starter models and selects qwen3.5:9b. This safeguard only changes automatic selection.

It does not make the larger models usable on N1X, reject an explicitly selected large model, or resolve the OpenClaw 1006 disconnect, embedded fallback, and model-timeout behavior tracked in issue #3707.

When Ollama reports a context length below 16384 and NEMOCLAW_CONTEXT_WINDOW is unset, NemoClaw writes a contextWindow of 16384 so the agent prompt and tool definitions fit better than the stock daemon default.

If the initial validation times out during a cold load, NemoClaw retries once with a 300-second probe budget. This retry also applies to tight-VRAM hosts where model warm-up can spill from GPU to CPU.

Release GPU Memory

When you switch away from Ollama, stop host services, or destroy an Ollama-backed sandbox, NemoClaw asks Ollama to unload each currently loaded model by sending keep_alive: 0. This cleanup runs on a best-effort basis and does not delete downloaded model files.