NemoClaw Inference Options
NemoClaw supports multiple inference providers. During onboarding, the wizard presents a numbered list of providers to choose from. Your selection determines where NemoClaw routes the agent’s inference traffic.
For OpenClaw onboarding, use nemoclaw onboard.
The provider flow is the same, with the NVIDIA Endpoints route available for OpenClaw Agent.
How Inference Routing Works
The agent inside the sandbox talks to inference.local.
It never connects to a provider directly.
OpenShell intercepts inference traffic on the host and forwards it to the provider you selected.
Provider credentials stay on the host.
The sandbox does not receive your API key.
Local Ollama and local vLLM do not require your host OPENAI_API_KEY.
NemoClaw uses provider-specific local tokens for those routes, and rebuilds of legacy local-inference sandboxes migrate away from stale OpenAI credential requirements.
Provider Status
Provider Options
The onboard wizard presents the following provider options by default.
The first six are always available.
Ollama appears when you have installed or started it on the host.
Local vLLM appears when NemoClaw detects a running vLLM server.
The managed install/start vLLM entry appears by default on DGX Spark and DGX Station, and appears on generic Linux NVIDIA GPU hosts after opt-in.
During interactive NVIDIA Endpoints onboarding, NemoClaw loads NVIDIA’s public featured model catalog once per onboarding session and reports progress before displaying the model picker.
It excludes retired or unsafe choices and corrects known catalog lag before displaying the result.
If the catalog is unavailable, malformed, or contains no safe model IDs, the wizard warns you and uses the bundled fallback list shown below.
Nemotron 3 Super remains the default when it is present; otherwise, the first live featured model becomes the interactive default.
If you set NEMOCLAW_MODEL to a safe custom model ID that is absent from the live catalog, it does not replace the live menu default.
Choose Other to use that value as the pre-filled manual entry; NemoClaw validates it against NVIDIA Endpoints before continuing.
NemoClaw neither displays nor accepts an unsafe NEMOCLAW_MODEL value as the manual-entry prefill.
Custom Endpoint URL Validation
Explicit endpoint URLs that NemoClaw saves through Hermes Provider setup, inference set, host-side config set, or a direct blueprint run must pass host-side SSRF validation.
NemoClaw rejects loopback, link-local, private, and internal addresses, including public hostnames that resolve to a private address.
For public HTTP URLs, NemoClaw stores the validated IP address so a downstream runtime cannot repeat DNS resolution and reach a different address.
NemoClaw rejects DNS-backed HTTPS URLs in these paths because it cannot pin the downstream peer address while preserving TLS SNI and host validation across the OpenShell runtime boundary.
Use an HTTPS IP-literal endpoint with a certificate valid for that address, or a public HTTP endpoint only when your deployment permits non-TLS traffic.
Managed provider defaults that do not supply an explicit custom endpoint through these paths are unaffected.
NemoClaw accepts http://host.openshell.internal:<port> only with an explicit port from 1024 through 65535; this narrow exception supports NemoClaw’s sandbox-to-host inference routes and is not a general private-endpoint bypass.
NVIDIA Endpoints and Hermes Provider use independent model catalogs, so a model can remain available through one provider after it leaves the other’s curated list. Curated-list updates affect new onboarding choices and do not rewrite existing sandbox configurations. Use Switch Inference Providers to move an existing sandbox before its configured model becomes unavailable.
Model Task-Fit Guide
Use this table as starter guidance when selecting a curated cloud model during onboarding. The provider catalog remains authoritative for exact context-window limits, availability, and current pricing. For NVIDIA Endpoints, the table covers the bundled fallback choices. The live featured model catalog can include additional models. The relative labels below are qualitative and compare models within the curated onboarding choices, not across every model a provider offers.
Choosing the Right Option for Nemotron
NVIDIA Nemotron models expose OpenAI-compatible APIs across every supported deployment surface, so two onboarding options can route to Nemotron.
For Option 3, the API key environment variable is COMPATIBLE_API_KEY. Set it to whatever credential your endpoint expects, or any non-empty placeholder if your endpoint does not require auth.
Model Router
The Model Router option uses the routed inference profile in nemoclaw-blueprint/blueprint.yaml.
When you select it, NemoClaw starts the router proxy on the host, waits for its health endpoint, registers the nvidia-router provider with OpenShell, and creates the sandbox with the same inference.local route the agent uses for other providers.
The sandbox does not call the router port directly.
The router model pool lives in nemoclaw-blueprint/router/pool-config.yaml.
Edit that file to define which models the router can choose from.
The default pool routes between NVIDIA-hosted Nemotron models and uses the tolerance value to choose the lowest-cost model whose predicted quality stays within the configured threshold.
The tolerance parameter controls the accuracy-cost tradeoff.
The router runs on the host, not inside the sandbox.
Credentials flow through the OpenShell provider system. The sandbox never sees raw API keys.
To use the router in scripted setup, set:
Host Python Requirement
The Model Router runs in a host-side virtual environment that NemoClaw creates during onboarding.
NemoClaw probes python3.13, python3.12, python3.11, python3.10, and bare python3, and adopts the first interpreter that satisfies both of:
- Version inside
[3.10, 3.14). ensurepip,pyexpat,ssl, andvenvall import without error.
If no candidate qualifies, onboarding aborts and prints the real failure for each candidate.
This surfaces issues like Homebrew python@3.14, whose pyexpat extension fails to load against the older system libexpat on macOS.
To pin a specific interpreter, set NEMOCLAW_MODEL_ROUTER_PYTHON to its absolute path before running nemoclaw onboard:
The pin is strict.
NemoClaw probes only that interpreter and aborts with the failure reason if it does not qualify, rather than silently falling back to a different interpreter on PATH.
NemoClaw rejects relative command names such as python3.12.
Use command -v python3.12 to find the absolute path.
If python -m venv itself fails for a probe-clean interpreter, for example a corrupt ensurepip seed, NemoClaw retries with the next healthy candidate when no pin is set.
With a pin set, the failure stops onboarding so you can fix or repoint the pinned interpreter.
Caveated Local Options
The following local inference options have caveats.
Local NIM and generic Linux managed vLLM install/start require NEMOCLAW_EXPERIMENTAL=1; DGX Spark and DGX Station managed vLLM entries appear by default.
An already-running vLLM server appears directly in the onboarding selection list.
For setup instructions, refer to Use a Local Inference Server.
Validation
NemoClaw validates the selected provider and model before creating the sandbox.
If credential validation fails, the wizard asks whether to re-enter the API key, choose a different provider, retry, or exit.
The wizard retries transient upstream validation failures before it reports a provider failure.
The nvapi- prefix check applies only to NVIDIA_INFERENCE_API_KEY.
Other provider credentials, such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and compatible endpoint keys, use provider-aware validation during retry.
Setup Details for Local and Compatible Providers
The sections below collect the detailed setup prompts and environment variables for local and compatible inference providers. Use them when the quickstart or local inference guide points you here for exact command shapes.
OpenAI-Compatible Server
This option works with any server that implements /v1/chat/completions, including vLLM, TensorRT-LLM, llama.cpp, LocalAI, and others.
For compatible endpoints, NemoClaw uses /v1/chat/completions by default.
This avoids a class of failures where local backends accept /v1/responses requests but silently drop the system prompt and tool definitions.
To opt in to /v1/responses, set NEMOCLAW_PREFERRED_API=openai-responses before running onboard.
Start your model server.
The examples below use vLLM, but any OpenAI-compatible server works.
Install vLLM first if it is not already on the host (for example, pip install vllm), then start it:
Run the onboard wizard.
When the wizard asks you to choose an inference provider, select Other OpenAI-compatible endpoint.
Enter the base URL of your local server, for example http://localhost:8000/v1 on the default Docker-driver topology.
If you are on the legacy containerized OpenShell gateway path and the gateway cannot reach host localhost, use the host-gateway URL for your setup, commonly http://host.openshell.internal:8000/v1.
For GGUF models, start a compatible server such as llama-server yourself and enter the model id that server reports from /v1/models.
The Ollama local path selects Ollama model tags; it does not accept a raw .gguf file path directly.
The wizard prompts for an API key.
If your server does not require authentication, enter any non-empty string (for example, dummy).
NemoClaw validates the endpoint by sending a test inference request before continuing.
The wizard probes /v1/chat/completions by default for the compatible-endpoint provider.
If you set NEMOCLAW_PREFERRED_API=openai-responses, NemoClaw probes /v1/responses instead and only selects it when the response includes the streaming events OpenClaw requires.
If a reasoning model returns only reasoning content before producing a final answer, NemoClaw retries the smoke request with a larger response budget.
Route, configuration, and authentication failures still fail immediately.
Non-Interactive Setup
Start your OpenAI-compatible server first; the endpoint in NEMOCLAW_ENDPOINT_URL must already be serving, or onboarding fails its validation probe.
Then set the following environment variables for scripted or CI/CD deployments.
Set NEMOCLAW_REASONING=true when the compatible endpoint serves a reasoning-only model.
Reasoning mode validates only /v1/chat/completions; it does not verify tool calling or
streaming. Enable it only when the endpoint supports the capabilities your agent needs.
Selecting the API Path
For the compatible-endpoint provider, /v1/chat/completions is the default.
NemoClaw tests streaming events during onboarding and uses chat completions without probing the Responses API.
To opt in to /v1/responses, set NEMOCLAW_PREFERRED_API before running onboard:
The wizard then probes /v1/responses and only selects it when streaming support is complete.
If the probe fails, the wizard falls back to /v1/chat/completions automatically.
You can use this variable in both interactive and non-interactive mode.
If you already onboarded and the sandbox is failing at runtime, re-run nemoclaw onboard to re-probe the endpoint and bake the correct API path into the image.
Refer to Switch Inference Models for more information.
Anthropic-Compatible Server
If your local server implements the Anthropic Messages API (/v1/messages), choose Other Anthropic-compatible endpoint during onboarding instead.
For non-interactive setup, use NEMOCLAW_PROVIDER=anthropicCompatible and set COMPATIBLE_ANTHROPIC_API_KEY.
vLLM
When vLLM is already running on localhost:8000, NemoClaw can detect it automatically and query the /v1/models endpoint to determine the loaded model.
On supported Linux hosts with NVIDIA GPUs, the onboard wizard can also install or start a managed vLLM container for you.
For an already-running vLLM server, run nemoclaw onboard and select Local vLLM [experimental] from the provider list.
If vLLM is already running, NemoClaw detects the running model and validates the endpoint.
When vLLM exposes runtime metadata such as max_model_len, NemoClaw uses that value for the contextWindow baked into openclaw.json unless you set NEMOCLAW_CONTEXT_WINDOW yourself.
If vLLM is not running and your host matches a DGX Spark or DGX Station managed profile, NemoClaw shows the Install vLLM or Start vLLM entry by default.
Generic Linux NVIDIA GPU hosts still require NEMOCLAW_EXPERIMENTAL=1 or NEMOCLAW_PROVIDER=install-vllm before the managed entry appears.
In interactive runs, the managed vLLM path lists the supported registry models for your host profile before it pulls weights.
Press Enter to use the default model, or choose a numbered entry to serve another validated model with its matching vllm serve flags.
NemoClaw pulls the vLLM image, downloads model weights into ~/.cache/huggingface, starts the nemoclaw-vllm container on localhost:8000, streams Hugging Face download progress, and polls /v1/models until the model is ready.
Managed DGX Spark and DGX Station profiles use the stable NGC nvcr.io/nvidia/vllm:26.05.post1-py3 container image.
If docker pull output stops making progress, a watchdog stops the stalled pull instead of failing slow but active downloads on a fixed wall-clock timeout.
If vLLM never becomes ready, NemoClaw prints a short tail of the vLLM container logs before exiting.
The first run can take 10 to 30 minutes.
Later runs reuse the cached image and model weights.
Managed vLLM uses these profiles:
NemoClaw forces the chat/completions API path for vLLM.
The vLLM /v1/responses endpoint does not run the --tool-call-parser, so tool calls arrive as raw text.
Non-Interactive Setup
Use an already-running vLLM server:
Install or start managed vLLM when NemoClaw detects a supported profile.
On DGX Spark and DGX Station, NEMOCLAW_PROVIDER=install-vllm is enough for non-interactive runs; add NEMOCLAW_EXPERIMENTAL=1 on generic Linux NVIDIA GPU hosts.
Non-interactive runs use the profile default unless you set NEMOCLAW_VLLM_MODEL.
NemoClaw records the model returned by vLLM’s /v1/models endpoint.
Start vLLM with the model you want before onboarding if you manage the server yourself.
Override the Managed-vLLM Model
Managed vLLM serves the profile default unless you choose a different registry entry in the interactive picker or set an override for automation.
Export NEMOCLAW_VLLM_MODEL=<slug> before invoking the installer to choose a different model without prompting.
NemoClaw uses the matching vllm serve flags, including the reasoning parser, tool-call parser, and --max-model-len.
Recognized slugs are:
The slug is case-insensitive; the full Hugging Face id is also accepted. An unrecognized value fails fast with a list of valid slugs.
Gated models require a Hugging Face token; export it before onboarding so NemoClaw can forward it into the managed vLLM container:
NemoClaw accepts HUGGING_FACE_HUB_TOKEN as an alternative.
The token check runs on the host before any docker pull, so a missing or empty token aborts onboarding before bandwidth is spent on a 401.
Add Managed-vLLM Serve Arguments
For advanced vLLM options that are not in the NemoClaw registry yet, export NEMOCLAW_VLLM_EXTRA_ARGS_JSON as a JSON array of individual non-blank vllm serve tokens.
NemoClaw trims and validates the array before pulling images or downloading models, shell-quotes each token, and appends the tokens after the registry defaults.
Use this for operator-owned tuning only. If the selected vLLM image does not support an argument, the managed container exits and NemoClaw prints the vLLM log tail.
NVIDIA NIM (Experimental)
NemoClaw can pull, start, and manage a NIM container on hosts with a NIM-capable NVIDIA GPU.
Set the experimental flag and run onboard.
Select Local NVIDIA NIM [experimental] from the provider list.
NemoClaw filters available models by GPU VRAM, pulls the NIM container image, starts it, and waits for it to become healthy before continuing.
On hosts with mixed NVIDIA GPU models, the preflight summary shows each detected GPU model and the total VRAM so you can confirm which device class the model selection used.
On Docker 29.x or containerd image-store hosts, NemoClaw resolves the host-platform manifest digest before pulling multi-architecture NIM images when the registry exposes an index.
It pulls repo@digest and retags the local image so NGC attestation metadata on other architectures does not block the selected platform.
If the registry does not expose a matching index, NemoClaw falls back to the tag pull.
NVIDIA hosts NIM container images on nvcr.io, and docker pull requires NGC registry authentication.
If Docker is not already logged in to nvcr.io, onboard prompts for an NGC API key and runs docker login nvcr.io over --password-stdin so the key is never written to disk or shell history.
The prompt masks the key during input and retries one time on a bad key before failing.
In non-interactive mode, onboard exits with login instructions if Docker is not already authenticated; run docker login nvcr.io yourself, then re-run nemoclaw onboard --non-interactive.
If NGC_API_KEY or NVIDIA_INFERENCE_API_KEY is already exported, NemoClaw passes it into the managed NIM container through the process environment instead of command-line arguments.
If the NIM container exits before the health endpoint becomes ready, onboarding stops early and prints the last container log lines.
After NIM becomes healthy, NemoClaw reads /v1/models and uses the served model id for validation when it differs from the catalog name.
Unsafe served ids are rejected instead of being written into the sandbox config.
NIM uses vLLM internally.
The same chat/completions API path restriction applies.
Non-Interactive Setup
To select a specific model, set NEMOCLAW_MODEL.
Timeout Configuration
Local inference requests use a default timeout of 180 seconds. Large prompts on hardware such as DGX Spark can exceed shorter timeouts, so NemoClaw sets a higher default for Ollama, vLLM, NIM, and compatible-endpoint setup.
To override the timeout, set the NEMOCLAW_LOCAL_INFERENCE_TIMEOUT environment variable before onboarding:
The value is in seconds.
NemoClaw bakes this setting into the sandbox at build time.
Changing it after onboarding requires re-running nemoclaw onboard.
NEMOCLAW_LOCAL_INFERENCE_TIMEOUT only governs the inference-server validation probe.
During local Ollama setup, NemoClaw treats host-side curl process timeouts as retryable probe failures and retries with a larger timeout before it reports a validation failure.
NemoClaw also retries Docker runtime detection with a longer docker info timeout before it chooses the local inference route.
The post-create readiness wait (image build, gateway upload, in-sandbox boot) has its own budget, NEMOCLAW_SANDBOX_READY_TIMEOUT, also defaulting to 180 seconds.
On hosts where the sandbox image takes minutes to build or upload, raise both settings together.
Examples include large quantized models, DGX Station first runs, and remote VMs over a slow link.
If onboard ends with Sandbox '<name>' was created but did not become ready within 180s, refer to Troubleshooting.
Next Steps
- Use a Local Inference Server for Ollama, vLLM, NIM, and compatible-endpoint setup details.
- Tool-Calling Reliability for deciding when Ollama is enough and when vLLM with a parser is safer.
- Switch Inference Models for changing the model at runtime without re-onboarding.