Choose a Local Inference Server

NemoClaw supports operator-run servers and NemoClaw-managed runtime profiles for local inference. Choose the option that matches your host, model, and operational needs.

The agent inside the sandbox sends inference traffic to inference.local. OpenShell intercepts that traffic and forwards it to the local endpoint configured during onboarding.

Compare the Options

Option	When to use it	Availability	Runtime API
Existing vLLM	You already run vLLM on `localhost:8000`.	Appears when NemoClaw detects the server.	`/v1/chat/completions`.
Managed vLLM	You want NemoClaw to pull an image, download model weights, and manage the server container.	Appears by default on DGX Spark and DGX Station, while generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`.	`/v1/chat/completions`.
Fixed vLLM profile	You need the fixed catalog-selected vLLM model and serving configuration.	Appears as option 2 in DGX Spark Express. Direct installation can use the dedicated installer flag. The profile does not appear in the provider menu.	`/v1/chat/completions`.
Existing llama.cpp	You already operate an authenticated llama.cpp server on loopback port `8081`.	Experimental. Always available for explicit selection and attaches only after cooperative fingerprinting succeeds.	`/v1/chat/completions`.
Managed llama.cpp	You want NemoClaw to acquire a verified GGUF file and manage an authenticated llama.cpp container.	Experimental. Appears on a qualified DGX Spark host and supports explicit non-interactive selection.	`/v1/chat/completions`.
NVIDIA NIM	You want NemoClaw to pull and manage a validated NIM container on a NIM-capable NVIDIA GPU.	Experimental and requires `NEMOCLAW_EXPERIMENTAL=1`.	`/v1/chat/completions`.

Managed vLLM uses host-specific model profiles and lets you select a supported registry model. NVIDIA NIM filters its available models by detected GPU VRAM.

Choose vLLM

Choose vLLM when you already operate a compatible server or want a managed container on a supported NVIDIA GPU host. NemoClaw forces the Chat Completions API path because the vLLM Responses endpoint does not run the configured tool-call parser.

Refer to Set Up vLLM.

Install a Fixed vLLM Profile

Use this feature-gated path to install the fixed catalog-selected vLLM model and runtime combination. The installer does not accept a provider or model override for this profile. DGX Spark Express offers this path as option 2 after the existing automatic managed-vLLM option.

Before you start, confirm these prerequisites:

Use a DGX Spark host with Linux on Arm64.
Confirm that Docker, the NVIDIA Container Toolkit, and the NVIDIA GPU driver are operational.
Stop any server that already uses port 8000.
Allow capacity for container images, model artifacts, and same-filesystem download staging.
Export HF_TOKEN only if the configured artifact source requires Hugging Face authentication.

NemoClaw uses HF_TOKEN only for model acquisition and does not write it to NemoClaw state. The value remains in the caller’s environment; run unset HF_TOKEN after installation when no other process needs it.

The installer downloads a pinned container image and a fixed catalog model from external registries. It runs the remaining onboarding steps non-interactively and can recreate the named sandbox when its recorded inference route differs. The runtime stores an owner-only bearer API key and model artifacts on the host.

$ curl -fsSL https://www.nvidia.com/nemoclaw.sh | \
>   NEMOCLAW_AGENT=langchain-deepagents-code \
>   NEMOCLAW_SANDBOX_NAME=my-assistant \
>   NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE=1 \
>   bash -s -- --local-model-runtime=vllm

The flag enables the dedicated vLLM profile gate, disables Express selection, and selects the matching catalog recipe. Selecting Express option 2 enables the same profile gate and selects the vLLM runtime. Do not combine this path with NEMOCLAW_PROVIDER or NEMOCLAW_MODEL. The profile rejects NEMOCLAW_VLLM_MODEL, NEMOCLAW_VLLM_PORT, and NEMOCLAW_VLLM_EXTRA_ARGS_JSON before installation.

The profile performs these actions:

Uses the pinned runtime image and fixed serving command from the catalog.
Stores model files in the host Hugging Face cache.
Publishes the authenticated server on 127.0.0.1:8000 and the exact private IPv4 gateway of the openshell-docker bridge, such as 172.18.0.1:8000.
Never publishes the Docker host port on 0.0.0.0 or ::.
Reuses the owner-only host-global managed-vLLM API key.

After the runtime passes its readiness check, onboarding registers the provider at https://inference.local inside the sandbox. OpenShell reaches the authenticated server through the private bridge binding, while host-side readiness and recovery use the loopback binding.

Verify the sandbox route:

$ nemo-deepagents my-assistant status
$ nemo-deepagents my-assistant doctor

Accept the result when status reports the inference route as reachable and doctor exits with status 0. Route reachability does not by itself establish successful model invocation.

Verify the bounded host publication:

$ docker network inspect \
>   --format '{{json .IPAM.Config}}' \
>   openshell-docker
$ docker container inspect \
>   --format '{{json (index .NetworkSettings.Ports "8000/tcp")}}' \
>   nemoclaw-vllm

The bridge result must contain exactly one private IPv4 Gateway value. The container result must contain exactly two bindings with HostPort set to 8000: one HostIp value of 127.0.0.1 and one that exactly matches the private bridge gateway. Reject the runtime if a binding uses 0.0.0.0, ::, an empty address, another host address, or an additional entry.

If installation stops after a model download, rerun the same installer command. The runtime reuses only an artifact that passes its recorded identity checks. If Docker reports a container-name conflict, inspect the resource labels before removing anything. Do not remove a resource that lacks the NemoClaw ownership label for this profile. Refer to Host Files and State before deleting a credential or shared cached artifact.

When you intend to remove the entire NemoClaw installation, run nemo-deepagents uninstall. Full uninstall verifies exact managed container ownership before it removes the runtime and its state. It preserves the shared Hugging Face cache used by vLLM. Add --delete-models only when you also accept deletion of NemoClaw-pulled Ollama models. If cleanup reports an ownership or Docker error, leave the state files in place, resolve the reported resource, and rerun uninstall. After a successful full uninstall, docker container inspect nemoclaw-vllm must report no object.

Choose llama.cpp

Choose llama.cpp when you already operate an authenticated server or want NemoClaw to materialize the experimental NVIDIA Nemotron recipe on DGX Spark. The existing-server path owns only provider registration and routing. The managed path owns the exact YAML-selected model, launch, Docker lifecycle, diagnostics, and cleanup.

Refer to Set Up llama.cpp.

Install Managed llama.cpp on DGX Spark

The managed procedure moved to Set Up llama.cpp.

Choose NVIDIA NIM

Choose NVIDIA NIM when you want a managed NIM container and your host has a NIM-capable NVIDIA GPU. The path is experimental, requires NGC registry access, and can fail when a selected image does not publish a manifest for the host architecture.

Refer to Set Up NVIDIA NIM.

Use Another Server

Use a custom endpoint when your server is not one of the managed local options. NemoClaw supports servers that expose an OpenAI-compatible API and supports compatible Anthropic routes with agent-specific runtime requirements.

Configure Inference Timeouts for slow local models and long sandbox startup times.
Verify the Inference Route after onboarding.